Revision 7fdf25fed954698d7bcf1c0099489ed4c8456fe6 authored by Dirk Roorda on 29 October 2023, 14:09:12 UTC, committed by Dirk Roorda on 29 October 2023, 14:09:12 UTC
1 parent c0e529a
tei.py
"""
# TEI import
You can convert any TEI source into TF by specifying a few details about the source.
Text-Fabric then invokes the `tf.convert.walker` machinery to produce a Text-Fabric
dataset out of the source.
Text-Fabric knows the TEI elements, because it will read and parse the complete
TEI schema. From this the set of complex, mixed elements is distilled.
If the TEI source conforms to a customised TEI schema, it will be detected and
the importer will read it and override the generic information of the TEI elements.
It is also possible to pass a choice of template and adaptation in a processing
instruction. This does not influence validation, but it may influence further
processing.
If the TEI consists of multiple source files, it is possible to specify different
templates and adaptations for different files.
The possible values for models, templates, and adaptations should be declared in
the configuration file.
For each model there should be corresponding schema in the schema directory,
either an `.rng` or an `.xsd` file.
The converter goes the extra mile: it generates a TF-app and documentation
(an *about.md* file and a *transcription.md* file), in such a way that the Text-Fabric
browser is instantly usable.
The TEI conversion is rather straightforward because of some conventions
that cannot be changed.
# Configuration and customization
We assume that you have a `programs` directory at the top-level of your repo.
In this directory we'll look for two optional files:
* a file `tei.yaml` in which you specify a bunch of values
Last, but not least, you can assemble all the input parameters needed to
get the conversion off the ground.
* a file `tei.py` in which you define custom functions that are executed at certain
specific hooks:
* `transform(text)` which takes a text string ar argument and delivers a
text string as result. The converter will call this on every TEI input
file it reads *before* feeding it to the XML parser.
* `beforeTag`: just before the walker starts processing the start tag of
a TEI element;
* `beforeChildren`: just after processing the start tag, but before processing
the element content (text and child elements);
* `afterChildren`: just after processing the complete element content
(text and child elements), but before processing the end tag of the
TEI element;
* `afterTag`: just after processing the end tag of a TEI element.
The `before` and `after` functions should take the following arguments
* `cv`: the walker converter object;
* `cur`: the dictionary with information that has gathered during the
conversion so far and that can be used to dump new information
into; it is nonlocal, i.e. all invocations of the hooks get the same
dictionary object passed to them;
* `xnode`: the lxml node corresponding to the TEI element;
* `tag`: the tag name of the element, without namespaces;
this is a bit redundant, because it can also be extracted from
the `xnode`, but it is convenient.
* `atts`: the attributes (names and values) of the element,
without namespaces;
this is a bit redundant, because it can also be extracted from
the `xnode`, but it is convenient.
These functions should not return anything, but they can write things to
the `cur` dictionary.
And they can create slots, nodes, and terminate them, in short, they
can do every `cv`-based action that is needed.
You can define these functions out of this context, but it is good to know
what information in `cur` is guaranteed to be available:
* `xnest`: the stack of xml tag names seen at this point;
* `tnest`: the stack of TF nodes built at this point;
* `tsiblings` (only if sibling nodes are being recorded): the list of
preceding TF nodes corresponding to the TEI sibling elements of the
current TEI element.
## Keys and values of the `tei.yaml` file
### generic
dict, optional `{}`
Metadata for all generated TF features.
The actual source version of the TEI files does not have to be stated here,
it will be inserted based on the version that the converter will actually use.
That version depends on the `tei` argument passed to the program.
The key under which the source version will be inserted is `teiVersion`.
### extra
dict, optional `{}`
Instructions and metadata for specific generated TF features, namely those that have not been
generated by the vanilla TEI conversion, but by extra code in one of the customised
hooks.
The dict is keyed by feature name, the values are again dicts.
These value dicts have a key meta under which any number of metadata key value
pairs, such as `description="xxx"`.
If you put the string «base» in such a filed, it will be expanded on the
basis of the contents of the `path` key, see below.
You must provide the key `valueType` and pass `int` or `str` there, depending on the
values of the feature.
You may provide extra keys, such as `conversionMethod="derived"`, so that other programs
can determine what to do with these features.
The information in this dict will also end up in the generated feature docs.
Besides the `meta` key, there may also be the keys `path`, and `nodeType`.
Together they contain an instruction to produce a feature value from element content
that can be found on the current stack of XML nodes and attributes.
The value found will be put in the feature in question
for the node of type specified in `nodeType` that is recently constructed.
Example:
```
extra:
letterid:
meta:
description: The identifier of a letter; «base»
valueType: str
conversionMethod: derived
conversionCode: tt
path:
- idno:
type: letterId
- altIdentifier
- msIdentifier
- msDesc
- sourceDesc
nodeType: letter
feature: letterid
```
The meaning is:
* if, while parsing the XML, I encounter an element `idno`,
* and if that element has an attribute `type` with value `letterId`,
* and if it has parent `altIdentifier`,
* and grandparent `msIdentifier`,
* and great-grandparent `msDesc`,
* and great-great-grandparent `sourceDesc`,
* then look up the last created node of type `letter`
* and get the text content of the current XML node (the `idno` one),
* and put it in the feature `letterid` for that node.
* Moreover, the feature `letterid` gets metadata as specified under the key `meta`,
where the `description` will be filled with the text
```
The identifier of a letter; the content is taken from sourceDesc/msDesc/msIdentifier/altIdentifier/idno[type=letterId]
```
### models
list, optional `[]`
Which TEI-based schemas are to be used.
For each model there should be an xsd or rng file with that name in the `schema`
directory. The `tei_all` schema is know to Text-Fabric, no need to specify that one.
We'll try a RelaxNG schema (`.rng`) first. If that exists, we use it for validation
with *jing*, and we also convert it with trang to an xsd schema, which we use for
analysing the schema: we want to know which elements are mixed and pure.
If there is no RelaxNG schema, we try an xsd schema (`.xsd`). If that exists,
we can do the analysis, and we will use it also for validation.
!!! note "Problems with RelaxNG validation"
RelaxNG validation is not always reliable when performed with lxml, or any tool
based on libxml, for that matter. That's why we try to avoid it. Even if we
translate the RelaxNG schema to an xsd schema by means of trang, the resulting
validation is not always reliable. So we use jing to validate the RelaxNG schema.
See also [jing-trang](https://code.google.com/archive/p/jing-trang/downloads).
### templates
list, optional `[]`
Which template(s) are to be used.
A template is just a keyword, associated with an xml file, that can be used to switch
to a specific kind of processing, such as `letter`, `bibliolist`, `artworklist`.
You may specify an element or processing instruction with an attribute
that triggers the template for the file in which it is found.
This will be retrieved from the file before XML parsing starts.
For example,
```
templateTrigger="?editem@template"
```
will read the file and extract the value of the `template` attribute of the `editem`
processing instruction and use that as the template for this file.
If no template is found in this way, the empty template is assumed.
### adaptations
list, optional `[]`
Which adaptations(s) are to be used.
An adaptation is just a keyword, associated with an xml file, that can be used to switch
to a specific kind of processing.
It is meant to trigger tweaks on top of the behaviour of a template.
You may specify an element or processing instruction with an attribute
that triggers the adaptation for the file in which it is found.
This will be retrieved from the file before XML parsing starts.
For example,
```
adaptationTrigger="?editem@adaptation"
```
will read the file and extract the value of the `adaptation` attribute of the `editem`
processing instruction and use that as the adaptation for this file.
If no adaptation is found in this way, the empty adaptation is assumed.
### prelim
boolean, optional `True`
Whether to work with the `pre` tf versions.
Use this if you convert TEI to a preliminary TF dataset, which will
receive NLP additions later on. That version will then lose the `pre`.
### wordAsSlot
boolean, optional `False`
Whether to take words as the basic entities (slots).
If not, the characters are taken as basic entities.
### parentEdges
boolean, optional `True`
Whether to create edges between nodes that correspond to XML elements and their parents.
### siblingEdges
boolean, optional `False`
Whether to create edges between nodes that correspond to XML elements and siblings.
Edges will be created between each sibling and its *preceding* siblings.
If you use these edges in the binary way, you can also find the following siblings.
The edges are labelled with the distance between the siblings, adjacent siblings
get distance 1.
!!! caution "Overwhelming space requirement"
If the corpus is divided into relatively few elements that each have very many
direct children, the number of sibling edges is comparable to the size of the
corpus squared. That means that the TF dataset will consist for 50-99% of
sibling edges!
An example is [ETCBC/nestle1904](https://github.com/ETCBC/nestle1904) (Greek New
Testament) where each book element has all of its sentences as direct children.
In that dataset, the siblings would occupy 40% of the size, and we have taken care
not to produce sibling edges for sentences.
### pageModel
dict, optional `False`
If not passed, or an empty dict, page model I is assumed.
A page model must be specified with the parameters relevant for the
model:
```
dict(
model="I",
)
```
(model I does not require any parameters)
or
```
dict(
model="II",
element="div",
attributes=dict(type=["original", "translation"]),
pbAtTop=True,
nodeType="page",
)
```
For model II, the default parameters are:
```
element="div",
pbAtTop=True,
nodeType="page",
attributes={},
```
Model I is the default, and nothing special happens to the `<pb>` elements.
In model II the `<pb>` elements translate to nodes of type `page`, which span
content, whereas the original `pb` elements just mark positions.
Instead of `page`, you can also specify another node type by the parameter `element`.
We assume that the material that the `<pb>` elements divide up is the material
that corresponds to their `<div>` parent element. Instead of `<div>`,
you can also specify another element in the parameter `element`.
If you want to restrict the parent elements of pages, you can do so by specifying
attributes, like `type="original"`. Then only parents that carry those attributes
will be chopped up into pages.
You can specify multiple values for each attribute. Elements that carry one of these
values are candidates for having their content divided into pages.
We assume that the material to be divided starts with a `<pb>` and we translate
it to a page element that we close either at the next `<pb>` or at the end of the `div`.
But if you specify `pbAtTop=False`, we assume that the `<pb>` marks the end of
the corresponding page element. We start the first page at the start of the enclosing
element. If there is material at between the last `<pb>` til the end of the enclosing
element, we generate an extra page node without features.
### sectionModel
dict, optional `{}`
If not passed, or an empty dict, section model I is assumed.
A section model must be specified with the parameters relevant for the
model:
```
dict(
model="II",
levels=["chapter", "chunk"],
element="head",
attributes=dict(rend="h3"),
)
```
(model I does not require the *element* and *attribute* parameters)
or
```
dict(
model="I",
levels=["folder", "file", "chunk"],
)
```
This section model (I) accepts a few other parameters:
```
backMatter="backmatter"
```
This is the name of the folder that should not be treated as an ordinary folder, but
as the folder with the sources for the backmatter, such as references, lists, indices,
bibliography, biographies, etc.
```
drillDownDivs=True
```
Whether the chunks are the immediate children of `body` elements, or whether
we should drill through all intervening `div` levels.
For model II, the default parameters are:
```
element="head"
levels=["chapter", "chunk"],
attributes={}
```
In model I, there are three section levels in total.
The corpus is divided in folders (section level 1), files (section level 2),
and chunks within files. The parameter `levels` allows you to choose names for the
node types of these section levels.
In model II, there are 2 section levels in total.
The corpus consists of a single file, and section nodes will be added
for nodes at various levels, mainly outermost `<div>` and `<p>` elements and their
siblings of other element types.
The section heading for the second level is taken from elements in the neighbourhood,
whose name is given in the parameter `element`, but only if they carry some attributes,
which can be specified in the `attributes` parameter.
### procins
boolean, optional `False`
If True, processing instructions will be treated.
Processing instruction `<?foo bar="xxx"?>` will be converted as if it were an empty
element named `foo` with attribute `bar` with value `xxx`.
# Usage
## Commandline
```sh
tf-fromtei tasks flags
```
## From Python
```python
from tf.convert.tei import TEI
T = TEI()
T.task(**tasks, **flags)
```
For a short overview the tasks and flags, see `HELP`.
## Tasks
We have the following conversion tasks:
1. `check`: makes and inventory of all XML elements and attributes used.
1. `convert`: produces actual TF files by converting XML files.
1. `load`: loads the generated TF for the first time, by which the precomputation
step is triggered. During precomputation some checks are performed. Once this
has succeeded, we have a workable Text-Fabric dataset.
1. `app`: creates or updates a corpus specific TF-app with minimal sensible settings,
plus basic documentation.
1. `apptoken`: updates a corpus specific TF-app from a character-based dataset
to a token-based dataset.
1. `browse`: starts the text-fabric browser on the newly created dataset.
Tasks can be run by passing any choice of task keywords to the
`TEI.task()` method.
## Note on versions
The TEI source files come in versions, indicated with a data.
The converter picks the most recent one, unless you specify an other one:
```python
tf-from-tei tei=-2 # previous version
tf-from-tei tei=0 # first version
tf-from-tei tei=3 # third version
tf-from-tei tei=2019-12-23 # explicit version
```
The resulting TF data is independently versioned, like `1.2.3` or `1.2.3pre`.
When the converter runs, by default it overwrites the most recent version,
unless you specify another one.
It looks at the latest version and then bumps a part of the version number.
```python
tf-fromtei tf=3 # minor version, 1.2.3 becomes 1.2.4; 1.2.3pre becomes 1.2.4pre
tf-fromtei tf=2 # intermediate version, 1.2.3 becomes 1.3.0
tf-fromtei tf=1 # major version, 1.2.3 becomes 2.0.0
tf-fromtei tf=1.8.3 # explicit version
```
## Examples
Exactly how you can call the methods of this module is demonstrated in the small
corpus of 14 letter by the Dutch artist Piet Mondriaan.
* [Mondriaan](https://nbviewer.org/github/annotation/mondriaan/blob/master/programs/convertExpress.ipynb).
"""
import sys
import collections
import re
from textwrap import dedent, wrap
from io import BytesIO
from subprocess import run
from importlib import util
from lxml import etree
from .helpers import (
setUp,
tweakTrans,
checkModel,
matchModel,
lookupSource,
NODE,
FILE,
PRE,
ZWSP,
XNEST,
TNEST,
TSIB,
WORD,
CHAR,
CONVERSION_METHODS,
CM_LIT,
CM_LITP,
CM_LITC,
CM_PROV,
)
from ..parameters import BRANCH_DEFAULT_NEW
from ..fabric import Fabric
from ..core.helpers import console, versionSort, mergeDict
from ..convert.walker import CV
from ..core.timestamp import AUTO, DEEP, TERSE
from ..core.command import readArgs
from ..core.files import (
abspath,
expanduser as ex,
unexpanduser as ux,
getLocation,
initTree,
baseNm,
dirNm,
dirExists,
dirContents,
fileExists,
fileCopy,
scanDir,
readYaml,
writeYaml,
)
from ..tools.xmlschema import Analysis
(HELP, TASKS, TASKS_EXCLUDED, PARAMS, FLAGS) = setUp("TEI")
CSS_REND = dict(
h1=(
"heading of level 1",
dedent(
"""
font-size: xx-large;
font-weight: bold;
margin-top: 3rem;
margin-bottom: 1rem;
"""
),
),
h2=(
"heading of level 2",
dedent(
"""
font-size: x-large;
font-weight: bold;
margin-top: 2rem;
margin-bottom: 1rem;
"""
),
),
h3=(
"heading of level 3",
dedent(
"""
font-size: large;
font-weight: bold;
margin-top: 1rem;
margin-bottom: 0.5rem;
"""
),
),
h4=(
"heading of level 4",
dedent(
"""
font-size: large;
font-style: italic;
margin-top: 1rem;
margin-bottom: 0.5rem;
"""
),
),
h5=(
"heading of level 5",
dedent(
"""
font-size: medium;
font-weight: bold;
font-variant: small-caps;
margin-top: 0.5rem;
margin-bottom: 0.25rem;
"""
),
),
h6=(
"heading of level 6",
dedent(
"""
font-size: medium;
font-weight: normal;
font-variant: small-caps;
margin-top: 0.25rem;
margin-bottom: 0.125rem;
"""
),
),
italic=(
"cursive font style",
dedent(
"""
font-style: italic;
"""
),
),
bold=(
"bold font weight",
dedent(
"""
font-weight: bold;
"""
),
),
underline=(
"underlined text",
dedent(
"""
text-decoration: underline;
"""
),
),
center=(
"horizontally centered text",
dedent(
"""
text-align: center;
"""
),
),
large=(
"large font size",
dedent(
"""
font-size: large;
"""
),
),
spaced=(
"widely spaced between characters",
dedent(
"""
letter-spacing: .2rem;
"""
),
),
margin=(
"in the margin",
dedent(
"""
position: relative;
top: -0.3em;
font-weight: bold;
color: #0000ee;
"""
),
),
above=(
"above the line",
dedent(
"""
position: relative;
top: -0.3em;
"""
),
),
below=(
"below the line",
dedent(
"""
position: relative;
top: 0.3em;
"""
),
),
small_caps=(
"small-caps font variation",
dedent(
"""
font-variant: small-caps;
"""
),
),
sub=(
"as subscript",
dedent(
"""
vertical-align: sub;
font-size: small;
"""
),
),
super=(
"as superscript",
dedent(
"""
vertical-align: super;
font-size: small;
"""
),
),
)
CSS_REND_ALIAS = dict(
italic="italics i",
bold="b",
underline="ul",
spaced="spat",
small_caps="smallcaps sc",
super="sup",
)
KNOWN_RENDS = set()
REND_DESC = {}
REFERENCING = dict(
ptr="target",
ref="target",
rs="ref",
)
def makeCssInfo():
"""Make the CSS info for the stylesheet."""
rends = ""
for (rend, (description, css)) in sorted(CSS_REND.items()):
aliases = CSS_REND_ALIAS.get(rend, "")
aliases = sorted(set(aliases.split()) | {rend})
for alias in aliases:
KNOWN_RENDS.add(alias)
REND_DESC[alias] = description
selector = ",".join(f".r_{alias}" for alias in aliases)
contribution = f"\n{selector} {{{css}}}\n"
rends += contribution
return rends
def getRefs(tag, atts, xmlFile):
refAtt = REFERENCING.get(tag, None)
result = []
if refAtt is not None:
refVal = atts.get(refAtt, None)
if refVal is not None and not refVal.startswith("http"):
for refv in refVal.split():
parts = refv.split("#", 1)
if len(parts) == 1:
targetFile = refv
targetId = ""
else:
(targetFile, targetId) = parts
if targetFile == "":
targetFile = xmlFile
result.append((refAtt, targetFile, targetId))
return result
class TEI:
def __init__(
self,
tei=PARAMS["tei"][1],
tf=PARAMS["tf"][1],
validate=PARAMS["validate"][1],
verbose=FLAGS["verbose"][1],
):
"""Converts TEI to TF.
For documentation of the resulting encoding, read the
[transcription template](https://github.com/annotation/text-fabric/blob/master/tf/convert/app/transcription.md).
Below we describe how to control the conversion machinery.
We adopt a fair bit of "convention over configuration" here, in order to lessen
the burden for the user of specifying so many details.
Based on current directory from where the script is called,
it defines all the ingredients to carry out
a `tf.convert.walker` conversion of the TEI input.
This function is assumed to work in the context of a repository,
i.e. a directory on your computer relative to which the input directory exists,
and various output directories: `tf`, `app`, `docs`.
Your current directory must be at
```
~/backend/org/repo/relative
```
where
* `~` is your home directory;
* `backend` is an online *backend* name,
like `github`, `gitlab`, `git.huc.knaw.nl`;
* `org` is an organisation, person, or group in the backend;
* `repo` is a repository in the `org`.
* `relative` is a directory path within the repo (0 or more components)
This is only about the directory structure on your local computer;
it is not required that you have online incarnations of your repository
in that backend.
Even your local repository does not have to be a git repository.
The only thing that matters is that the full path to your repo can be parsed
as a sequence of *home*/*backend*/*org*/*repo*/*relative*.
Relative to this directory the program expects and creates
input/output directories.
## Input directories
### `tei`
*Location of the TEI-XML sources.*
**If it does not exist, the program aborts with an error.**
Several levels of subfolders are assumed:
1. the version of the source (this could be a date string).
2. volumes/collections of documents. The subfolder `__ignore__` is ignored.
3. the TEI documents themselves, conforming to the TEI schema or some
customisation of it.
### `schema`
*TEI or other XML schemas against which the sources can be validated.*
They should be `.xsd` or `.rng` files.
!!! note "Multiple `.xsd` files"
When you started with a `.rng` file and used `tf.tools.xmlschema` to
convert it to `xsd`, you may have got multiple `.xsd` files.
One of them has the same base name as the original `.rng` file,
and you should pass that name. It will import the remaining `.xsd` files,
so do not throw them away.
We use these files as custom TEI schemas,
but to be sure, we still analyse the full TEI schema and
use the schemas here as a set of overriding element definitions.
## Output directories
### `report`
Directory to write the results of the `check` task to: an inventory
of elements/attributes encountered, and possible validation errors.
If the directory does not exist, it will be created.
The default value is `.` (i.e. the current directory in which
the script is invoked).
### `tf`
The directory under which the text-fabric output file (with extension `.tf`)
are placed.
If it does not exist, it will be created.
The tf files will be generated in a folder named by a version number,
passed as `tfVersion`.
### `app` and `docs`
Location of additional TF-app configuration and documentation files.
If they do not exist, they will be created with some sensible default
settings and generated documentation.
These settings can be overriden in the `app/config_custom.yaml` file.
Also a default `display.css` file and a logo are added.
Custom content for these files can be provided in files
with `_custom` appended to their base name.
### `docs`
Location of additional documentation.
This can be generated or had-written material, or a mixture of the two.
Parameters
----------
tei: string, optional ""
If empty, use the latest version under the `tei` directory with sources.
Otherwise it should be a valid integer, and it is the index in the
sorted list of versions there.
* `0` or `latest`: latest version;
* `-1`, `-2`, ... : previous version, version before previous, ...;
* `1`, `2`, ...: first version, second version, ....
* everything else that is not a number is an explicit version
If the value cannot be parsed as an integer, it is used as the exact
version name.
tf: string, optional ""
If empty, the tf version used will be the latest one under the `tf`
directory. If the parameter `prelim` was used in the initialization of
the TEI object, only versions ending in `pre` will be taken into account.
If it can be parsed as the integers 1, 2, or 3 it will bump the latest
relevant tf version:
* `0` or `latest`: overwrite the latest version
* `1` will bump the major version
* `2` will bump the intermediate version
* `3` will bump the minor version
* everything else is an explicit version
Otherwise, the value is taken as the exact version name.
verbose: integer, optional -1
Produce no (-1), some (0) or many (1) orprogress and reporting messages
"""
self.good = True
(backend, org, repo, relative) = getLocation()
if any(s is None for s in (backend, org, repo, relative)):
console(
"Not working in a repo: "
f"backend={backend} org={org} repo={repo} relative={relative}"
)
self.good = False
return
if verbose == 1:
console(
f"Working in repository {org}/{repo}{relative} in backend {backend}"
)
base = ex(f"~/{backend}")
repoDir = f"{base}/{org}/{repo}"
refDir = f"{repoDir}{relative}"
programDir = f"{refDir}/programs"
schemaDir = f"{refDir}/schema"
convertSpec = f"{programDir}/tei.yaml"
convertCustom = f"{programDir}/tei.py"
self.schemaDir = schemaDir
settings = readYaml(asFile=convertSpec, plain=True)
customKeys = set(
"""
transform
beforeTag
beforeChildren
afterChildren
afterTag
""".strip().split()
)
functionType = type(lambda x: x)
if fileExists(convertCustom):
hooked = []
try:
spec = util.spec_from_file_location("teicustom", convertCustom)
code = util.module_from_spec(spec)
sys.path.insert(0, dirNm(convertCustom))
spec.loader.exec_module(code)
sys.path.pop(0)
for method in customKeys:
if not hasattr(code, method):
continue
func = getattr(code, method)
typeFunc = type(func)
if typeFunc is not functionType:
console(
f"custom member {method} should be a function, "
f"but it is a {typeFunc.__name__}"
)
continue
methodC = f"{method}Custom"
setattr(self, methodC, func)
hooked.append(method)
except Exception as e:
print(str(e))
for method in customKeys:
if not hasattr(self, method):
methodC = f"{method}Custom"
setattr(self, methodC, None)
if verbose >= 0:
console("With custom behaviour hooked in at:")
for method in hooked:
methodC = f"{method}Custom"
console(f"\t{methodC} = {ux(convertCustom)}.{method}")
generic = settings.get("generic", {})
extra = settings.get("extra", {})
models = settings.get("models", [])
templates = settings.get("templates", [])
templateTrigger = settings.get("templateTrigger", None)
adaptations = settings.get("adaptations", [])
adaptationTrigger = settings.get("adaptationTrigger", None)
prelim = settings.get("prelim", True)
wordAsSlot = settings.get("wordAsSlot", True)
parentEdges = settings.get("parentEdges", True)
siblingEdges = settings.get("siblingEdges", True)
procins = settings.get("procins", False)
pageModel = settings.get("pageModel", {})
pageModel = checkModel("page", pageModel)
if not pageModel:
self.good = False
return
makePageElems = pageModel["model"] == "II"
pageProperties = pageModel.get("properties", None)
pageModel = pageModel["model"]
self.makePageElems = makePageElems
self.pageModel = pageModel
self.pageProperties = pageProperties
sectionModel = settings.get("sectionModel", {})
sectionModel = checkModel("section", sectionModel)
if not sectionModel:
self.good = False
return
sectionProperties = sectionModel.get("properties", None)
sectionModel = sectionModel["model"]
self.sectionModel = sectionModel
self.sectionProperties = sectionProperties
self.generic = generic
self.extra = extra
self.models = models
self.templates = templates
self.adaptations = adaptations
if templateTrigger is None:
self.templateAtt = None
self.templateTag = None
else:
(tag, att) = templateTrigger.split("@")
self.templateAtt = att
self.templateTag = tag
if adaptationTrigger is None:
self.adaptationAtt = None
self.adaptationTag = None
else:
(tag, att) = adaptationTrigger.split("@")
self.adaptationAtt = att
self.adaptationTag = tag
templateTag = self.templateTag
templateAtt = self.templateAtt
adaptationTag = self.adaptationTag
adaptationAtt = self.adaptationAtt
triggers = {}
self.triggers = triggers
for (kind, theAtt, theTag) in (
("template", templateAtt, templateTag),
("adaptation", adaptationAtt, adaptationTag),
):
triggerRe = None
if theAtt is not None and theTag is not None:
tagPat = re.escape(theTag)
triggerRe = re.compile(
rf"""<{tagPat}\b[^>]*?{theAtt}=['"]([^'"]+)['"]"""
)
triggers[kind] = triggerRe
self.A = Analysis(verbose=verbose)
self.readSchemas()
self.prelim = prelim
self.wordAsSlot = wordAsSlot
self.parentEdges = parentEdges
self.siblingEdges = siblingEdges
self.procins = procins
reportDir = f"{refDir}/report"
appDir = f"{refDir}/app"
docsDir = f"{refDir}/docs"
teiDir = f"{refDir}/tei"
tfDir = f"{refDir}/tf"
teiVersions = sorted(dirContents(teiDir)[1], key=versionSort)
nTeiVersions = len(teiVersions)
if tei in {"latest", "", "0", 0} or str(tei).lstrip("-").isdecimal():
teiIndex = (0 if tei == "latest" else int(tei)) - 1
try:
teiVersion = teiVersions[teiIndex]
except Exception:
absIndex = teiIndex + (nTeiVersions if teiIndex < 0 else 0) + 1
console(
(
f"no item in {absIndex} in {nTeiVersions} source versions "
f"in {ux(teiDir)}"
)
if len(teiVersions)
else f"no source versions in {ux(teiDir)}",
error=True,
)
self.good = False
return
else:
teiVersion = tei
teiPath = f"{teiDir}/{teiVersion}"
reportPath = f"{reportDir}/{teiVersion}"
if not dirExists(teiPath):
console(
f"source version {teiVersion} does not exists in {ux(teiDir)}",
error=True,
)
self.good = False
return
teiStatuses = {tv: i for (i, tv) in enumerate(reversed(teiVersions))}
teiStatus = teiStatuses[teiVersion]
teiStatusRep = (
"most recent"
if teiStatus == 0
else "previous"
if teiStatus == 1
else f"{teiStatus - 1} before previous"
)
if teiStatus == len(teiVersions) - 1 and len(teiVersions) > 1:
teiStatusRep = "oldest"
if verbose >= 0:
console(f"TEI data version is {teiVersion} ({teiStatusRep})")
tfVersions = sorted(dirContents(tfDir)[1], key=versionSort)
if prelim:
tfVersions = [tv for tv in tfVersions if tv.endswith(PRE)]
latestTfVersion = (
tfVersions[-1] if len(tfVersions) else ("0.0.0" + (PRE if prelim else ""))
)
if tf in {"latest", "", "0", 0}:
tfVersion = latestTfVersion
vRep = "latest"
elif tf in {"1", "2", "3", 1, 2, 3}:
bump = int(tf)
parts = latestTfVersion.split(".")
def getVer(b):
return (
int(parts[b].removesuffix(PRE))
if prelim and b == len(parts) - 1
else int(parts[b])
)
def setVer(b, val):
parts[b] = f"{val}{PRE}" if prelim and b == len(parts) - 1 else f"{val}"
if bump > len(parts):
console(
f"Cannot bump part {bump} of latest TF version {latestTfVersion}",
error=True,
)
self.good = False
return
else:
b1 = bump - 1
old = getVer(b1)
setVer(b1, old + 1)
for b in range(b1 + 1, len(parts)):
setVer(b, 0)
tfVersion = ".".join(str(p) for p in parts)
vRep = (
"major" if bump == 1 else "intermediate" if bump == 2 else "minor"
)
vRep = f"next {vRep}"
else:
tfVersion = tf
status = "existing" if dirExists(f"{tfDir}/{tfVersion}") else "new"
vRep = f"explicit {status}"
tfPath = f"{tfDir}/{tfVersion}"
if verbose >= 0:
console(f"TF data version is {tfVersion} ({vRep})")
console(
f"Processing instructions are {'treated' if procins else 'ignored'}"
)
self.refDir = refDir
self.teiVersion = teiVersion
self.teiPath = teiPath
self.tfVersion = tfVersion
self.tfPath = tfPath
self.reportPath = reportPath
self.tfDir = tfDir
self.appDir = appDir
self.docsDir = docsDir
self.backend = backend
self.org = org
self.repo = repo
self.relative = relative
levelNames = sectionProperties["levels"]
self.levelNames = levelNames
self.chunkLevel = levelNames[-1]
if sectionModel == "II":
self.chapterSection = levelNames[0]
self.chunkSection = levelNames[1]
else:
self.folderSection = levelNames[0]
self.fileSection = levelNames[1]
self.chunkSection = levelNames[2]
self.backMatter = sectionProperties.get("backMatter", None)
chunkSection = self.chunkSection
intFeatures = {"empty", chunkSection}
self.intFeatures = intFeatures
if siblingEdges:
intFeatures.add("sibling")
slotType = WORD if wordAsSlot else CHAR
self.slotType = slotType
sectionFeatures = ",".join(levelNames)
sectionTypes = ",".join(levelNames)
textFeatures = "{str}{after}" if wordAsSlot else "{ch}"
otext = {
"fmt:text-orig-full": textFeatures,
"sectionFeatures": sectionFeatures,
"sectionTypes": sectionTypes,
}
self.otext = otext
featureMeta = dict(
str=dict(
description="the text of a word or token",
conversionMethod=CM_LITC,
conversionCode=CONVERSION_METHODS[CM_LITC],
),
after=dict(
description="the text after a word till the next word",
conversionMethod=CM_LITC,
conversionCode=CONVERSION_METHODS[CM_LITC],
),
empty=dict(
description="whether a slot has been inserted in an empty element",
conversionMethod=CM_PROV,
conversionCode=CONVERSION_METHODS[CM_PROV],
),
is_meta=dict(
description="whether a slot or word is in the teiHeader element",
conversionMethod=CM_LITC,
conversionCode=CONVERSION_METHODS[CM_LITC],
),
is_note=dict(
description="whether a slot or word is in the note element",
conversionMethod=CM_LITC,
conversionCode=CONVERSION_METHODS[CM_LITC],
),
)
if not wordAsSlot:
featureMeta["extraspace"] = dict(
description=(
"whether a space has been added after a character, "
"when it is in the direct child of a pure XML element"
),
conversionMethod=CM_LITP,
conversionCode=CONVERSION_METHODS[CM_LITP],
)
featureMeta["ch"] = dict(
description="the unicode character of a slot",
conversionMethod=CM_LITC,
conversionCode=CONVERSION_METHODS[CM_LITC],
)
if parentEdges:
featureMeta["parent"] = dict(
description="edge between a node and its parent node",
conversionMethod=CM_LITP,
conversionCode=CONVERSION_METHODS[CM_LITP],
)
if siblingEdges:
featureMeta["sibling"] = dict(
description=(
"edge between a node and its preceding sibling nodes; "
"labelled with the distance between them"
),
conversionMethod=CM_LITP,
conversionCode=CONVERSION_METHODS[CM_LITP],
)
featureMeta[chunkSection] = dict(
description=f"number of a {chunkSection} within a document",
conversionMethod=CM_PROV,
conversionCode=CONVERSION_METHODS[CM_PROV],
)
if sectionModel == "II":
chapterSection = self.chapterSection
featureMeta[chapterSection] = dict(
description=f"name of {chapterSection}",
conversionMethod=CM_PROV,
conversionCode=CONVERSION_METHODS[CM_PROV],
)
else:
folderSection = self.folderSection
fileSection = self.fileSection
featureMeta[folderSection] = dict(
description=f"name of source {folderSection}",
conversionMethod=CM_PROV,
conversionCode=CONVERSION_METHODS[CM_PROV],
)
featureMeta[fileSection] = dict(
description=f"name of source {fileSection}",
conversionMethod=CM_PROV,
conversionCode=CONVERSION_METHODS[CM_PROV],
)
self.featureMeta = featureMeta
generic["sourceFormat"] = "TEI"
generic["version"] = tfVersion
generic["teiVersion"] = teiVersion
generic["schema"] = "TEI" + (" + " + (" + ".join(models))) if models else ""
extraInstructions = []
for (feat, featSpecs) in extra.items():
featMeta = featSpecs.get("meta", {})
if "valueType" in featMeta:
if featMeta["valueType"] == "int":
intFeatures.add(feat)
del featMeta["valueType"]
featPath = featSpecs.get("path", None)
featPathRep = "" if featPath is None else "the content is taken from "
featPathLogical = []
sep = ""
for comp in reversed(featPath):
if type(comp) is str:
featPathRep += f"{sep}{comp}"
featPathLogical.append((comp, None))
else:
for (tag, atts) in comp.items():
# there is only one item in this dict
featPathRep += f"{sep}{tag}["
featPathRep += ",".join(
f"{att}={v}" for (att, v) in sorted(atts.items())
)
featPathRep += "]"
featPathLogical.append((tag, atts))
sep = "/"
featureMeta[feat] = {
k: v.replace("«base»", featPathRep) for (k, v) in featMeta.items()
}
nodeType = featSpecs.get("nodeType", None)
if nodeType is not None and featPath:
extraInstructions.append(
(list(reversed(featPathLogical)), nodeType, feat)
)
self.extraInstructions = tuple(extraInstructions)
self.verbose = verbose
self.validate = validate
myDir = dirNm(abspath(__file__))
self.myDir = myDir
def readSchemas(self):
schemaDir = self.schemaDir
models = self.models
A = self.A
schemaFiles = dict(rng={}, xsd={})
self.schemaFiles = schemaFiles
modelInfo = {}
self.modelInfo = modelInfo
modelXsd = {}
self.modelXsd = modelXsd
modelInv = {}
self.modelInv = modelInv
for model in [None] + models:
for kind in ("rng", "xsd"):
schemaFile = (
A.getBaseSchema()[kind]
if model is None
else f"{schemaDir}/{model}.{kind}"
)
if fileExists(schemaFile):
schemaFiles[kind][model] = schemaFile
if (
kind == "rng"
or kind == "xsd"
and model not in schemaFiles["rng"]
):
modelInfo[model] = schemaFile
if model in schemaFiles["rng"] and model not in schemaFiles["xsd"]:
schemaFileXsd = f"{schemaDir}/{model}.xsd"
A.fromrelax(schemaFiles["rng"][model], schemaFileXsd)
schemaFiles["xsd"][model] = schemaFileXsd
baseSchema = schemaFiles["xsd"][None]
modelXsd[None] = baseSchema
modelInv[(baseSchema, None)] = None
for model in models:
override = schemaFiles["xsd"][model]
modelXsd[model] = override
modelInv[(baseSchema, override)] = model
def getSwitches(self, xmlPath):
verbose = self.verbose
models = self.models
adaptations = self.adaptations
templates = self.templates
triggers = self.triggers
A = self.A
text = None
found = {}
for (kind, allOfKind) in (
("model", models),
("adaptation", adaptations),
("template", templates),
):
if text is None:
with open(xmlPath, encoding="utf8") as fh:
text = fh.read()
found[kind] = None
if kind == "model":
result = A.getModel(text)
if result is None or result == "tei_all":
result = None
else:
result = None
triggerRe = triggers[kind]
if triggerRe is not None:
match = triggerRe.search(text)
result = match.group(1) if match else None
if result is not None and result not in allOfKind:
if verbose >= 0:
console(f"unavailable {kind} {result} in {ux(xmlPath)}")
result = None
found[kind] = result
return (found["model"], found["adaptation"], found["template"])
def getParser(self):
"""Configure the lxml parser.
See [parser options](https://lxml.de/parsing.html#parser-options).
Returns
-------
object
A configured lxml parse object.
"""
procins = self.procins
return etree.XMLParser(
remove_blank_text=False,
collect_ids=False,
remove_comments=True,
remove_pis=not procins,
huge_tree=True,
)
def getXML(self):
"""Make an inventory of the TEI source files.
Returns
-------
tuple of tuple | string
If section model I is in force:
The outer tuple has sorted entries corresponding to folders under the
TEI input directory.
Each such entry consists of the folder name and an inner tuple
that contains the file names in that folder, sorted.
If section model II is in force:
It is the name of the single XML file.
"""
verbose = self.verbose
teiPath = self.teiPath
sectionModel = self.sectionModel
if verbose == 1:
console(f"Section model {sectionModel}")
if sectionModel == "I":
backMatter = self.backMatter
IGNORE = "__ignore__"
xmlFilesRaw = collections.defaultdict(list)
with scanDir(teiPath) as dh:
for folder in dh:
folderName = folder.name
if folderName == IGNORE:
continue
if not folder.is_dir():
continue
with scanDir(f"{teiPath}/{folderName}") as fh:
for file in fh:
fileName = file.name
if not (
fileName.lower().endswith(".xml") and file.is_file()
):
continue
xmlFilesRaw[folderName].append(fileName)
xmlFiles = []
hasBackMatter = False
for folderName in sorted(xmlFilesRaw):
if folderName == backMatter:
hasBackMatter = True
else:
fileNames = xmlFilesRaw[folderName]
xmlFiles.append((folderName, tuple(sorted(fileNames))))
if hasBackMatter:
fileNames = xmlFilesRaw[backMatter]
xmlFiles.append((backMatter, tuple(sorted(fileNames))))
xmlFiles = tuple(xmlFiles)
return xmlFiles
if sectionModel == "II":
xmlFile = None
with scanDir(teiPath) as fh:
for file in fh:
fileName = file.name
if not (fileName.lower().endswith(".xml") and file.is_file()):
continue
xmlFile = fileName
break
return xmlFile
def checkTask(self):
"""Implementation of the "check" task.
It validates the TEI, but only if a schema file has been passed explicitly
when constructing the `TEI()` object.
Then it makes an inventory of all elements and attributes in the TEI files.
If tags are used in multiple namespaces, it will be reported.
!!! caution "Conflation of namespaces"
The TEI to TF conversion does construct node types and attributes
without taking namespaces into account.
However, the parsing process is namespace aware.
The inventory lists all elements and attributes, and many attribute values.
But is represents any digit with `n`, and some attributes that contain
ids or keywords, are reduced to the value `x`.
This information reduction helps to get a clear overview.
It writes reports to the `reportPath`:
* `errors.txt`: validation errors
* `elements.txt`: element/attribute inventory.
"""
if not self.good:
return
verbose = self.verbose
procins = self.procins
validate = self.validate
modelInfo = self.modelInfo
modelInv = self.modelInv
modelXsd = self.modelXsd
A = self.A
teiPath = self.teiPath
reportPath = self.reportPath
docsDir = self.docsDir
sectionModel = self.sectionModel
if verbose == 1:
console(f"TEI to TF checking: {ux(teiPath)} => {ux(reportPath)}")
if verbose >= 0:
console(
f"Processing instructions are {'treated' if procins else 'ignored'}"
)
console(f"XML validation will be {'performed' if validate else 'skipped'}")
kindLabels = dict(
format="Formatting Attributes",
keyword="Keyword Attributes",
rest="Remaining Attributes and Elements",
)
getStore = lambda: collections.defaultdict( # noqa: E731
lambda: collections.defaultdict(collections.Counter)
)
analysis = {x: getStore() for x in kindLabels}
errors = []
tagByNs = collections.defaultdict(collections.Counter)
refs = collections.defaultdict(lambda: collections.Counter())
ids = collections.defaultdict(lambda: collections.Counter())
parser = self.getParser()
baseSchema = modelXsd[None]
overrides = [
override for (model, override) in modelXsd.items() if model is not None
]
A.getElementInfo(baseSchema, overrides, verbose=verbose)
elementDefs = A.elementDefs
initTree(reportPath)
initTree(docsDir)
nProcins = 0
def analyse(root, analysis, xmlFile):
FORMAT_ATTS = set(
"""
dim
level
place
rend
""".strip().split()
)
KEYWORD_ATTS = set(
"""
facs
form
function
lang
reason
type
unit
who
""".strip().split()
)
TRIM_ATTS = set(
"""
id
key
target
value
""".strip().split()
)
NUM_RE = re.compile(r"""[0-9]""", re.S)
def nodeInfo(xnode):
nonlocal nProcins
if procins and isinstance(xnode, etree._ProcessingInstruction):
target = xnode.target
tag = f"?{target}"
ns = ""
nProcins += 1
else:
qName = etree.QName(xnode.tag)
tag = qName.localname
ns = qName.namespace
atts = {etree.QName(k).localname: v for (k, v) in xnode.attrib.items()}
tagByNs[tag][ns] += 1
if len(atts) == 0:
kind = "rest"
analysis[kind][tag][""][""] += 1
else:
idv = atts.get("id", None)
if idv is not None:
ids[xmlFile][idv] += 1
for (refAtt, targetFile, targetId) in getRefs(tag, atts, xmlFile):
refs[xmlFile][(targetFile, targetId)] += 1
for (k, v) in atts.items():
kind = (
"format"
if k in FORMAT_ATTS
else "keyword"
if k in KEYWORD_ATTS
else "rest"
)
dest = analysis[kind]
if kind == "rest":
vTrim = "X" if k in TRIM_ATTS else NUM_RE.sub("N", v)
dest[tag][k][vTrim] += 1
else:
words = v.strip().split()
for w in words:
dest[tag][k][w.strip()] += 1
for child in xnode.iterchildren(
tag=(etree.Element, etree.ProcessingInstruction)
if procins
else etree.Element
):
nodeInfo(child)
nodeInfo(root)
def writeErrors():
"""Write the errors to a file."""
errorFile = f"{reportPath}/errors.txt"
nErrors = 0
nFiles = 0
with open(errorFile, "w", encoding="utf8") as fh:
prevFolder = None
prevFile = None
for (folder, file, line, col, kind, text) in errors:
newFolder = prevFolder != folder
newFile = newFolder or prevFile != file
if newFile:
nFiles += 1
if kind == "error":
nErrors += 1
indent1 = f"{folder}\n\t" if newFolder else "\t"
indent2 = f"{file}\n\t\t" if newFile else "\t"
loc = f"{line or ''}:{col or ''}"
text = "\n".join(wrap(text, width=80, subsequent_indent="\t\t\t"))
fh.write(f"{indent1}{indent2}{loc} {kind or ''} {text}\n")
prevFolder = folder
prevFile = file
if nErrors:
console(
f"{nErrors} validation error(s) in {nFiles} file(s) "
f"written to {errorFile}"
)
else:
if verbose >= 0:
if validate:
console("Validation OK")
else:
console("No validation performed")
def writeNamespaces():
errorFile = f"{reportPath}/namespaces.txt"
nErrors = 0
nTags = len(tagByNs)
with open(errorFile, "w", encoding="utf8") as fh:
for (tag, nsInfo) in sorted(
tagByNs.items(), key=lambda x: (-len(x[1]), x[0])
):
label = "OK"
nNs = len(nsInfo)
if nNs > 1:
nErrors += 1
label = "XX"
for (ns, amount) in sorted(
nsInfo.items(), key=lambda x: (-x[1], x[0])
):
fh.write(
f"{label} {nNs:>2} namespace for "
f"{tag:<16} : {amount:>5}x {ns}\n"
)
if procins:
plural = "" if nProcins == 1 else "s"
console(f"{nProcins} processing instruction{plural} encountered.")
console(
f"{nTags} tags of which {nErrors} with multiple namespaces "
f"written to {errorFile}"
if verbose >= 0 or nErrors
else "Namespaces OK"
)
def writeReport():
reportFile = f"{reportPath}/elements.txt"
with open(reportFile, "w", encoding="utf8") as fh:
fh.write(
"Inventory of tags and attributes in the source XML file(s).\n"
"Contains the following sections:\n"
)
for label in kindLabels.values():
fh.write(f"\t{label}\n")
fh.write("\n\n")
infoLines = 0
def writeAttInfo(tag, att, attInfo):
nonlocal infoLines
nl = "" if tag == "" else "\n"
tagRep = "" if tag == "" else f"<{tag}>"
attRep = "" if att == "" else f"{att}="
atts = sorted(attInfo.items())
(val, amount) = atts[0]
fh.write(
f"{nl}\t{tagRep:<18} " f"{attRep:<11} {amount:>5}x {val}\n"
)
infoLines += 1
for (val, amount) in atts[1:]:
fh.write(
f"""\t{'':<7}{'':<18} {'"':<18} {amount:>5}x {val}\n"""
)
infoLines += 1
def writeTagInfo(tag, tagInfo):
nonlocal infoLines
tags = sorted(tagInfo.items())
(att, attInfo) = tags[0]
writeAttInfo(tag, att, attInfo)
infoLines += 1
for (att, attInfo) in tags[1:]:
writeAttInfo("", att, attInfo)
for (kind, label) in kindLabels.items():
fh.write(f"\n{label}\n")
for (tag, tagInfo) in sorted(analysis[kind].items()):
writeTagInfo(tag, tagInfo)
if verbose >= 0:
console(f"{infoLines} info line(s) written to {reportFile}")
def writeElemTypes():
elemsCombined = {}
modelSet = set()
for (schemaOverride, eDefs) in elementDefs.items():
model = modelInv[schemaOverride]
modelSet.add(model)
for (tag, (typ, mixed)) in eDefs.items():
elemsCombined.setdefault(tag, {}).setdefault(model, {})
elemsCombined[tag][model]["typ"] = typ
elemsCombined[tag][model]["mixed"] = mixed
tagReport = {}
for (tag, tagInfo) in elemsCombined.items():
tagLines = []
tagReport[tag] = tagLines
if None in tagInfo:
teiInfo = tagInfo[None]
teiTyp = teiInfo["typ"]
teiMixed = teiInfo["mixed"]
teiTypRep = "??" if teiTyp is None else typ
teiMixedRep = (
"??" if teiMixed is None else "mixed" if teiMixed else "pure"
)
mds = ["TEI"]
for model in sorted(x for x in tagInfo if x is not None):
info = tagInfo[model]
typ = info["typ"]
mixed = info["mixed"]
if typ == teiTyp and mixed == teiMixed:
mds.append(model)
else:
typRep = (
"" if typ == teiTyp else "??" if typ is None else typ
)
mixedRep = (
""
if mixed == teiMixed
else "??"
if mixed is None
else "mixed"
if mixed
else "pure"
)
tagLines.append((tag, [model], typRep, mixedRep))
tagLines.insert(0, (tag, mds, teiTypRep, teiMixedRep))
else:
for model in sorted(tagInfo):
info = tagInfo[model]
typ = info["typ"]
mixed = info["mixed"]
typRep = "??" if typ is None else typ
mixedRep = (
"??" if mixed is None else "mixed" if mixed else "pure"
)
tagLines.append((tag, [model], typRep, mixedRep))
reportFile = f"{reportPath}/types.txt"
with open(reportFile, "w", encoding="utf8") as fh:
for tag in sorted(tagReport):
tagLines = tagReport[tag]
for (tag, mds, typ, mixed) in tagLines:
model = ",".join(mds)
fh.write(f"{tag:<18} {model:<18} {typ:<7} {mixed:<5}\n")
if verbose >= 0:
console(
f"{len(elemsCombined)} tag(s) type info written to {reportFile}"
)
def writeIdRefs():
reportIdFile = f"{reportPath}/ids.txt"
reportRefFile = f"{reportPath}/refs.txt"
ih = open(reportIdFile, "w", encoding="utf8")
rh = open(reportRefFile, "w", encoding="utf8")
refdIds = collections.Counter()
missingIds = set()
totalRefs = 0
totalRefsU = 0
totalResolvable = 0
totalResolvableU = 0
totalDangling = 0
totalDanglingU = 0
seenItems = set()
for (file, items) in refs.items():
rh.write(f"{file}\n")
resolvable = 0
resolvableU = 0
dangling = 0
danglingU = 0
for (item, n) in sorted(items.items()):
totalRefs += n
if item in seenItems:
newItem = False
else:
seenItems.add(item)
newItem = True
totalRefsU += 1
(target, idv) = item
if target not in ids or idv not in ids[target]:
status = "dangling"
dangling += n
if newItem:
missingIds.add((target, idv))
danglingU += 1
else:
status = "ok"
resolvable += n
refdIds[(target, idv)] += n
if newItem:
resolvableU += 1
rh.write(f"\t{status:<10} {n:>5} x {target} # {idv}\n")
msgs = (
f"\tDangling: {dangling:>4} x {danglingU:>4}",
f"\tResolvable: {resolvable:>4} x {resolvableU:>4}",
)
for msg in msgs:
rh.write(f"{msg}\n")
totalResolvable += resolvable
totalResolvableU += resolvableU
totalDangling += dangling
totalDanglingU += danglingU
if verbose >= 0:
console(f"Refs written to {reportRefFile}")
msgs = (
f"\tresolvable: {totalResolvableU:>4} in {totalResolvable:>4}",
f"\tdangling: {totalDanglingU:>4} in {totalDangling:>4}",
f"\tALL: {totalRefsU:>4} in {totalRefs:>4} ",
)
for msg in msgs:
console(msg)
totalIds = 0
totalIdsU = 0
totalIdsM = 0
totalIdsRefd = 0
totalIdsRefdU = 0
totalIdsUnused = 0
for (file, items) in ids.items():
totalIds += len(items)
ih.write(f"{file}\n")
unique = 0
multiple = 0
refd = 0
refdU = 0
unused = 0
for (item, n) in sorted(items.items()):
nRefs = refdIds.get((file, item), 0)
if n == 1:
unique += 1
else:
multiple += 1
if nRefs == 0:
unused += 1
else:
refd += nRefs
refdU += 1
status1 = f"{n}x"
plural = "" if nRefs == 1 else "s"
status2 = f"{nRefs}ref{plural}"
ih.write(f"\t{status1:<8} {status2:<8} {item}\n")
msgs = (
f"\tUnique: {unique:>4}",
f"\tNon-unique: {multiple:>4}",
f"\tUnused: {unused:>4}",
f"\tReferenced: {refd:>4} x {refdU:>4}",
)
for msg in msgs:
ih.write(f"{msg}\n")
totalIdsU += unique
totalIdsM += multiple
totalIdsRefdU += refdU
totalIdsRefd += refd
totalIdsUnused += unused
if verbose >= 0:
console(f"Ids written to {reportIdFile}")
msgs = (
f"\treferenced: {totalIdsRefdU:>4} by {totalIdsRefd:>4}",
f"\tnon-unique: {totalIdsM:>4}",
f"\tunused: {totalIdsUnused:>4}",
f"\tALL: {totalIdsU:>4} in {totalIds:>4}",
)
for msg in msgs:
console(msg)
def writeDoc():
teiUrl = "https://tei-c.org/release/doc/tei-p5-doc/en/html"
elUrlPrefix = f"{teiUrl}/ref-"
attUrlPrefix = f"{teiUrl}/REF-ATTS.html#"
docFile = f"{docsDir}/elements.md"
with open(docFile, "w", encoding="utf8") as fh:
fh.write(
dedent(
"""
# Element and attribute inventory
Table of contents
"""
)
)
for label in kindLabels.values():
labelAnchor = label.replace(" ", "-")
fh.write(f"*\t[{label}](#{labelAnchor})\n")
fh.write("\n")
tableHeader = dedent(
"""
| element | attribute | value | amount
| --- | --- | --- | ---
"""
)
def writeAttInfo(tag, att, attInfo):
tagRep = " " if tag == "" else f"[{tag}]({elUrlPrefix}{tag}.html)"
attRep = " " if att == "" else f"[{att}]({attUrlPrefix}{att})"
atts = sorted(attInfo.items())
(val, amount) = atts[0]
valRep = f"`{val}`" if val else ""
fh.write(
"| "
+ (
" | ".join(
str(x)
for x in (
tagRep,
attRep,
valRep,
amount,
)
)
)
+ "\n"
)
for (val, amount) in atts[1:]:
valRep = f"`{val}`" if val else ""
fh.write(f"""| | | {valRep} | {amount}\n""")
def writeTagInfo(tag, tagInfo):
tags = sorted(tagInfo.items())
(att, attInfo) = tags[0]
writeAttInfo(tag, att, attInfo)
for (att, attInfo) in tags[1:]:
writeAttInfo("", att, attInfo)
for (kind, label) in kindLabels.items():
fh.write(f"## {label}\n{tableHeader}")
for (tag, tagInfo) in sorted(analysis[kind].items()):
writeTagInfo(tag, tagInfo)
fh.write("\n")
def filterError(msg):
return msg == (
"Element 'graphic', attribute 'url': [facet 'pattern'] "
"The value '' is not accepted by the pattern '\\S+'."
)
def doXMLFile(xmlPath):
tree = etree.parse(xmlPath, parser)
root = tree.getroot()
xmlFile = baseNm(xmlPath)
ids[xmlFile][""] = 1
analyse(root, analysis, xmlFile)
xmlFilesByModel = collections.defaultdict(list)
if sectionModel == "I":
i = 0
for (xmlFolder, xmlFiles) in self.getXML():
console(f"Start folder {xmlFolder}:")
j = 0
cr = ""
nl = True
for xmlFile in xmlFiles:
i += 1
j += 1
if j > 20:
cr = "\r"
nl = False
xmlPath = f"{teiPath}/{xmlFolder}/{xmlFile}"
(model, adapt, tpl) = self.getSwitches(xmlPath)
mdRep = model or "TEI"
tplRep = tpl or ""
adRep = adapt or ""
label = f"{mdRep:<12} {tplRep:<12} {adRep:<12}"
console(f"{cr}{i:>4} {label} {xmlFile:<50}", newline=nl)
xmlFilesByModel[model].append(xmlPath)
console("")
console(f"End folder {xmlFolder}")
elif sectionModel == "II":
xmlFile = self.getXML()
if xmlFile is None:
console("No XML files found!")
return False
xmlPath = f"{teiPath}/{xmlFile}"
(model, adapt, tpl) = self.getSwitches(xmlPath)
xmlFilesByModel[model].append(xmlPath)
good = True
for (model, xmlPaths) in xmlFilesByModel.items():
if verbose >= 0:
console(f"{len(xmlPaths)} {model or 'TEI'} file(s) ...")
thisGood = True
if validate:
console("\tValidating ...")
schemaFile = modelInfo.get(model, None)
if schemaFile is None:
console(f"\t\tNo schema file for {model}")
if good is not None and good is not False:
good = None
continue
(thisGood, info, theseErrors) = A.validate(schemaFile, xmlPaths)
for line in info:
if verbose >= 0:
console(f"\t\t{line}")
if not thisGood:
good = False
errors.extend(theseErrors)
if verbose >= 0:
console("\tMaking inventory ...")
for xmlPath in xmlPaths:
doXMLFile(xmlPath)
if not good:
self.good = False
console("")
writeErrors()
writeReport()
writeElemTypes()
writeDoc()
writeNamespaces()
writeIdRefs()
# SET UP CONVERSION
def getConverter(self):
"""Initializes a converter.
Returns
-------
object
The `tf.convert.walker.CV` converter object, initialized.
"""
verbose = self.verbose
tfPath = self.tfPath
silent = AUTO if verbose == 1 else TERSE if verbose == 0 else DEEP
TF = Fabric(locations=tfPath, silent=silent)
return CV(TF, silent=silent)
# DIRECTOR
def getDirector(self):
"""Factory for the director function.
The `tf.convert.walker` relies on a corpus dependent `director` function
that walks through the source data and spits out actions that
produces the TF dataset.
The director function that walks through the TEI input must be conditioned
by the properties defined in the TEI schema and the customised schema, if any,
that describes the source.
Also some special additions need to be programmed, such as an extra section
level, word boundaries, etc.
We collect all needed data, store it, and define a local director function
that has access to this data.
Returns
-------
function
The local director function that has been constructed.
"""
TEI_HEADER = "teiHeader"
TEXT_ANCESTOR = "text"
TEXT_ANCESTORS = set(
"""
front
body
back
group
""".strip().split()
)
CHUNK_PARENTS = TEXT_ANCESTORS | {TEI_HEADER}
CHUNK_ELEMS = set(
"""
facsimile
fsdDecl
sourceDoc
standOff
""".strip().split()
)
PASS_THROUGH = set(
"""
TEI
""".strip().split()
)
# CHECKING
HY = "\u2010" # hyphen
IN_WORD_HYPHENS = {HY, "-"}
procins = self.procins
verbose = self.verbose
teiPath = self.teiPath
wordAsSlot = self.wordAsSlot
parentEdges = self.parentEdges
siblingEdges = self.siblingEdges
featureMeta = self.featureMeta
intFeatures = self.intFeatures
transform = getattr(self, "transformCustom", None)
chunkLevel = self.chunkLevel
modelInv = self.modelInv
modelInfo = self.modelInfo
modelXsd = self.modelXsd
A = self.A
transformFunc = (
(lambda x: BytesIO(x.encode("utf-8")))
if transform is None
else lambda x: BytesIO(transform(x).encode("utf-8"))
)
parser = self.getParser()
baseSchema = modelInfo[None]
overrides = [
override for (model, override) in modelInfo.items() if model is not None
]
baseSchema = modelXsd[None]
overrides = [
override for (model, override) in modelXsd.items() if model is not None
]
A.getElementInfo(baseSchema, overrides, verbose=-1)
refs = collections.defaultdict(lambda: collections.defaultdict(set))
ids = collections.defaultdict(dict)
# WALKERS
WHITE_TRIM_RE = re.compile(r"\s+", re.S)
NON_NAME_RE = re.compile(r"[^a-zA-Z0-9_]+", re.S)
NOTE_LIKE = set(
"""
note
""".strip().split()
)
EMPTY_ELEMENTS = set(
"""
addSpan
alt
anchor
anyElement
attRef
binary
caesura
catRef
cb
citeData
classRef
conversion
damageSpan
dataFacet
default
delSpan
elementRef
empty
equiv
fsdLink
gb
handShift
iff
lacunaEnd
lacunaStart
lb
link
localProp
macroRef
milestone
move
numeric
param
path
pause
pb
ptr
redo
refState
specDesc
specGrpRef
symbol
textNode
then
undo
unicodeProp
unihanProp
variantEncoding
when
witEnd
witStart
""".strip().split()
)
NEWLINE_ELEMENTS = set(
"""
ab
addrLine
cb
l
lb
lg
list
p
pb
seg
table
u
""".strip().split()
)
CONTINUOUS_ELEMENTS = set(
"""
choice
""".strip().split()
)
def makeNameLike(x):
return NON_NAME_RE.sub("_", x).strip("_")
def walkNode(cv, cur, xnode):
"""Internal function to deal with a single element.
Will be called recursively.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
The subdict `cur["node"]` is used to store the currently generated
nodes by node type.
xnode: object
An lxml element node.
"""
if procins and isinstance(xnode, etree._ProcessingInstruction):
target = xnode.target
tag = f"?{target}"
else:
tag = etree.QName(xnode.tag).localname
atts = {etree.QName(k).localname: v for (k, v) in xnode.attrib.items()}
beforeTag(cv, cur, xnode, tag, atts)
cur[XNEST].append((tag, atts))
curNode = beforeChildren(cv, cur, xnode, tag, atts)
if curNode is not None:
if parentEdges:
if len(cur[TNEST]):
parentNode = cur[TNEST][-1]
cv.edge(curNode, parentNode, parent=None)
cur[TNEST].append(curNode)
if siblingEdges:
if len(cur[TSIB]):
siblings = cur[TSIB][-1]
nSiblings = len(siblings)
for (i, sib) in enumerate(siblings):
cv.edge(sib, curNode, sibling=nSiblings - i)
siblings.append(curNode)
cur[TSIB].append([])
for child in xnode.iterchildren(
tag=(etree.Element, etree.ProcessingInstruction)
if procins
else etree.Element
):
walkNode(cv, cur, child)
afterChildren(cv, cur, xnode, tag, atts)
if curNode is not None:
xmlFile = cur["xmlFile"]
for (refAtt, targetFile, targetId) in getRefs(tag, atts, xmlFile):
refs[refAtt][(targetFile, targetId)].add(curNode)
idVal = atts.get("id", None)
if idVal is not None:
ids[xmlFile][idVal] = curNode
if len(cur[TNEST]):
cur[TNEST].pop()
if siblingEdges:
if len(cur[TSIB]):
cur[TSIB].pop()
cur[XNEST].pop()
afterTag(cv, cur, xnode, tag, atts)
def isChapter(cur):
"""Whether the current element counts as a chapter node.
## Model I
Not relevant: there are no chapter nodes inside an XML file.
## Model II
Chapters are the highest section level (the only lower level is chunks).
Chapters come in two kinds:
* the TEI header;
* the immediate children of `<text>`
except `<front>`, `<body>`, `<back>`, `<group>`;
* the immediate children of
`<front>`, `<body>`, `<back>`, `<group>`.
Parameters
----------
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
Returns
-------
boolean
"""
sectionModel = self.sectionModel
if sectionModel == "II":
nest = cur[XNEST]
nNest = len(nest)
if nNest > 0 and nest[-1][0] in EMPTY_ELEMENTS:
return False
outcome = nNest > 0 and (
nest[-1][0] == TEI_HEADER
or (
nNest > 1
and (
nest[-2][0] in TEXT_ANCESTORS
or nest[-2][0] == TEXT_ANCESTOR
and nest[-1][0] not in TEXT_ANCESTORS
)
)
)
if outcome:
cur["chapterElems"].add(nest[-1][0])
return outcome
return False
def isChunk(cur):
"""Whether the current element counts as a chunk node.
It depends on the section model, but also on the template.
Note that we only can have distinct templates if we deal with
multiple files, so only when we are in section model I.
## Model I
Chunks are the lowest section level (the higher levels are folders
and then files)
The default is that chunks are the immediate children of the
`<teiHeader>` and the `<body>`
elements; a few other elements also count as chunks.
However, if `drillDownDivs` is True and if the chunk appears to be
a `<div>` element, we drill further down, until we arrive at a
non-`<div>` element.
But in specific templates we have different rules:
### `bibliolist`:
* The TEI Header is a chunk, and nothing inside the TEI header is a chunk;
* Everything at level 5, except `<listBibl>` is a chunk;
* The children of `<listBibl>` are chunks (the `<bibl>` elements
and a few others), provided they are at level 6.
### `artworklist`
* The TEI Header is a chunk, and nothing inside the TEI header is a chunk;
* Everything at level 5 is a chunk.
## Model II
Chunks are the lowest section level (the only higher level is chapters).
Chunks are the immediate children of the chapters, and they come in two
kinds: the ones that are `<p>` elements, and the rest.
Deviation from this rule:
* If a chapter is a mixed content node, then it is also a chunk.
and its subelements are not chunks
Parameters
----------
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
Returns
-------
boolean
"""
sectionModel = self.sectionModel
nest = cur[XNEST]
nNest = len(nest)
model = cur["model"]
if nNest == 0:
return False
thisTag = nest[-1][0]
if sectionModel == "II":
if nNest == 1:
outcome = False
else:
parentTag = nest[-2][0]
meChptChnk = (
isChapter(cur) and thisTag not in cur["pureElems"][model]
)
if meChptChnk:
outcome = True
elif parentTag == TEI_HEADER:
outcome = True
elif nNest <= 2:
outcome = False
elif parentTag not in cur["pureElems"][model]:
outcome = False
else:
grandParentTag = nest[-3][0]
outcome = (
grandParentTag in TEXT_ANCESTORS
and thisTag not in EMPTY_ELEMENTS
) or (
grandParentTag == TEXT_ANCESTOR
and parentTag not in TEXT_ANCESTORS
)
elif sectionModel == "I":
template = cur["template"]
if template == "bibliolist":
if thisTag == TEI_HEADER:
outcome = True
elif any(n[0] == TEI_HEADER for n in nest[0:-1]):
outcome = False
elif nNest not in {5, 6}:
outcome = False
else:
parentTag = nest[-2][0]
if nNest == 5:
outcome = thisTag != "listBibl"
else:
outcome = parentTag == "listBibl"
elif template == "artworklist":
if thisTag == TEI_HEADER:
outcome = True
elif any(n[0] == TEI_HEADER for n in nest[0:-1]):
outcome = False
else:
outcome = nNest == 5
else:
if thisTag in CHUNK_ELEMS:
outcome = True
elif nNest == 1:
outcome = False
else:
sectionProperties = self.sectionProperties
drillDownDivs = sectionProperties["drillDownDivs"]
parentTag = nest[-2][0]
if drillDownDivs:
if thisTag == "div":
outcome = False
else:
dParentTag = None
for ancestor in reversed(nest[0:-1]):
if ancestor[0] != "div":
dParentTag = ancestor[0]
break
outcome = (
dParentTag in CHUNK_PARENTS
and thisTag not in EMPTY_ELEMENTS
) or (
dParentTag == TEXT_ANCESTOR
and thisTag not in TEXT_ANCESTORS
)
else:
outcome = (
parentTag in CHUNK_PARENTS
and thisTag not in EMPTY_ELEMENTS
) or (
parentTag == TEXT_ANCESTOR
and thisTag not in TEXT_ANCESTORS
)
if outcome:
cur["chunkElems"].add(nest[-1][0])
return outcome
def isPure(cur):
"""Whether the current tag has pure content.
Parameters
----------
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
Returns
-------
boolean
"""
nest = cur[XNEST]
model = cur["model"]
return (
len(nest) == 0
or len(nest) > 0
and nest[-1][0] in cur["pureElems"][model]
)
def isEndInPure(cur):
"""Whether the current end tag occurs in an element with pure content.
If that is the case, then it is very likely that the end tag also
marks the end of the current word.
And we should not strip spaces after it.
Parameters
----------
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
Returns
-------
boolean
"""
nest = cur[XNEST]
model = cur["model"]
return len(nest) > 1 and nest[-2][0] in cur["pureElems"][model]
def hasMixedAncestor(cur):
"""Whether the current tag has an ancestor with mixed content.
We use this in case a tag ends in an element with pure content.
We should then add whitespace to separate it from the next
element of its parent.
If the whole stack of element has pure content, we add
a newline, because then we are probably in the TEI header,
and things are most clear if they are on separate lines.
But if one of the ancestors has mixed content, we are typically
in some structured piece of information within running text,
such as change markup. In this case we want to add merely a space.
And we should not strip spaces after it.
Parameters
----------
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
Returns
-------
boolean
"""
nest = cur[XNEST]
model = cur["model"]
return any(n[0] in cur["mixedElems"][model] for n in nest[0:-1])
def hasContinuousAncestor(cur):
"""Whether an ancestor tag is a continuous pure element.
A continuous pure element is an element whose child elements do not
imply word separation, e.g. `<choice>`.
Parameters
----------
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
Returns
-------
boolean
"""
nest = cur[XNEST]
return any(n[0] in CONTINUOUS_ELEMENTS for n in nest[0:-1])
def startWord(cv, cur, ch):
"""Start a word node if necessary.
Whenever we encounter a character, we determine
whether it starts or ends a word, and if it starts
one, this function takes care of the necessary actions.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
ch: string
A single character, the next slot in the result data.
"""
curWord = cur[NODE][WORD]
if not curWord:
prevWord = cur["prevWord"]
if prevWord is not None:
cv.feature(prevWord, after=cur["afterStr"])
if ch is not None:
if wordAsSlot:
curWord = cv.slot()
else:
curWord = cv.node(WORD)
cur[NODE][WORD] = curWord
addSlotFeatures(cv, cur, curWord)
if ch is not None:
cur["wordStr"] += ch
def finishWord(cv, cur, ch, spaceChar):
"""Terminate a word node if necessary.
Whenever we encounter a character, we determine
whether it starts or ends a word, and if it ends
one, this function takes care of the necessary actions.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
ch: string
A single character, the next slot in the result data.
spaceChar: string | void
If None, no extra space or newline will be added.
Otherwise, the spaceChar (a single space or newline will be added).
"""
curWord = cur[NODE][WORD]
if curWord:
cv.feature(curWord, str=cur["wordStr"])
if not wordAsSlot:
cv.terminate(curWord)
cur[NODE][WORD] = None
cur["wordStr"] = ""
cur["prevWord"] = curWord
cur["afterStr"] = ""
if ch is not None:
cur["afterStr"] += ch
if spaceChar is not None:
cur["afterStr"] = cur["afterStr"].rstrip() + spaceChar
if not wordAsSlot:
addSpace(cv, cur, spaceChar)
cur["afterSpace"] = True
else:
cur["afterSpace"] = False
def addEmpty(cv, cur):
"""Add an empty slot.
We also terminate the current word.
If words are slots, the empty slot is a word on its own.
Returns
-------
node
The empty slot
"""
finishWord(cv, cur, None, None)
startWord(cv, cur, ZWSP)
emptyNode = cur[NODE][WORD]
cv.feature(emptyNode, empty=1)
if not wordAsSlot:
emptyNode = cv.slot()
cv.feature(emptyNode, ch=ZWSP, empty=1)
finishWord(cv, cur, None, None)
return emptyNode
def addSlotFeatures(cv, cur, s):
"""Add generic features to a slot.
Whenever we encounter a character, we add it as a new slot, unless
`wordAsSlot` is in force. In that case we suppress the triggering of a
slot node.
If needed, we start/terminate word nodes as well.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
s: slot
A previously added (slot) node
"""
if cur["inHeader"]:
cv.feature(s, is_meta=1)
if cur["inNote"]:
cv.feature(s, is_note=1)
for (r, stack) in cur.get("rend", {}).items():
if len(stack) > 0:
cv.feature(s, **{f"rend_{r}": 1})
def addSlot(cv, cur, ch):
"""Add a slot.
Whenever we encounter a character, we add it as a new slot, unless
`wordAsSlot` is in force. In that case we suppress the triggering of a
slot node.
If needed, we start/terminate word nodes as well.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
ch: string
A single character, the next slot in the result data.
"""
if ch in {"_", None} or ch.isalnum() or ch in IN_WORD_HYPHENS:
startWord(cv, cur, ch)
else:
finishWord(cv, cur, ch, None)
if wordAsSlot:
s = cur[NODE][WORD]
elif ch is None:
s = None
else:
s = cv.slot()
cv.feature(s, ch=ch)
if s is not None:
addSlotFeatures(cv, cur, s)
def addSpace(cv, cur, spaceChar):
"""Adds a space or a new line.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
spaceChar: string
The character to add (supposed to be either a space or a newline).
Only meant for the case where slots are characters.
Suppressed when not in a lowest-level section.
"""
if chunkLevel in cv.activeTypes():
s = cv.slot()
cv.feature(s, ch=spaceChar, extraspace=1)
addSlotFeatures(cv, cur, s)
def endPage(cv, cur):
"""Ends a page node.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
"""
pageProperties = self.pageProperties
pageType = pageProperties["nodeType"]
slots = cv.linked(cur[NODE][pageType])
empty = len(slots) == 0
if empty:
lastSlot = addEmpty(cv, cur)
if cur["inNote"]:
cv.feature(lastSlot, is_note=1)
cv.terminate(cur[NODE][pageType])
def beforeTag(cv, cur, xnode, tag, atts):
"""Actions before dealing with the element's tag.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
xnode: object
An lxml element node.
tag: string
The tag of the lxml node.
"""
beforeTagCustom = getattr(self, "beforeTagCustom", None)
if beforeTagCustom is not None:
beforeTagCustom(cv, cur, xnode, tag, atts)
def beforeChildren(cv, cur, xnode, tag, atts):
"""Actions before dealing with the element's children.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
xnode: object
An lxml element node.
tag: string
The tag of the lxml node.
atts: string
The attributes of the lxml node, with namespaces stripped.
"""
makePageElems = self.makePageElems
if makePageElems:
pageProperties = self.pageProperties
pageType = pageProperties["nodeType"]
isPageContainer = makePageElems and matchModel(
pageProperties, tag, atts
)
inPage = cur["inPage"]
pbAtTop = pageProperties["pbAtTop"]
if isPageContainer:
cur["inPage"] = True
if pbAtTop:
# material before the first pb in the container is not in a page
pass
else:
# the page starts with the container
cur[NODE][pageType] = cv.node(pageType)
sectionModel = self.sectionModel
sectionProperties = self.sectionProperties
if sectionModel == "II":
chapterSection = self.chapterSection
chunkSection = self.chunkSection
if isChapter(cur):
cur["chapterNum"] += 1
cur["prevChapter"] = cur[NODE].get(chapterSection, None)
cur[NODE][chapterSection] = cv.node(chapterSection)
cv.link(cur[NODE][chapterSection], cur["danglingSlots"])
value = {chapterSection: f"{cur['chapterNum']} {tag}"}
cv.feature(cur[NODE][chapterSection], **value)
cur["chunkPNum"] = 0
cur["chunkONum"] = 0
cur["prevChunk"] = cur[NODE].get(chunkSection, None)
cur[NODE][chunkSection] = cv.node(chunkSection)
cv.link(cur[NODE][chunkSection], cur["danglingSlots"])
cur["danglingSlots"] = set()
cur["infirstChunk"] = True
# N.B. A node can count both as chapter and as chunk,
# e.g. a <trailer> sibling of the chapter <div>s
# A trailer has mixed content, so its subelements aren't typical chunks.
if isChunk(cur):
if cur["infirstChunk"]:
cur["infirstChunk"] = False
else:
cur[NODE][chunkSection] = cv.node(chunkSection)
cv.link(cur[NODE][chunkSection], cur["danglingSlots"])
cur["danglingSlots"] = set()
if tag == "p":
cur["chunkPNum"] += 1
cn = cur["chunkPNum"]
else:
cur["chunkONum"] -= 1
cn = cur["chunkONum"]
value = {chunkSection: cn}
cv.feature(cur[NODE][chunkSection], **value)
if matchModel(sectionProperties, tag, atts):
heading = etree.tostring(
xnode, encoding="unicode", method="text", with_tail=False
).replace("\n", " ")
value = {chapterSection: heading}
cv.feature(cur[NODE][chapterSection], **value)
chapterNum = cur["chapterNum"]
console(f"\rchapter {chapterNum:>4} {heading:<50}", newline=False)
else:
chunkSection = self.chunkSection
if isChunk(cur):
cur["chunkNum"] += 1
cur["prevChunk"] = cur[NODE].get(chunkSection, None)
cur[NODE][chunkSection] = cv.node(chunkSection)
cv.link(cur[NODE][chunkSection], cur["danglingSlots"])
cur["danglingSlots"] = set()
value = {chunkSection: cur["chunkNum"]}
cv.feature(cur[NODE][chunkSection], **value)
if tag == TEI_HEADER:
cur["inHeader"] = True
if sectionModel == "II":
value = {chapterSection: "TEI header"}
cv.feature(cur[NODE][chapterSection], **value)
if tag in NOTE_LIKE:
cur["inNote"] = True
finishWord(cv, cur, None, None)
curNode = None
if makePageElems:
if inPage and tag == "pb":
if pbAtTop:
if cur[NODE][pageType] is not None:
endPage(cv, cur)
cur[NODE][pageType] = cv.node(pageType)
if len(atts):
cv.feature(cur[NODE][pageType], **atts)
else:
if cur[NODE][pageType] is not None:
if len(cur["pageAtts"]):
cv.feature(cur[NODE][pageType], **cur["pageAtts"])
endPage(cv, cur)
cur[NODE][pageType] = cv.node(pageType)
cur["pageAtts"] = atts
if tag not in PASS_THROUGH:
cur["afterSpace"] = False
cur[NODE][tag] = cv.node(tag)
curNode = cur[NODE][tag]
if wordAsSlot:
if cur[NODE][WORD]:
cv.link(curNode, [cur[NODE][WORD][1]])
if len(atts):
cv.feature(curNode, **atts)
if "rend" in atts:
rValue = atts["rend"]
r = makeNameLike(rValue)
if r:
cur.setdefault("rend", {}).setdefault(r, []).append(True)
beforeChildrenCustom = getattr(self, "beforeChildrenCustom", None)
if beforeChildrenCustom is not None:
beforeChildrenCustom(cv, cur, xnode, tag, atts)
if not hasattr(xnode, "target") and xnode.text:
textMaterial = WHITE_TRIM_RE.sub(" ", xnode.text)
if isPure(cur):
if textMaterial and textMaterial != " ":
console(
"WARNING: Text material at the start of "
f"pure-content element <{tag}>"
)
stack = "-".join(n[0] for n in cur[XNEST])
console(f"\tElement stack: {stack}")
console(f"\tMaterial: `{textMaterial}`")
else:
for ch in textMaterial:
addSlot(cv, cur, ch)
return curNode
def afterChildren(cv, cur, xnode, tag, atts):
"""Node actions after dealing with the children, but before the end tag.
Here we make sure that the newline elements will get their last slot
having a newline at the end of their `after` feature.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
xnode: object
An lxml element node.
tag: string
The tag of the lxml node.
atts: string
The attributes of the lxml node, with namespaces stripped.
"""
chunkSection = self.chunkSection
makePageElems = self.makePageElems
if makePageElems:
pageProperties = self.pageProperties
pageType = pageProperties["nodeType"]
pageProperties = self.pageProperties
sectionModel = self.sectionModel
if sectionModel == "II":
chapterSection = self.chapterSection
extraInstructions = self.extraInstructions
if len(extraInstructions):
lookupSource(cv, cur, extraInstructions)
isChap = isChapter(cur)
isChnk = isChunk(cur)
afterChildrenCustom = getattr(self, "afterChildrenCustom", None)
if afterChildrenCustom is not None:
afterChildrenCustom(cv, cur, xnode, tag, atts)
if makePageElems:
isPageContainer = matchModel(pageProperties, tag, atts)
inPage = cur["inPage"]
hasFinishedWord = False
if makePageElems and inPage and tag == "pb":
pass
if tag not in PASS_THROUGH:
curNode = cur[TNEST][-1]
slots = cv.linked(curNode)
empty = len(slots) == 0
newLineTag = tag in NEWLINE_ELEMENTS
if (
newLineTag
or isEndInPure(cur)
and not hasContinuousAncestor(cur)
and not empty
and not cur["afterSpace"]
):
spaceChar = "\n" if newLineTag or not hasMixedAncestor(cur) else " "
finishWord(cv, cur, None, spaceChar)
hasFinishedWord = True
slots = cv.linked(curNode)
empty = len(slots) == 0
if empty:
lastSlot = addEmpty(cv, cur)
if cur["inHeader"]:
cv.feature(lastSlot, is_meta=1)
if cur["inNote"]:
cv.feature(lastSlot, is_note=1)
# take care that this empty slot falls under all sections
# for folders and files this is already guaranteed
# We need only to watch out for chapters and chunks
if cur[NODE].get(chunkSection, None) is None:
prevChunk = cur.get("prevChunk", None)
if prevChunk is None:
cur["danglingSlots"].add(lastSlot[1])
else:
cv.link(prevChunk, lastSlot)
if sectionModel == "II":
if cur[NODE].get(chapterSection, None) is None:
prevChapter = cur.get("prevChapter", None)
if prevChapter is None:
cur["danglingSlots"].add(lastSlot[1])
else:
cv.link(prevChapter, lastSlot)
cv.terminate(curNode)
if isChnk:
if not hasFinishedWord:
finishWord(cv, cur, None, "\n")
cv.terminate(cur[NODE][chunkSection])
if sectionModel == "II":
if isChap:
if not hasFinishedWord:
finishWord(cv, cur, None, "\n")
cv.terminate(cur[NODE][chapterSection])
if makePageElems and isPageContainer:
pbAtTop = pageProperties["pbAtTop"]
if pbAtTop:
# the page ends with the container
if cur[NODE][pageType] is not None:
endPage(cv, cur)
else:
# material after the last pb is not in a page
if cur[NODE][pageType] is not None:
cv.delete(cur[NODE][pageType])
cur["inPage"] = False
def afterTag(cv, cur, xnode, tag, atts):
"""Node actions after dealing with the children and after the end tag.
This is the place where we proces the `tail` of an lxml node: the
text material after the element and before the next open/close
tag of any element.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
cur: dict
Various pieces of data collected during walking
and relevant for some next steps in the walk.
xnode: object
An lxml element node.
tag: string
The tag of the lxml node.
atts: string
The attributes of the lxml node, with namespaces stripped.
"""
if tag == TEI_HEADER:
cur["inHeader"] = False
elif tag in NOTE_LIKE:
cur["inNote"] = False
if tag not in PASS_THROUGH:
if "rend" in atts:
rValue = atts["rend"]
r = makeNameLike(rValue)
if r:
cur["rend"][r].pop()
if xnode.tail:
tailMaterial = WHITE_TRIM_RE.sub(" ", xnode.tail)
if isPure(cur):
if tailMaterial and tailMaterial != " ":
elem = cur[XNEST][-1][0]
console(
"WARNING: Text material after "
f"<{tag}> in pure-content element <{elem}>"
)
stack = "-".join(cur[XNEST][0])
console(f"\tElement stack: {stack}-{tag}")
console(f"\tMaterial: `{tailMaterial}`")
else:
for ch in tailMaterial:
addSlot(cv, cur, ch)
afterTagCustom = getattr(self, "afterTagCustom", None)
if afterTagCustom is not None:
afterTagCustom(cv, cur, xnode, tag, atts)
def director(cv):
"""Director function.
Here we program a walk through the TEI sources.
At every step of the walk we fire some actions that build TF nodes
and assign features for them.
Because everything is rather dynamic, we generate fairly standard
metadata for the features, namely a link to the tei website.
Parameters
----------
cv: object
The convertor object, needed to issue actions.
"""
makePageElems = self.makePageElems
if makePageElems:
pageProperties = self.pageProperties
pageType = pageProperties["nodeType"]
sectionModel = self.sectionModel
A = self.A
elementDefs = A.elementDefs
cur = {}
cur["pureElems"] = {
modelInv[schemaOverride]: {
x for (x, (typ, mixed)) in eDefs.items() if not mixed
}
for (schemaOverride, eDefs) in elementDefs.items()
}
cur["mixedElems"] = {
modelInv[schemaOverride]: {
x for (x, (typ, mixed)) in eDefs.items() if mixed
}
for (schemaOverride, eDefs) in elementDefs.items()
}
cur[NODE] = {}
if sectionModel == "I":
folderSection = self.folderSection
fileSection = self.fileSection
i = 0
for (xmlFolder, xmlFiles) in self.getXML():
console(f"Start folder {xmlFolder}:")
cur[NODE][folderSection] = cv.node(folderSection)
value = {folderSection: xmlFolder}
cv.feature(cur[NODE][folderSection], **value)
j = 0
cr = ""
nl = True
for xmlFile in xmlFiles:
i += 1
j += 1
if j > 20:
cr = "\r"
nl = False
cur["xmlFile"] = xmlFile
xmlPath = f"{teiPath}/{xmlFolder}/{xmlFile}"
(model, adapt, tpl) = self.getSwitches(xmlPath)
cur["model"] = model
cur["template"] = tpl
cur["adaptation"] = adapt
modelRep = model or "TEI"
tplRep = tpl or ""
adRep = adapt or ""
label = f"{modelRep:<12} {adRep:<12} {tplRep:<12}"
console(
f"{cr}{i:>4} {label} {xmlFile:<50}",
newline=nl,
)
cur[NODE][fileSection] = cv.node(fileSection)
ids[xmlFile][""] = cur[NODE][fileSection]
value = {fileSection: xmlFile.removesuffix(".xml")}
cv.feature(cur[NODE][fileSection], **value)
if tpl:
cur[NODE][tpl] = cv.node(tpl)
cv.feature(cur[NODE][tpl], **value)
with open(xmlPath, encoding="utf8") as fh:
text = fh.read()
if transformFunc is not None:
text = transformFunc(text)
tree = etree.parse(text, parser)
root = tree.getroot()
if makePageElems:
cur[NODE][pageType] = None
cur["inPage"] = False
cur["pageAtts"] = None
cur[NODE][WORD] = None
cur["inHeader"] = False
cur["inNote"] = False
cur[XNEST] = []
cur[TNEST] = []
cur[TSIB] = []
cur["chunkNum"] = 0
cur["prevChunk"] = None
cur["danglingSlots"] = set()
cur["prevWord"] = None
cur["wordStr"] = ""
cur["afterStr"] = ""
cur["afterSpace"] = True
cur["chunkElems"] = set()
walkNode(cv, cur, root)
addSlot(cv, cur, None)
if tpl:
cv.terminate(cur[NODE][tpl])
cv.terminate(cur[NODE][fileSection])
console("")
console(f"End folder {xmlFolder}")
cv.terminate(cur[NODE][folderSection])
elif sectionModel == "II":
xmlFile = self.getXML()
if xmlFile is None:
console("No XML files found!")
return False
xmlPath = f"{teiPath}/{xmlFile}"
(cur["model"], cur["adaptation"], cur["template"]) = self.getSwitches(
xmlPath
)
with open(f"{teiPath}/{xmlFile}", encoding="utf8") as fh:
cur["xmlFile"] = xmlFile
text = fh.read()
if transformFunc is not None:
text = transformFunc(text)
tree = etree.parse(text, parser)
root = tree.getroot()
if makePageElems:
cur[NODE][pageType] = None
cur["inPage"] = False
cur["pageAtts"] = None
cur[NODE][WORD] = None
cur["inHeader"] = False
cur["inNote"] = False
cur[XNEST] = []
cur[TNEST] = []
cur[TSIB] = []
cur["chapterNum"] = 0
cur["chunkPNum"] = 0
cur["chunkONum"] = 0
cur["prevChunk"] = None
cur["prevChapter"] = None
cur["danglingSlots"] = set()
cur["prevWord"] = None
cur["wordStr"] = ""
cur["afterStr"] = ""
cur["afterSpace"] = True
cur["chunkElems"] = set()
cur["chapterElems"] = set()
for child in root.iterchildren(tag=etree.Element):
walkNode(cv, cur, child)
addSlot(cv, cur, None)
console("")
if verbose >= 0:
console("Resolving links into edges ...")
unresolvedRefs = {}
unresolved = 0
unresolvedUnique = 0
resolved = 0
resolvedUnique = 0
for (att, attRefs) in refs.items():
feature = f"link_{att}"
edgeFeat = {feature: None}
for ((targetFile, targetId), sourceNodes) in attRefs.items():
nSourceNodes = len(sourceNodes)
targetNode = ids[targetFile].get(targetId, None)
if targetNode is None:
unresolvedRefs.setdefault(targetFile, set()).add(targetId)
unresolvedUnique += 1
unresolved += nSourceNodes
else:
for sourceNode in sourceNodes:
cv.edge(sourceNode, targetNode, **edgeFeat)
resolvedUnique += 1
resolved += nSourceNodes
if verbose >= 0:
console(f"\t{resolvedUnique} in {resolved} reference(s) resolved")
if unresolvedRefs:
console(
f"\t{unresolvedUnique} in {unresolved} reference(s): "
"could not be resolved"
)
if verbose == 1:
for (targetFile, targetIds) in sorted(unresolvedRefs.items()):
examples = " ".join(sorted(targetIds)[0:3])
console(f"\t\t{targetFile}: {len(targetIds)} x: {examples}")
for fName in featureMeta:
if not cv.occurs(fName):
cv.meta(fName)
for fName in cv.features():
if fName not in featureMeta:
if fName.startswith("rend_"):
r = fName[5:]
cv.meta(
fName,
description=f"whether text is to be rendered as {r}",
valueType="int",
conversionMethod=CM_LITC,
conversionCode=CONVERSION_METHODS[CM_LITC],
)
intFeatures.add(fName)
elif fName.startswith("link_"):
r = fName[5:]
cv.meta(
fName,
description=(
f"links to node identified by xml:id in attribute {r}"
),
valueType="str",
conversionMethod=CM_LITP,
conversionCode=CONVERSION_METHODS[CM_LITP],
)
else:
cv.meta(
fName,
description=f"this is TEI attribute {fName}",
valueType="str",
conversionMethod=CM_LIT,
conversionCode=CONVERSION_METHODS[CM_LIT],
)
levelConstraints = ["note < chunk, p", "salute < opener, closer"]
if "chapterElems" in cur:
for elem in cur["chapterElems"]:
levelConstraints.append(f"{elem} < chapter")
if "chunkElems" in cur:
for elem in cur["chunkElems"]:
levelConstraints.append(f"{elem} < chunk")
levelConstraints = "; ".join(levelConstraints)
cv.meta("otext", levelConstraints=levelConstraints)
if verbose == 1:
console("source reading done")
return True
return director
def convertTask(self):
"""Implementation of the "convert" task.
It sets up the `tf.convert.walker` machinery and runs it.
Returns
-------
boolean
Whether the conversion was successful.
"""
if not self.good:
return
procins = self.procins
verbose = self.verbose
slotType = self.slotType
generic = self.generic
otext = self.otext
featureMeta = self.featureMeta
makePageElems = self.makePageElems
pageModel = self.pageModel
intFeatures = self.intFeatures
if makePageElems:
pageProperties = self.pageProperties
pbAtTop = pageProperties["pbAtTop"] if makePageElems else None
tfPath = self.tfPath
teiPath = self.teiPath
if verbose == 1:
console(f"TEI to TF converting: {ux(teiPath)} => {ux(tfPath)}")
if makePageElems:
pbRep = (
f" with pb elements at the {'top' if pbAtTop else 'bottom'}"
"of the page"
)
console(f"Page model {pageModel}{pbRep}")
if verbose >= 0:
console(
f"Processing instructions are {'treated' if procins else 'ignored'}"
)
initTree(tfPath, fresh=True, gentle=True)
cv = self.getConverter()
self.good = cv.walk(
self.getDirector(),
slotType,
otext=otext,
generic=generic,
intFeatures=intFeatures,
featureMeta=featureMeta,
generateTf=True,
)
def loadTask(self):
"""Implementation of the "load" task.
It loads the tf data that resides in the directory where the "convert" task
deliver its results.
During loading there are additional checks. If they succeed, we have evidence
that we have a valid TF dataset.
Also, during the first load intensive precomputation of TF data takes place,
the results of which will be cached in the invisible `.tf` directory there.
That makes the TF data ready to be loaded fast, next time it is needed.
Returns
-------
boolean
Whether the loading was successful.
"""
if not self.good:
return
tfPath = self.tfPath
verbose = self.verbose
silent = AUTO if verbose == 1 else TERSE if verbose == 0 else DEEP
if not dirExists(tfPath):
console(f"Directory {ux(tfPath)} does not exist.")
console("No tf found, nothing to load")
self.good = False
return
TF = Fabric(locations=[tfPath], silent=silent)
allFeatures = TF.explore(silent=True, show=True)
loadableFeatures = allFeatures["nodes"] + allFeatures["edges"]
api = TF.load(loadableFeatures, silent=silent)
if api:
if verbose >= 0:
console(f"max node = {api.F.otype.maxNode}")
self.good = True
return
self.good = False
# APP CREATION/UPDATING
def appTask(self, tokenBased=False):
"""Implementation of the "app" task.
It creates/updates a corpus-specific app plus specific documentation files.
There should be a valid TF dataset in place, because some
settings in the app derive from it.
It will also read custom additions that are present in the target app directory.
These files are:
* `about_custom.md`:
A markdown file with specific colofon information about the dataset.
In the generated file, this information will be put at the start.
* `transcription_custom.md`:
A markdown file with specific encoding information about the dataset.
In the generated file, this information will be put at the start.
* `config_custom.yaml`:
A yaml file with config data that will be *merged* into the generated
config.yaml.
* `app_custom.py`:
A python file with named snippets of code to be inserted
at corresponding places in the generated `app.py`
* `display_custom.css`:
Additonal css definitions that will be appended to the generated
`display.css`.
If the TF app for this resource needs custom code, this is the way to retain
that code between automatic generation of files.
Returns
-------
boolean
Whether the operation was successful.
"""
if not self.good:
return
verbose = self.verbose
refDir = self.refDir
myDir = self.myDir
procins = self.procins
wordAsSlot = self.wordAsSlot
parentEdges = self.parentEdges
siblingEdges = self.siblingEdges
sectionModel = self.sectionModel
sectionProperties = self.sectionProperties
tfVersion = self.tfVersion
# key | parentDir | file | template based
# if parentDir is a tuple, the first part is the parentDir of the source
# end the second part is the parentDir of the destination
itemSpecs = (
("about", "docs", "about.md", False),
("trans", ("app", "docs"), "transcription.md", False),
("logo", "app/static", "logo.png", True),
("display", "app/static", "display.css", False),
("config", "app", "config.yaml", False),
("app", "app", "app.py", False),
)
genTasks = {
s[0]: dict(parentDir=s[1], file=s[2], justCopy=s[3]) for s in itemSpecs
}
cssInfo = makeCssInfo()
version = tfVersion.removesuffix(PRE) if tokenBased else tfVersion
def createConfig(sourceText, customText):
text = sourceText.replace("«version»", f'"{version}"')
settings = readYaml(text=text, plain=True)
settings.setdefault("provenanceSpec", {})["branch"] = BRANCH_DEFAULT_NEW
if tokenBased:
if "typeDisplay" in settings and "word" in settings["typeDisplay"]:
del settings["typeDisplay"]["word"]
customSettings = (
{} if not customText else readYaml(text=customText, plain=True)
)
mergeDict(settings, customSettings)
text = writeYaml(settings)
return text
def createDisplay(sourceText, customText):
"""Copies and tweaks the display.css file of an TF app.
We generate css code for a certain text formatting styles,
triggered by `rend` attributes in the source.
"""
css = sourceText.replace("«rends»", cssInfo)
return f"{css}\n\n{customText}\n"
def createApp(sourceText, customText):
"""Copies and tweaks the app.py file of an TF app.
The template app.py provides text formatting functions.
It retrieves text from features, but that is dependent on
the settings of the conversion, in particular whether we have words as
slots or characters.
Depending on that we insert some code in the template.
The template contains the string `F.matérial`, and it will be replaced
by something like
```
F.ch.v(n)
```
or
```
f"{F.str.v(n)}{F.after.v(n)}"
```
That's why the variable `materialCode` in the body gets a rather
unusual value: it is interpreted later on as code.
"""
materialCode = (
"""f'{F.str.v(n) or ""}{F.after.v(n) or ""}'"""
if wordAsSlot or tokenBased
else '''F.ch.v(n) or ""'''
)
rendValues = repr(KNOWN_RENDS)
code = sourceText.replace("F.matérial", materialCode)
code = code.replace('"rèndValues"', rendValues)
hookStartRe = re.compile(r"^# DEF (import|init|extra)\s*$", re.S)
hookEndRe = re.compile(r"^# END DEF\s*$", re.S)
hookInsertRe = re.compile(r"^\t# INSERT (import|init|extra)\s*$", re.S)
custom = {}
section = None
for line in (customText or "").split("\n"):
line = line.rstrip()
if section is None:
match = hookStartRe.match(line)
if match:
section = match.group(1)
custom[section] = []
else:
match = hookEndRe.match(line)
if match:
section = None
else:
custom[section].append(line)
codeLines = []
for line in code.split("\n"):
line = line.rstrip()
match = hookInsertRe.match(line)
if match:
section = match.group(1)
codeLines.extend(custom.get(section, []))
else:
codeLines.append(line)
return "\n".join(codeLines) + "\n"
def createTranscription(sourceText, customText):
"""Copies and tweaks the transcription.md file for a TF corpus."""
org = self.org
repo = self.repo
relative = self.relative
intFeatures = self.intFeatures
extra = self.extra
def metaRep(feat, meta):
valueType = "int" if feat in intFeatures else "str"
description = meta.get("description", "")
extraFieldRep = "\n".join(
f"* `{field}`: {value}"
for (field, value) in meta.items()
if field not in {"description", "valueType"}
)
return dedent(
f"""
{description}
The values of this feature have type {valueType}.
{extraFieldRep}
"""
)
extra = "\n\n".join(
f"## {feat}\n{metaRep(feat, meta)}\n" for (feat, meta) in extra.items()
)
text = (
dedent(
f"""
# Corpus {org} - {repo}{relative}
"""
)
+ tweakTrans(
sourceText,
procins,
wordAsSlot,
parentEdges,
siblingEdges,
tokenBased,
sectionModel,
sectionProperties,
REND_DESC,
extra,
)
+ dedent(
"""
## See also
* [about](about.md)
"""
)
)
return f"{text}\n\n{customText}\n"
def createAbout(sourceText, customText):
org = self.org
repo = self.repo
relative = self.relative
generic = self.generic
if tokenBased:
generic["version"] = version
generic = "\n\n".join(
f"## {key}\n\n{value}\n" for (key, value) in generic.items()
)
return f"{customText}\n\n{sourceText}\n\n" + (
dedent(
f"""
# Corpus {org} - {repo}{relative}
"""
)
+ generic
+ dedent(
"""
## Conversion
Converted from TEI to Text-Fabric
## See also
* [transcription](transcription.md)
"""
)
)
extraRep = " with NLP output " if tokenBased else ""
if verbose >= 0:
console(f"App updating {extraRep} ...")
for (name, info) in genTasks.items():
parentDir = info["parentDir"]
(sourceBit, targetBit) = (
parentDir if type(parentDir) is tuple else (parentDir, parentDir)
)
file = info[FILE]
fileParts = file.rsplit(".", 1)
if len(fileParts) == 1:
fileParts = [file, ""]
(fileBase, fileExt) = fileParts
if fileExt:
fileExt = f".{fileExt}"
targetDir = f"{refDir}/{targetBit}"
itemTarget = f"{targetDir}/{file}"
itemCustom = f"{targetDir}/{fileBase}_custom{fileExt}"
itemPre = f"{targetDir}/{fileBase}_orig{fileExt}"
justCopy = info["justCopy"]
teiDir = f"{myDir}/{sourceBit}"
itemSource = f"{teiDir}/{file}"
# If there is custom info, we do not have to preserve the previous version.
# Otherwise we save the target before overwriting it; # unless it
# has been saved before
preExists = fileExists(itemPre)
targetExists = fileExists(itemTarget)
customExists = fileExists(itemCustom)
msg = ""
if justCopy:
if targetExists:
msg = "(already exists, not overwritten)"
safe = False
else:
msg = "(copied)"
safe = True
else:
if targetExists:
if customExists:
msg = "(generated with custom info)"
else:
if preExists:
msg = "(no custom info, older orginal exists)"
else:
msg = "(no custom info, original preserved)"
fileCopy(itemTarget, itemPre)
else:
msg = "(created)"
initTree(targetDir, fresh=False)
if justCopy:
if safe:
fileCopy(itemSource, itemTarget)
else:
if fileExists(itemSource):
with open(itemSource, encoding="utf8") as fh:
sourceText = fh.read()
else:
sourceText = ""
if fileExists(itemCustom):
with open(itemCustom, encoding="utf8") as fh:
customText = fh.read()
else:
customText = ""
targetText = (
createConfig
if name == "config"
else createApp
if name == "app"
else createDisplay
if name == "display"
else createTranscription
if name == "trans"
else createAbout
if name == "about"
else fileCopy # this cannot occur because justCopy is False
)(sourceText, customText)
with open(itemTarget, "w", encoding="utf8") as fh:
fh.write(targetText)
if verbose >= 0:
console(f"\t{ux(itemTarget):30} {msg}")
if verbose >= 0:
console("Done")
else:
console(f"App updated{extraRep}")
# START the TEXT-FABRIC BROWSER on this CORPUS
def browseTask(self):
"""Implementation of the "browse" task.
It gives a shell command to start the text-fabric browser on
the newly created corpus.
There should be a valid TF dataset and app configuration in place
Returns
-------
boolean
Whether the operation was successful.
"""
if not self.good:
return
org = self.org
repo = self.repo
relative = self.relative
backend = self.backend
tfVersion = self.tfVersion
backendOpt = "" if backend == "github" else f"--backend={backend}"
versionOpt = f"--version={tfVersion}"
versionOpt = ""
try:
run(
(
f"text-fabric {org}/{repo}{relative}:clone --checkout=clone "
f"{versionOpt} {backendOpt}"
),
shell=True,
)
except KeyboardInterrupt:
pass
def task(
self,
check=False,
convert=False,
load=False,
app=False,
apptoken=False,
browse=False,
verbose=None,
validate=None,
):
"""Carry out any task, possibly modified by any flag.
This is a higher level function that can execute a selection of tasks.
The tasks will be executed in a fixed order: check, convert, load, app,
apptoken, browse.
But you can select which one(s) must be executed.
If multiple tasks must be executed and one fails, the subsequent tasks
will not be executed.
Parameters
----------
check: boolean, optional False
Whether to carry out the "check" task.
convert: boolean, optional False
Whether to carry out the "convert" task.
load: boolean, optional False
Whether to carry out the "load" task.
app: boolean, optional False
Whether to carry out the "app" task.
apptoken: boolean, optional False
Whether to carry out the "apptoken" task.
browse: boolean, optional False
Whether to carry out the "browse" task"
verbose: integer, optional -1
Produce no (-1), some (0) or many (1) orprogress and reporting messages
validate: boolean, optional True
Whether to perform XML validation during the check task
Returns
-------
boolean
Whether all tasks have executed successfully.
"""
if verbose is not None:
verboseSav = self.verbose
self.verbose = verbose
if validate is not None:
self.validate = validate
if not self.good:
return False
for (condition, method, kwargs) in (
(check, self.checkTask, {}),
(convert, self.convertTask, {}),
(load, self.loadTask, {}),
(app, self.appTask, {}),
(apptoken, self.appTask, dict(tokenBased=True)),
(browse, self.browseTask, {}),
):
if condition:
method(**kwargs)
if not self.good:
break
if verbose is not None:
self.verbose = verboseSav
return self.good
def main():
(good, tasks, params, flags) = readArgs(
"tf-fromtei", HELP, TASKS, PARAMS, FLAGS, notInAll=TASKS_EXCLUDED
)
if not good:
return False
T = TEI(**params, **flags)
T.task(**tasks, **flags)
return T.good
if __name__ == "__main__":
sys.exit(0 if main() else 1)
Computing file changes ...
