v3 API: general XPath 2.0 mechanism, generateDS true reverse mapping, ocrd-filter #21
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I initially published the first form of builtin processor
ocrd-filter
on OCR-D#1240 directly, but since this is new functionality and involves lots of other changes, I rebased and split this off into this for easier reviewing.The idea behind
ocrd-filter
is that the user gets to write powerful XPath expressions as runtime parameters, and the processor takes care of the removal from PAGE (including ReadingOrder update, and optionally saving images for those segments that did get removed for quick visual inspection).To make this as expressive as possible, we need
pixelarea
or concatenatedtextequiv
, but more to come surely)For the former I initially (see first commits) experimented with lxml's builtin
etree.FunctionNamespace
, but this turned out to be quite buggy. (It crashes with segmentation errors if using the global namespace registration with a namespace prefix, even in single-threaded mode. It did work using local namespace registration, though.) I briefly looked at SaxonC-HE, but found it does not allow for extension functions in Python (only in Java). So I ended up with pure-Python elementpath, which is slower, but really powerful – and easy to use.Then I figured it would be really helpful (for ocrd-filter, but also other processors) if our
OcrdPage.revmap
actually did contain a reverse lookup mechanism (from tree node to generated DOM object). And since generateds (after v2.40) does now support that, while it does not have the problems with simple type enums anymore, I decided to try and update ocrd_page_generateds again – and it worked. So now we can really dopage.xpath()
→page.revmap
→page.pcgts
.I placed the first two extension functions under
ocrd_models.xpath_functions
(as we might also want to write some for METS or MODS or whatever), but this is just an idea ATM.Besides more XPath extension functions (e.g. a function for the ratio of foreground pixels when binarized derived images are present) I am also planning on extending generated PcGtsType via user methods directly (e.g. a method for TextEquiv consistency across the hierarchy, and another for Coords projection)...