Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v3 API: general XPath 2.0 mechanism, generateDS true reverse mapping, ocrd-filter #21

Open
wants to merge 13 commits into
base: new-processor-api
Choose a base branch
from

Conversation

bertsky
Copy link
Owner

@bertsky bertsky commented Sep 16, 2024

I initially published the first form of builtin processor ocrd-filter on OCR-D#1240 directly, but since this is new functionality and involves lots of other changes, I rebased and split this off into this for easier reviewing.

The idea behind ocrd-filter is that the user gets to write powerful XPath expressions as runtime parameters, and the processor takes care of the removal from PAGE (including ReadingOrder update, and optionally saving images for those segments that did get removed for quick visual inspection).

To make this as expressive as possible, we need

  1. custom functions (like pixelarea or concatenated textequiv, but more to come surely)
  2. XPath 2.0 operators and functions

For the former I initially (see first commits) experimented with lxml's builtin etree.FunctionNamespace, but this turned out to be quite buggy. (It crashes with segmentation errors if using the global namespace registration with a namespace prefix, even in single-threaded mode. It did work using local namespace registration, though.) I briefly looked at SaxonC-HE, but found it does not allow for extension functions in Python (only in Java). So I ended up with pure-Python elementpath, which is slower, but really powerful – and easy to use.

Then I figured it would be really helpful (for ocrd-filter, but also other processors) if our OcrdPage.revmap actually did contain a reverse lookup mechanism (from tree node to generated DOM object). And since generateds (after v2.40) does now support that, while it does not have the problems with simple type enums anymore, I decided to try and update ocrd_page_generateds again – and it worked. So now we can really do page.xpath()page.revmappage.pcgts.

I placed the first two extension functions under ocrd_models.xpath_functions (as we might also want to write some for METS or MODS or whatever), but this is just an idea ATM.

Besides more XPath extension functions (e.g. a function for the ratio of foreground pixels when binarized derived images are present) I am also planning on extending generated PcGtsType via user methods directly (e.g. a method for TextEquiv consistency across the hierarchy, and another for Coords projection)...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant