New processor API

[bertsky]

This is a first attempt at implementing #322 – without the actual error handling or parallel processing (so more or less refactoring and deprecation). It also enables fixing #579 in the process.

[bertsky]

A couple of points worth discussing:

• In contrast to the formulation in change API to get page-level parallelization everywhere #322, there is no single process_page now. We have too many cases to consider of what processors may need to do.
  I decided to approach this by splitting up into process_page_file (taking an OcrdFile as input and adding the result to the workspace with nothing in return) and process_page_pcgts (taking a parsed OcrdPage as input and returning the modified OcrdPage): The latter must be overriden, the former can (but uses the latter in its default implementation).
  That should be sufficiently general to be useful in adopting the new API, but let's see if we can indeed bring all processors into that paradigm. (For now I just provided the API adaptation for the builtin dummy processor.)
• We have a design choice of providing a trunk definition in the superclass throwing NotImplementedError vs. not having the method at all. The latter can quickly be tested via hasattr, the former must be caught. For the pure PAGE processing function, I think it makes most sense to use the former.
• To avoid the whole chdir business once and for all, I made process_workspace take the workspace as a mandatory argument, and deprecate passing a workspace in the constructor.
• I did not sufficiently separate processing from non-processing context in the constructor, yet. The goal here was to first find a viable path in that direction.
  Thus, I did add a setup method already, but the constructor still determines non-processing or processing context. I believe we should just use methods for that e.g. show_resources instead of taking that as a kwarg in the constructor. But it's an even larger change. (Hence the fixme comments about deprecating this.)
• All of our actual (external) processors use workspace.download_file() on their input files, so that's worth including in the superclass behaviour. However, since most of our processor tests (in core) do not process any actual files, which I did not want to rewrite completely, I had to add an option download_files=False to avoid attempting that.

[MehmedGIT]

Looks like a great start.

[MehmedGIT]

I hope the Pool workers would be also controllable by an environment variable to ease resource distribution in HPC environments.

[bertsky]

That's the idea, exactly.

[bertsky]

BTW, I consider it bad practice that we always load the ocrd-tool.json in the inheriting constructor.

It should really be a class attribute. And assuming it is always placed in the same spot of the Python distribution (which is currently the case but not strictly required as our specs only require it in the repository root), one can even load it in the superclass (so nothing would have to be done in the inheriting code).

[bertsky]

Another backwards compatibility problem is behaviour for --overwrite: our v2 processors use their own Workspace.add_file calls, without setting force=ocrd_utils.config.OCRD_EXISTING_OUTPUT == 'OVERWRITE'. They expect the workspace to implicitly handle that via Workspace.overwrite_mode, which I removed in favour of the config mechanism.

Any ideas how we can best get a workaround for that (at least for some grace period)?

[bertsky]

Another backwards compatibility problem is behaviour for --overwrite: our v2 processors use their own Workspace.add_file calls, without setting force=ocrd_utils.config.OCRD_EXISTING_OUTPUT == 'OVERWRITE'. They expect the workspace to implicitly handle that via Workspace.overwrite_mode, which I removed in favour of the config mechanism.

Perhaps, in v3, instead of using force=ocrd_utils.config.OCRD_EXISTING_OUTPUT == 'OVERWRITE' whereever we can control, we should rather do if force or ocrd_utils.config.OCRD_EXISTING_OUTPUT == 'OVERWRITE' within add_file generally?

EDIT: I did exactly that – in hindsight, it also seems natural.