Dependencies: Make installing plugin packages optional

[sphuber]

Fixes #233
Fixes #290

Installing all plugin packages is quite costly and most users are not likely to want to use all codes. Therefore, an optional requirement is created for each code that implements the common workflow interface.

To make it easy to still install all plugin packages, the all_plugins optional requirement group adds the union of all the plugin optional requirements. The downside is that this requires duplicating the requirements and risks getting out of sync.

Built in [all] support was proposed in PEP 426 but this was rejected: https://peps.python.org/pep-0426

[sphuber]

@bosonie what do you think of this proposal? As mentioned in the commit, having to duplicate the requirements in pyproject.toml is not ideal, but as far as I can tell, there is no dynamic way of providing the all_plugins extras, and I think it is necessary to have it because forcing someone to type out all extras if they want support for all plugins is shit UX.

Was thinking about further improvements:

• ✅ Done: Improve the docs to mention the extras (installation is anyway missing and will be adding this soon, see documentation should mention how to install the software #290 )
• ✅ Done: Add custom pre-commit hook that somehow checks the dependency requirements for the extras are consistent, to prevent them getting out-of-sync
• Find a way to catch ModuleNotFoundError that is likely to be thrown if a plugin is not installed and the user tries to use its implementation. Would be great if we could reliably catch this and point to the docs on how to install the prerequisites for a particular plugin
• ✅ Done: Add tests to ensure that certain parts of the code are functioning even without any plugins installed

[sphuber]

@bosonie I have added a pre-commit hook check for the consistency of the all_plugins extras. In addition, I have also added tests to make sure the plugin-independent parts of that package can be imported.

[bosonie]

@sphuber I had a look at all the commits from my phone. It looks all good to me. The strategy is the same I would implement.
I would have loved to have more time to test it, but I will not be able before Saturday.
Up to you if you want to wait (and maybe sort the last point regarding the NonImplementedError) or you prefer to merge and we will sort any possible problem later on.

[sphuber]

@sphuber I had a look at all the commits from my phone. It looks all good to me. The strategy is the same I would implement. I would have loved to have more time to test it, but I will not be able before Saturday. Up to you if you want to wait (and maybe sort the last point regarding the NonImplementedError) or you prefer to merge and we will sort any possible problem later on.

Thanks for having a look @bosonie . It can wait till next week. It would be great to have another pair of eyes on it and if you can test it a bit.

Regarding catching ModuleNotFoundErrors: the tricky question is where to put it. I though about subclassing the WorkflowFactory of aiida-core and catching the ImportError. Since we then have the entry point string, which should include the plugin name, we can automatically format the help message to say the should run pip install aiida-common-workflows[plugin_name]. This would require that all code in ACWF consistently uses the custom factory, but that is doable.

However, this does not catch everything. If the code manually imports a plugin module, e.g., from aiida_siesta import workflows, then this cannot be automatically caught. We would have to wrap this in a try/catch ourselves and print the message. This is certainly doable, but it will require some care on our part to make sure that every time a plugin import is added, it is done within this try/catch.

Thinking about it now, it probably still makes sense to do it, even though it won't be perfect and there is a risk for mistakes slipping through the cracks. But in the majority of cases it will give the user actual useful instructions for them to solve the problem themselves and to avoid them coming to the forums with the same questions. I will add another commit on top here later today I think.

[sphuber]

Ok @bosonie @giovannipizzi , I now also added the last idea of using a custom workflow factory to print pip install instructions when a plugin is requested that is not installed.

This PR is now complete and ready for final review.

[giovannipizzi]

Thanks @sphuber ! I gave a look to the code and your comments - I think it implements all I think it should. I'm approving it, but since @bosonie said he would have given a careful check tomorrow (and I didn't check the code myself) I suggest we wait to merge until next week, in case he has specific comments.

[bosonie]

Hi @sphuber. I've been reviewing the code and I have to report a missing change that is necessary to complete the final feature that raises a better explained MissingEntryPointError.
When running through the CLI, this is the function called to load the workchains:

aiida-common-workflows/src/aiida_common_workflows/plugins/entry_point.py

Lines 34 to 42 in 9a2c9b1

	def load_workflow_entry_point(workflow: str, plugin_name: str):
	"""Load the entry point for the given plugin implementation of a certain common workflow.
	:param workflow: the name of the common workflow.
	:param plugin_name: name of the plugin implementation.
	:return: the workchain class of the plugin implementation of the common workflow.
	"""
	prefix = f'{PACKAGE_PREFIX}.{workflow}.{plugin_name}'
	return entry_point.load_entry_point('aiida.workflows', prefix)

This function must be modified to use the new aiida_common_workflow.plugins.factories.WorkflowFactory.

I'll continue a bit tomorrow to test few things but I believe the rest is all good.

[bosonie]

@sphuber I had a look at the documentation and here you should change the import.

aiida-common-workflows/docs/source/workflows/base/relax/index.rst

Line 39 in 303f3df

from aiida.plugins import WorkflowFactory

[sphuber]

@sphuber I had a look at the documentation and here you should change the import.

aiida-common-workflows/docs/source/workflows/base/relax/index.rst

Line 39 in 303f3df

from aiida.plugins import WorkflowFactory

Done. I also fixed the docs in a separate PR, so now all checks are passing.

[bosonie]

Great! Thanks @sphuber