Investigate packaging genie_python [timebox: 2.5 days]

[zsoltkebel]

This Ticket should instead be a discussion based on points raised in comments

I suggested investigating whether we could turn genie_python helper functions into a pip installable package from vcs for better maintenance and clarity.

I suggest revisiting this ticket in planning and deciding priority etc.

Work done

On branch ticket8381-package in the genie_python repository.

• ticket branch created and requirements.txt file trimmed to the package that are used by genie_python code. We had a lot of seemingly unused dependencies (pygame?), some of which were severely outdated (e.g. https://pypi.org/project/ipython_genutils/)
  • pygame might be used for collision avoidance on some instruments. not in genie python itself
• Note that some environment variables have to be set for the CaChannel package to install correctly through pip.

Going further

• Dependencies seemingly unused here might be necessary on instruments, somehow separate this. Idially we don't want genie_python to list any dependency that it doesn't use directly.
• Do we need caRepeater.exe in our Python installation directory? One of the install steps is moving this into there. We already have one at EPICS dir. do we need both? Could not find usage of it within genie_python code.
• Perhaps separate the python install from genie_python helper functions (package)

discussed ~1:21 20/60/24/

planning 00:00 2024/10/31

[FreddieAkeroyd]

Should we consider splitting some parts of genie python that are server specific and/or shared into separate packages e.g. i think we have things like CaChannel wrappers that may be common to both client and server, but scripting stuff is not needed by inst_servers, which could run in their own virtual environment separate from genie python

[FreddieAkeroyd]

regarding caRepeater, it is launched by the EPICS ca.dll which needs to find it in your PATH. As the epics site packages is not in the PATH, i think it got copied to next to python.exe so it would be found. However we now build cachannel with our own epics that i modified to add the directory where ca.dll is loaded from to the internal search path, so it would probably work without it next to python.exe now

[Tom-Willemsen]

This likely needs splitting into a series of tickets - I think this is too big to just do as one ticket.

Some likely subtasks which I think we need/want to do as part of this (in no particular order) - each of which could be a sensible ticket in itself:

• Ensure genie_python can be made public (may need some git history manipulation)
  • See discussion in teams "technical" channel.
• Create an isiscomputinggroup or similar organisation in pypi
  • Or we could use a private pypi server, but for sharing with other groups I think we should prefer to put everything on pypi directly
• Decide our python version support policy
  • What we currently do: just one version, but that's not great for sharing with other groups and limits the benefit that we get from packaging genie_python
  • "Industry standard" for scientific python is SPEC00 which typically covers 3 recent python versions
  • If we need to support old(er) linux distributions e.g. RHEL, that may mean we need to support quite old python versions
  • Something else.
  • n.b. ask mantid, idaas etc what their python version support policy is for comparison
  • And then actually make genie_python follow it & run at least the unit tests against those versions.
• Work out what to do with our customized builds - may need to build and package these ourselves, or contribute our changes upstream:
  • EPICS libraries - complexity as we use local-ish builds - was mostly done in Use epicscorelibs for python modules #8453
  • smslib - currently has a complex build, but complexity could be removed relatively easily. Ask in technical channel on teams if unsure what this means. Genie_python: Use server to send alerts #5570 .
  • ode - we build our own wheels currently. However it's only actually used by collision avoidance monitor, which in turn isn't being used in practice. Best solution may be to remove ODE as a genie_python dependency, and add it as a dependency of a CAM venv explicitly, installing a wheel from the share.
  • readline - we apply a manual patch currently. Our dependency is quite outdated. We may need to search for a compatible, supported alternative. Might no longer be required at all. e.g. https://github.com/pyreadline3/pyreadline3 may help.
  • Ensure these are all installable as "standard" python modules, not using manual robocopies / env variables / relying on compilers or EPICS being present etc
• Ensure our builds & dependencies are .bat-script (and .sh) free, using standard cross-platform python packaging mechanisms not manual robocopies and environment variable sets etc.
• Separate library from distribution.
  • The end result of installing the distribution should look identical to currently, but the sources should reside in two separate repositories - one for the library and one for the distribution.
  • The "library" repo should use the src layout as specified here for it's repository structure.
  • The requirements of the library repo should only be what the genie_python library itself uses
  • The "distribution" repo should contain the requirements.txt for requirements not directly needed by the genie_python library (e.g. things like scipy which we provide for users, but do not directly depend on)
  • Give library and distribution different names so that we don't get confused when we talk about "genie_python" whether we're talking about library or distribution.
• Move to PEP621 (pyproject.toml) for genie_python
  • See ibex_bluesky_core for an example
  • Use setuptools_scm to auto-generate version metadata from git tags as per i_b_c
• Set up build to automatically push new releases of genie_python to pypi
  • Follow pattern ibex_bluesky_core uses for this
• Generate tickets to start making installable packages for other modules (e.g. IocTestFramework) would be a good candidate so that we can use those as dependencies of individual support modules
• Thin out dependencies of the distribution, in principle leaving only things that scientists will want to use. Our code should move towards .venvs, into which we can install whatever we need without making it a global dependency of the distribution
• Sort out genie_python's unit tests to consistently not try to do i/o , and especially not i/o on module import
  • This will dramatically speed up unit tests
• In distribution, get rid of contextlib2 and other python-2 only libraries (after checking/removing usage in other repos)
• Make genie_python's docs better, they exist but are extremely minimal.
• Move distribution creation towards a better mechanism. Environment management mechanism uktena#2

[Tom-Willemsen]

For the reviewer, sorry, this ticket is going to be fiddly to review as it largely involved a split of an existing repo, with git history rewriting, and a very scattered set of preparatory changes all over the place, so there are not "nice" pull requests to look at for this change. I have done my best to explain what needs to be reviewed below.

---

Context

The meat of this ticket was to split the genie_python repo into two (library vs distribution), while doing the necessary git/jenkins/github/pypi bookkeeping:

• genie, the library:
  • An installable pip package, genie_python
  • The genie_python module itself, i.e. the result of from genie_python import genie in python
  • Unit tests and library documentation
• uktena, the distribution:
  • A python.exe at a specific version
  • Many pre-installed libraries, including numpy, scipy, genie_python, ibex_bluesky_core, and various others.
  • The scripts required to bundle this all up and put it on the share in a similar format to before

Having done the rather fiddly splitting work, we then got the following mostly "for free", or at least much more easily than was possible before:

• PEP621 packaging
• PyPI releasing
• CI on GHA (which doesn't depend on the whole distribution, so is much much faster)
• Docs building on GHP (mirroring to shadow)
• etc

Preparatory steps

Some of the preparatory work on which these changes rely (mostly for context in case anyone refers back to this in future - reviewer, you do not need to review these):

• Use epicscorelibs for python modules #8453
• https://github.com/ISISComputingGroup/genie_python_archive/pull/447
• https://github.com/ISISComputingGroup/genie_python_archive/pull/446
• Remove collision avoidance monitor EPICS-inst_servers#400
• https://github.com/ISISComputingGroup/genie_python_archive/pull/448

---

To review docs:

• Check https://github.com/ISISComputingGroup/ibex_developers_manual/wiki/Building-and-installing-uktena
• Check other relevant dev-facing pages have been updated to separate library and distribution where possible

To review uktena:

• Check you can install the uktena python distribution into c:\instrument\apps\python3 following documentation
• Check that the Jenkins CI setup makes sense (simply moved from previous setup - so just check it's "working")
• Check builds are getting put in kits$\ICP
• Check the general "shape" of the installed package is the same as before
• Start an ibex server using uktena and do some basic smoke testing to ensure nothing is "obviously" broken
• Verify that first-time developer instructions have been updated appropriately

To review genie:

• Check that you can install a development version of genie into uktena using pip editable installs
• Check that you can develop using a standalone venv, with just genie installed into it without all of uktena's dependencies. i.e.

git clone https://github.com/IsisComputingGroup/genie.git c:\instrument\dev\genie
cd c:\instrument\dev\genie
python -m venv .venv
.venv\Scripts\activate
python -m pip install -e .[dev]
pytest

• Check that optional dependency groups make sense
• Check that documentation setup works:
  • Should appear on both shadow and github pages
• Check that the github actions CI setup makes sense
• Check that PyPI publishing setup and docs makes sense (i.e. that they generally follow documented best-practice)
• Check metadata in pyproject.toml makes sense
• Verify that no .bat or .sh scripts exist in the genie repository
• Review & merge Remove irrelevant files genie#2 (if not already merged)
• Review & merge Refactor out shared script checker util genie#5 (if not already merged)
• Review & merge Use refactored script checker utils from g_p system_tests#117 (if not already merged)

Follow-up tickets

• Move to better environment management mechanism: Environment management mechanism uktena#2 (i.e. make uktena not a .bat repository)
• Python version support: Python version support policy genie#3
• Remove obsolete backports: Remove old backports & python 2 shims uktena#3
• Split out matplotlib backend Split out matplotlib backend to separate package genie#4
• Various others to be generated over time

---

Post-review

Make a PyPI release of genie_python at release 15.1.0