Shared python environment on network?

[RyanConway91]

I recently tried using a shared network python environment for a PESTPP run across an in-house compute pool (~12 servers). I use:
CALL"X:\X\X\python_env\Scripts\activate.bat" at the start of a .bat that launches all my workers on each machine. It seems to work fine, but randomly some runs fail with a python error (the error I have seen is something about matplotlib not getting a lock on some library). I set up PESTPP to try to rerun failed runs, and all failed runs were re-ran successfully, so I am guessing this error is from having so many workers (~300) calling the python environment at the same time.

So I think I like this approach, it is much easier to make one python environment on the network rather than clone a venv all over my compute pool. Anyone aware of any risks in doing this?

[jtwhite79]

I think most people try to isolate the python env being used at least on the physical host if not even in the individual worker dirs. Have you seen conda pack? A lot of people are using that to help get python shipped out to the workers...otherwise, there used to be these pre-compiled wheels for python on windows that you can use to build a portable python distro (yuck!). @mnfienen probably has the scoop on the last and greatest approach...

[mnfienen]

hey @RyanConway91 - I've done both approaches ... sharing an env across a network and distributing it via copies. In both cases, I highly recommend using conda pack to zip up your conda/mamba made env. Then, you can either unzip it on each worker or path to it.

But....rather than calling the activate.bat script (I agree, in your original post, that sounds like an issue), you can unzip the conda pack directory in a central location and just add X:\X\X\ptyhon_env\bin to the path. Looks like you are on windows so should work like:
SET PATH=X:\X\X\ptyhon_env\bin;%PATH% at the top of your worker.bat script.

[RyanConway91]

Okay, so pointing each worker individually to the python environment is "safer" than initially activating the environment and then launching all workers within that activated environment? I initially did what you suggested above (called the environment in each worker.bat) but found it was much faster to call the environment once when the workers are initially launched. I'll switch back to calling the environment from each worker.bat and see if the errors go away. Thanks!

[mnfienen]

Yeah - I think it's safer because activate.bat does more than just update the path, but if you are running scripts in the background, you really just need to control which command-line based python env is running. I think the activate.bat script may cause some conflicts (a little out of my depth on the details, but I've launched 500 workers in the cloud all pointing to the same python_env/bin directory without issues).

post up if that doesn't do it though

[MoiseRousseau]

My approach is to (re)create a Python virtual environment for each of my agent with virtualenv. You can afterward source the environment and install the minimal dependencies to run your workflow with pip install -r requirements.txt where requirements.txt contains the dependencies. Example of requirements.txt:

matplotlib==3.8.2
pandas==2.2.0

Like this, every agent is independent.