cuda version is very old: updating to 11.4or newer to allow python installs of cuda-toolkit

[nimzodisaster]

The cuda version doesnt suppoort some newer gpus, ant 10.1 is not as easy to install as the newer versions of cuda which allow conda environment installs of specific cuda versions. e.g. https://anaconda.org/nvidia/cuda-toolkit

How dependent is the code onthsoe specific versions of cuda, and would upgrading the scripts require take much?

[hvgazula]

Hi @nimzodisaster I managed to run the scripts in the newer versions with minimal changes. For more information, see the discussion at #90, and the upgraded code is in https://github.com/MGH-LEMoN/Photo-SynthSeg/tree/synthseg_tf2.15

[nimzodisaster]

Awesome, thanks!
when you already work hard to manage python/package environments, not wanting to change the system's default cuda, nivida drivers is a priority :)

[hvgazula]

Awesome, thanks! when you already work hard to manage python/package environments, not wanting to change the system's default cuda, nivida drivers is a priority :)

Correct. With the changes I made, you should able to move your work to your preferred (newer) versions of python and tensorflow (until 2.15) based on your driver version.

[BBillot]

Sorry for the late answer (I took some holidays), but tit sounds like @hvgazula was able to point you in the right direction! :)

[nimzodisaster]

Hi @nimzodisaster I managed to run the scripts in the newer versions with minimal changes. For more information, see the discussion at #90, and the upgraded code is in https://github.com/MGH-LEMoN/Photo-SynthSeg/tree/synthseg_tf2.15

just a clarification, the commit master...MGH-LEMoN:Photo-SynthSeg:synthseg_tf2.15
is just to get things running on python 3.11.0?

Would you recommend using that requirements file rather than the one you posted in the other thread?

[hvgazula]

The two rek files you are referring to (in the thread and the commit) are the same.

[hvgazula]

Hi @nimzodisaster I managed to run the scripts in the newer versions with minimal changes. For more information, see the discussion at #90, and the upgraded code is in https://github.com/MGH-LEMoN/Photo-SynthSeg/tree/synthseg_tf2.15

just a clarification, the commit master...MGH-LEMoN:Photo-SynthSeg:synthseg_tf2.15 is just to get things running on python 3.11.0?

Would you recommend using that requirements file rather than the one you posted in the other thread?

and not just for python...but also optimized for newer versions of TF (with explicit garbage collection)

[nimzodisaster]

ahh. I am having the hardest time getting those packages installed. Its always dying on tensorrt-lib not finding a compatible version (with pip).
I used pyenv to install python 3.11.0
and used your require:
pip install -r requirements_python3.11.txt

Hi @nimzodisaster I managed to run the scripts in the newer versions with minimal changes. For more information, see the discussion at #90, and the upgraded code is in https://github.com/MGH-LEMoN/Photo-SynthSeg/tree/synthseg_tf2.15

just a clarification, the commit master...MGH-LEMoN:Photo-SynthSeg:synthseg_tf2.15 is just to get things running on python 3.11.0?
Would you recommend using that requirements file rather than the one you posted in the other thread?

and not just for python...but also optimized for newer versions of TF (with explicit garbage collection)

Unfortunately, when I try to pip install that requirement file its get hung up on tensorrt and tensorrt-libsn with either an Bad Params error or a message about no compatible version available.

Im using pyenv for a 3.11.0 python install. Not sure whats going on, but I've been trying this or that for several hours now with no luck.

[hvgazula]

Hi @nimzodisaster! I am sorry you are having issues with installing the modules. It wasn't clear to me what OS you are working in. Also, I don't use pyenv and it is difficult to suggest a solution without looking at the exact error trace. Nevertheless, here's what I found in my notes that may resolve your installation issues (I just tested it again in Rocky Linux/CentOS and it worked alright for me).

conda create -n ss2.15 python=3.11 pip -y
pip install --extra-index-url https://pypi.nvidia.com tensorrt_libs
pip install tensorflow[and-cuda]==2.15 nibabel scipy

Once again, I am sorry about your experience. I shall update the README.md (in my fork) to reflect this installation step.

[hvgazula]

@nimzodisaster I updated the requirements file as well. So, a simple pip install -r requirements_python3.11.txt should do now.

[nimzodisaster]

Thanks! I've just relied on pip, pyenv, and venv, not conda. FYI: Ubuntu 24.04 Nvidia 560 open kernel

No need to say sorry. You are offering help for free and version hell is just part of python

[nimzodisaster]

I cloned your branch (for simplicity's sake) installed anaconda, created the conda environment, activated it, then pip installed the requirement file. And it installed! THANK YOU!

In your opinion is anaconda better for data science installs than venv (virtualenv) for resolving complex dependencies? And I used pip this time, but would it have been better (generally) to use conda install?

[hvgazula]

Hello! With my updated requirements file, I hoped you would simply run pip install -r requirements_python3.11.txt in your pyenv. And that is exactly what I did today. I installed pyenv, created a 3.11 shim followed it up by creating a virtual environment, and finally ran pip install from the requirements file. It all worked beautifully. So, you switching to anaconda is a bit surprising :). If you did so to avoid additional roadblocks, that's understandable.

Coming to your question- when I first started in data science, I used virtualenv/venv/pyenv but it felt too cumbersome. Then anaconda came along and made life easy with the environment handling, package management, etc. and I stuck with it since then. (I could be wrong but) I also remember conflicts with pip install and conda install in the initial days. Today, it has gotten better to the point where I use conda to "create" an environment and pip for installing python modules (I rarely use conda install). Of course, conda can install non-python packages too but that's beside the point. pip is usually good at resolving dependency conflicts barring the odd ones like tensorrt.

Bottomline: With my current experiment using pyenv, I do not see much difference between using pyenv or conda in my pythonic workflow. Would I switch to pyenv? No. Would I recommend you switch to conda? Probably no :). From my POV, they are fundamentally the same but zealots may disagree :). I hope this helps.

If your issue is resolved, please consider closing this. If you have issues with the updated fork, don't hesitate to open a new one (in the fork) and I will respond from there. Good luck. 👍

[nimzodisaster]

Thanks for the solid practical info. Very useful.