multistatesampler issue during yank lysozyme example

[camattelaer]

Hi

I'm trying to run the T4 lysozyme example with p-xylene in yank. However, during the replica exchange simulation it is throwing an error which i am finding difficult to troubleshoot (see attached message corresponding to the error).
error.txt

At the beginning it mentions KeyError: (1579084505162490359, -6828962564961262775) but i am unaware what this key does or where it comes from.

At the end it also throws Exception: Error loading CUDA module: CUDA_ERROR_UNSUPPORTED_PTX_VERSION (222) which should be related towards an unsupported driver version with the installed cudatoolkit in the conda environment, but openmm works fine on cuda when testing one of their examples (see screenshot).

conda list output below:

# packages in environment at /home/camattelaer/Programs/anaconda3/envs/yank:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
alabaster                 0.7.13             pyhd8ed1ab_0    conda-forge
amberlite                 22.0                     pypi_0    pypi
ambertools                22.0             py37h461e982_3    conda-forge
amberutils                21.0                     pypi_0    pypi
anyio                     3.7.1              pyhd8ed1ab_0    conda-forge
argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0           py37h540881e_2    conda-forge
arpack                    3.7.0                hdefa2d7_2    conda-forge
astunparse                1.6.3              pyhd8ed1ab_0    conda-forge
attrs                     23.2.0             pyh71513ae_0    conda-forge
babel                     2.14.0             pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 2.0.0              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.12.3             pyha770c72_0    conda-forge
bleach                    6.1.0              pyhd8ed1ab_0    conda-forge
blosc                     1.21.5               hc2324a3_1    conda-forge
boost-cpp                 1.74.0               h75c5d50_8    conda-forge
bottleneck                1.3.5            py37hda87dfa_0    conda-forge
brotli-python             1.0.9            py37hd23a5d3_7    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.28.1               hd590300_0    conda-forge
ca-certificates           2024.2.2             hbcca054_0    conda-forge
cerberus                  1.3.5              pyhd8ed1ab_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1           py37h43b0acd_1    conda-forge
cftime                    1.6.2            py37hc105733_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
clusterutils              0.3.1                    py37_1    omnia
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.2.2              pyhd8ed1ab_0    conda-forge
cudatoolkit               10.1.243            h6d9799a_13    conda-forge
curl                      8.8.0                he654da7_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cython                    0.29.32          py37hd23a5d3_0    conda-forge
dataclasses               0.8                pyhc8e2a94_3    conda-forge
debugpy                   1.6.3            py37hd23a5d3_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
docopt                    0.6.2                      py_1    conda-forge
docutils                  0.17.1           py37h89c1867_2    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
fftw                      3.3.10          nompi_hc118613_108    conda-forge
fftw3f                    3.3.4                         2    omnia
freetype                  2.12.1               h267a509_2    conda-forge
hdf4                      4.2.15               h9772cbc_5    conda-forge
hdf5                      1.12.2          nompi_h4df4325_101    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.7                pyhd8ed1ab_0    conda-forge
imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.4           py37h89c1867_0    conda-forge
importlib_metadata        4.11.4               hd8ed1ab_0    conda-forge
importlib_resources       6.0.0              pyhd8ed1ab_0    conda-forge
ipykernel                 6.16.2             pyh210e3f2_0    conda-forge
ipython                   7.33.0           py37h89c1867_0    conda-forge
ipython_genutils          0.2.0              pyhd8ed1ab_1    conda-forge
ipywidgets                8.1.2              pyhd8ed1ab_1    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h0b41bf4_3    conda-forge
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
jupyter                   1.0.0             pyhd8ed1ab_10    conda-forge
jupyter_client            7.4.9              pyhd8ed1ab_0    conda-forge
jupyter_console           6.5.1              pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1           py37h89c1867_0    conda-forge
jupyter_server            1.23.4             pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.3.0              pyhd8ed1ab_1    conda-forge
jupyterlab_widgets        3.0.10             pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4            py37h7cecad7_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
latexcodec                2.0.1              pyh9f0ad1d_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.40                 h55db66e_0    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libaec                    1.1.3                h59595ed_0    conda-forge
libblas                   3.9.0           20_linux64_openblas    conda-forge
libcblas                  3.9.0           20_linux64_openblas    conda-forge
libcurl                   8.8.0                hca28451_0    conda-forge
libdeflate                1.10                 h7f98852_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 13.2.0               h77fa898_7    conda-forge
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            13.2.0               h69a702a_7    conda-forge
libgfortran5              13.2.0               hca663fb_7    conda-forge
libgomp                   13.2.0               h77fa898_7    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblapack                 3.9.0           20_linux64_openblas    conda-forge
libnetcdf                 4.8.1           nompi_h261ec11_106    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.25          pthreads_h413a1c8_0    conda-forge
libpng                    1.6.43               h2797004_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsqlite                 3.45.3               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               hc0a3c3a_7    conda-forge
libtiff                   4.3.0                h0fcbabc_4    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.4.0                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.10.3               hca2bb57_4    conda-forge
libzip                    1.10.1               h2629f0a_3    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              hd590300_1001    conda-forge
markupsafe                2.1.1            py37h540881e_1    conda-forge
matplotlib                3.3.2                         0    conda-forge
matplotlib-base           3.3.2            py37h4f6019d_1    conda-forge
matplotlib-inline         0.1.7              pyhd8ed1ab_0    conda-forge
mdtraj                    1.9.7            py37hcc56668_2    conda-forge
mistune                   3.0.2              pyhd8ed1ab_0    conda-forge
mmpbsa-py                 16.0                     pypi_0    pypi
mpich                     3.2                           4    omnia
mpiplus                   v0.0.2             pyhd8ed1ab_0    conda-forge
nbclassic                 1.0.0              pyhb4ecaf3_1    conda-forge
nbclient                  0.7.0              pyhd8ed1ab_0    conda-forge
nbconvert                 7.6.0              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.6.0              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          7.6.0              pyhd8ed1ab_0    conda-forge
nbformat                  5.8.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.5                  h59595ed_0    conda-forge
nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
netcdf-fortran            4.6.0           nompi_he1eeb6f_102    conda-forge
netcdf4                   1.6.1           nompi_py37hb61e06c_100    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
notebook                  6.5.7              pyha770c72_0    conda-forge
notebook-shim             0.2.4              pyhd8ed1ab_0    conda-forge
numexpr                   2.8.3           py37h85a3170_100    conda-forge
numpy                     1.21.6           py37h976b520_0    conda-forge
numpydoc                  1.7.0              pyhd8ed1ab_0    conda-forge
olefile                   0.47               pyhd8ed1ab_0    conda-forge
openjpeg                  2.5.0                h7d73246_0    conda-forge
openmm                    7.4.2           py37_cuda101_rc_1    omnia
openmmtools               0.20.0                   py37_0    omnia
openmoltools              0.8.7                    py37_0    omnia
openssl                   3.3.0                h4ab18f5_3    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
packmol                   1!18.013                      0    omnia
packmol-memgen            1.2.3rc0                 pypi_0    pypi
pandas                    1.3.5            py37h8c16a72_0  
pandoc                    3.2                  ha770c72_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parmed                    3.4.3            py37hd23a5d3_2    conda-forge
parso                     0.8.4              pyhd8ed1ab_0    conda-forge
pdb4amber                 22.0                     pypi_0    pypi
pdbfixer                  1.6                        py_1    omnia
perl                      5.32.1          7_hd590300_perl5    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.4.0            py37h0f21c89_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
prometheus_client         0.17.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.42             pyha770c72_0    conda-forge
prompt_toolkit            3.0.42               hd8ed1ab_0    conda-forge
psutil                    5.9.3            py37h540881e_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pybtex                    0.24.0             pyhd8ed1ab_2    conda-forge
pybtex-docutils           1.0.2            py37h89c1867_1    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pymbar                    3.0.3            py37h3010b51_3    conda-forge
pyparsing                 3.1.2              pyhd8ed1ab_0    conda-forge
pyrsistent                0.18.1           py37h540881e_1    conda-forge
pysocks                   1.7.1            py37h89c1867_5    conda-forge
pytables                  3.7.0            py37h7d129aa_2    conda-forge
python                    3.7.12          hf930737_100_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.19.1             pyhd8ed1ab_0    conda-forge
python_abi                3.7                     4_cp37m    conda-forge
pytraj                    2.0.6                    pypi_0    pypi
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0              py37h540881e_4    conda-forge
pyzmq                     24.0.1           py37h0c0c2a8_0    conda-forge
qtconsole-base            5.4.4              pyha770c72_0    conda-forge
qtpy                      2.4.1              pyhd8ed1ab_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.32.2             pyhd8ed1ab_0    conda-forge
sander                    22.0                     pypi_0    pypi
scipy                     1.7.3            py37hf2a6cf1_0    conda-forge
send2trash                1.8.3              pyh0d859eb_0    conda-forge
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.2.0                hdb0a2a9_1    conda-forge
sniffio                   1.3.1              pyhd8ed1ab_0    conda-forge
snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sphinx                    5.3.0              pyhd8ed1ab_0    conda-forge
sphinxcontrib-applehelp   1.0.4              pyhd8ed1ab_0    conda-forge
sphinxcontrib-bibtex      2.6.2              pyhd8ed1ab_0    conda-forge
sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
sphinxcontrib-htmlhelp    2.0.1              pyhd8ed1ab_0    conda-forge
sphinxcontrib-jsmath      1.0.1              pyhd8ed1ab_0    conda-forge
sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
sphinxcontrib-serializinghtml 1.1.5              pyhd8ed1ab_2    conda-forge
sqlite                    3.45.3               h2c6b66d_0    conda-forge
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
terminado                 0.17.1             pyh41d4057_0    conda-forge
tinycss2                  1.3.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tornado                   6.2              py37h540881e_0    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.7.1                hd8ed1ab_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
urllib3                   2.2.1              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.10             pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
websocket-client          1.6.1              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.10             pyhd8ed1ab_0    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.9                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxt                1.3.0                hd590300_1    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yank                      0.25.2                   py37_2    omnia
zeromq                    4.3.5                h75354e8_4    conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

Kind regards and thank you for any help

CA

[mikemhenry]

Sorry for the delayed response.

What is the output of python -m openmm.testInstallation

Also, openmm 7.4.2 is quite old at this point (3 years and 7 months). Is there a reason you are using a version that old? Python 3.7 has been EOL'ed for ~4 years and hasn't had security updates in ~1 year. What GPU are you using and can you post the full output of nvidia-smi

It is possible that the simulation that worked didn't use the CUDA platform which is why it didn't throw an error.