Topaz training erro

[wuhucryoem]

When I do a topaz training, it show me there haven't some file or directory, but don't show me the concrete file or directory.Like that:

Traceback (most recent call last):
File "/home/amax/miniconda3/envs/topaz/bin/topaz", line 33, in
sys.exit(load_entry_point('topaz-em==0.2.5', 'console_scripts', 'topaz')())
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/main.py", line 148, in main
args.func(args)
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/train.py", line 695, in main
, save_prefix=save_prefix, use_cuda=use_cuda, output=output)
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/train.py", line 577, in fit_epochs
, use_cuda=use_cuda, output=output)
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/train.py", line 552, in fit_epoch
for X,Y in data_iterator:
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
idx, data = self._get_data()
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data
success, data = self._try_get_data()
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/multiprocessing/queues.py", line 113, in get
return _ForkingPickler.loads(res)
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 289, in rebuild_storage_fd
fd = df.detach()
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
with _resource_sharer.get_connection(self._id) as conn:
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
c = Client(address, authkey=process.current_process().authkey)
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/multiprocessing/connection.py", line 487, in Client
c = SocketClient(address)
File "/home/amax/miniconda3/envs/topaz/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory

[tbepler]

We'll look into this. It looks like this may actually be a multiprocessing error in our data loader, but could you also provide the command you used to run topaz train so we can take a look at it?

[tbepler]

It worked on another computer? Can you share your conda env there and also what OS you're using on each computer?

[wuhucryoem]

This computer, which have this erro, using CentOS Linux release 7.8.2003 (Core). And this computer is before, maybe someone use it to install other package, so that I can't use topaz now.
Another computer, which is ok, using Unbutu 20.04.3.
And conda env is not conda list? I don't konw it means.(sorry)

[tbepler]

Yeah, I meant the conda list for the environment on the computer where it worked. We'll look into this more, but I think it will be difficult for us to diagnose the issue without direct access to the computer. At least it works on your other computer, so hopefully you can just use topaz there in the meantime!

[wuhucryoem]

Ok, the conda list for the environment on the computer where it worked, like taht:
(topaz) wucang@R9000P2021H:~$ conda list

packages in environment at /home/wucang/anaconda3/envs/topaz:

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
_pytorch_select 0.2 gpu_0
blas 1.0 mkl
ca-certificates 2023.05.30 h06a4308_0
certifi 2021.5.30 py36h06a4308_0
cffi 1.14.6 py36h400218f_0
cudatoolkit 11.1.74 h6bb024c_0 nvidia
dataclasses 0.8 pyh4f3eec9_6
freetype 2.12.1 h4a9f257_0
future 0.18.2 py36_1
intel-openmp 2022.1.0 h9e868ea_3769
joblib 1.0.1 pyhd3eb1b0_0
jpeg 9e h5eee18b_1
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libdeflate 1.17 h5eee18b_0
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgfortran-ng 7.5.0 ha8ba4b0_17
libgfortran4 7.5.0 ha8ba4b0_17
libgomp 11.2.0 h1234567_1
libpng 1.6.39 h5eee18b_0
libstdcxx-ng 11.2.0 h1234567_1
libtiff 4.5.0 h6a678d5_2
libuv 1.44.2 h5eee18b_0
libwebp-base 1.2.4 h5eee18b_1
lz4-c 1.9.4 h6a678d5_0
mkl 2020.2 256
mkl-service 2.3.0 py36he8ac12f_0
mkl_fft 1.3.0 py36h54f3939_0
mkl_random 1.1.1 py36h0573a6f_0
ncurses 6.4 h6a678d5_0
ninja 1.10.2 h06a4308_5
ninja-base 1.10.2 hd09550d_5
numpy 1.19.2 py36h54aff64_0
numpy-base 1.19.2 py36hfa32c7d_0
olefile 0.46 py36_0
openjpeg 2.4.0 h3ad879b_0
openssl 1.1.1u h7f8727e_0
pandas 1.1.5 py36ha9443f7_0
pillow 8.3.1 py36h2c7a002_0
pip 21.2.2 py36h06a4308_0
pycparser 2.21 pyhd3eb1b0_0
python 3.6.13 h12debd9_1
python-dateutil 2.8.2 pyhd3eb1b0_0
pytorch 1.8.2 py3.6_cuda11.1_cudnn8.0.5_0 pytorch-lts
pytz 2021.3 pyhd3eb1b0_0
readline 8.2 h5eee18b_0
scikit-learn 0.24.2 py36ha9443f7_0
scipy 1.5.2 py36h0b6359f_0
setuptools 58.0.4 py36h06a4308_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.41.2 h5eee18b_0
threadpoolctl 2.2.0 pyh0d69192_0
tk 8.6.12 h1ccaba5_0
topaz 0.2.5 py_0 tbepler
torchaudio 0.8.2 py36 pytorch-lts
torchvision 0.2.1 py36_0
typing_extensions 4.1.1 pyh06a4308_0
wheel 0.37.1 pyhd3eb1b0_0
xz 5.4.2 h5eee18b_0
zlib 1.2.13 h5eee18b_0
zstd 1.5.5 hc292b87_0

[DarnellGranberry]

I took a look through the package lists and I've listed the unique packages in each conda env below.

CentOS Env with Error

bzip2 1.0.8 h7b6447c_0 defaults
cudatoolkit 11.3.1 h2bc3f7f_2 defaults
ffmpeg 4.3 hf484d3e_0 pytorch
gmp 6.2.1 h295c915_3 defaults
gnutls 3.6.15 he1e5248_0 defaults
lame 3.100 h7b6447c_0 defaults
libiconv 1.16 h7f8727e_2 defaults
libidn2 2.3.4 h5eee18b_0 defaults
libtasn1 4.19.0 h5eee18b_0 defaults
libunistring 0.9.10 h27cfd23_0 defaults
nettle 3.7.3 hbbd107a_1 defaults
openh264 2.1.1 h4ff587b_0 defaults
pytorch 1.10.2 py3.6_cuda11.3_cudnn8.2.0_0 pytorch
pytorch-mutex 1.0 cuda pytorch
torchvision 0.11.3 py36_cu113 pytorch

Functioning Ubuntu Environment

_pytorch_select 0.2 gpu_0
cffi 1.14.6 py36h400218f_0
cudatoolkit 11.1.74 h6bb024c_0 nvidia
ninja 1.10.2 h06a4308_5
ninja-base 1.10.2 hd09550d_5
pycparser 2.21 pyhd3eb1b0_0
pytorch 1.8.2 py3.6_cuda11.1_cudnn8.0.5_0 pytorch-lts
torchaudio 0.8.2 py36 pytorch-lts
torchvision 0.2.1 py36_0

It looks like the main differences are in your CUDA versions but that doesn't seem like a big issue.

[DarnellGranberry]

This error comes after we create the dataloader, as we get items from it, so I don't think it should involve reading any files. If anything, I think there would be an error in writing the output or saving the model.

I just ran the tutorial code in a notebook and it worked fine. Could you perhaps create a new conda environment, install Topaz there, and try again?

[wuhucryoem]

I have tried several times to reinstall Topaz, none of them worked.

[DarnellGranberry]

Could you try running the same command, but setting num_workers to 0 (the training and Torch default)?

[wuhucryoem]

Sorry, I reinstalled the system because I urgently need to use it.