Simulation stopped after 5 model years with strange error message

[l975421700]

Dear developers

May I ask a question about a failed simulation on ollie? I am using esm_tools to run ECHAM6 standalone simulation. I run it with yaml script here: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_m_411_4.9/run_20050101-20050131/scripts/pi_echam6_1m.yaml. The simulation is installed here: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_m_411_4.9.

It was successfully run for 5 year, but in the sixth year, it gives error message as shown here: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_m_411_4.9/run_20050101-20050131/log/pi_m_411_4.9_echam_compute_16562441.log.

The first few error messages are copied below:

Module for impi, version 2018.4.274 loaded.
216: slurmstepd: error: couldn't chdir to `/work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_m_411_4.9/run_20050101-20050131/work': No such file or directory: going to /tmp instead
222: slurmstepd: error: execve(): /tmp/./echam6_0720_4.9: No such file or directory
216: slurmstepd: error: Unable to create TMPDIR [/work/ollie/tmp]: Permission denied
216: slurmstepd: error: Setting TMPDIR to /tmp
218: slurmstepd: error: execve(): /tmp/./echam6_0720_4.9: No such file or directory

I checked that the directory indeed exists. Can you please advise how to solve this issue?

Regards, Qinggang

[denizural]

Hi Qinggang,
this looks to me like an Ollie problem. Recall that we had some issues on Ollie due to high temperature. It looks like the previous run finished at 11:09 but it did not start the next job for 3 hours. The folder which is said to be non-existent exists currently and it looks like it is created earlier. I speculate that this error is due to a technical problem on Ollie.
You can make a warm restart from your previous successful simulation and continue from there.

[l975421700]

Hi Deniz

Thanks a lot. I'll take this suggestion and make a warm start to solve it.

Regards, Qinggang

[l975421700]

Dear developers

I just found that even I warm start another simulation, it will fail with the same error message.

The experiment is installed here: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/test.
The yaml script is here: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/test/scripts/pi_echam6_1d_restart.yaml.
And the error message appears here in the log file: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/test/log/test_echam_compute_20000101-20000131_16562701.log.
The error messages are again copied below:

Module for impi, version 2018.4.274 loaded.
180: slurmstepd: error: couldn't chdir to `/work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/test/run_20000101-20000131/work': No such file or directory: going to /tmp instead
199: slurmstepd: error: execve(): /tmp/./echam6_0711_4.3: No such file or directory
184: slurmstepd: error: execve(): /tmp/./echam6_0711_4.3: No such file or directory
180: slurmstepd: error: Unable to create TMPDIR [/work/ollie/tmp]: Permission denied
180: slurmstepd: error: Setting TMPDIR to /tmp
187: slurmstepd: error: execve(): /tmp/./echam6_0711_4.3: No such file or directory

Could it be because I used too much storage on ollie? Allowed 3TB and used 20TB, I already tried to clean up a bit. But 3TB is far from enough, maybe I should request for more storage.

Regards, Qinggang

[mandresm]

That is definitely a possibility. Maybe you can ask for more storage, but also a good question is if it's expected that your simulations take that much disk space.

[l975421700]

Yes, it is expected. Now I cannot even run a simulation for one month.

[l975421700]

Hi, I can run simulations after cleaning up spaces. So I assume this is an HPC problem rather than related to esm_tools. Unless it happens again even when I do not occupy too much space, this issue is closed for me. Thanks.