Rocoto issues on hera

[JessicaMeixner-NOAA]

Anyone else having rocoto issues on hera? I have a lot of my crons for rocoto break. A job will say "submitting" or "running" forever. You can try to rewind or force complete and there are no changes to the status of the job. I have asked around and havne't found anyone else who's had this problem, but as it just happened to me again, for the 4-5th time the past 2 weeks I thought I'd ask here. I do see that in https://rdhpcs-common-docs.rdhpcs.noaa.gov/wiki/index.php?title=Rocoto_-_The_Workflow_Manager#Rocoto_Best_Practices that it recommends pointing to /apps/rocoto/default/bin/rocotorun i don't know if that will help or not? It seems i'm using the latest version otherwise?

[DavidHuber-NOAA]

Indeed, /apps/rocoto/default/bin/rocotorun is version 1.3.7, which is the latest.

I had a similar issue like that on Hera shortly after it was upgraded to Rocky 8, which was the cause for the upgrade to 1.3.7 across systems. Hera's system Ruby was very slow, so they had to upgrade it. However, newer versions of Ruby were resulting in buggy interactions with the experiment's database and would sometimes corrupt the database, but this would result in "UNAVAILABLE" statuses for jobs. @christopherwharrop-noaa resolved that with upgrades to the libxml, libxml-ruby, and sqlite3 libraries built within the rocoto package and some fixes within the rocoto source code. If you want to try an older version of rocoto without those bug fixes and an older version of Ruby, you can try this:

module use /home/David.Huber/modules
module load rocoto/1.3.6_mine

If that works for you, then perhaps there is still an issue in 1.3.7 that needs to be resolved.

[christopherwharrop-noaa]

Make certain you are using version 1.3.7 which is also the default. If that does not work, please open a ticket (rdhpcs.rocoto.help@noaa.gov) which describes the behavior you are observing and includes the full rocotorun command you are using, including full paths to your XML and database file. If you are seeing the behavior a handful of times in a two week period, it may be difficult to reliably reproduce the conditions causing the behavior, so if you have an active instance of that, save copies of the database file and XML and point to them in your ticket.

[JessicaMeixner-NOAA]

Thanks! If I see this happen again, I will save additional information and open a ticket. I am using 1.3.7, this happened not too long after the switch to that. I did try going back to 1.3.6 to see if that would help fix things at times but it did not.

[christopherwharrop-noaa]

That is great information, thank you. The issue with version 1.3.6 is that it is built against the system default Ruby, which, for reasons we don't understand, is extremely slow and memory intensive. Version 1.3.7 was built against a much newer version of Ruby to address that problem. Updating to the newer Ruby exposed some incompatibilities with a few libraries which then also needed to be updated with a couple minor tweaks to the source.

The bottom line is that if you observed that same problem with 1.3.6, then the enhancements in 1.3.7 are not likely to fix it. That information is very helpful.

One of the challenges is that sometimes problems only appear in production scale workflows and only under very specific, rare, circumstances, making testing and reproducing failures difficult. Anything you can provide that captures the failure "in the moment" and which can be preserved for further testing and inspection is extremely valuable.

[aerorahul]

@JessicaMeixner-NOAA
Are you still experiences issues with Rocoto on Hera?

[christopherwharrop-noaa]

The issues you are experiencing appear to be very hard to reproduce, and we have not received reports from other users. The only way to get any traction on this issue is if we have a reproducer, which means we need you to catch the problem as it is happening and capture the XML and data base files, make copies of them, and immediately open a ticket with pointers to the exact details of the behaviors and pointers to the files. Once the problem has disappeared, it is pretty much impossible to figure out what was happening.

[JessicaMeixner-NOAA]

I have not had time to reproduce the erorr on hera. I've been out most of the last week. I did notice that the less I checked on my runs, the more consistent they were and I didn't have time to continuously restart things, so I just stopped looking and things mostly went better. We have however had a similar issue on hercules for another set-up and have submitted a ticket to rdhpcs.rocoto.help@noaa.gov based on that. If I encounter the issue again on hera I will save the information and submit an additional ticket.

[christopherwharrop-noaa]

What are you doing to "check on your runs"? If there is a relationship between "checking on the runs" and initiation of the behavior, then it would be helpful to know exactly what commands you are running. The bottom line, though, is that unless we catch it in the act, and leave it in the failed state, there is very little hope of ever finding out what the problem is.

[JessicaMeixner-NOAA]

Checking on my runs by using rocotorun/rocotostat commads outside of the crontab. I will save any future issue so that we can help recreate the issue. I have not had a chance to make runs recently due to leave and other tasks, but if/when I run into an issue I will save all relevant information to share. The good news is for hera, I seem to be the only one who has this issue. For hercules - information has been shared - let me know if I need to repost it here.

[christopherwharrop-noaa]

The information for the hercules ticket only has a path to the XML file and doesn't have any reference to the output, or a detailed description of the behavior (i.e. error messages, log output, commands (with output) showing the behavior, etc. Please post that info into the hercules ticket. It might also help to know how you are running the rocotostat command (which options) in case that is relevant. But the most important thing is to catch it in the act and preserve that state so that I can debug it to figure out why job states are not advancing properly.