Staged cable improve resubmission and restart handling

[Whyborn]

Improvements to staged_cable driver

I made some improvements to the staged_cable driver to hopefully make it more easily understandable, and more easily shareable. Summary of changes:

• Changed the way older restarts are handled. Previously, when setting up the current work directory, we would also run through older restart directories to retrieve any "unique" restarts (any with file name not matching a current restart). I've changed it so that at the end of a stage, we grab any "unique" restart from the previous restarts and copy it into the current restart work directory. Quick example: we've finished stage 1, which created restarts climate_rst.nc and cable_rst.nc, which live in {archive}/restart000. We run stage 2, which created restarts casa_rst.nc and cable_nc.nc. We look back at {archive}/restart000, see that there's no new version of climate_rst.nc, so copy that to the work restart directory. The contents of the work restart directory are then moved to {archive}/restart001. This makes restarts much more easily shareable.
• Changed the way the remaining number of runs are set. Previously, on initialisation of the model, the configuration log was read and the environment variable dictating the number of runs was modified. I realised I can set expt.n_runs instead of the environment variable, which is persistent. Now the configuration log work is only done where necessary in setup() and archive```, rather than every initialisation.

Coming with this is the removal of a fair bit of now-dead code.

Edit: One thing I forgot to mention is the handling of potentially new namelists. I would like it to be possible for people adding new science (or perhaps cable configurations I don't know about yet) to be able to use the tool. Unfortunately, I can't know a priori which namelists will be required in these instances. To get around this, I could simply set

self.optional_config_files = glob.glob('*/nml')

rather than the predefined list of namelists, but it feels like a bit of an abuse of process. Thoughts?

[pep8speaks]

Hello @Whyborn! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-12-04 04:17:35 UTC

[coveralls]

coverage: 58.784% (+0.4%) from 58.364%
when pulling 7a399f5 on staged-cable-improve-resubmission-and-restart-handling
into 27aac37 on master.

[aidanheerdegen]

I wanted to add Claire to the reviewers, but she doesn't appear in the list of reviewers.

I've made a team for ACCESS-NRI and this is rectified

[aidanheerdegen]

Looks fine to me, but I've not looked at the logic deeply, just had a skim and commented on things that stood out to me.

[aidanheerdegen]

I think it should be possible to just make links, which will resolve to their original location IIRC.

[Whyborn]

What about the scenario where we want to share a restart directory with someone else, or perhaps even publicly somewhere, so that they can use it as a start point for simulations? I think it's likely enough that the original run would be somewhere that others won't have read permissions to, or even potentially on other machines.

[ccarouge]

I haven't looked into the code changes but I don't see how making a copy solves problems of read permissions? You need read access to be able to copy files. So links would just be as well.

[Whyborn]

I'm thinking of a situation where someone (e.g. ACCESS-NRI) has run an expensive spin-up configuration and want to provide the restarts as a starting point for others in the community. The configuration may have 20+ stages, but the full suite of restarts may contain a restart from the 1st stage, one from the 5th stage and the rest from the most recent stage. A supported payu workflow is to take a restart from elsewhere and effectively plug it in, so payu transparently restarts from this restart (I think this is true- I have memory of this being demonstrated on the training day).

A scenario where I run a configuration to create a starting point for community science. I'll likely save the data to my non-permanent scratch space. I want to take the most recent payu set of restarts and store it someone accessible on gdata. If the necessary older restarts are only known by symlink, then they'll disappear relatively soon and whichever user takes this payu restart may not have access to my scratch space where the symlinked files live.

[aidanheerdegen]

You can use cp with the -L option to "dereference" links, so the destination will contain all the files referenced by links

https://superuser.com/a/216920

[ccarouge]

It looks ok to me.

I had to go read the whole file to remind myself of what was going on here, so I have a comment on something you haven't changed. Feel free to ignore.

In this block of code:

payu/payu/models/staged_cable.py

Lines 178 to 188 in b69b26d

	for namelist in namelists:
	write_target = os.path.join(self.work_input_path, namelist)
	stage_nml = os.path.join(self.control_path, stage_name, namelist)
	if os.path.isfile(os.path.join(self.control_path, namelist)):
	# Instance where there is a master and stage namelist
	with open(stage_nml) as stage_nml_f:
	stage_namelist = f90nml.read(stage_nml_f)
	master_nml = os.path.join(self.control_path, namelist)
	f90nml.patch(master_nml, stage_namelist, write_target)

I think it would be more readable if you define master_nml at the same place as stage_nml and use it in the IF condition on file existence (l.182):

if os.path.isfile(master_nml):