Packaging of folders before copying/moving/linking #1062
Closed
JanStreffing
started this conversation in
Ideas & Feature Requests
Replies: 1 comment
-
This resulted in a new plugin that tars binary restart files for FESOM2. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I recently started an AWI-CM3 production run on aleph without netcdf FESOM2 restarts. Only binary. Normally we move the binary restart files from the old to the new run folder at the end of a run. It's not viable to copy them out first, because it's too many files. (because time to copy is long and number of files goes towards file limit).
I tried a parallel packing like this:
tar cf - np1280 | pigz -p 30 > archive.tar.gz
. This was quite fast. We could then move this archive out and move the unpacked files to the new work folder.How would this best be done? plugin script, modification of python backend?
Beta Was this translation helpful? Give feedback.
All reactions