Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use stochastic restart patterns on rerun #3077

Open
wants to merge 3 commits into
base: develop
Choose a base branch
from

Conversation

WalterKolczynski-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA commented Nov 7, 2024

Description

The stochastic pattern restart files were not being copied into the input directory when restarting the model after a segment/failure. These files are now copied in and the stochini flag set to .true. in the namelist on a rerun.

The files are NOT copied in for non-rerun warm starts.

Also removes the restriction that stochastic physics cannot be run on member 0, as this is desired down the line. Additional settings are added to the fcst and efcs configs to retain the current behavior.

Resolves #2937

Type of change

  • Bug fix (fixes something broken)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

Multi-segment test on Hercules

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

The stochastic pattern restart files were not being copied into the
input directory when restarting the model after a segment/failure.
These files are now copied in. Additional changes appear to be
needed for them to be used by UFS.

The files are NOT copied in for non-rerun warm starts.

Refs NOAA-EMC#2937
Adds the namelist flag so the stochastic pattern restart files
get read when doing a RERUN.

Refs NOAA-EMC#2937
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me. No comments.
Please invite review from @pjpegion @NeilBarton-NOAA

@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Nov 13, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Nov 13, 2024
@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96_atm3DVar FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96_atm3DVar_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d9209edc/logs/2021032412/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_d9209edc/logs/2024022318/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmDA_d9209edc/logs/2021122018/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_d9209edc/logs/2021122012/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem000_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem001_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem002_seg1.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48_S2SWA_gefs_d9209edc

@@ -47,7 +47,19 @@ FV3_postdet() {
|| ( echo "FATAL ERROR: Unable to copy FV3 IC, ABORT!"; exit 1 )
done

if [[ "${RERUN}" != "YES" ]]; then
if [[ "${RERUN}" == "YES" ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bingfu-NOAA @EricSinsky-NOAA Is there a time that we want to start the model with these files? i.e., use them in more ways than a RERUN?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For 00Z, we will stop at 16-day forecast, then warm start the model to run 16-48 days forecast segment. We will need those files at that time besides a RERUN.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Subsequent segments count as RERUNs inside the system.

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 13, 2024
@emcbot
Copy link

emcbot commented Nov 13, 2024

CI Failed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3077


Experiment C48mx500_3DVarAOWCDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:45:45 UTC 2024
Experiment C48mx500_3DVarAOWCDA_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d9209edc/logs/2021032412/gdas_fcst_seg0.log
Experiment C96_atm3DVar_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:45:50 UTC 2024
Experiment C96_atm3DVar_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96_atm3DVar_d9209edc/logs/2021122018/gdas_fcst_seg0.log
Experiment C96C48_hybatmDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:56 UTC 2024
Experiment C96C48_hybatmDA_d9209edc Terminated: *FAIL*
Experiment C96C48_ufs_hybatmDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:57 UTC 2024
Experiment C96C48_ufs_hybatmDA_d9209edc Terminated: *FAIL*
Experiment C96C48_hybatmaerosnowDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:59 UTC 2024
Experiment C96C48_hybatmaerosnowDA_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmDA_d9209edc/logs/2021122018/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_d9209edc/logs/2024022318/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_d9209edc/logs/2021122012/gdas_fcst_seg0.log
Experiment C48_S2SWA_gefs_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Wed Nov 13 15:37:41 UTC 2024
Experiment C48_S2SWA_gefs_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem000_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem001_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem002_seg1.log
Experiment C96_S2SWA_gefs_replay_ics_d9209edc Completed 1 Cycles: *SUCCESS* at Wed Nov 13 17:25:52 UTC 2024
Experiment C48_ATM_d9209edc Completed 2 Cycles: *SUCCESS* at Wed Nov 13 18:45:20 UTC 2024
Experiment C48_S2SW_d9209edc Completed 2 Cycles: *SUCCESS* at Wed Nov 13 23:13:39 UTC 2024

export DO_SKEB="NO"
export DO_SHUM="NO"
export DO_LAND_PERT="NO"
export DO_CA="NO"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this config file being used by mem000 in ensemble?, if so, then DO_CA should be YES.

export DO_SKEB="NO"
export DO_SHUM="NO"
export DO_LAND_PERT="NO"
export DO_CA="NO"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GFS is using CA, so DO_CA should be YES.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

stage atm_stoch.res.nc and ocn_stoch.res.nc during segment forecasts
5 participants