segmentation fault in post_fv3.f90?

[benjamin-cash]

I just hit the following in one of my test runs, and was wondering if this could be related to a workflow setting I need to change for Frontera:

af read_xml,name=postxconfig-NT.txt ifhr=          36  num_pset=           2
 post_fname=./GFSPRS.GrbF36
 PROCESS starts
 PROCESS MDLFLD done
 PROCESS MDL2P done
 PROCESS MDL2SIGMA done
 PROCESS MDL2SIGMA2 done
 PROCESS MDL2AGL done
 PROCESS SURFCE done
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
libc.so.6          00002B6F8EFDC520  Unknown               Unknown  Unknown
ufs_model.x        0000000003006035  fdlvl_                    397  FDLVL.f
ufs_model.x        0000000002F1AD42  miscln_                   822  MISCLN.f
ufs_model.x        0000000002BD3CD2  process_                  112  PROCESS.f
ufs_model.x        0000000001A01ADC  post_fv3_mp_post_         206  post_fv3.F90
ufs_model.x        00000000019C0282  module_wrt_grid_c        2036  module_wrt_grid_comp.F90

Has anyone run into this error, or have a suggestion as to what I might change to avoid it?

[KateFriedman-NOAA]

@benjamin-cash Can you share the log that the error came from? Does the error happen on reruns of the job?

[benjamin-cash]

fcst_mem010_seg0.log.crash.txt

I'm pretty sure I have had it happen on multiple runs from the same directory.

[benjamin-cash]

@KateFriedman-NOAA - I thought I had included this information in the original post but apparently not - I've been able to run the model out to 48 hours if I set the run length to 48 hours, but when I set it to 744 hours I got this crash in the write step. I haven't tried any other run lengths yet.

[DavidHuber-NOAA]

Looking at the log, it seems there is an instability. I see several messages starting at the first time step like this:

  (zap_snow_temperature)zap_snow_temperature: temperature out of bounds!
  (zap_snow_temperature)k:           1
  (zap_snow_temperature)zTsn:  4.539375917082573E-002
  (zap_snow_temperature)Tmin:  -100.000000000000
  (zap_snow_temperature)Tmax:  1.382988002948323E-004
  (zap_snow_temperature)zqsn:  -110188452.245251

This may be causing indexes inside of the inline post (specifically, FDLVL.f) to point out of the bounds of some arrays (just guessing).

Also, I noticed that the forecast is restarting from the 24-hour restart file. This is probably because the first forecast you ran failed at about the same time (~35 hours), so the 24-hour forecast state was used to restart the forecast.

It would be good to know when these warning messages started to appear in your initial run (starting at forecast hour 0). Would you be able to share the log from the first forecast you ran that failed?

I suspect that the snow parameters are being initialized to a non-physical state, but I may know more if I can see the first log.

[benjamin-cash]

I'm not certain I have the initial log file still - let me make a clean run of this setup for you and I will post that file. I do know that those zap_snow_temperature warnings appear pretty frequently, so I imagine they are there from the beginning.

[benjamin-cash]

@DavidHuber-NOAA - My test run in a clean directory is now out past hour 72, so I think the problem might have been that I tried to keep running in the directory with the previous crash and it attempted to restart. I'm going to go ahead and close this and will re-open if the issue appears again.