UndefRefError on write with JDL2

[bornhoft]

Hi all, I am new to CES, but I am interested in using it for a project. I am working through the tutorials and have come across the same error in both the Sinusoid and Lorenz examples. Specifically, when I run julia calibrate.jl I get the following error:

Error encountered while save FileIO.File{FileIO.DataFormat{:JLD2}, String}("/<userpath>/CalibrateEmulateSample.jl/examples/Sinusoid/output/calibrate_results.jld2").

Fatal error:
ERROR: LoadError: UndefRefError: access to undefined reference

Any assistance would be greatly appreciated. I may have an issue on my build, but thought this would be the best place to receive support.

[odunbar]

Hi @bornhoft thanks for getting in touch and for using the package!
Sorry for the delayed response, i was on vacation and getting back into the swing of things. Basically I did not find your error to be reproducible here so I have a list of some things below and what i did to try to reproduce your error that perhaps you can compare to your own procedure!

Let me know whether this helps!

First impressions and things to try

• you mention calling julia calibrate.jl but did you mean julia --project calibrate.jl? missing the --project flag will use your machines native julia environment, rather than the local one given by the Project.toml file in the Sinusoid directory. I'd highly recommend doing the latter. (see my procedure below). If in doubt of a bad install, delete the localManifest.toml and re-instantiate/build etc.

• Are the packages you use the same as the ones listed below under the package status call? e.g. JLD2 is v0.5.3?

• JLD2 can be brittle, e.g. changing the JLD2/Julia versions between save and load has been know to cause some issues. have you tried removing all the files in output and rerunning, so it really is writing a new "save" file?

• Could you try perhaps commenting out things being saved to see if one in particular is causing an issue. i.e. removing (string,value) pairs, or replacing the saved objects with a value of 1, from here:
  

  CalibrateEmulateSample.jl/examples/Sinusoid/calibrate.jl

  Lines 155 to 170 in e2eba85

  	"inputs",
  	u_stored,
  	"outputs",
  	g_stored,
  	"constrained_inputs",
  	ϕ_stored,
  	"prior",
  	prior,
  	"eki",
  	ensemble_kalman_process,
  	"N_ensemble",
  	N_ensemble,
  	"N_iterations",
  	N_iterations,
  	"rng",
  	rng,

My procedure of trying to reproduce issue (unsuccessfully)

julia version v1.10.5

1. Clone CES
2. Navigate cd CalibrateEmulateSample.jl/examples/Sinusoid/
3. Call julia --project and then

julia> ]
(Sinusoid) > instantiate

4. if PyCall has some issue then do (in-line with our installation instructions)

julia> using Pkg
julia> ENV["PYTHON"]=""
julia> Pkg.build("PyCall")

My package status then gives:

julia> ]
(Sinusoid) pkg> status
Status `~/Dropbox/Caltech/CESjl/tmp/CalibrateEmulateSample.jl/examples/Sinusoid/Project.toml`
  [95e48a1f] CalibrateEmulateSample v0.5.3
  [31c24e10] Distributions v0.25.111
⌅ [aa8a2aa5] EnsembleKalmanProcesses v1.1.7
  [033835bb] JLD2 v0.5.3
  [91a5bcdd] Plots v1.40.8

5. I then run the EKI calibration with

julia> include("calibrate.jl")

(equivalent to julia --project calibrate.jl)
and get output:

Observations:[6.1546719095984805, 6.422583812292371]
Ensemble mean: [3.0513053859865864; 6.3689317470332725;;]
True parameters: [3.0, 7.0]
┌ Warning: Attempting to store EnsembleKalmanProcesses.ParameterDistributions.var"#11#14"{Int64}.
│ JLD2 only stores functions by name.
│  This may not be useful for anonymous functions.
└ @ JLD2 ~/.julia/packages/JLD2/gi4yM/src/data/writing_datatypes.jl:449
... # other warnings

So it produces some kind of JLD2 warnings but runs through. Indeed I can then call

julia> include("emulate.jl")
... # printouts
julia> include("sample.jl")
... # printouts

[bornhoft]

Hi @odunbar, no worries at all. Thanks for the response!

• I was using julia calibrate.jl instead of julia --project calibrate.jl. This didn't seem to have an immediate effect, but it is useful after I got my fix.
• I'm using JLD2 v0.5.4

After trying your first few suggestions, I rebuilt using your approach above directly. The only change I needed to make was to point the ENV[PYTHON]="<my conda python> to my specific conda python. This ended up working without issue.

The instructions I followed were found here. The ones you posted above are slightly different. Maybe this difference had something to do with it? Either way, I have successfully run the Sinusoid and Lorenz examples.

Thank you for the detailed assistance!