Hypothesis testing with PESTPP-DA

[laat0003]

Hi,

I am in the process of running an hypothesis test akin to the one presented latest version of the manual. I get through the first cycle fine but I keep failing during one of the subsequent cycles. I think that it may have to do with realizations failing. Is there a minimum number of realizations required for a cycle? The error I get when the analysis stalls is a get_eigen() error with a realization number not found.

I am also curious about the role of NOPTMAX in the direct hypothesis approach with PESTPP-DA. I have it set at 3, which I know gives me a good result with IES and therefore cycle 0 in the hypothesis test runs as expected. However, when cycle 1 initiates it uses the end of cycle 0 as the starting ensemble, is this correct behavior? If so, wouldn't that invalidate a direct comparison between realizations in different cycles given that they've effectively had more updates. I assumed it would reuse the initial ensemble of cycle 0 and run the smoother through NOPTMAX iterations with the increased hypothesis observation weight for the cycle.

Sorry to be pain with all the questions.
Cheers,
Tariq

[jtwhite79]

actually can you post the .rec and .log file that caused the error? We are aiming to trap issues with pestpp-da before it gets to that point but there are a lot of blindspots with pestpp-da since its the first tool to go beyond the batch-model-run approach.

In concept, to do the direct hypothesis testing with pestpp-da, you would use a weight-cycle table and increase the weight of the observation you want to force toward a given value across each cycle (so cycle 0 would have zero weight for this observation, leading to a standard posterior ensemble). Then the subsequent cycles would start with the previous cycle's parameter ensemble and update it so that the obsval for the observation of interest is being matched better and better. In this way its similar to the pareto mode pest in that this weight change causes pestpp-da to crawl along the trade-off between the fitting the standard observations and fitting the "bad outcome" value for prediction. You could also do this same thing by running a single cycle/pestpp-ies several times with different weight values in the control file. But I think there is some value in crawling from point to point along that tradeoff surface, both in terms of efficiency and coherence along that surface.

In pestpp-da, NOPTMAX is the number of iterations each cycle; pestpp-da also accepts a da_noptmax_schedule which is a two-columns-of-integers ascii file for how many iterations you want for each cycle. So, like pareto mode, you could do 3 iterations for cycle 0 and then maybe 1 or 2 for the following cycles. It will probably take some experimenting to find the right weight progression and the noptmax schedule...

[laat0003]

Thanks for the rapid response. I've used pareto mode a couple of times two years ago so I can appreciate that crawling along the surface is more efficient. Thanks for reminding me about the scheduling on noptmax. Its exactly what is needed in this case, I think. In my eagerness to try this out I must have missed that one.

What is the recommendation for prior data conflict? I can eliminate all conflicts for the prior in cycle 0 but I see a large number of conflicts appear when cycle 1 starts because it is effectively using the posterior of cycle 0. I usually use drop conflicts with IES but was concerned that I wouldn't be able to do direct comparisons with phi if the conflicts differ between cycles.

I've attached the files from a scaled down version of the analysis, which I was running for trouble-shooting purposes after it failed the first time. I'm only using 56 ensemble members here and the error was reproduced albeit much earlier than before.

hd2da.pfm.txt
hd2da.rec.txt

hd2da.rmr.txt

[jtwhite79]

I found the bug and Im testing a fix for it now - it was related to the base realization failing in a previous cycle and then on the next cycle, the initialization realized the base was missing and tried to re-add it...#badtimes

Prior-data conflict is important in sequential just like it is in batch, but I think you want to keep all non-zero weighted obs for the hypothesis testing because if obs start dropping, then that will change the shape of the tradeoff. But this is all a new approach so I dont have a good feel for this stuff yet!

[laat0003]

No need to put yourself out on my behalf. There is no urgency to get the bug sorted. The hypothesis test was just an extension of a project I am working on where the client has some recurring in-house debates about site conceptualisation. I thought the test would help the discussion out in some way. Big thanks again for your time. It is very much appreciated.