@ESMValGroup/esmvaltool-developmentteam any interest?

@matthew-mizielinski, @Jon-Lillis I'm told you might have some candidate workflows?

Yep - I've got a standalone jupyter notebook that demonstrates the problem of failing to to save a large cube. Could be related to this issue, or could just be me being silly. Happy to share it if it helps.

@piotr-florek-mohc, fancy building a CDDS environment with an update to iris, which would be good to do for CDDS v3.0 anyway, and rerunning some of the N512 tests in the SPICE testing?

@trexfeathers, note that our tests will do a fair amount of IO over an extended job (6-12 hours), but is only a single threaded application. We use iris to read data, but the writing out is done via the PCMDI/cmor library.

I ran one of our benchmarks we used for comparative testing of SPICE and Azure, and it OOMed during processing of one of larger, 3D diagnostics (cl).

Command terminated by signal 9
	Command being timed: "mip_convert mip_convert.cfg.2024-04-09T1721Z -a -s ap6 -l mip_convert --datestamp 2024-04-09T1721Z --mip_era CMIP6"
	User time (seconds): 179.49
	System time (seconds): 331.44
	Percent of CPU this job got: 17%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 48:29.64
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 24798960
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 3892
	Minor (reclaiming a frame) page faults: 1626138
	Voluntary context switches: 508683
	Involuntary context switches: 136769
	Swaps: 0
	File system inputs: 533102424
	File system outputs: 766432
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Since I had to update quite a few dependencies to install the rc, I will try to replicate the environment with older iris and check if it works better.

@piotr-florek-mohc thanks

fail, repeating with 40G now

Piotr will post some of the details later, but initial results suggest we need to increase the amount of memory allocated for this test job by a factor of 7 in order to get our test case to work. The maximum resident set size has gone from 22 GB shown above, to 154 GB when changing from iris 3.4.1 to the 3.9.0rc*. We're re-running a number of reference jobs now to confirm that it isn't something on the system that has unexpectedly changed to give rise to this, and also looking into some of the darker corners of our code base to understand what is going on.

Note that the MIP Convert tool being used for this test doesn't make any direct use of dask and we do not use multiple threads. While data is read in by iris, it is written out via the CMOR library to enforce CMIP standards compliance so we are not using any of the iris netcdf saver code.

*In stepping up to the newer iris version we've had to update the python version to 3.11 and it is not impossible that other dependencies changing in our conda environment could be involved in this expansion of memory use. We were planning on picking up iris v3.7.1 in the near future, and we'll hopefully run some similar test cases in the next week or two.

Just in case anyone comes across this again; there was an error in our testing that resulted in loading 6 times more data than in our reference case which was not due to this change to iris (see further details from Piotr below).

Can I clarify if your scripts are using Iris saving at any point? That's the only thing we changed in 3.9

Sorry for skim-reading, you clearly already told us that you are NOT.

Good morning, Matt reminded me this morning that there might have been a code change in our codebase (subannual slicing) that I was sure hadn't been deployed, but on double checking I found out I did actually partially implement it for the purpose of SPICE/Azure benchmarks (long story short: I couldn't get it working for Gregorian calendar, but since tests were to be done using our standard model configuration, I could get away with 360 calendar). I forgot to patch it when doing the 3.9.x installation, and that explains why the code OOMs now.

I merged the change and I restarted the benchmark, fingers crossed it will behave more reasonably now with loading 6 times less data into memory. Apologies for all the confusion.

I've run our tests with a fixed workflow and can confirm that the silly behaviour encountered before does not occur. The maxRSS is still a bit higher than before (28G vs 22G) but not egregiously so. Once again, apologies for incurring confusion!