Repository navigation
Write seasonal leaderboard RMSEs to CSV artifacts - #1950
petebachant wants to merge 9 commits into
Conversation
|
we should add the changes of #1897 (ar/boxplot_ilamb_rmse) - which ensure we use the same RMSE computation as ILAMB for the boxplot, and we use the global data (not just a region) |
|
Also, if we want to be comparable, we should force with CRUJRA and use the prognostic LAI model |
|
if we want to track our ERA5 RMSE to the log, that's different - then we should use our RMSE computation, and keep the ERA5 |
…the boxplot values
|
From Claude: Thanks, that makes sense. The goal here is a time series of our own RMSE against ERA5, so I've followed your last comment: the PR no longer touches |
|
Thank @AlexisRenchon and sorry for the Claude slop auto-reply. I think I'm going to switch this over to exporting a CSV to the Buildkite artifacts as well. |
|
No problem for the Claude replies - they do contain useful information! I think that sounds good, keep track of our ERA5 RMSE in a log. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1950 +/- ##
=======================================
Coverage 86.54% 86.54%
=======================================
Files 77 77
Lines 8807 8807
=======================================
Hits 7622 7622
Misses 1185 1185 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are identified for the seasonal metrics CSV output and artifact-upload changes.
No open issues 🎉
What changed in this PR
Saves seasonal leaderboard metrics as CSV artifacts and prints them to job logs, enabling RMSE and bias tracking across runs without changing plots or the separate ILAMB-style boxplot calculation.
Changes:
- Writes one row per variable and season using the same time-averaged maps as the leaderboard plots.
- Records
NaNmetrics for seasons without data and logs the complete table. - Adds CSV artifact uploads to the five relevant long-run jobs.
| File | Description |
|---|---|
ext/land_sim_vis/leaderboard/leaderboard.jl |
Adds CSV and log output for seasonal RMSE and bias. |
.buildkite/longruns_gpu/pipeline.yml |
Uploads CSV artifacts alongside existing plot artifacts. |
Review effort: Balanced
From Claude: Pete asked me to open this so the land model's RMSE values are saved as data, which gives a time series of them across runs, and I wrote the change.
compute_seasonal_leaderboardbuilds time-averaged simulation and observation maps for each variable and season, and the global RMSE and bias of those maps only appear in the plots. This also writes<data source>_leaderboard_rmse.csvnext to the plots (ERA5_leaderboard_rmse.csvandILAMB_leaderboard_rmse.csv), one row per variable and season:The same table is printed to the job log, since Buildkite ages artifacts out sooner than logs and it's only about 50 rows per data source.
The values come from
ClimaAnalysis.global_rmseandClimaAnalysis.global_biason the same land-masked, time-averaged maps the seasonal plots use, so they match the RMSE and bias in the figure titles and are ClimaLand's own RMSE definition. Seasons without data areNaN.rmse_boxplots.jlisn't touched, so this doesn't overlap with #1897.The long-run jobs that make leaderboard plots (Snowy Land + P Model, its optimal-LAI variant, the 19-year versions of both, and the global bucket run) only upload
*png, so this adds a*csvglob to theirartifact_paths.clima-perf mirrors the weekly
climaland-long-runsjobs, so with these numbers it can build a time series of each RMSE and bias instead of them being read off the plots by hand. Nothing else changes; the plots are the same.Testing: after merging main, I ran
write_rmseson synthetic lon/lat/time fields with a known uniform offset and some missing cells, passed through_land_intersection_maskand time-averaged as in the leaderboard, plus a season with no data. The RMSE, bias, andNaNvalues are as expected, and the logged table matches the file. I haven't run the long-run leaderboard locally. JuliaFormatter 2.10.1 makes no changes.🤖 Generated with Claude Code