Skip to content

Write seasonal leaderboard RMSEs to CSV artifacts - #1950

Open
petebachant wants to merge 9 commits into
mainfrom
pb/print-leaderboard-rmses
Open

petebachant wants to merge 9 commits into
mainfrom
pb/print-leaderboard-rmses

Conversation

@petebachant

@petebachant petebachant commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

From Claude: Pete asked me to open this so the land model's RMSE values are saved as data, which gives a time series of them across runs, and I wrote the change.

compute_seasonal_leaderboard builds time-averaged simulation and observation maps for each variable and season, and the global RMSE and bias of those maps only appear in the plots. This also writes <data source>_leaderboard_rmse.csv next to the plots (ERA5_leaderboard_rmse.csv and ILAMB_leaderboard_rmse.csv), one row per variable and season:

short_name,season,data_source,rmse,bias,units
lhf,ANN,ERA5,10.3,-1.2,"W m^-2"

The same table is printed to the job log, since Buildkite ages artifacts out sooner than logs and it's only about 50 rows per data source.

The values come from ClimaAnalysis.global_rmse and ClimaAnalysis.global_bias on the same land-masked, time-averaged maps the seasonal plots use, so they match the RMSE and bias in the figure titles and are ClimaLand's own RMSE definition. Seasons without data are NaN. rmse_boxplots.jl isn't touched, so this doesn't overlap with #1897.

The long-run jobs that make leaderboard plots (Snowy Land + P Model, its optimal-LAI variant, the 19-year versions of both, and the global bucket run) only upload *png, so this adds a *csv glob to their artifact_paths.

clima-perf mirrors the weekly climaland-long-runs jobs, so with these numbers it can build a time series of each RMSE and bias instead of them being read off the plots by hand. Nothing else changes; the plots are the same.

Testing: after merging main, I ran write_rmses on synthetic lon/lat/time fields with a known uniform offset and some missing cells, passed through _land_intersection_mask and time-averaged as in the leaderboard, plus a season with no data. The RMSE, bias, and NaN values are as expected, and the logged table matches the file. I haven't run the long-run leaderboard locally. JuliaFormatter 2.10.1 makes no changes.

🤖 Generated with Claude Code

@AlexisRenchon

Copy link
Copy Markdown
Member

we should add the changes of #1897 (ar/boxplot_ilamb_rmse) - which ensure we use the same RMSE computation as ILAMB for the boxplot, and we use the global data (not just a region)

@AlexisRenchon

Copy link
Copy Markdown
Member

Also, if we want to be comparable, we should force with CRUJRA and use the prognostic LAI model

@AlexisRenchon

Copy link
Copy Markdown
Member

if we want to track our ERA5 RMSE to the log, that's different - then we should use our RMSE computation, and keep the ERA5

@petebachant petebachant changed the title Print RMSE boxplot values to the log Print seasonal leaderboard RMSEs to the log Nov 4, 2026
@petebachant

Copy link
Copy Markdown
Member Author

From Claude: Thanks, that makes sense. The goal here is a time series of our own RMSE against ERA5, so I've followed your last comment: the PR no longer touches rmse_boxplots.jl (so it doesn't overlap with #1897) and drops the other-model numbers, which aren't comparable as you point out. It now prints from compute_seasonal_leaderboard instead, using ClimaAnalysis.global_rmse and global_bias on the same time-averaged maps the seasonal plots use, for each variable, season, and data source (ERA5 and ILAMB). I've updated the description to match.

@petebachant

Copy link
Copy Markdown
Member Author

Thank @AlexisRenchon and sorry for the Claude slop auto-reply. I think I'm going to switch this over to exporting a CSV to the Buildkite artifacts as well.

@petebachant petebachant changed the title Print seasonal leaderboard RMSEs to the log Write seasonal leaderboard RMSEs to CSV artifacts Nov 4, 2026
@AlexisRenchon

Copy link
Copy Markdown
Member

No problem for the Claude replies - they do contain useful information! I think that sounds good, keep track of our ERA5 RMSE in a log.

@codecov

codecov Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.54%. Comparing base (ba89f1f) to head (098a4b4).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1950   +/-   ##
=======================================
  Coverage   86.54%   86.54%           
=======================================
  Files          77       77           
  Lines        8807     8807           
=======================================
  Hits         7622     7622           
  Misses       1185     1185           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the seasonal metrics CSV output and artifact-upload changes.

No open issues 🎉

What changed in this PR

Saves seasonal leaderboard metrics as CSV artifacts and prints them to job logs, enabling RMSE and bias tracking across runs without changing plots or the separate ILAMB-style boxplot calculation.

Changes:

  • Writes one row per variable and season using the same time-averaged maps as the leaderboard plots.
  • Records NaN metrics for seasons without data and logs the complete table.
  • Adds CSV artifact uploads to the five relevant long-run jobs.
File Description
ext/​land_sim_vis/​leaderboard/​leaderboard.jl Adds CSV and log output for seasonal RMSE and bias.
.buildkite/​longruns_gpu/​pipeline.yml Uploads CSV artifacts alongside existing plot artifacts.

Review effort: Balanced

@petebachant
petebachant marked this pull request as ready for review November 6, 2026 00:16

@AlexisRenchon AlexisRenchon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@petebachant
petebachant enabled auto-merge (squash) November 6, 2026 17:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants