Skip to content

Run the Store evals on two agent models and show them side by side - #31

Merged
artogahr merged 16 commits into
mainfrom
feat/multi-model-report
Oct 19, 2026
Merged

Run the Store evals on two agent models and show them side by side#31
artogahr merged 16 commits into
mainfrom
feat/multi-model-report

Conversation

@artogahr

@artogahr artogahr commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Closes #30.

The suite can now run the same scenarios on several agent models each morning and the report shows them side by side. Models and the judge are OpenRouter moving aliases; the Actors record the concrete model behind each alias so a move is visible instead of silent.

Actors

  • Runner preflights every model in the matrix (not only the first) and records the model the proxy answered with as resolvedModel on the experiment, in OUTPUT and on every per-turn generation (Langfuse prices by that name).
  • Judge resolves its alias once per batch, stamps judgeModelResolved into every score, and compares the resolved model for idempotency, so an alias move re-judges rather than being skipped. JUDGE_IMPL_VERSION stays 0.4.0: verdict logic is unchanged and a bump would re-score the online judge's backlog.
  • model input is free text again (the enum rejected every newer id). Descriptions explain aliases.
  • Dockerfile pins Claude Code 2.1.300 (the agent version is part of the measurement; the installer pulled latest) and bumps Codex to 0.162.1.

Report

  • One canonical attempt per task, day and model. Model tabs, a models line with alias moves, per-model chart series with grouped move markers, mixed day cells (k/n), per-model evidence with the prompt shown once.
  • Smaller page: scores trimmed to what the page shows, trace URL prefix factored out, pass results without the judge comment unless a disagreement explains it. Live single-model page 2.75 MB to 1.66 MB.
  • Single-model rendering is unchanged apart from the models line and the pass wording.

Rollout (after merge, separate)

Langfuse price definitions for the new concrete ids, one re-judge with Sonnet-latest under candidate, then the daily schedule flips to ~anthropic/claude-haiku-latest + ~openai/gpt-luna-latest with ~anthropic/claude-sonnet-latest as judge.

Tests: contract 26, judge 204, runner 54, report 28.

https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e

@artogahr
feat(report): model slots in the data layer, canonical run per day an…
0770a79 
…d model, smaller payload

Observations carry the configured model alias as their slot (legacy
anthropic/claude-haiku-4.5 maps into ~anthropic/claude-haiku-latest with the
pinned id kept as resolvedModel), the resolved model when the runner recorded
one, and the judge model from score metadata. Canonical selection is per day
and model slot. The compact page model lists model slots with first day and
resolved ids per day, the judge per day, and strips the shared trace URL
prefix; score comments are embedded only where the page shows them. The
rendered page is unchanged with one model; the embedded JSON shrinks by 31%.

Part of #30.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
feat(report): show three agent models side by side
8334dab 
Model becomes a scope next to Team and Period. The All-models view keeps
the same four columns: each task cell shows a cross-model badge (2/3
passed) with a per-model line, the 7-day track shows k/n when models
disagreed, rows sort by failing attempts, and each evidence card shows the
prompt once with failing models expanded and passing ones collapsed. Charts
draw one line per model; before/after pools only models with eligible
results on both sides and always lists every model. A models line and
chart markers show when an alias or the judge resolved to a new version.
A single model renders as before. The embedded JSON drops a further 17%
by keeping disagreement scores and pass boilerplate out.

Part of #30.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
chore(report): format client-v2 test
fe2c2e5 
Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
feat(actors): resolve moving model aliases and record the concrete model
0093e6f 
Runner: preflight every model in the matrix (not only the first) and record
the model the proxy answered with as resolvedModel on the experiment
metadata and in OUTPUT. The model input is free text again so any
OpenRouter id or ~vendor/model-latest alias works.

Judge: resolve the judge alias once per batch, stamp judgeModelResolved into
every score, and compare the resolved model for idempotency so an alias
move re-judges instead of being skipped. JUDGE_IMPL_VERSION 0.4.1.

Dockerfile: pin Claude Code 2.1.300 (the agent version is part of the
measurement) and bump Codex to 0.162.1.

Refs #30

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
fix(judge): keep JUDGE_IMPL_VERSION at 0.4.0, format llm.ts
e091835 
The impl version is part of every online score id, so a bump would make
the online judge re-score its backlog. Verdict logic is unchanged; the
resolved-model metadata is additive, so the version stays.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
feat(report): drop the judge comment on passing results
99968ad 
A passing result's comment restates the pass. Keep it only when the judge
and the checks disagreed, where it explains the call. The page says "The
task passed" / "Passed" instead. Live single-model page 1.85 MB -> 1.66 MB;
the saving scales with the number of models.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
feat(runner): per-turn generations carry the resolved model
f269b8a 
Langfuse prices a generation by matching its model name, so the alias
(~anthropic/claude-haiku-latest) must not be what the generations carry.
The harness gets the preflight's resolved id and stamps it on every turn
generation; the agent is still launched with the configured id and the
trace version keeps it for dashboards.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
chore(runner): two alias examples in the model description
e4e4415 
Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
artogahr requested a review from RobertCrupa October 16, 2026 11:01
@artogahr
feat(report): show the models line only when a model moved in the period
2ac83e0 
The tabs name the models and "How this is measured" carries the resolved
ids and the judge, so the standing "Models in latest run" line only took
space. The move notice stays, and only on days a move is in the window.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
feat(report): metric help and model updates open from the window line
a45e172 
Both were standalone lines. They are now inline dropdowns after
"N days available in this view", each opening a small panel under the
line; the model updates one appears only when a model moved in the period.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
fix(report): clearer chart markers, no overlap, no scroll jump on mar…
a6897fd 
…ker change

Marker labels now read "Haiku model changed 10/13" / "Haiku and judge
models changed 10/13" / "marker 10/14"; a label that would overlap the
previous one moves to a second row. A re-render scrolls to the evidence or
the how section only when that target is new, so choosing a change-marker
date no longer jumps to an open task.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
feat(report): chart marker labels on hover only
4c4c09a 
The dashed lines stay; a short label ("Model update 10/13", "Marker
10/14") shows on hover or focus. Markers on the same day share one hover
target with stacked labels. Details are in the Model updates menu.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
feat(report): split day cells when models disagree
06ca943 
In the all-models view a day where the models disagree is drawn as one
colour segment per model in tab order, so the cell shows which model
failed. The legend names the order. The k/n count stays in the tooltip
and the date detail.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr
style(report): split cell halves match the whole cells
313c70c 
Same pale fills, borders and symbols as single-outcome cells, at half
width, instead of solid colour blocks.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
@artogahr

artogahr commented Oct 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Preview: https://claude.ai/artifact/DW261NJKUjThbMT81puutN?sk=fBjqbAsbRUfnn5RyW_3T9g

@artogahr
fix(report): outcomes column sized to its cells below 1050px
e0a3024 
The fixed 205px column clipped the seventh day cell at viewports between
820 and 1050px; the column now takes its content width.

Claude-Session: https://claude.ai/code/session_01Gb2b7DMXcfsDdz6ALUWA3e
RobertCrupa
RobertCrupa reviewed Oct 16, 2026
View reviewed changes

@RobertCrupa RobertCrupa left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran a test against the mcp evals and it seems to break the langfuse traces, I no longer see the judge trace.

Here is the experiment

@artogahr artogahr mentioned this pull request Oct 16, 2026
Merged
@artogahr
artogahr added this pull request to stack #38 October 16, 2026 12:41
@artogahr

artogahr commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Could be because of #28, I'm running some tests now

@artogahr artogahr mentioned this pull request Oct 16, 2026
Closed
@artogahr
fix(runner): always pass the runner's Langfuse keys to the judge
cc1d898 
The judge got the keys only when they came in as inputs. A runner that used
its own environment keys for another project (a local run, or a deployment
with different keys than the judge) sent the judge to its own project, where
the dataset run does not exist: 0 items, no scores, no error. Now the judge
always receives the set the runner resolved; in production both Actors share
the keys, so nothing changes there.

(cherry picked from commit 6d213fe)
@artogahr

artogahr commented Oct 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

As I understand @RobertCrupa your local run failed since the keys were passed to the judge actor only if they were given to the eval actor as inputs. I changed it so that the keys are always passed on, even if they were given within the environment. Can you test again please?

This was caused by #28 as I thought, since the default judge changed

edit: this change also has some overlap with #15 but I think it's fine to include here

@artogahr
artogahr requested a review from RobertCrupa October 19, 2026 08:05
@artogahr
artogahr merged commit 9490117 into main Oct 19, 2026
4 checks passed
@artogahr
artogahr deleted the feat/multi-model-report branch October 19, 2026 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@RobertCrupa RobertCrupa RobertCrupa approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Run the Store evals on current models

3 participants

@artogahr @RobertCrupa @apify-service-account