Repository navigation
feat: write online scores idempotently and keep them past retention - #4
Merged
Merged
Conversation
RobertCrupa
added this pull request to stack #6
September 9, 2026 15:14
RobertCrupa
force-pushed
the
feat/online-evals-score-writing
branch
from
October 7, 2026 09:00
4a4cefb to
77f215f
Compare
RobertCrupa
removed this pull request from stack #6
October 7, 2026 09:08
RobertCrupa
added this pull request to stack #26
October 7, 2026 09:08
vojtechj-apify
requested changes
Oct 7, 2026
vojtechj-apify
left a comment
vojtechj-apify
left a comment
There was a problem hiding this comment.
The two-copy design and the idempotent ids work well, and verifying every API assumption live shows. I'd fix the first three inline comments before the schedule goes live: environment isolation, the mostly-failed batch, and writing per trace.
Question: the rollup dataset name is permanent once it's created, since there's no public delete. Can we explicitly agree on apify-ai-online-rollups before the first prod run?
RobertCrupa
force-pushed
the
feat/online-evals-score-writing
branch
2 times, most recently
from
October 8, 2026 13:27
32eff63 to
21088f7
Compare
…ai-team#270) Every verdict from the online judge is written to Langfuse twice: once on the trace, once under an invented datasetRunId apify-ai-online-YYYY-MM-DD so the score survives the 30-day trace sweep. Score ids are readable, <traceId>-<scoreName>-p<promptVersion>-i<JUDGE_IMPL_VERSION>-<judgeModel>, with a -run suffix on the archival copy; a bumped version writes beside the old score. Sampled traces already judged under the same version are dropped before judging, so a re-run spends nothing on the LLM. A daily rollup (pass rate and n per score, coverage counters) is upserted as dataset item rollup-YYYY-MM-DD in apify-ai-online-rollups, merging runs of the same day. The checkpoint write moves out of selectTraces (writeCheckpoint option) and happens only after the scores and the rollup are written; a rollup failure fails the run and leaves the window to be retried. OUTPUT gains scoresSkipped, scoresWritten, failedToWrite and rollupItemId.
…the window day (ai-team#270 review) A batch whose writes all failed was reported as success: the rollup came from every judged verdict, the checkpoint advanced and the run exited 0, so a Langfuse outage silently lost the window and its LLM spend. writeOnlineScores now returns the verdicts that were written, only those feed the rollup, the rollup coverage gains scoresWritten and failedToWrite, and an all-failed batch skips the rollup, keeps the checkpoint and exits non-zero. Run ids and rollup items are keyed on the UTC date of the window start, not the write, so a backfill lands on its own day; the daily run therefore labels D-1 when it writes at ~06:0x on D (README says so for the #271 monitor). The rollup is skipped when nothing was sampled or judged. README and comments now describe the ReplacingMergeTree semantics (same-day rewrite replaces the row, cross-day retry of a half-written trace leaves two rows) and the retry double count of the window counters. A round-trip test pins writer and reader to each other.
… a judge failure (ai-team#270 round 3) A live verification against langfuse.apify.dev (2026-09-09) confirmed every API assumption. datasets.create is idempotent by name, so ensureRollupDataset calls it unconditionally; the events_only hedges become "verified live"; the README records the experimentId read filter and the experiment subject kind, that ANNOTATION needs a configId (API is the only usable source), that BOOLEAN scores read back as JSON booleans in environment default, and that no public API deletes a dataset (ROLLUP_DATASET_NAME is permanent; pinned by a test). isNotFound stays on statusCode, now pinned by a test that constructs the client's real NotFoundError (@langfuse/core added as a devDependency for it). finishOnlineRun gains the judge-side twin guard: when every trace given to the judge failed to judge, no rollup, no checkpoint, AllJudgementsFailedError and a non-zero exit. Window override docs say the ISO 8601 value needs an offset.
…cs claim (ai-team#270 follow-ups) The all-judgements-failed guard is judged === 0 && failedToJudge > 0, which is what sampled - scoresSkipped > 0 && judged === 0 meant, so the extra scoresSkipped option on finishOnlineRun goes. The README no longer claims the metrics aggregation of the run copy was verified live: read-back via experimentId and the subject kind were, aggregation under an invented run is the one open inference, and the working endpoint in events_only mode is GET /api/public/v2/metrics. The two wordings for the held checkpoint are one.
main bumped JUDGE_IMPL_VERSION to 0.4.0 in #13. The score-id test and the README example still used 0.1.0. The old stack made the same change (to 0.3.0) in its merge of the PoC branch.
RobertCrupa
force-pushed
the
feat/online-evals-score-writing
branch
from
October 8, 2026 13:31
21088f7 to
325292b
Compare
apify-service-account
added
the
tested
vojtechj-apify
self-requested a review
October 8, 2026 13:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes apify/ai-team#270. Part of apify/ai-team#249, stacked on the #269 PR.
Every Langfuse API assumption was verified live on 2026-09-09 with throwaway objects (score create and same-day replace, the run subject without a run, exactly-one-subject, EVAL rejected, datasets and items in events_only mode, v3 read with a traceId list). One inference remains and is marked as such: metrics-API aggregation of run-attached scores. No public API deletes a dataset, so the rollup dataset name is permanent once created.