fix: make the online eval run safe for production

[RobertCrupa]

Part of apify/ai-team#249, stacked on the #271 PR. Fixes three blockers that a production-readiness review of the online evals stack found. The unit tests did not catch any of them.

1. The stack did not merge into main (fixed by the rebase, not in this PR)

The stack was based on poc/workflow-runner, but main got the PoC as squash PRs #7 to #13. A test merge conflicted in 21 files. PRs #1 to #5 are now rebased onto main. Two old merge commits had quietly adapted the stack to main; the rebase redoes them as commits:

• fix: read the online judge reply from judgeLlmCall's result (in the #269 PR). On main, judgeLlmCall returns {json, usage, ...}. Without this, every online judgement fails with InvalidJudgeReplyError. The build passed; only typecheck caught it.
• test: pin the online score id to JUDGE_IMPL_VERSION 0.4.0 (in the #270 PR).

2. The checkpoint did not carry over between runs

actorCheckpointStore() used Actor.getValue / Actor.setValue, so the run's default key-value store. Every platform run gets a new default store, so every scheduled run read null and fell back to now - 24h. Results: the traffic from 05:27 to 06:00 UTC was never judged, and a failed or timed-out run lost its window (the documented retry had no effect).

• The checkpoint is now in the named store apify-ai-online-state, one record per environment (ONLINE_CHECKPOINT-<environment>), so a staging run never moves the prod checkpoint.
• New test test/checkpoint-store.test.ts runs against the SDK's real local storage and changes the default store id between "runs", like the platform. It fails on the old code (expected null to deeply equal {...}) and passes now.
• JUDGE_IMPL_VERSION is not bumped: it is part of every score id, and scoring did not change.

3. The scheduled run would use the wrong Langfuse project

Online mode must read "Apify AI Agent". The Actor env keys belong to "MCP Agent Evals" and must stay there, because the workflow runner starts the judge without keys. The schedule sent no keys and its README said to put them in the Actor env, which breaks one mode or the other.

• New guard (src/online-project.ts): before any other Langfuse call, online mode resolves the keys' project with GET /api/public/projects and fails the run unless its name or id equals the new langfuseProject input (default Apify AI Agent). Dataset-run mode does not change. 8 tests.
• The schedule now runs an Apify Task (RUN_ACTOR_TASK). schedule/online-task.json is the POST /v2/actor-tasks body with placeholders; the keys go into the Task's secret inputs, typed in the Console. .gitignore blocks filled-in *.local.json copies.
• Compatible with feat: Langfuse project keys as mandatory Actor input #15 (keys as required input). After feat: Langfuse project keys as mandatory Actor input #15, drop the sentences about the env-key fallback.

Verification

On the top of the stack with this PR: npm ci, build, npm test for contract (26), judge (179), workflow-runner (42) and tools (8), scenarios:check, judge typecheck, and Prettier all pass. ESLint is clean on the changed files; the remaining errors are older lines from main.

Not verified yet

• The guard default is the project name. The project id and whether production has apify-ai traces since apify-ai-agent v0.3.4 are not checked yet: that needs the "Apify AI Agent" keys.
• Whether POST /v2/actor-tasks encrypts isSecret input. The README avoids the question: type the keys in the Console.
• Nothing is created on Apify or in Langfuse by this PR.

[vojtechj-apify]

All three fixes are right, and the checkpoint test that switches the default store between runs is a good reproduction of the bug.

• The guard defaults to the project name. Has it been run with the real "Apify AI Agent" keys yet?
• "Prettier passes" holds only for the new files. A few files inherited from the base commit are flagged. That goes away with the rebase onto main.
• Merge order with #15: both change how the Langfuse keys are handled, and they conflict. Which one goes first?

[RobertCrupa]

Addressed every open review thread on #1 to #5 and #25 here (commits 3982874..3354391), each thread has a reply with its commit. Verified end to end: unit tests (195), typecheck, and two cloud runs of build 0.1.2 of eval-judge in online mode over the prod window 2026-10-07 11:00 to 12:00 UTC with maxItems: 5: run 1 judged and wrote 5 traces and created rollup-2026-10-07 (https://console.apify.com/actors/runs/2MqlKJtws8cujvsC0); run 2 over the same window skipped the already-judged trace, wrote 2, failed 2 on reconstruction (tolerated, under the 50% hold) and merged into the rollup (https://console.apify.com/actors/runs/SV1a7mspsDqisn01y). Both score copies read back with environment: prod and the run copies under apify-ai-online-2026-10-07.