Repository navigation
Compare the MCP and API surfaces in one run - #35
RobertCrupa wants to merge 1 commit into
Conversation
A surface is how the agent reaches Apify (surfaces/<name>.yaml). Every model x surface x skill set is its own experiment, so the compare view shows MCP and raw API side by side on the same scenarios. - contract 1.3.0: surface schema, classification of tool calls into surfaces, `surfacesUsed` in the evidence, the `surface.used` check. - runner: `mcp` (default) and `api` (curl, jq, WebFetch, APIFY_TOKEN) surfaces. Evidence finds runs in Bash and WebFetch output, by the curl session marker for run-sync calls, and reads each run's INPUT. - judge: reads the trajectory against the agent setup; fix areas `surface` and `skill-instructions` (RUBRIC_VERSION 1203-draft-4, JUDGE_IMPL_VERSION 0.6.0). - tools: a scenario's `surfaces` list now reaches the dataset item, so the runner can skip the scenario on other surfaces.
| const command = String((call.name === 'Bash' ? input.command : input.url) ?? ''); | ||
| const origin = surfacesOfCall(call.name, call.input).includes('cli') ? 'cli' : 'api'; | ||
| const actor = cliActor(command) ?? apiActor(command); | ||
| const text = `${command}\n${res?.content ?? ''}`; |
There was a problem hiding this comment.
When the curl command contains a { (any -d '{...}'), parseJson on command+output fails: the whole string isn't JSON, and the fallback slice from the first { starts inside the command. Pretty-printed API output then fails the line-by-line pass too, so the run is never found. The test only passes because it uses -d @file and compact JSON. Parse res.content on its own and scan the command separately for URLs.
| emitTimeline(span, r.timeline, r.startedAt, prompt, harness.model); | ||
|
|
||
| const traceId = span.otelSpan.spanContext().traceId; | ||
| const logRef = await ctx.artifactStore.putLog(traceId, r.rawStdout); |
There was a problem hiding this comment.
This stores the full session log without redaction. On the api surface the session env has APIFY_TOKEN, and the agent will at some point run curl -v or echo $APIFY_TOKEN. The token then sits in the stored log and in the tool observations. redact() is only applied to the harness's stderr at 607. It should run on this log and on tool results as well.
3 participants
Stack 3 of 4, on top of #34.
A surface is how the agent reaches Apify (
surfaces/<name>.yaml). Each model × surface × skill set is its own experiment, so the compare view shows the MCP server and the raw API side by side on the same scenarios. The CLI surface comes in #36.What changes
mcp,cliorapi,surfacesUsedin the evidence, and thesurface.usedcheck.mcpis the default and keeps the old behaviour.apigives the agent curl, jq, WebFetch andAPIFY_TOKEN, and no MCP server.INPUT, soapify.inputchecks work on every surface.mcp, the traceversionis<model> · <surface>. Onmcpit stays the model.surfaceandskill-instructions.RUBRIC_VERSIONgoes to 1203-draft-4 andJUDGE_IMPL_VERSIONto 0.6.0.scenarios/README.mdsaid a scenario can setsurfaces: [mcp], but the sync dropped the field, so every scenario ran on every surface. The sync now copies it onto the dataset item. A new test reproduced the bug before the fix.Tests: contract 42, judge 197, workflow-runner 94, tools 10.
scenarios:checkpasses.