Repository navigation
Compare MCP, CLI and API surfaces and agent skills in one run - #32
Open
RobertCrupa wants to merge 15 commits into
Open
RobertCrupa wants to merge 15 commits into
RobertCrupa wants to merge 15 commits into
Conversation
Dataset items can carry the MCP server the agent talks to (mcp: stdio or
http, ${VAR} placeholders for secrets) and the skills a suite may inject
(agentSkills). The agent span records mcpServer, agentSkills and
skillsLoaded. tool.called takes a name alternation safely and a max, so a
check without a name becomes a call budget. Contract 1.2.0.
mcp.ts resolves the item's server (Apify by default, narrowed with ?tools=) into the Claude Code mcp-config, allowedTools patterns, secrets to redact and a tools/list client for preflight and schema snapshots. skills.ts turns the skillSets / skillCombinations inputs into variants; each (model, skill set) is its own experiment, the skills are copied into the session as project skills and the session runs with --setting-sources project so the developer's own skills never leak into a 'none' variant. OUTPUT lists the variants by pass rate. The image installs @notionhq/notion-mcp-server and copies skills/.
The sync stamps the profile's MCP server and injectable skills on every item and validates that each skill exists under skills/. The judge honours a profile's own forcedFixAreas mapping (judge impl 0.5.0).
… check A surface (surfaces/<name>.yaml) says how the agent reaches Apify: MCP, the CLI or raw HTTP to the API. Contract 1.3 adds its schema, the span fields surface, surfaceKind, surfacesUsed and cliVersion, an item-level surfaces filter, the run origin in the evidence, and a surface.used check that flags when an agent falls back to another surface.
Each run takes a surfaces input (surfaces/<name>.yaml) and runs one experiment per model x surface x skill set. The surface builds the session: MCP tools or none, Bash, WebFetch, a pinned Apify CLI on PATH (cliVersion overrides it for testing a build), extra env and a prompt suffix. Evidence no longer depends on MCP: runs are found in CLI and API output (every Console and API URL form, run objects in JSON), runs that no output names are found by the session marker in the curl user agent, and run input comes from the run's INPUT record, so apify.input checks work on every surface. Skills now go into the working directory, the only setting source. maxSessions refuses runs that grow too large (exit 15).
…ub ref Vendors four skills from apify/agent-skills at d522584 (the deprecated apify-generate-output-schema is left out). The skillsRef input downloads a public GitHub ref once per run and uses it as the skills root, so a skill author can evaluate a branch without a runner build; the run records the resolved commit.
…nt had The facts block names the agent setup (surface, CLI version, injected and loaded skills) and the Apify surfaces the agent used, and the prompt tells the judge to read tool choices against it. The store profile gains the surface and skill-instructions fix areas, lists apify-ultimate-scraper as an injectable skill, and a failed surface.used check forces the surface fix area. RUBRIC_VERSION 1203-draft-4, JUDGE_IMPL_VERSION 0.6.0.
…ment surfaces The image installs apify-cli 1.10.0 in /opt/apify-cli (CLI_ROOT), not on PATH: only sessions of a CLI surface get it. Other versions install at run time. surfaces/ is copied next to skills/ (SURFACES_DIR). The README says how to compare surfaces and skill sets and how to test a CLI, skill, MCP or API change.
Scenarios say what the user wants, not how to reach Apify, so one scenario runs on every surface; surfaces: [mcp] limits the few that check an MCP tool. Adds the surface.used check to the table.
…runs Found in a local smoke run: - Apify CLI 1.10.0 ignores APIFY_TOKEN in the environment; it reads only its own login (on macOS the Keychain). The runner now logs each CLI surface in once (apify login --token, file credentials, throwaway HOME) and every session copies that login through its own apify wrapper, so a session never reaches the developer's login and the cli surface works in the cloud. - On surfaces without the CLI the wrapper makes apify a missing command, as in the image. - apify call -s prints only items. The wrapper preloads a module that adds the session marker to the CLI's api.apify.com user agent, and the session-window lookup accepts CLI runs as well as API runs. - Sessions always run with --strict-mcp-config: without an MCP config, a local session picked up the developer's claude.ai connectors.
The cli surface now has mcpc (0.7.0) next to the Apify CLI. Each session runs it through a wrapper with its state in the session directory, connected to the Apify MCP server as @apify before the agent starts and closed after (every mcpc session holds a bridge process). mcpc is installed without optional dependencies, which leaves out its keychain module, so its credentials stay in that file on every platform. mcpcVersion tests another build; the preflight connects once. mcpc commands count as the cli surface. Its readable tool output names only datasetId=..., so the evidence also finds runs through the dataset that recorded them (actRunId) and reads runId from mcpc --json output.
An agent polled its run with mcpc @apify tools-call get-actor-run runId:='"..."' and timed out before any output named the run, so the evidence missed it.
A cloud matrix on a 5-run account had 86 of 126 sessions where the platform
refused the agent's Actor run ("exceed your limit of 5 concurrent Actor
runs"); they were scored as wrong-actor or fail against the surface.
- A session whose tool output shows that refusal is now inconclusive
(infra reason, contract evidence session.runLimitHit).
- Preflight warns when concurrency + this run exceed the account's
maxConcurrentActorJobs and suggests a concurrency that fits.
- After the evidence is frozen, runs the agent left running are aborted:
they only cost money and hold run slots.
- scenarioIds input: run only the named scenarios.
apify-service-account
added
the
tested
…ndoring The runner downloads apify/agent-skills once per run at the commit pinned in DEFAULT_SKILLS_REF. skillsRef and SKILLS_DIR still override it, and a run without skills downloads nothing. A full commit skips the GitHub API call. The pinned commit is byte-identical to the copy this removes. scenarios:check now only checks that agentSkills entries are skill names; the runner fails on an unknown name before any session starts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The runner can now compare how the agent reaches Apify (MCP, CLI, raw API) and which agent skills it has. Every model × surface × skill set is its own experiment on the same Langfuse dataset, so the compare view shows them side by side. Scenarios stay surface-agnostic.
Surfaces
surfaces/<name>.yaml:mcp(the default and the old behaviour),cli(Apify CLI plusmcpcconnected to the Apify MCP server, in Bash, no MCP tools of the agent's own) andapi(curl and WebFetch withAPIFY_TOKEN).surfaces, pluscliVersionandmcpcVersionto test another CLI or mcpc build without editing the surface file.surface.usedcheck. A scenario can limit itself withsurfaces: [mcp].Agent skills
apify/agent-skillsonce per run at the commit pinned inDEFAULT_SKILLS_REF(d522584, a full SHA, so no GitHub API call). A run without skills downloads nothing. Profiles list the skills a suite may use (agentSkills).skillsRefinput, thenSKILLS_DIR(a local directory), then the pin. An unknown skill name fails the run before any session starts;scenarios:checkonly checks the name format.skillSets(none,a,a+b),skillCombinations(every subset) andskillsRef(take skills from another GitHub ref, to test a change before it is merged).--setting-sources project, so personal skills do not leak into thenonevariant. The trace recordsagentSkillsandskillsLoadedand gets the tagskills:<label>.Profiles and judge
mcp), plusagentSkillsandforcedFixAreas.JUDGE_IMPL_VERSION0.4.0 → 0.6.0,RUBRIC_VERSION1203-draft-3 → 1203-draft-4.Guard
maxSessions(default 400) stops a run with exit 15 when models × surfaces × skill sets × scenarios × repeats is larger. The full store-actors suite on 3 surfaces × 2 skill sets is 366 sessions.Example input
{ "datasetName": "store-actors", "surfaces": ["mcp", "cli", "api"], "skillSets": ["none", "apify-ultimate-scraper"], "subjects": ["compass/crawler-google-places"], "itemLimit": 2 }Tests: shared 237, contract 42, report 11, judge 197, workflow-runner 108, tools 9.
scenarios:checkpasses.npm run lintfails onmaintoo (no ESLint config).