From 1c058da869006c2d929458bb54ed27a2a1f079b9 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 19 Oct 2026 13:56:47 +0000 Subject: [PATCH 1/2] docs: Record git, mcpc, release and eval pitfalls Rules that agents and contributors hit while shipping 0.18.4: - AGENTS.md: one worktree per branch, no parallel git across worktrees, fetch before push, never rewrite someone else's branch. - DEVELOPMENT.md: stdio env under mcpc, auto-injected tools in expected tool lists, hosted itemsSchema differs between requests. - RELEASE.md: merge PRs one at a time; check a hosted rollout on several new connections. - evals/README.md: reading scores through the v3 API, TELEMETRY_ENV=DEV for report-problem cases, cases that expect removed behavior. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011QiQRhqm3wKyv6KFLncAqG --- AGENTS.md | 2 ++ DEVELOPMENT.md | 6 ++++++ RELEASE.md | 2 ++ evals/README.md | 11 +++++++++++ 4 files changed, 21 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 25d5414c9..a7db9513e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,6 +25,8 @@ Conventional Commits for all three. Branch: `type/short-desc` (e.g. `fix/connect Use `git mv` (not `mv` + `rm`) when renaming files so git records a rename rather than delete+create. +Working on several branches at once: one worktree per branch, run git as `git -C `, and never run git commands for two worktrees in parallel. Before pushing, fetch the branch; if someone else pushed, merge or reapply on top. Never rebase, amend or force-push a branch someone else pushed to. + ## Verification (mandatory) After every code change, run `pnpm run type-check`, `pnpm run lint`, `pnpm run test:unit`, `pnpm run format` and `pnpm run check:agents`. diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 393920064..a1c756ab7 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -250,6 +250,12 @@ mcpc --json @stdio tools-call search-actors keywords:="scraper" | jq ‘.content - `get-dataset-items` — test field filtering with dot notation, non-existent dataset - `search-apify-docs` / `fetch-apify-docs` — test relevant and non-existent queries +#### Before you compare results + +- **stdio env:** mcpc starts a stdio server with only the MCP SDK's default variables (`HOME`, `PATH`, …) plus the config's `env`. List `APIFY_TOKEN` and, behind a proxy, `HTTPS_PROXY` and `NODE_EXTRA_CA_CERTS` there. +- **Expected tool list:** a `?tools=` selection with `get-actor-run` or an Actor tool also serves the auto-injected tools (⚡ in the [README tool list](./README.md)), including `abort-actor-run`. +- **Comparing sessions:** on the hosted server, direct Actor tools' `outputSchema` embeds an item schema from the `ActorStore` (`enrichActorToolOutputSchemas`), and it can differ between requests. Compare `outputSchema` without `storages.datasets.default.itemsSchema`. + ### Testing with MCPJam (optional) diff --git a/RELEASE.md b/RELEASE.md index c4b47fe5d..da28a5f1e 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -39,4 +39,6 @@ That completes the package release. To roll the new version out to `mcp.apify.co ## Notes - The changelog commit carries `[skip ci]` so it does not retrigger CI. +- Merge PRs to `master` one at a time: wait for the CI (master) run's **Update changelog** job before the next merge. That job commits only if `master` still points at the commit that triggered it, so a merge seconds later makes it fail and skips that run's beta publish. Re-running an older CI (master) run also cancels the queued run of a newer merge (concurrency group `release`). +- To check a hosted rollout, read `serverInfo.version` from `initialize` on several new connections. During a rollout some connections still reach the previous version. - When you change a tool contract here, update the matching integration tests in the hosted-server repo in the same release window — drift there has broken hosted releases before. diff --git a/evals/README.md b/evals/README.md index dd07af392..9172c0e89 100644 --- a/evals/README.md +++ b/evals/README.md @@ -360,6 +360,8 @@ Results are recorded in Langfuse, not to a local file. Each run: - **Scores** each agent item: `mcp_agent_judge` (`1` on a PASS verdict, comment = judge reason) and `tool_errors` (count of unexpected failed tool calls, comment lists every failure with expected ones marked, `0` on a clean item) together form the gate, and `total_tokens` is the agent tokens billed (omitted when the provider reported no usage so an unmeasured run cannot look like a free one; an item whose agent run was retried reports only the second attempt). Each tool-call item scores `first_tool_match` alone (`1`/`0`, comment names the captured call and the verdict). - **Scores the run** with `pass_rate`: passed trials over requested trials (`requestedIds.length * iterations`), so runs stay comparable even when trials were dropped. +To read scores outside the UI, call `GET /api/public/v3/scores?name=&fields=core,details` on `LANGFUSE_BASE_URL` with the public and secret key as basic auth. The v1 and v2 score endpoints return 404 on that instance. + ### Concurrency `--concurrency` maps to the SDK's `maxConcurrency`, which runs **sequential batches** of that size rather than a rolling window: one slow test stalls the rest of its batch. @@ -444,6 +446,15 @@ above). The snapshot holds the same fields flattened, one object per case, in th - Reduce `maxTurns` to fail faster - Try a different agent model +### Eval runs send real problem reports +**Symptom:** `report-problem` calls from an eval run reach the production Segment source.
+**Cause:** The eval server inherits the runner's environment, and telemetry defaults to `PROD` (`src/telemetry.ts`).
+**Solution:** Run cases where the agent can call `report-problem` (e.g. cases with `failTools`) with `TELEMETRY_ENV=DEV`. + +### A case expects behavior your change removes on purpose +**Symptom:** A case fails because it checks for behavior the PR removes on purpose, e.g. a call that tool texts no longer ask for.
+**Solution:** Don't change the case in the same PR. Name the case and the reason in the PR, and open an issue for the case owner ([#1603](https://github.com/apify/apify-mcp-server/issues/1603) is an example). + ## CI CI runs two Langfuse tiers: From 1a436034d30ac659e1eea1b41b9e3912209c7520 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 19 Oct 2026 14:06:47 +0000 Subject: [PATCH 2/2] docs: Drop the release notes and the mcpc comparison section They described one release, not the repo: the repo's .mcp.json already passes the stdio env, the README already lists the auto-injected tools, and the itemsSchema difference needs an issue rather than a doc note. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011QiQRhqm3wKyv6KFLncAqG --- DEVELOPMENT.md | 6 ------ RELEASE.md | 2 -- 2 files changed, 8 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index a1c756ab7..393920064 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -250,12 +250,6 @@ mcpc --json @stdio tools-call search-actors keywords:="scraper" | jq ‘.content - `get-dataset-items` — test field filtering with dot notation, non-existent dataset - `search-apify-docs` / `fetch-apify-docs` — test relevant and non-existent queries -#### Before you compare results - -- **stdio env:** mcpc starts a stdio server with only the MCP SDK's default variables (`HOME`, `PATH`, …) plus the config's `env`. List `APIFY_TOKEN` and, behind a proxy, `HTTPS_PROXY` and `NODE_EXTRA_CA_CERTS` there. -- **Expected tool list:** a `?tools=` selection with `get-actor-run` or an Actor tool also serves the auto-injected tools (⚡ in the [README tool list](./README.md)), including `abort-actor-run`. -- **Comparing sessions:** on the hosted server, direct Actor tools' `outputSchema` embeds an item schema from the `ActorStore` (`enrichActorToolOutputSchemas`), and it can differ between requests. Compare `outputSchema` without `storages.datasets.default.itemsSchema`. - ### Testing with MCPJam (optional) diff --git a/RELEASE.md b/RELEASE.md index da28a5f1e..c4b47fe5d 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -39,6 +39,4 @@ That completes the package release. To roll the new version out to `mcp.apify.co ## Notes - The changelog commit carries `[skip ci]` so it does not retrigger CI. -- Merge PRs to `master` one at a time: wait for the CI (master) run's **Update changelog** job before the next merge. That job commits only if `master` still points at the commit that triggered it, so a merge seconds later makes it fail and skips that run's beta publish. Re-running an older CI (master) run also cancels the queued run of a newer merge (concurrency group `release`). -- To check a hosted rollout, read `serverInfo.version` from `initialize` on several new connections. During a rollout some connections still reach the previous version. - When you change a tool contract here, update the matching integration tests in the hosted-server repo in the same release window — drift there has broken hosted releases before.