diff --git a/AGENTS.md b/AGENTS.md index 25d5414c9..a7db9513e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,6 +25,8 @@ Conventional Commits for all three. Branch: `type/short-desc` (e.g. `fix/connect Use `git mv` (not `mv` + `rm`) when renaming files so git records a rename rather than delete+create. +Working on several branches at once: one worktree per branch, run git as `git -C `, and never run git commands for two worktrees in parallel. Before pushing, fetch the branch; if someone else pushed, merge or reapply on top. Never rebase, amend or force-push a branch someone else pushed to. + ## Verification (mandatory) After every code change, run `pnpm run type-check`, `pnpm run lint`, `pnpm run test:unit`, `pnpm run format` and `pnpm run check:agents`. diff --git a/evals/README.md b/evals/README.md index dd07af392..9172c0e89 100644 --- a/evals/README.md +++ b/evals/README.md @@ -360,6 +360,8 @@ Results are recorded in Langfuse, not to a local file. Each run: - **Scores** each agent item: `mcp_agent_judge` (`1` on a PASS verdict, comment = judge reason) and `tool_errors` (count of unexpected failed tool calls, comment lists every failure with expected ones marked, `0` on a clean item) together form the gate, and `total_tokens` is the agent tokens billed (omitted when the provider reported no usage so an unmeasured run cannot look like a free one; an item whose agent run was retried reports only the second attempt). Each tool-call item scores `first_tool_match` alone (`1`/`0`, comment names the captured call and the verdict). - **Scores the run** with `pass_rate`: passed trials over requested trials (`requestedIds.length * iterations`), so runs stay comparable even when trials were dropped. +To read scores outside the UI, call `GET /api/public/v3/scores?name=&fields=core,details` on `LANGFUSE_BASE_URL` with the public and secret key as basic auth. The v1 and v2 score endpoints return 404 on that instance. + ### Concurrency `--concurrency` maps to the SDK's `maxConcurrency`, which runs **sequential batches** of that size rather than a rolling window: one slow test stalls the rest of its batch. @@ -444,6 +446,15 @@ above). The snapshot holds the same fields flattened, one object per case, in th - Reduce `maxTurns` to fail faster - Try a different agent model +### Eval runs send real problem reports +**Symptom:** `report-problem` calls from an eval run reach the production Segment source.
+**Cause:** The eval server inherits the runner's environment, and telemetry defaults to `PROD` (`src/telemetry.ts`).
+**Solution:** Run cases where the agent can call `report-problem` (e.g. cases with `failTools`) with `TELEMETRY_ENV=DEV`. + +### A case expects behavior your change removes on purpose +**Symptom:** A case fails because it checks for behavior the PR removes on purpose, e.g. a call that tool texts no longer ask for.
+**Solution:** Don't change the case in the same PR. Name the case and the reason in the PR, and open an issue for the case owner ([#1603](https://github.com/apify/apify-mcp-server/issues/1603) is an example). + ## CI CI runs two Langfuse tiers: