Skip to content

fix(find-tools): say capabilities take the address, not the id - #536

Merged
KrisOei merged 4 commits into
firecrawl:mainfrom
Ben2W:find-tools-capability-addresses
Oct 15, 2026
Merged

KrisOei merged 4 commits into
firecrawl:mainfrom
Ben2W:find-tools-capability-addresses

Conversation

@Ben2W

@Ben2W Ben2W commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

firecrawl_find_tools lists each tool with id: "apps-apple-com/apps/app", provider: "apps-apple-com" and capability: "apps/app". The capabilities parameter says "Exact capability IDs", so an agent holding an id passes it and gets invalid_option; discovery only accepts the address (apps/app), scoped by providers.

This describes both parameters by the listing field they take, with an example. Descriptions only; no behaviour change.

Evidence

I benchmarked description variants with an agent loop behind a stdio proxy in front of firecrawl-mcp@3.29.1 that replaced only the descriptions. Firecrawl responses were recorded once and replayed, so every variant saw the same data. Tasks were lookups on Alexandria-covered pages: the page scrape was refused and the matching catalogue items were shown, id included, with no ready-made call.

This PR's wording, 12 lookup tasks × 3 repeats (re-run 2026-10-15 after the review rewording):

Model Text First contract lookup right id passed as a capability
GPT-6 Luna current 21/28 8
GPT-6 Luna this PR 26/26 0
GPT-6.1 Sol current 5/30 28
GPT-6.1 Sol this PR 32/32 0

The pre-review wording ("Capability addresses, as in each listed tool's capability field") ran in the same batch and did the same (Luna 29/29, Sol 30/30, no id passed), as it had on 2026-10-14 (Luna 15/25 → 27/27, Sol 12/30 → 34/34).

An earlier version of the capabilities text also said "not its provider-prefixed id". It did the same over 19 tasks × 3 repeats (Luna 14/30 → 33/33, Sol 21/42 → 47/47). Combined with the description change in the draft #537, it did the same across 15 more models from 10 labs (table there). Naming the field turned out to be enough, so the warning was dropped.

Caveats

  • The mistake needs an agent that has an id but no ready-made call. firecrawl_search (toolDetail: "summary"/"full") and firecrawl_scrape (domainTools: true) list id alongside a raw next. Agents that followed nextTool from find_tools listings, or came from compact search results (no id), were right with the current text too. So the production rate is lower than the table.
  • Exchange's invalid_option now names the fix, so most agents recover after one extra call. A few gave up and scraped instead.
  • The final wording was checked on Luna and Sol only.

Tests

Contract snapshots regenerated with UPDATE_CONTRACT_SNAPSHOTS=1; only these two descriptions and the find_tools definition hash change. pnpm test: 168/168 pass.

🤖 Generated with Claude Code

find_tools lists each tool with id "apps-apple-com/apps/app", provider
"apps-apple-com" and capability "apps/app". The capabilities parameter
said "Exact capability IDs", so agents holding an id passed it and got
invalid_option; discovery only accepts the address, scoped by providers.
Describe both parameters by the listing field they take, with an example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 10 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Auto-approved: Documentation-only change clarifying firecrawl_find_tools parameter descriptions, especially that capabilities takes capability addresses rather than provider-prefixed ids; contract snapshots updated and tests pass. No behavior change.

View guided diff | Re-trigger cubic

Drop "not its provider-prefixed `id`" from the capabilities description.
Naming the `capability` field with an example is enough: on GPT-6 Luna
and GPT-6.1 Sol both wordings got every first contract lookup right.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 10 files (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Auto-approved: Documentation-only change clarifying the firecrawl_find_tools providers and capabilities parameter descriptions (addresses/slugs rather than IDs), with contract snapshots regenerated and tests passing. No behavior or validation changes.
Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack

View guided diff | Re-trigger cubic

@Ben2W

Ben2W commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Some more context:

When I was testing Alexandria over MCP, agents kept calling firecrawl_find_tools with the wrong parameters. Each tool in the listing comes back with three fields: id: "apps-apple-com/apps/app", provider: "apps-apple-com" and capability: "apps/app".

The capabilities parameter was described as "Exact capability IDs", so agents assumed the id was the capability ID, passed it in, and got invalid_option.

At first this was a big problem because the error was generic. In my test environments, agents gave up on Alexandria about half the time. The error now explains the fix, so most agents recover after one extra call, but that call is still wasted.

The PR describes providers and capabilities by the listing field each one takes and adds an example. It only changes descriptions. In my runs, agents got their first lookup right only about half the time before; with this change, GPT-6 Luna and GPT-6.1 Sol got it right every time.

Ben and others added 2 commits October 15, 2026 11:39
Resolve contract snapshot conflicts with firecrawl#538/firecrawl#539 by regenerating them
from the merged source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review feedback: "Capability addresses" and "Provider slugs" are new
terms for the agent to map. Say where the values come from instead:
"Values from each listed tool's `capability` field", and the same for
`provider`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 10 files (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Auto-approved: Documentation-only fix updating the firecrawl_find_tools providers and capabilities descriptions to refer to listed tool field values rather than IDs, with regenerated contract snapshots; no validation or behavior changes.

View guided diff | Re-trigger cubic

@KrisOei
KrisOei merged commit d705793 into firecrawl:main Oct 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants