Skip to content

feat(find-tools): lead the description with the lookup sequence - #537

Draft
Ben2W wants to merge 3 commits into
firecrawl:mainfrom
Ben2W:find-tools-description
Draft

Ben2W wants to merge 3 commits into
firecrawl:mainfrom
Ben2W:find-tools-description

Conversation

@Ben2W

@Ben2W Ben2W commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Draft, stacked on #536. Review only the last commit (e1f3017); the first two are #536.

Rewrites the firecrawl_find_tools description to lead with the lookup sequence and the field rule:

Find Alexandria data providers and read their contracts; free, and executes nothing. {"urls": [""]} lists tools covering a site, {"query": ""} searches, no arguments browses categories. Each listed tool has separate provider and capability fields: read its contract with {"providers": [provider], "capabilities": [capability]} (never the combined id), then run it with firecrawl_scrape {"alexandria": {provider, capability, options}}. Alexandria covers companies, people, jobs, filings, public records, real estate, places, prices, package registries, news, research and more. Prefer normal firecrawl_search for a data task; it already returns matching providers with web results. Use this tool for a full contract, catalogue browsing, or before scraping the same fields from several pages.

992 → 821 characters. It also defines the description once in alexandria.ts; server.ts and the search surface had identical copies.

Evidence

Same setup as #536: description-only overrides, replayed Firecrawl responses, 19 tasks. First contract lookup right, current text → this text with #536. These runs used #536's earlier wording, which also said "not its provider-prefixed id"; #536 has since dropped that clause after it checked out the same on Luna and Sol.

Model Current With this
Claude Haiku 5.5 2/15 31/31
Grok 4.7 5/22 32/32
Qwen3.8 Max 7/20 24/24
DeepSeek V4.1 Flash 6/15 21/21
Claude Opus 5.5 10/25 27/27
Muse Spark 1.3 18/30 39/39
Hy4 preview 13/21 24/24
Mistral Large 4 9/14 21/22
Gemini 3.8 Flash 11/17 23/23
Gemini 4 Argon 19/28 33/33
Step 5 preview 17/18 22/22
MiMo-V2.6-Pro 17/18 17/17
GLM 5.3 3/7 11/11
GPT-6 Luna (3 repeats) 14/30 36/36
GPT-6.1 Sol (3 repeats) 21/42 47/47
GPT-6.1 Sol, live MCP (3 repeats) 22/44 47/47

The models routed through OpenRouter ran 1–2 repeats. Claude Sonnet 5.5 and Kimi K3 rarely looked up a contract (they ran the provider directly), so they are left out. Judged accuracy on Luna and Sol: 0.99 → 0.99 (Luna), 0.99 → 0.97 (Sol, replay) and 0.96 → 0.99 (Sol, live); the Sol dips are a model declining to quote a transcript. The other models weren't judged.

What this adds over #536 alone: on Sol, 0.37 fewer tool calls per run in replay and 0.32 live, against 0.21 for the parameter text alone. On Luna, the parameter text alone did as well or slightly better (0.25 vs 0.16).

Why a draft

  • tests/mcp-description-budget.test.mjs pins "Prefer normal firecrawl_search" as routing copy from your AX runs. The tested text ended "firecrawl_search already returns matching providers with web results; use this tool for …". This PR keeps your sentence instead, so this exact wording wasn't run.
  • Dropped from the current text: "nextTool identifies further discovery or pagination", "Use firecrawl_search when you also need web results", and the long category list (now shortened). Pagination, category browsing and search → find_tools routing weren't covered by these tasks.
  • Glad to cut this down to one added sentence on the field rule, or to close it if fix(find-tools): say capabilities take the address, not the id #536 is enough.

pnpm test: 166/166 pass, with regenerated contract snapshots.

🤖 Generated with Claude Code

Ben2W and others added 2 commits October 14, 2026 14:42
find_tools lists each tool with id "apps-apple-com/apps/app", provider
"apps-apple-com" and capability "apps/app". The capabilities parameter
said "Exact capability IDs", so agents holding an id passed it and got
invalid_option; discovery only accepts the address, scoped by providers.
Describe both parameters by the listing field they take, with an example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop "not its provider-prefixed `id`" from the capabilities description.
Naming the `capability` field with an example is enough: on GPT-6 Luna
and GPT-6.1 Sol both wordings got every first contract lookup right.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rewrite the firecrawl_find_tools description to say how to find a
provider (urls / query / browse), read its contract with the listed
tool's provider and capability fields (never the id), and run it with
firecrawl_scrape. Keeps "Prefer normal firecrawl_search for a data task".
992 -> 821 characters.

Define the description once in alexandria.ts; server.ts and the search
surface carried identical copies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant