Skip to content

feat: add fastCRW search tool#94

Open
us wants to merge 1 commit into
SkyworkAI:mainfrom
us:feat/add-fastcrw
Open

feat: add fastCRW search tool#94
us wants to merge 1 commit into
SkyworkAI:mainfrom
us:feat/add-fastcrw

Conversation

@us

@us us commented Jun 14, 2026

Copy link
Copy Markdown

What

Adds a fastCRW search tool (CRWSearch), alongside FirecrawlSearch.

New crw_search.py, registered alongside Brave/Bing/Google.

Why

fastCRW is a fully open-source web-scraping engine (AGPL, single ~8 MB Rust binary) that outperforms Firecrawl on Firecrawl's own benchmark dataset and runs entirely locally — no cloud dependency.

More open — actually works on protected/JS-heavy sites when self-hosted. Anti-bot/stealth bypass, BYO-proxy + rotation, and JS rendering all ship in the open core. Firecrawl's OSS edition gates its stealth engine (fire-engine) behind a cloud-only flag, so its self-host silently falls back to plain fetch/Playwright and can't reach Cloudflare-protected or JS-heavy pages. fastCRW handles Cloudflare JS challenges, UA rotation, SPA rendering, and an HTTP→headless→proxy fallback ladder — one binary, no multi-service stack, no cloud required.

Faster and higher recall on Firecrawl's own benchmark dataset. Truth-recall 63.74 % vs 56.04 %; median latency p50 ~1.9 s vs ~2.3 s. RAM footprint ~6 MB.

Search is built on SearXNG, not an alternative to it. SearXNG is the metasearch aggregator underneath; crw adds a quality layer on top: query expansion (multi-variant rewrite), content-aware reranking (re-scoring by fetched content instead of SearXNG's content-blind ordering), and category routing (research queries fan out to arxiv / semantic scholar / google scholar, code queries to GitHub). So you get SearXNG's breadth plus a measurable accuracy layer — all open-source (AGPL) and self-hostable.

Flat pricing: 1 credit = 1 page, no 4× stealth surcharge, no billed-on-failure. Key via CRW_API_KEY (free tier at https://fastcrw.com/dashboard); self-host base URL supported.

Firecrawl-API compatibility is the reason this integration is a tiny additive diff — FirecrawlSearch is untouched. Happy to adjust to project conventions; I maintain fastCRW and can provide free credits for testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant