fix(crawl): wait out rate-limited status polls and page results only once the crawl ends

[erikengervall]

firecrawl_crawl starts a crawl and then polls GET /v2/crawl/:id until it ends. Two things in that loop made the tool fail while the crawl itself was fine:

• Each poll read every result page. getCrawlStatusWithOrigin follows next to collect all documents, and the loop called it on every tick, so a crawl with several result pages issued several status reads every couple of seconds and quickly used up the crawl-status rate limit.
• A rate-limited poll failed the tool. The 429 surfaced as Request failed with status code 429, the crawl ID was lost, and the agent typically started the same crawl again.

Now the loop reads only the first status page while the crawl runs and collects result pages once, after it reaches completed, failed or cancelled. A 429 on a status read waits for Retry-After (or retry_after_seconds, default 10s, capped at 60s) and keeps polling; other errors still fail as before.

Test: tests/mcp-crawl-polling.test.mjs serves a running crawl with a second result page, a 429 on the second status read, then completion; it fails on main with the 429 and passes here (full suite passes).

---

Summary by cubic

Fixes firecrawl_crawl polling so a healthy crawl no longer fails the tool.

• Polls only the first status page while the crawl runs and collects result pages once it reaches completed, failed or cancelled, reusing the terminal poll's response as the first result page.
• A 429 on a status read now waits for Retry-After (seconds or HTTP-date, or retry_after_seconds, default 10s, capped at 60s) and keeps polling; other errors still fail as before.

^{Written for commit 1e39bf9. Summary will update on new commits.}

[cubic-dev-ai]

All reported issues were addressed across 2 files

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Auto-approved: Makes crawl polling resilient: polls only the first status page while running, collects result pages once at termination, and retries 429 status reads via Retry-After; the added test pins this behavior, so the change is bounded and clearly beneficial.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>