feat(batch-scrape): add a batch-scrape command and JSON extraction guidance

README.md

@@ -703,6 +703,67 @@ firecrawl crawl https://example.com --wait -o crawl-results.json --pretty
 
 ---
 
+### `batch-scrape` - Scrape a list of URLs
+
+Scrape many known URLs as one async job. Unlike multi-URL `scrape`, every requested format is kept and results come back as JSON.
+
+```bash
+# Start a batch scrape (returns job ID)
+firecrawl batch-scrape https://example.com https://example.org
+
+# Read URLs from a file (one per line; blank lines and # comments ignored)
+firecrawl batch-scrape --urls-file urls.txt
+
+# Wait for results
+firecrawl batch-scrape --urls-file urls.txt --format markdown,links --wait --progress
+
+# Check status (returns documents scraped so far)
+firecrawl batch-scrape <job-id>
+
+# Cancel a job
+firecrawl batch-scrape <job-id> --cancel
+```
+
+#### Batch Scrape Options
+
+| Option                         | Description                                     |
+| ------------------------------ | ----------------------------------------------- |
+| `--urls-file <path>`           | File with one URL per line                      |
+| `-f, --format <formats>`       | Output format(s), comma-separated               |
+| `--only-main-content`          | Extract only main content                       |
+| `--wait-for <ms>`              | Wait time before scraping each page             |
+| `--max-age <milliseconds>`     | Maximum age of cached content in milliseconds   |
+| `--schema <json>`              | JSON schema for extraction (adds `json` format) |
+| `--schema-file <path>`         | Path to JSON schema file                        |
+| `--prompt <text>`              | Extraction prompt (adds `json` format)          |
+| `--scrape-options <json>`      | JSON scrape options applied to every URL        |
+| `--scrape-options-file <path>` | Path to scrape options JSON file                |
+| `--wait`                       | Wait for the job to complete                    |
+| `--progress`                   | Show progress while waiting                     |
+| `--timeout <seconds>`          | Timeout when waiting                            |
+| `--poll-interval <seconds>`    | Status check interval (default: 2)              |
+| `--webhook <url-or-json>`      | Webhook URL or configuration                    |
+| `--max-concurrency <n>`        | Max concurrent scrapes for this job             |
+| `--ignore-invalid-urls`        | Skip invalid URLs instead of failing the job    |
+| `--status`                     | Check status of a job by job ID                 |
+| `--cancel`                     | Cancel an active job by job ID                  |
+| `-o, --output <path>`          | Save output to file                             |
+| `--pretty`                     | Pretty print JSON output                        |
+
+#### Examples
+
+```bash
+# Extract the same fields from every page
+firecrawl batch-scrape --urls-file products.txt --wait \
+  --schema '{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"string"}}}' \
+  -o products.json --pretty
+
+# Notify a webhook instead of waiting
+firecrawl batch-scrape --urls-file urls.txt --webhook https://example.com/hooks/firecrawl
+```
+
+---
+
 ### `credit-usage` - Check your credits
 
 ```bash
@@ -1026,6 +1087,9 @@ firecrawl https://example.com --format markdown,links,images
 ```bash
 # Just pass multiple URLs -- results are saved to .firecrawl/
 firecrawl scrape https://firecrawl.dev https://firecrawl.dev/blog https://docs.firecrawl.dev
+
+# Many URLs, other formats, or JSON extraction: run one batch job
+firecrawl batch-scrape --urls-file urls.txt --format markdown,links --wait -o results.json
 ```
 
 ### Combine with other tools

package.json

@@ -1,6 +1,6 @@
 {
   "name": "firecrawl-cli",
-  "version": "1.26.17",
+  "version": "1.26.18",
   "publishConfig": {
     "tag": "latest"
   },

skills/firecrawl-scrape/SKILL.md

@@ -24,7 +24,7 @@ firecrawl scrape "<url>" --only-main-content -o .firecrawl/page.md
 # Wait for JS to render, then scrape
 firecrawl scrape "<url>" --wait-for 3000 -o .firecrawl/page.md
 
-# Multiple URLs (markdown only; each saved to .firecrawl/; -o is ignored)
+# A few URLs (markdown only; each saved to .firecrawl/; -o is ignored)
 firecrawl scrape https://example.com https://example.com/blog https://example.com/docs
 
 # Get markdown and links together
@@ -38,6 +38,40 @@ Run `firecrawl scrape --help` for the full option list.
 
 **Done when:** the page content or provider result has been checked for errors and inspected in bounded sections to answer the request. Preserve source links and disclose partial results.
 
+## Structured JSON from a page
+
+Pass a JSON schema with the `json` format to get typed fields instead of markdown:
+
+```bash
+firecrawl scrape "<url>" --format json \
+  --schema '{"type":"object","properties":{"plan":{"type":"string"},"price":{"type":"string"}},"required":["plan"]}' \
+  -o .firecrawl/pricing.json
+
+# Large schemas: keep them in a file
+firecrawl scrape "<url>" --format json --schema-file schema.json -o .firecrawl/page.json
+```
+
+The result is under `.json` in the output. `--schema` only applies when `json` is in `--format`; add other formats with commas (`--format markdown,json`). For the same schema across many URLs, use `batch-scrape --schema` below.
+
+## Many known URLs
+
+Use `firecrawl batch-scrape` for a list of known URLs when you need formats other than markdown, JSON extraction, a webhook, or a job you can check later. Keep the multi-URL `scrape` above for a few pages of markdown.
+
+```bash
+# Start a job (prints a job ID), then check it later
+firecrawl batch-scrape https://example.com/a https://example.com/b
+firecrawl batch-scrape <job-id>
+
+# Wait and save every document as JSON (urls.txt: one URL per line, # comments ok)
+firecrawl batch-scrape --urls-file urls.txt --format markdown,links --wait -o .firecrawl/batch.json
+
+# Same extraction for every URL (--schema or --prompt adds the json format)
+firecrawl batch-scrape --urls-file urls.txt --wait \
+  --schema '{"type":"object","properties":{"title":{"type":"string"}}}' -o .firecrawl/batch.json
+```
+
+Results are in `.data[]`, one document per URL with `metadata.sourceURL`. Add `--webhook <url>` to get notified, `--ignore-invalid-urls` to skip bad entries, and `--cancel` with a job ID to stop a job. Run `firecrawl batch-scrape --help` for all options.
+
 ## Find tools, inspect inputs, and get help
 
 Use the CLI help to check supported options rather than guessing:
@@ -98,7 +132,7 @@ The cap applies to each PDF, not the whole command or total credits. Extra forma
 
 - **Prefer plain scrape over `--query`.** Scrape to a file, then use `grep`, `head`, or read the markdown directly — you can search and reason over the full content yourself. Use `--query` only when you want a single targeted answer without saving the page (costs 5 extra credits).
 - **Scrape handles static pages and JS-rendered SPAs.** Escalate to `interact` when the page needs interaction (clicks, form fills, pagination) or scrape misses content.
-- Multiple URLs are scraped concurrently — check `firecrawl --status` for your concurrency limit. This mode saves markdown only and ignores `-o`; other requested formats are dropped. If markdown wasn't requested, the whole JSON response is written into the `.md` file.
+- Multiple URLs are scraped concurrently — check `firecrawl --status` for your concurrency limit. This mode saves markdown only and ignores `-o`; other requested formats are dropped. If markdown wasn't requested, the whole JSON response is written into the `.md` file. Use `batch-scrape` when you need other formats or JSON extraction.
 - Single format outputs raw content. Multiple formats (e.g., `--format markdown,links`) output JSON.
 - Always quote URLs — shell interprets `?` and `&` as special characters.
 - Naming convention: `.firecrawl/{site}-{path}.md`

skills/firecrawl/SKILL.md

@@ -39,6 +39,7 @@ Follow this escalation pattern:
 | Answer a coding question    | `developer`           | Issues, merged PRs, READMEs, and docs — not a general web page  |
 | Find US law or regulation   | `gov`                 | US statutes, regulations, codes, and court opinions             |
 | Get a page's content        | `scrape`              | Have a URL, page is static or JS-rendered                       |
+| Scrape a list of known URLs | `batch-scrape`        | Many URLs, non-markdown formats, JSON extraction, or a webhook  |
 | Find URLs within a site     | `map`                 | Need to locate a specific subpage                               |
 | Bulk extract a site section | `crawl`               | Need many pages (e.g., all /docs/)                              |
 | AI-powered data extraction  | `agent`               | Need structured data from complex sites                         |
@@ -78,6 +79,7 @@ A client context/output error does not prove the provider failed. Keep the reque
 - **Finding research papers (biomedical, clinical, or scientific literature; PubMed, bioRxiv, medRxiv, arXiv)** -> [firecrawl-research-index](../firecrawl-research-index/SKILL.md). Use the paper index instead of scraping PubMed or Google Scholar by hand; `search --categories research` is a website filter, not the paper index.
 - **Answering a library, API, error, or known-bug question from issues, merged PRs, READMEs, or docs** -> [firecrawl-developer-index](../firecrawl-developer-index/SKILL.md)
 - **Scraping a known URL** -> [firecrawl-scrape](../firecrawl-scrape/SKILL.md)
+- **Scraping a list of known URLs as one job (batch scrape) or extracting JSON with a schema** -> [firecrawl-scrape](../firecrawl-scrape/SKILL.md)
 - **Finding URLs on a known site** -> [firecrawl-map](../firecrawl-map/SKILL.md)
 - **Bulk extraction from a docs section or site** -> [firecrawl-crawl](../firecrawl-crawl/SKILL.md)
 - **AI-powered structured extraction from complex sites** -> [firecrawl-agent](../firecrawl-agent/SKILL.md)

src/__tests__/cli-aliases.test.ts

@@ -22,6 +22,7 @@ describe('CLI compatibility aliases', { timeout: 30000 }, () => {
       scrape.handleAllScrapeCommand = (_url, options) => print(options);
       require('./dist/commands/parse').handleParseCommand = print;
       require('./dist/commands/crawl').handleCrawlCommand = print;
+      require('./dist/commands/batch-scrape').handleBatchScrapeCommand = print;
       require('./dist/commands/agent').handleAgentCommand = print;
       process.argv = [process.execPath, ${JSON.stringify(cliPath)}, ...${JSON.stringify(args)}];
       require(${JSON.stringify(cliPath)});
@@ -103,7 +104,7 @@ describe('CLI compatibility aliases', { timeout: 30000 }, () => {
     expect(alias.stdout).toContain('"formats":["markdown","links"]');
   });
 
-  testWithBuiltCli.each(['crawl', 'agent'])(
+  testWithBuiltCli.each(['crawl', 'batch-scrape', 'agent'])(
     'preserves %s job status',
     (command) => {
       const result = run([