AI-driven CLI tool — turns websites into command-line interfaces.
deepscrape is a blazing-fast CLI tool that scrapes, crawls, extracts, and transforms web content into structured data — all from your terminal.
- Configurable depth — crawl from 1 to unlimited levels deep
- Concurrent crawling — adaptive rate-limiting with politeness settings
- Sitemap support — auto-discover URLs from
sitemap.xml - Robots.txt — respects robots.txt with caching
- File downloads — download PDFs, images, documents during crawl
- Domain filtering — stay on-site or follow external links
- HTML → Markdown — clean article extraction
- JSON & CSV parsing — extract structured data
- PDF text extraction — read text from PDFs
- Image metadata — EXIF and format detection
| Command | Description |
|---|---|
deepscrape crawl <url> |
Deep crawl websites with configurable options |
deepscrape extract <url> |
Extract all content and files from a webpage |
deepscrape download <url> |
Download files with naming strategies |
deepscrape weread <url> |
Read web content with AI-powered summarization |
deepscrape readwise |
Readwise & Reader CLI for highlights and reading list |
deepscrape doctor |
Run diagnostics checks |
Grab the latest binary from the Releases page.
git clone https://github.com/RootBugs/deepscrape.git
cd deepscrape
cargo build --release
./target/release/deepscrape# Crawl a website 2 levels deep
deepscrape crawl https://example.com --depth 2
# Download all PDFs from a page
deepscrape download https://example.com/docs
# Extract content as markdown
deepscrape extract https://example.com/articledeepscrape/
├── crates/
│ ├── deepscrape-ai/ # AI-powered content generation
│ ├── deepscrape-browser/ # Headless browser automation
│ ├── deepscrape-cli/ # CLI entry point & commands
│ ├── deepscrape-core/ # Core types & abstractions
│ ├── deepscrape-crawl/ # Web crawling engine
│ ├── deepscrape-discovery/ # CLI discovery & YAML parsing
│ ├── deepscrape-external/ # External CLI integration
│ ├── deepscrape-output/ # Output formatting (JSON, YAML, CSV, Markdown)
│ └── deepscrape-pipeline/ # Fetch-transform pipeline
└── Cargo.toml
MIT — see LICENSE.
Built with ❤️ and Rust.