firecrawl-crawl
Bulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk extract", or needs content from many pages on the same site. Handles depth limits, path filtering, and concurrent extraction.
Install
Install this Claude and Codex skill with the command below. The command stays visible even when copy support is unavailable.
git clone https://github.com/firecrawl/cli /tmp/cli && ln -s /tmp/cli/skills/firecrawl-crawl ~/.claude/skills/firecrawl-crawlFrom README
firecrawl crawl Bulk extract content from a website. Crawls pages following links up to a depth/limit. When to use You need content from many pages on a site (e.g., all /docs/) You want to extract an entire site section Step 4 in the workflow escalation pattern: search → scrape → map → crawl → interact Options | Option | Description | | ------------------------- | ------------------------------------------- | | --wait | Wait for crawl to complete before returning | | --progress | Show progress while waiting | | --limit | Max pages to crawl | | --max-depth | Max link depth to follow | | --include-paths | Only crawl URLs matching these paths | | --exclude-paths | Skip URLs matching these paths | | --delay | Delay between requests | | --max-concurrency | Max parallel crawl workers | | --pretty | Pretty print JSON output | | -o, --output | Output file path | Tips Always use --wait when you need the results immediately. Without it, crawl returns a job ID for async polling.
More from this repo
firecrawl
Search, scrape, and interact with the web via the Firecrawl CLI.
firecrawl-agent
AI-powered autonomous data extraction that navigates complex sites and returns structured JSON.
firecrawl-download
Download an entire website as local files — markdown, screenshots, or multiple formats per page.