crawl
Autonomously crawl an entire website starting from a URL, following links to discover and scrape pages
Install
With your agent
Install OMGSkills skill "olostep/olostep-claude-code-plugin:skills/crawl" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/olostep/olostep-claude-code-plugin /tmp/olostep-claude-code-plugin && ln -s /tmp/olostep-claude-code-plugin/skills/crawl ~/.claude/skills/crawlFrom README
Olostep Crawl Autonomously discover and scrape entire websites by following links from a starting URL. Instructions When the user wants to crawl a website (via $ARGUMENTS or in conversation): Use the mcpolostepcreatecrawl tool with the starting URL. Default to maxpages: 10 unless the user specifies otherwise. Default to markdown output format unless specified. Inform the user that crawling follows links automatically.
More from this repo
answers
Get AI-powered answers with citations and sources from the web using Olostep
batch
Scrape up to 10,000 URLs in a single batch operation using Olostep
debug-error
Used to search the live web for a specific error message and find the real fix — from GitHub issues, StackOverflow, or...