# crawl

> Claude/Codex skill by [@olostep](https://omgskills.com/library/olostep/index.md) · 1 stars · Updated 2026-03-25

Autonomously crawl an entire website starting from a URL, following links to discover and scrape pages

Tags: ai, ai-agents, claude-code, olostep, web-scraping, web-search

## Install

```sh
git clone https://github.com/olostep/olostep-claude-code-plugin /tmp/olostep-claude-code-plugin && ln -s /tmp/olostep-claude-code-plugin/skills/crawl ~/.claude/skills/crawl
```

## From README

Olostep Crawl Autonomously discover and scrape entire websites by following links from a starting URL. Instructions When the user wants to crawl a website (via $ARGUMENTS or in conversation): Use the mcpolostepcreatecrawl tool with the starting URL. Default to maxpages: 10 unless the user specifies otherwise. Default to markdown output format unless specified. Inform the user that crawling follows links automatically.

## Links

- HTML page: https://omgskills.com/library/olostep/olostep-claude-code-plugin/skills/crawl/
- GitHub: https://github.com/olostep/olostep-claude-code-plugin
- Author: [@olostep](https://omgskills.com/library/olostep/index.md)

## More from this repo

- [answers](https://omgskills.com/library/olostep/olostep-claude-code-plugin/skills/answers/index.md) — Get AI-powered answers with citations and sources from the web using Olostep
   - [@olostep](https://omgskills.com/library/olostep/index.md) · 1 stars
- [batch](https://omgskills.com/library/olostep/olostep-claude-code-plugin/skills/batch/index.md) — Scrape up to 10,000 URLs in a single batch operation using Olostep
   - [@olostep](https://omgskills.com/library/olostep/index.md) · 1 stars
- [debug-error](https://omgskills.com/library/olostep/olostep-claude-code-plugin/skills/debug-error/index.md) — Used to search the live web for a specific error message and find the real fix — from GitHub issues, StackOverflow, or...
   - [@olostep](https://omgskills.com/library/olostep/index.md) · 1 stars