# tavily-crawl

> Claude/Codex skill by [@tavily-ai](https://omgskills.com/library/tavily-ai/index.md) · 489 stars · Updated 2026-09-04

Crawl websites and extract content from multiple pages via the Tavily CLI. Use this skill when the user wants to crawl a site, download documentation, extract an entire docs section, bulk-extract pages, save a site as local markdown files, or says "crawl", "get all the pages", "download the docs", "extract everything under /docs", "bulk extract", or needs content from many pages on the same domain. Supports depth/breadth control, path filtering, semantic instructions, and saving each page as a local markdown file.

## Install

```sh
git clone https://github.com/tavily-ai/skills /tmp/skills && ln -s /tmp/skills/skills/tavily-crawl ~/.claude/skills/tavily-crawl
```

## From README

tavily crawl Crawl a website and extract content from multiple pages. Supports saving each page as a local markdown file. Before running Crawl requires authentication. Run the requested command directly when tvly is already authenticated; do not add a status check to every invocation. If tvly is missing, follow the tavily-cli . If an installed CLI reports an authentication error, use tvly login for authentication only, or tvly init --skip-skills when guided verification is also useful. Browser-based OAuth is preferred when an interactive user can complete it. --no-browser prints the sign-in link instead of opening it, but still waits for a localhost callback. In an unattended agent or CI environment, leave authentication to the user or use a securely provided TAVILYAPIKEY. Do not start a second login immediately after guided has completed.

## Links

- HTML page: https://omgskills.com/library/tavily-ai/skills/tavily-crawl/
- GitHub: https://github.com/tavily-ai/skills
- Author: [@tavily-ai](https://omgskills.com/library/tavily-ai/index.md)

## More from this repo

- [tavily-best-practices](https://omgskills.com/library/tavily-ai/skills/tavily-best-practices/index.md) — Used to build production-ready Tavily integrations with best practices baked in.
   - [@tavily-ai](https://omgskills.com/library/tavily-ai/index.md) · 489 stars
- [tavily-cli](https://omgskills.com/library/tavily-ai/skills/tavily-cli/index.md) — Set up, authenticate, update, troubleshoot, or choose between Tavily CLI web commands.
   - [@tavily-ai](https://omgskills.com/library/tavily-ai/index.md) · 489 stars
- [tavily-dynamic-search](https://omgskills.com/library/tavily-ai/skills/tavily-dynamic-search/index.md) — Programmatic Tavily search with context isolation.
   - [@tavily-ai](https://omgskills.com/library/tavily-ai/index.md) · 489 stars