Cloudflare:Crawl

lastmod 2026-03-12created 2026-03-121 edits

Crawl an entire website with a single API call

About

• Cloudflare offers a new /crawl endpoint for Browser Rendering in open beta, which crawls an entire website with just one API call
• Submit a starting URL and it automatically navigates the pages, renders them with a headless browser, and returns the results in HTML, Markdown, or JSON format
• Provides structured JSON output powered by Workers AI, scope controls such as crawl depth, page count limits, and wildcard patterns, plus incremental crawling, static mode, and more
• Respects robots.txt rules and also supports crawl-delay to prevent abnormal traffic
• Useful for model training, building RAG pipelines, and site-wide content research and monitoring

See also

• Cloudflare

Favorite site

• Crawl entire websites with a single API call using Browser Rendering · Changelog
• Cloudflare's /crawl: crawl an entire website with a single API call | GeekNews