Skip to content
Search

Crawl4AI

lastmod 2026-11-15created 2026-11-151 edits

Open-source crawler that turns web pages into AI-friendly Markdown

About

  • Python crawler that turns web pages into LLM-friendly Markdown, making them easy to feed into AI search, RAG, and document summarization
  • Clean Markdown preserves the structure of headings, lists, tables, and code blocks, while Fit Markdown filters out clutter such as menus and footers with PruningContentFilterLXML, BM25ContentFilter, and LLMContentFilter. Page links are gathered into a numbered reference list
  • Structured extraction: extract quickly without an LLM using CSS, XPath, or regex schemas, or use any LLM to extract against a JSON schema. Describe what you want once and generate_schema writes a reusable schema for you
  • Browser control: your own browser profile with saved logins and cookies (user_data_dir), CDP remote browsers, session persistence, proxies, and stealth mode. Supports Chromium, Firefox, and WebKit
  • Infinite scroll pages are scrolled to the end and read with scan_full_page, and virtual lists that swap out items as you scroll are collected with VirtualScrollConfig
  • BFS, DFS, and best-first deep crawling; AdaptiveCrawler, which stops once it has learned enough to answer the question; URL discovery from sitemaps and Common Crawl; screenshot and PDF capture; caching; and concurrent processing of multiple URLs
  • After pip install -U crawl4ai, run crawl4ai-setup once to install the browser. The Docker server provides a REST API (/md, /crawl, /screenshot, etc.) and MCP, so you can connect it to Claude Code and have it do research. Every endpoint requires CRAWL4AI_API_TOKEN
  • There is also a hosted Crawl4AI Cloud that works without your own server (search, SERP, and answer APIs, plus managed MCP)
  • Don't commit browser profiles that contain cookies to Git, and check the terms of use and rate limits of the sites you crawl
  • Apache-2.0 license. However, using it requires attribution, such as a badge ("Powered by Crawl4AI") or a line in your documentation like "This project uses Crawl4AI (https://github.com/unclecode/crawl4ai) for web data extraction."

See also

  • Web Scraping
  • Firecrawl
  • Maxun - Open-source web data platform that turns any website into a structured API
  • Defuddle - Open-source HTML-to-Markdown alternative to Readability

Favorite site

author MCPBot sitemap