Crawl4AI

Connect an agent via an official MCP server G

Agent access

Drive the product through a documented public API G

Agent access

Plug MCP servers into this product so it can use their tools G

Agent access

Delegate tasks to a built-in AI assistant inside the product G

Agentic features

The documented rate limit (requests per second or minute) enforced on my API key before throttling kicks in G

Api quality

Run the product headlessly / in CI for automation G

Agent access

Use an official CLI G

Agent access

Build against official SDKs G

Agent access

Apply a preset configuration tuned for research agents that returns structured, citable output C

Agentic features

Operate the product with natural-language commands G

Agentic features

Get AI-generated insights and suggestions from my data inside the product G

Agentic features

Set up automations that run autonomously in the background G

Agentic features

Download a machine-readable API spec (OpenAPI or equivalent) G

Api quality

Explore an interactive API reference with runnable examples G

Api quality

Issue scoped/least-privilege API credentials for an agent G

Agent access

Point an agent at llms.txt or agent-oriented docs G

Agent access

Rely on versioned APIs with a documented deprecation policy G

Api quality

Subscribe to events via webhooks G

Agent access

Test against a sandbox environment without touching production data G

Api quality

Crawl an entire website and get content from all its pages with one request C

Site crawling

Get clean LLM-ready text directly instead of dealing with blocking, rendering, and messy HTML myself C

Llm ready output

Receive scraped content as clean markdown instead of raw HTML C

Content formats

Self-host the core product G

Export all of my data in open formats and leave G

Batch scrape thousands of URLs asynchronously C

Batch processing

Extract specific fields from a page using CSS or XPath selector rules C

Selector extraction

Extract structured data from a page using natural language instructions instead of writing selectors C

Ai extraction

Receive scraped content as structured JSON C

Content formats

Scrape a web page with a single API call and get its raw HTML back C

Basic scraping

Spin up many concurrent scraping sessions to gather data at scale C

Concurrency

Use an undetected browser mode to bypass sophisticated bot detection systems C

Block evasion

Render JavaScript-heavy single-page applications and get the fully rendered HTML C

Headless rendering

Route requests through a rotating pool of proxy IPs to avoid blocks C

Proxy rotation

Export my scraped data and job configurations in a portable format to migrate to another provider without lock-in C

Migration lock in

Use premium residential or datacenter proxies to bypass sites that are hard to scrape C

Proxy rotation

Define rules that trigger actions automatically on events G

Prevent my data from being used to train AI models G

Search the web and get full page content from results in a single call instead of just links and snippets C

Search integration

Whether exceeding my plan's monthly credit or request quota triggers overage charges or a hard cutoff G

Cost transparency

Script page interactions like clicking, filling inputs, and scrolling before content is returned C

Interactive automation

Set a spending cap or usage alert so proxy/credit consumption doesn't silently blow past my budget G

Cost transparency

Self-host an open-source version of the scraper instead of relying on a hosted cloud service C

Deployment flexibility

Perform bulk operations across many items at once G

Plug in a local or self-hosted LLM as the extraction backend instead of a cloud-only model C

Ai extraction

Run a deep crawl using a breadth-first strategy with a configurable maximum page limit C

Site crawling

Deploy the scraping service via a Docker container for production use C

Deployment flexibility

Reuse a persistent browser profile with saved cookies and login state across multiple requests C

Session persistence

Automatically retry through a chain of different proxies when anti-bot detection blocks a request C

Block evasion

Have an LLM read a page and decide what structured fields to pull out without pre-written selectors C

Ai extraction

Pass my own session cookies so the API fetches pages requiring authentication C

Session persistence

Read the product's source under an open license G

Rely on adaptive crawling that automatically stops once enough information has been gathered to answer my query C

Ai driven crawling

Resume a crashed deep crawl from a saved checkpoint instead of restarting from scratch C

Fault tolerance

Do everything through the API that I can do in the UI G

Pass a JSON schema so the API returns structured data matching that schema C

Ai extraction

Choose exactly which output format is returned, such as markdown, HTML, text, or frontmatter C

Content formats

Have an agent automatically get past a CAPTCHA, login, or form wall without my manual intervention C

Block evasion

Keep interacting with an already-scraped page, clicking and filling forms to reach content behind a login wall C

Interactive automation

Request semantically chunked output instead of one large content blob, so it feeds cleanly into a retrieval pipeline C

Llm ready output

Choose where my data is stored (region/residency) G

Monitor job performance, validate data quality, and receive alerts when something fails C

Scheduling monitoring

Automatically detect and filter personally identifiable information out of scraped content before it reaches storage C

Data safety

Build and deploy custom serverless scraping scripts on the platform without managing my own infrastructure C

Deployment flexibility

Build scrapers using popular open-source automation libraries like Playwright, Puppeteer, Selenium, or Scrapy C

Library compatibility

Configure the crawler to respect robots.txt rules and target-site rate limits automatically C

Crawl compliance

Connect the scraping API to no-code automation platforms like n8n or Zapier through a prebuilt connector C

Integrations

Control data retention and deletion G

Have the API wait for a specific selector to appear before returning the rendered page C

Headless rendering

Instantly discover all URLs on a website without fully crawling it C

Site crawling

Request a proxy from a specific country to get geolocation-appropriate content C

Proxy rotation

Route multiple requests through the same proxy IP using a session identifier to maintain a consistent identity P

Proxy rotation

Schedule recurring jobs or workflows G

Schedule scraping jobs to run automatically at specific times C

Scheduling monitoring

The maximum concurrent sessions or requests allowed on my pricing tier and the cost to raise that cap G

Plan scale limits

Access a managed remote browser sandbox for interactive, manual browsing workflows C

Interactive automation

Capture a screenshot of a full page or a specific selected area C

Visual capture

Check a public status page showing uptime history and past incident postmortems before committing to the service G

Operational transparency

Extract text content from PDFs, Word, Excel, and PowerPoint files without hosting them myself C

Document extraction

Get automatic captions for images on a page so a text-only model can reason about visual content C

Multimodal extraction

Let the API automatically pick the cheapest configuration that still succeeds C

Cost optimization

Monitor target pages for content changes, such as price or listing updates, and get notified as they happen C

Scheduling monitoring

Opt out of telemetry and usage tracking G

Run a ready-made scraper from a marketplace instead of building one from scratch C

Quickstart

Set how much reasoning effort an autonomous agent spends on a data-gathering task (low, medium, high) C

Cost optimization

Share scrapers with teammates and manage organizations and role-based permissions P

Collaboration

Whether failed, blocked, or empty-result requests still consume my billing quota G

Cost transparency

Apply different crawl configurations to different URL patterns within a single batch job C

Batch processing

Extract data from very large tables using intelligent chunking so it fits within processing limits C

Structured data handling

Monitor live system metrics and worker/browser pool status through a real-time dashboard C

Scheduling monitoring

Trade off latency against completeness by controlling exactly when content is returned C

Performance tuning

Start building immediately using a library of ready-made project templates C

Quickstart

Block ads on the target page to speed up scraping requests C

Cost optimization

Block images and CSS resources by default to reduce bandwidth and speed up requests C

Cost optimization

Control the browser viewport width and height when rendering a page C

Render configuration

Publish my custom scraper to a public marketplace and earn revenue when others use it P

Quickstart

Version, review, and roll back my automations G