Web Scraping APIs — procurement report
ProductArena · rankings as of 2026-09-15 · evidence as of 2026-09-15 · 8 products · 96 judged requirements · 768 judged cells
Methodology: Every product is judged against a shared taxonomy of user stories using cited evidence — hands-on probes > repository code > independent community sources > vendor claims — never opinion. Full writeup: https://ultrametric.ai/productarena/methodology
Leaderboard
| # | Product | PA Score | Coverage score | Applicable cells | Confidence |
|---|---|---|---|---|---|
| 1 | Apify | 35.6 | 22.5 | 96/96 | D |
| 2 | Context.dev | 35.1 | 24.7 | 94/96 | C |
| 3 | Firecrawl | 31.5 | 26.4 | 95/96 | C |
| 4 | Crawl4AI | 29.6 | 26.1 | 90/96 | B |
| 5 | Browserbase | 27.2 | 19.3 | 95/96 | C |
| 6 | Jina Reader | 22.9 | 22.1 | 94/96 | C |
| 7 | Riveter | 22.3 | 15.2 | 95/96 | D |
| 8 | ScrapingBee | 18.8 | 24.0 | 94/96 | C |
PA Score = agent-readiness blend (see methodology). Coverage score = weighted share of judged requirements met. Confidence = how much of the score rests on tested vs claimed evidence (A–D).
Uncertainty note
This arena is currently a close race: Apify (35.6) vs Context.dev (35.1), a gap of 0.5 PA Score. The ordering was re-checked with extra judge samples: 38 decisive cells were triple-judged and 6 came back unstable. Treat the #1/#2 ordering as contested — shortlist both.
Buyer checklist (RFP)
The arena's 96 judged user stories as requirements, grouped by theme. Priorities mirror the story weights our scoring uses (3 = must-have, 2 = should-have, 1 = nice-to-have). Interactive version with per-requirement verdicts for the top products: /arena/web-scraping/checklist
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
- ai-native userPlug MCP servers into this product so it can use their toolsmust-have
- ai-native userConnect an agent via an official MCP servermust-have
- ai-native userDrive the product through a documented public APImust-have
- ai-native userDelegate tasks to a built-in AI assistant inside the productmust-have
- data-engineerThe documented rate limit (requests per second or minute) enforced on my API key before throttling kicks inmust-have
- ai-native userPoint an agent at llms.txt or agent-oriented docsshould-have
- ai-native userRun the product headlessly / in CI for automationshould-have
- ai-native userUse an official CLIshould-have
- ai-native userIssue scoped/least-privilege API credentials for an agentshould-have
- ai-native userBuild against official SDKsshould-have
- ai-native userSubscribe to events via webhooksshould-have
- ai-native userGet AI-generated insights and suggestions from my data inside the productshould-have
- ai-native userSet up automations that run autonomously in the backgroundshould-have
- ai-native userOperate the product with natural-language commandsshould-have
- ai-native userApply a preset configuration tuned for research agents that returns structured, citable outputshould-have
- ai-native userExplore an interactive API reference with runnable examplesshould-have
- ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)should-have
- ai-native userRely on versioned APIs with a documented deprecation policyshould-have
- ai-native userTest against a sandbox environment without touching production datanice-to-have
Anti bot — getting past bot defenses — CAPTCHAs, fingerprinting, blocksAnti bot
Getting past bot defenses — CAPTCHAs, fingerprinting, blocks
- developerUse an undetected browser mode to bypass sophisticated bot detection systemsmust-have
- developerUse premium residential or datacenter proxies to bypass sites that are hard to scrapemust-have
- developerRoute requests through a rotating pool of proxy IPs to avoid blocksmust-have
- ai-native userHave an agent automatically get past a CAPTCHA, login, or form wall without my manual interventionshould-have
- data-engineerAutomatically retry through a chain of different proxies when anti-bot detection blocks a requestshould-have
- developerRequest a proxy from a specific country to get geolocation-appropriate contentshould-have
- developerRoute multiple requests through the same proxy IP using a session identifier to maintain a consistent identityshould-have
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
- ai-native userDefine rules that trigger actions automatically on eventsmust-have
- ai-native userPerform bulk operations across many items at onceshould-have
- ai-native userSchedule recurring jobs or workflowsshould-have
- ai-native userVersion, review, and roll back my automationsnice-to-have
Dev experience — day-to-day developer experience — setup friction, docs, debugging, iteration speedDev experience
Day-to-day developer experience — setup friction, docs, debugging, iteration speed
- developerExport my scraped data and job configurations in a portable format to migrate to another provider without lock-inmust-have
- developerShare scrapers with teammates and manage organizations and role-based permissionsshould-have
- developerBuild and deploy custom serverless scraping scripts on the platform without managing my own infrastructureshould-have
- developerDeploy the scraping service via a Docker container for production useshould-have
- developerSelf-host an open-source version of the scraper instead of relying on a hosted cloud serviceshould-have
- developerConnect the scraping API to no-code automation platforms like n8n or Zapier through a prebuilt connectorshould-have
- developerBuild scrapers using popular open-source automation libraries like Playwright, Puppeteer, Selenium, or Scrapyshould-have
- developerRun a ready-made scraper from a marketplace instead of building one from scratchshould-have
- developerPublish my custom scraper to a public marketplace and earn revenue when others use itnice-to-have
- developerStart building immediately using a library of ready-made project templatesnice-to-have
Extraction quality — how faithfully content is extracted — structure, fidelity, edge casesExtraction quality
How faithfully content is extracted — structure, fidelity, edge cases
- developerExtract structured data from a page using natural language instructions instead of writing selectorsmust-have
- developerScrape a web page with a single API call and get its raw HTML backmust-have
- developerSearch the web and get full page content from results in a single call instead of just links and snippetsmust-have
- developerExtract specific fields from a page using CSS or XPath selector rulesmust-have
- developerPass a JSON schema so the API returns structured data matching that schemashould-have
- ai-native userHave an LLM read a page and decide what structured fields to pull out without pre-written selectorsshould-have
- developerPlug in a local or self-hosted LLM as the extraction backend instead of a cloud-only modelshould-have
- data-engineerAutomatically detect and filter personally identifiable information out of scraped content before it reaches storageshould-have
- data-engineerExtract text content from PDFs, Word, Excel, and PowerPoint files without hosting them myselfshould-have
- ai-native userGet automatic captions for images on a page so a text-only model can reason about visual contentshould-have
- data-engineerExtract data from very large tables using intelligent chunking so it fits within processing limitsnice-to-have
Js rendering — handling JavaScript-heavy pages — rendering, waiting, dynamic contentJs rendering
Handling JavaScript-heavy pages — rendering, waiting, dynamic content
- developerRender JavaScript-heavy single-page applications and get the fully rendered HTMLmust-have
- developerScript page interactions like clicking, filling inputs, and scrolling before content is returnedmust-have
- developerHave the API wait for a specific selector to appear before returning the rendered pageshould-have
- developerAccess a managed remote browser sandbox for interactive, manual browsing workflowsshould-have
- developerKeep interacting with an already-scraped page, clicking and filling forms to reach content behind a login wallshould-have
- developerPass my own session cookies so the API fetches pages requiring authenticationshould-have
- developerReuse a persistent browser profile with saved cookies and login state across multiple requestsshould-have
- developerControl the browser viewport width and height when rendering a pagenice-to-have
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
- ai-native userExport all of my data in open formats and leavemust-have
- ai-native userSelf-host the core productmust-have
- ai-native userDo everything through the API that I can do in the UIshould-have
- ai-native userRead the product's source under an open licenseshould-have
Output formats — stories about output formats in this arenaOutput formats
Stories about output formats in this arena
- developerReceive scraped content as clean markdown instead of raw HTMLmust-have
- developerReceive scraped content as structured JSONmust-have
- ai-native userGet clean LLM-ready text directly instead of dealing with blocking, rendering, and messy HTML myselfmust-have
- developerChoose exactly which output format is returned, such as markdown, HTML, text, or frontmattershould-have
- ai-native userRequest semantically chunked output instead of one large content blob, so it feeds cleanly into a retrieval pipelineshould-have
- developerCapture a screenshot of a full page or a specific selected areashould-have
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
- developerWhether exceeding my plan's monthly credit or request quota triggers overage charges or a hard cutoffmust-have
- developerSet a spending cap or usage alert so proxy/credit consumption doesn't silently blow past my budgetmust-have
- developerLet the API automatically pick the cheapest configuration that still succeedsshould-have
- ai-native userSet how much reasoning effort an autonomous agent spends on a data-gathering task (low, medium, high)should-have
- developerWhether failed, blocked, or empty-result requests still consume my billing quotashould-have
- data-engineerThe maximum concurrent sessions or requests allowed on my pricing tier and the cost to raise that capshould-have
- developerBlock ads on the target page to speed up scraping requestsnice-to-have
- developerBlock images and CSS resources by default to reduce bandwidth and speed up requestsnice-to-have
- developerTrade off latency against completeness by controlling exactly when content is returnednice-to-have
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
- ai-native userPrevent my data from being used to train AI modelsmust-have
- ai-native userChoose where my data is stored (region/residency)should-have
- ai-native userControl data retention and deletionshould-have
- ai-native userOpt out of telemetry and usage trackingshould-have
Scale reliability — behavior under load — scaling limits, uptime, failure handlingScale reliability
Behavior under load — scaling limits, uptime, failure handling
- data-engineerBatch scrape thousands of URLs asynchronouslymust-have
- data-engineerSpin up many concurrent scraping sessions to gather data at scalemust-have
- developerCrawl an entire website and get content from all its pages with one requestmust-have
- ai-native userRely on adaptive crawling that automatically stops once enough information has been gathered to answer my queryshould-have
- data-engineerConfigure the crawler to respect robots.txt rules and target-site rate limits automaticallyshould-have
- data-engineerResume a crashed deep crawl from a saved checkpoint instead of restarting from scratchshould-have
- data-engineerCheck a public status page showing uptime history and past incident postmortems before committing to the serviceshould-have
- data-engineerMonitor target pages for content changes, such as price or listing updates, and get notified as they happenshould-have
- data-engineerMonitor job performance, validate data quality, and receive alerts when something failsshould-have
- developerSchedule scraping jobs to run automatically at specific timesshould-have
- data-engineerRun a deep crawl using a breadth-first strategy with a configurable maximum page limitshould-have
- developerInstantly discover all URLs on a website without fully crawling itshould-have
- developerApply different crawl configurations to different URL patterns within a single batch jobnice-to-have
- developerMonitor live system metrics and worker/browser pool status through a real-time dashboardnice-to-have
Pricing signals
Extracted verbatim from each vendor's own pricing page — never converted, averaged, or derived. Products whose page prints no unit price are recorded as unclear, honestly.
| Product | Headline price | Unit | As of |
|---|---|---|---|
| Apify | $19 | month (entry plan) | 2026-09-07 |
| Firecrawl | $5 | 1k pages | 2026-09-07 |
| Crawl4AI | pricing unclear — Crawl4AI is a free open-source tool with no printed pricing; its Cloud API is in closed beta with no published prices. | 2026-09-07 | |
| Browserbase | $1 | 1k pages | 2026-09-07 |
| Jina Reader | pricing unclear — The page prices Reader API usage in tokens with a free token allowance and prints no per-page, per-1k-pages, per-browser-hour, or monthly plan figures without arithmetic conversion. | 2026-09-07 | |
| ScrapingBee | $19 | month (entry plan) | 2026-09-07 |
Appendix: recorded probes
Hands-on probe recordings — transcripts/videos a human can replay, the strongest evidence tier. Watch them at https://ultrametric.ai/productarena/proofs
- Context.dev
curl -si https://api.context.dev/v1/web/scrape/markdownterminal · recorded 2026-09-15 · exit 0 - Context.dev
curl -si -X POST https://mcp.context.dev/mcp -H 'Content-Type: application/json' -d '<jsonrpc initialize>'terminal · recorded 2026-09-15 · exit 0 - Context.dev
curl -s https://www.context.dev/openapi.json | head -c 300terminal · recorded 2026-09-15 · exit 0 - Firecrawl
playwright: load https://docs.firecrawl.dev/api-reference/endpoint/scrape and locate the documented POST https://api.firecrawl.dev/v2/scrape endpointvideo · recorded 2026-09-03 · exit 0 - Riveter
curl -si -X POST https://api.riveterhq.com/v1/quick_search -H 'Content-Type: application/json' -d '{"query":"productarena probe"}'terminal · recorded 2026-09-14 · exit 0 - Riveter
curl -s https://docs.riveterhq.com/openapi.yaml | head -3terminal · recorded 2026-09-14 · exit 0 - Riveter
mktemp -d && uv venv && uv pip install riveter-sdk && python -c "import riveter; print('PA_PROBE_OK riveter-sdk', version('riveter-sdk'))"terminal · recorded 2026-09-14 · exit 0
Cite as: ProductArena by Ultrametric Inc, Web Scraping APIs arena, rankings as of 2026-09-15 — https://ultrametric.ai/productarena/arena/web-scraping
License: © 2026 Ultrametric Inc. Brief quotation of individual verdicts, scores, or evidence excerpts is permitted with attribution to "ProductArena by Ultrametric Inc (ultrametric.ai/productarena)", as is use of the data to evaluate, contest, or contribute corrections. Bulk copying, redistribution, or use to build competing datasets requires prior written permission (see DATA-LICENSE in the repository).
No liability: rankings, verdicts, and scores are research outputs derived from the cited evidence at a point in time, provided "as is", without warranties. Ultrametric Inc accepts no responsibility for procurement, purchasing, or other decisions made in reliance on them — verify against the cited evidence before acting (https://ultrametric.ai/productarena/terms).