ScrapingBee vs Context.dev
ScrapingBee
ScrapingBee
Context.dev wins · 20–27 (47 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to Context.devScrapingBee has a confirmed live llms.txt at the documented URL (HTTP 200) serving an index of its documentation, directly satisfying the story of pointing an agent at agent-oriented docs; it also has an official MCP server further supporting agentic access. Missing for 10: a per-page markdown/docs.md mirror (probe shows 404) and an OpenAPI spec discoverable at standard paths, which would round out machine-readable doc coverage.
- [probe] “PROBE llms.txt: HTTP 200 at https://www.scrapingbee.com/llms.txt # ScrapingBee Documentation > Official documentation index for ScrapingBee…”
- [probe] “PROBE docs-md: HTTP 404 at https://www.scrapingbee.com/documentation.md”
- [probe] “official MCP server documented at https://mcp.scrapingbee.com/”
Context.dev has a confirmed live llms.txt at docs.context.dev/llms.txt (HTTP 200, agent-oriented index of docs), plus agent-oriented docs, MCP server, CLI, and a coding-agent skill install guide, directly enabling an agent to be pointed at agent-native documentation. Missing for 10: independent third-party confirmation that agents successfully consume the llms.txt in practice beyond the probe check.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.context.dev/llms.txt # Context.dev - [The go-to web data API](https://docs.context.dev/introductio…”
- [claimed-docs] “Connect your AI client to Context.dev tools for live web and company data.”
- [claimed-docs] “Call Context.dev from your terminal and use JSON responses in scripts or CI.”
- [claimed-docs] “Teach your coding agent how to choose and use the Context.dev API.”
- [probe] “official MCP server documented at https://mcp.context.dev/mcp”
- [probe] “official CLI documented at https://docs.context.dev/install-cli”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to Context.devScrapingBee is a pure API service (API key + HTTP calls) designed for programmatic use, and it ships an official CLI (scrapingbee-cli) enabling scripted/headless invocation suitable for CI pipelines. Missing for 10: explicit CI/CD pipeline examples (e.g., GitHub Actions), and no independent hands-on confirmation of CLI use in automation contexts.
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
- [probe] “official CLI documented at https://github.com/ScrapingBee/scrapingbee-cli”
Context.dev ships a CLI explicitly documented for scripting and CI use ('Call Context.dev from your terminal and use JSON responses in scripts or CI'), backed by a full REST API with OpenAPI spec, async batch jobs for long-running headless crawls, and documented rate-limit/timeout handling suited to automated pipelines. Missing for 10: no explicit CI/CD pipeline example (e.g., GitHub Actions), and no independent/community confirmation of headless CI usage beyond vendor docs.
- [claimed-docs] “Call Context.dev from your terminal and use JSON responses in scripts or CI.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “`return-partial` | Return usable completed work with a completion marker. If no usable result exists, fail without a charge.”
- [claimed-docs] “Authenticated API responses expose these headers when a per-minute limit applies”
- [probe] “PROBE openapi: HTTP 200 at https://docs.context.dev/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.context.dev/install-cli”
ai-native userConnect an agent via an official MCP server
weight 3 · round to Context.devScrapingBee is a scraping API/service (not itself an agent), so publishing an official MCP server is a valid axis; a probe confirms an official MCP server is documented at mcp.scrapingbee.com. However, there is no first-party documentation detail on the MCP server's tool set, setup, or independent/community corroboration of it working. Missing for 10: detailed official docs on MCP server capabilities/setup, independent hands-on confirmation.
- [probe] “official MCP server documented at https://mcp.scrapingbee.com/”
Context.dev is a web-data API (not itself an agent), so the MCP-server axis applies, and it publishes an official hosted MCP endpoint (mcp.context.dev/mcp) plus install docs for connecting AI clients to its tools for live web/company data. Missing for 10: independent/hands-on verification of the MCP server working in practice beyond first-party docs and a probe confirming the endpoint exists.
- [probe] “official MCP server documented at https://mcp.context.dev/mcp”
- [claimed-docs] “Connect your AI client to Context.dev tools for live web and company data.”
ai-native userUse an official CLI
weight 2 · round to ScrapingBeeEvidence shows an official CLI repo published by ScrapingBee (scrapingbee-cli on GitHub), directly satisfying the story. However, there's no documentation excerpt describing CLI features, installation, or usage details, nor independent community corroboration of its use. Missing for 10: detailed CLI documentation/usage examples, independent/hands-on validation, and confirmation of active maintenance.
- [probe] “official CLI documented at https://github.com/ScrapingBee/scrapingbee-cli”
Docs and probe confirm an official CLI exists ('Call Context.dev from your terminal and use JSON responses in scripts or CI') with a dedicated install page, supporting agentic/CI workflows. However, there's no independent/hands-on corroboration of the CLI's functionality or depth beyond first-party docs. Missing for 10: independent verification/hands-on review of CLI usage, details on CLI command coverage vs the full API surface.
- [claimed-docs] “Call Context.dev from your terminal and use JSON responses in scripts or CI.”
- [probe] “official CLI documented at https://docs.context.dev/install-cli”
ai-native userDrive the product through a documented public API
weight 3 · round to Context.devScrapingBee's entire product is a documented public REST API with extensive parameter documentation (docs-1 to docs-15) and an llms.txt index for AI discoverability (probe-1), plus official CLI and MCP server (probe-4, probe-5) enabling agentic access. Missing for 10: a machine-readable OpenAPI/Swagger spec (probe-2 and probe-3 both 404) and independent hands-on confirmation of API integration ease.
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
- [probe] “PROBE llms.txt: HTTP 200 at https://www.scrapingbee.com/llms.txt # ScrapingBee Documentation > Official documentation index for ScrapingBee…”
- [probe] “official MCP server documented at https://mcp.scrapingbee.com/”
- [probe] “official CLI documented at https://github.com/ScrapingBee/scrapingbee-cli”
- [probe] “PROBE openapi: all candidate paths 404 (https://www.scrapingbee.com/openapi.json, https://www.scrapingbee.com/swagger.json, https://www.scra…”
Context.dev is fundamentally an API product with a public OpenAPI spec, documented endpoints (crawl, extract, screenshot, brand data, auth), API key management, rate-limit headers, plus a CLI and MCP server built on top of the same API — clear evidence of a documented, drivable public API for AI-native consumption. Missing for 10: independent third-party developer confirmation of full API coverage beyond docs/probes.
- [probe] “PROBE openapi: HTTP 200 at https://docs.context.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [claimed-docs] “Choose **Restricted** when an integration needs only selected operations; a restricted key with no permissions cannot call the API.”
- [claimed-docs] “Call Context.dev from your terminal and use JSON responses in scripts or CI.”
- [claimed-docs] “Authenticated API responses expose these headers when a per-minute limit applies”
- [claimed-docs] “discover → register → deliver setup link & code to the user → user completes claim in browser → poll for access_token → call API.”
- [probe] “official CLI documented at https://docs.context.dev/install-cli”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to Context.devScrapingBeenone0/10Evidence only shows a single API key model for authentication with no mention of scoped, restricted-permission, or per-agent credential issuance; the community note about using two API keys does not indicate least-privilege scoping. Missing for 10: any documentation of scoped/restricted API keys, role-based permissions, or credential issuance mechanisms for agents.
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
- [community] “cool idea, but I don't like how I need to use two separate API keys to connect to this API (which relies on ScrapingBee for scraping) - can …”
Docs explicitly describe restricted API keys scoped to selected operations only, with a no-permission key unable to call the API at all, directly supporting least-privilege credential issuance for agents; the OAuth-like device flow (discover→register→claim→poll) also supports scoped token issuance per client. missing for 10: no evidence of fine-grained scoping beyond operation-level (e.g., resource/data scoping), and no independent/hands-on confirmation of restricted-key behavior in production.
- [claimed-docs] “Choose **Restricted** when an integration needs only selected operations; a restricted key with no permissions cannot call the API.”
- [claimed-docs] “discover → register → deliver setup link & code to the user → user completes claim in browser → poll for access_token → call API.”
ai-native userBuild against official SDKs
weight 2 · round drawnScrapingBeenone0/10The evidence pack documents ScrapingBee's REST API parameters, an official CLI, and an MCP server, but contains no mention of official SDKs (e.g., Python, Node.js, PHP client libraries) that AI-native developers could build against. Absence of evidence for this applicable capability warrants a 'none' verdict.
Context.devnone0/10The evidence shows an OpenAPI spec, CLI, MCP server, and 'skill' for coding agents, but there is no mention of official SDK client libraries (e.g., Python, JS, Go packages) for Context.dev. Missing for 10: explicit official SDK packages/documentation, language-specific client libraries, versioning/release notes for SDKs.
- [probe] “PROBE openapi: HTTP 200 at https://docs.context.dev/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.context.dev/install-cli”
- [claimed-docs] “Call Context.dev from your terminal and use JSON responses in scripts or CI.”
ai-native userSubscribe to events via webhooks
weight 2 · round to Context.devScrapingBeenone0/10No evidence of any webhook subscription or event notification system in ScrapingBee's documentation; all evidence covers synchronous scraping API parameters, proxies, and rendering options with no mention of webhooks or event-driven callbacks.
Context.dev supports monitoring pages/sitemaps/datasets and receiving 'signed change events' on a schedule, which functions as a webhook-like event delivery mechanism, but the docs never explicitly describe a subscribe/webhook API, event types, delivery retries, or webhook management endpoints. missing for 10: explicit webhook subscription/management API docs, event schema/type documentation, delivery reliability/retry details, and independent confirmation of webhook functionality.
- [claimed-docs] “Watch a page, sitemap, or structured dataset on a schedule and receive signed change events.”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round to ScrapingBeeScrapingBee's `ai_query` parameter lets users ask AI to extract specific information from scraped pages, which is a limited AI capability applied to data the product handles, but it's user-directed extraction rather than proactive AI-generated insights or suggestions surfaced inside the product. Missing for 10: evidence of automated insight generation, trend/anomaly detection, or suggestion features beyond on-demand query-based extraction.
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
Context.devnone0/10Context.dev is a data-extraction/scraping API (Markdown, structured JSON extraction, screenshots, brand data) intended to feed external AI agents and applications, but there is no evidence of the product itself surfacing AI-generated insights, recommendations, or analysis inside a Context.dev interface — it delivers raw/structured data, not in-product AI insight generation.
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
- [claimed-docs] “retrieve brand profiles with logos, colors, descriptions, and social links through the same API.”
- [claimed-docs] “Connect your AI client to Context.dev tools for live web and company data.”
ai-native userSet up automations that run autonomously in the background
weight 2 · round to Context.devScrapingBeenone0/10ScrapingBee is an on-demand scraping API/CLI/MCP server; evidence shows only synchronous request-response scraping calls, with no scheduling, triggers, or background job/automation orchestration features documented.
Context.dev supports background automation via async batch crawling that runs as a tracked job until completion, and scheduled monitoring of pages/sitemaps/datasets that emits signed change events without user intervention — both run autonomously once configured. However, there's no evidence of a broader automation/workflow engine (e.g., chaining actions, triggering downstream agent tasks, retries/orchestration) beyond these two specific background job types. Missing for 10: evidence of workflow chaining or agent-triggered automation, independent confirmation of monitoring reliability, and details on scheduling flexibility.
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Watch a page, sitemap, or structured dataset on a schedule and receive signed change events.”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round drawnScrapingBeenone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
ai-native userOperate the product with natural-language commands
weight 2 · round drawnScrapingBee's `ai_query` parameter lets users specify what to extract from a page using natural language, and there is a documented official MCP server (mcp.scrapingbee.com) that would let AI agents invoke ScrapingBee via natural-language tool calls. However, the core product interface remains a structured REST API with many typed parameters, not a natural-language command interface itself. Missing for 10: evidence of a chat/NL interface for configuring scrapes beyond ai_query, and independent confirmation the MCP server supports full natural-language operation.
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
- [probe] “official MCP server documented at https://mcp.scrapingbee.com/”
Context.dev ships an official MCP server ('Connect your AI client to Context.dev tools for live web and company data') and an agent 'skill' file that teaches coding agents how to call the API, which together let AI-native users issue natural-language requests that get translated into API calls; there is also a CLI for scripted/terminal use. However, all natural-language operation is mediated through third-party AI clients (Claude, agents) rather than a native NL interface in Context.dev itself, and no community/hands-on evidence confirms this NL workflow works smoothly in practice. Missing for 10: first-party or independent evidence of actual natural-language usage/output quality via the MCP or skill integration, and any native chat/NL interface within the product itself.
- [claimed-docs] “Connect your AI client to Context.dev tools for live web and company data.”
- [claimed-docs] “Teach your coding agent how to choose and use the Context.dev API.”
- [claimed-docs] “Call Context.dev from your terminal and use JSON responses in scripts or CI.”
- [probe] “official MCP server documented at https://mcp.context.dev/mcp”
- [probe] “official CLI documented at https://docs.context.dev/install-cli”
ai-native userApply a preset configuration tuned for research agents that returns structured, citable output
weight 2 · round to ScrapingBeeScrapingBee offers markdown output (return_page_markdown), AI-driven extraction (ai_query), and structured extraction (extract_rules) which can produce citable, structured output usable by research agents, plus an MCP server for agentic integration. However, there is no evidence of a dedicated 'preset configuration tuned for research agents' — no named research-agent mode, no citation metadata, and no documentation bundling these features into a single agent-oriented preset. missing for 10: a documented research-agent preset/mode, citation/source-attribution output, and evidence of agent-specific tuning beyond generic AI extraction params.
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
- [claimed-docs] “return_page_markdown [boolean] (false) Return the page content in markdown format”
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
- [probe] “official MCP server documented at https://mcp.scrapingbee.com/”
Context.devnone0/10Context.dev is a web scraping/data extraction API with structured extraction, crawling, and monitoring features, but there is no evidence of a preset or configuration profile specifically tuned for 'research agents' that returns structured, citable output (e.g., with source attribution/citations). The extraction guide supports JSON Schema output but nothing about citation tracking or a research-agent preset.
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnScrapingBeenone0/10The evidence pack shows only static parameter documentation and no mention of an interactive API reference, live 'try it' console, or runnable code examples; probes even show no OpenAPI/swagger spec and a 404 on a machine-readable docs endpoint, suggesting no interactive explorer exists.
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
- [probe] “PROBE docs-md: HTTP 404 at https://www.scrapingbee.com/documentation.md”
- [probe] “PROBE openapi: all candidate paths 404 (https://www.scrapingbee.com/openapi.json, https://www.scrapingbee.com/swagger.json, https://www.scra…”
Context.devnone0/10Evidence confirms docs, guides, and an OpenAPI spec exist, but nothing indicates an interactive reference with runnable/try-it-out examples (no Swagger/Redoc playground, no 'try it' feature mentioned).
- [probe] “PROBE openapi: HTTP 200 at https://docs.context.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round to Context.devScrapingBeenone0/10Direct probes for an OpenAPI/Swagger spec at all standard locations returned 404, and no evidence pack item shows a downloadable machine-readable API spec being offered elsewhere; only an llms.txt index and human-readable docs exist.
A probe confirms a live OpenAPI JSON spec at docs.context.dev/openapi.json (HTTP 200, contains 'openapi' key), directly satisfying the machine-readable spec requirement, alongside first-party docs describing the API surface. Missing for 10: independent third-party corroboration of spec completeness/versioning beyond the probe check.
- [probe] “PROBE openapi: HTTP 200 at https://docs.context.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round drawnScrapingBeenone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnScrapingBeenone0/10No evidence of API versioning scheme or a documented deprecation policy; OpenAPI spec probes 404 and docs don't mention versioning/deprecation terms at all.
data-engineerThe documented rate limit (requests per second or minute) enforced on my API key before throttling kicks in
weight 3 · round to Context.devScrapingBeenone0/10No evidence pack item mentions a documented rate limit (requests per second/minute) or throttling behavior for API keys; documentation excerpts cover scraping parameters and features but not concurrency/rate-limit thresholds.
Docs confirm a per-minute rate limit exists and that authenticated responses expose rate-limit headers, but no specific numeric threshold (requests/sec or /min) is given in the evidence. Missing for 10: the actual documented numeric limit value, guidance on limits per plan/key tier, and confirmation via headers example showing remaining/limit values.
- [claimed-docs] “Authenticated API responses expose these headers when a per-minute limit applies”
Anti bot — getting past bot defenses — CAPTCHAs, fingerprinting, blocksAnti bot
Getting past bot defenses — CAPTCHAs, fingerprinting, blocks
Block evasion
ai-native userHave an agent automatically get past a CAPTCHA, login, or form wall without my manual intervention
weight 2 · round to ScrapingBeeScrapingBee provides premium proxies to bypass hard-to-scrape sites and JS 'scenario' scripting to interact with pages (e.g., click/fill forms), which could support login flows, but there is no explicit claim or evidence of automatic CAPTCHA solving or a documented login-automation workflow that removes manual intervention entirely. missing for 10: explicit CAPTCHA-solving mechanism, documented login/form-wall bypass workflow, and independent evidence of successful autonomous bypass.
- [claimed-docs] “If you want to interact with pages you want to scrape before we return your the HTML you can add JavaScript scenario to your API call.”
- [claimed-docs] “premium_proxy [boolean] (false) Use premium proxies to bypass difficult to scrape websites”
- [claimed-docs] “session_id [integer] ("") Route multiple API requests through the same IP address”
Context.devnone0/10Context.dev is a web scraping/crawling/data-extraction API; there is no evidence of CAPTCHA-solving, login/session automation, or form-wall bypass capability. Community comments even question its handling of restricted/anti-scraping sites, and no docs describe login or CAPTCHA handling.
- [community] “Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies? AKA unusable for…”
- [community] “\"Websites can opt out of our service, and we respect these requests and add them to our block list.\" I.e: robots.txt already exists and is…”
- [community] “Are you using residential proxies? How do you handle websites that don't want to be scraped. EG if I start passing in Linkedin pages what is…”
- [claimed-docs] “Click, wait, or scroll before scraping or extracting a page, then check which interactions succeeded.”
data-engineerAutomatically retry through a chain of different proxies when anti-bot detection blocks a request
weight 2 · round to ScrapingBeeScrapingBee offers premium_proxy and country_code parameters and an 'auto' mode that picks the cheapest configuration that succeeds, implying some automatic fallback/retry logic, but there's no explicit documentation of a chained multi-proxy retry mechanism specifically triggered by anti-bot detection. missing for 10: explicit documentation of automatic retry chains across multiple proxies upon anti-bot block detection, and independent verification of this retry behavior.
- [claimed-docs] “premium_proxy [boolean] (false) Use premium proxies to bypass difficult to scrape websites”
- [claimed-docs] “country_code [string] ("") Premium proxy geolocation”
- [claimed-docs] “mode [string] ("") Let ScrapingBee pick the cheapest configuration that succeeds. Only value is auto”
Context.devnone0/10No documentation or evidence describes proxy rotation, proxy-chain retries, or anti-bot bypass mechanisms; a community comment explicitly notes the homepage never mentions IP rotation or residential proxies, reinforcing the absence of this capability.
- [community] “Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies? AKA unusable for…”
- [community] “Are you using residential proxies? How do you handle websites that don't want to be scraped. EG if I start passing in Linkedin pages what is…”
developerUse an undetected browser mode to bypass sophisticated bot detection systems
weight 3 · round to ScrapingBeeScrapingBee's docs mention `premium_proxy` explicitly for bypassing 'difficult to scrape websites' and headless browser rendering with JS scenarios, which implies anti-bot capability, but the evidence never uses 'undetected browser' or 'stealth mode' terminology or details specific bot-detection bypass techniques (fingerprint spoofing, TLS/JA3 evasion, etc.). Missing for 10: explicit stealth/undetected-mode documentation, technical detail on fingerprint evasion, and independent verification that it defeats sophisticated bot detection.
- [claimed-docs] “premium_proxy [boolean] (false) Use premium proxies to bypass difficult to scrape websites”
- [claimed-docs] “If you want to interact with pages you want to scrape before we return your the HTML you can add JavaScript scenario to your API call.”
- [claimed-docs] “Our headless browsers will wait for the CSS / Xpath selector passed in the parameter before returning the HTML.”
Context.devnone0/10No evidence in the pack claims an 'undetected browser' or anti-bot-bypass mode; the docs describe scraping, crawling, screenshots, and browser actions but never mention stealth/anti-detection techniques, and community comments explicitly question whether the product uses rotating/residential IPs at all, suggesting no such capability is documented.
- [claimed-docs] “Click, wait, or scroll before scraping or extracting a page, then check which interactions succeeded.”
- [community] “Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies? AKA unusable for…”
- [community] “Are you using residential proxies? How do you handle websites that don't want to be scraped. EG if I start passing in Linkedin pages what is…”
Proxy rotation
developerRequest a proxy from a specific country to get geolocation-appropriate content
weight 2 · round to ScrapingBeeScrapingBee's docs explicitly document a `country_code` parameter for premium proxy geolocation, directly enabling country-specific proxy requests, alongside `premium_proxy` to enable this feature. missing for 10: independent/hands-on confirmation of geolocation accuracy and no list of supported countries in the evidence.
- [claimed-docs] “premium_proxy [boolean] (false) Use premium proxies to bypass difficult to scrape websites”
- [claimed-docs] “country_code [string] ("") Premium proxy geolocation”
Context.devnone0/10No documentation or feature mentions country-specific proxy selection or geolocation control; community comments even question whether Context.dev uses rotating/residential proxies at all, suggesting no such capability exists.
- [community] “Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies? AKA unusable for…”
- [community] “Are you using residential proxies? How do you handle websites that don't want to be scraped. EG if I start passing in Linkedin pages what is…”
developerUse premium residential or datacenter proxies to bypass sites that are hard to scrape
weight 3 · round to ScrapingBeeDocs explicitly document `premium_proxy` for bypassing hard-to-scrape sites, plus `country_code` for geolocation and `session_id` for sticky IP sessions, directly matching the story. Missing for 10: explicit distinction/documentation of residential vs datacenter proxy types and independent third-party validation of bypass success rates.
- [claimed-docs] “premium_proxy [boolean] (false) Use premium proxies to bypass difficult to scrape websites”
- [claimed-docs] “country_code [string] ("") Premium proxy geolocation”
- [claimed-docs] “session_id [integer] ("") Route multiple API requests through the same IP address”
Context.devnone0/10No documentation or product page mentions residential/datacenter proxies, IP rotation, or anti-bot bypass infrastructure; community comments explicitly note the absence of any proxy mention and question whether the product can handle high-value/anti-scraping targets like LinkedIn.
- [community] “Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies? AKA unusable for…”
- [community] “Are you using residential proxies? How do you handle websites that don't want to be scraped. EG if I start passing in Linkedin pages what is…”
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
developerRoute requests through a rotating pool of proxy IPs to avoid blocks
weight 3 · round to ScrapingBeeDocs confirm premium/rotating proxy usage (premium_proxy, country_code) to bypass blocks, plus session_id to pin a single IP when needed, indicating an underlying rotating proxy pool by default with control options. Missing for 10: no independent/hands-on evidence confirming rotation effectiveness against real anti-bot defenses, and no explicit documentation describing pool size or rotation algorithm.
- [claimed-docs] “premium_proxy [boolean] (false) Use premium proxies to bypass difficult to scrape websites”
- [claimed-docs] “country_code [string] ("") Premium proxy geolocation”
- [claimed-docs] “session_id [integer] ("") Route multiple API requests through the same IP address”
Context.devnone0/10No documentation or product page mentions proxy IP rotation, residential proxies, or anti-blocking infrastructure; a community comment on Hacker News explicitly notes the homepage never mentions 'ip' and questions whether rotating/residential proxies are used at all.
- [community] “Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies? AKA unusable for…”
- [community] “Are you using residential proxies? How do you handle websites that don't want to be scraped. EG if I start passing in Linkedin pages what is…”
developerRoute multiple requests through the same proxy IP using a session identifier to maintain a consistent identity
weight 2 · round to ScrapingBeeOfficial docs explicitly document a `session_id` parameter to route multiple API requests through the same proxy IP, directly matching the story. Missing for 10: independent/hands-on corroboration of session persistence behavior beyond first-party docs.
- [claimed-docs] “session_id [integer] ("") Route multiple API requests through the same IP address”
Context.devnone0/10No documentation or product page mentions session-based IP persistence, sticky sessions, or proxy identity management; the crawl/scrape/extract guides only cover content retrieval, not proxy control. A community comment even flags the total absence of any IP/residential-proxy discussion on the site, reinforcing that this capability isn't offered.
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Click, wait, or scroll before scraping or extracting a page, then check which interactions succeeded.”
- [community] “Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies? AKA unusable for…”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to Context.devScrapingBeenone0/10The evidence pack documents single-page scraping parameters (JS scenarios, extraction rules, proxies, screenshots) but never mentions a batch/bulk API endpoint, concurrent job submission, or a mechanism to process many URLs/items in one call.
Docs describe genuine bulk capability: async crawl jobs processing up to 25,000 pages in the background with progress tracking, plus a smaller 500-page synchronous crawl mode, which cover bulk operations across many web pages. However, evidence doesn't show bulk operations across arbitrary item sets (e.g., batch brand lookups, batch document parsing, or bulk extraction across a list of disparate items) beyond website crawling, and there's no independent/hands-on corroboration of large-scale batch reliability. Missing for 10: evidence of bulk/batch endpoints beyond crawling (e.g., batch document conversion, batch structured extraction across arbitrary item lists), and third-party validation of large-scale batch performance.
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to Context.devScrapingBeenone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
Context.dev supports watching a page, sitemap, or dataset on a schedule and receiving signed change events, which functions as an event-trigger mechanism, but this is presented as a single monitoring feature rather than a general rule-definition system with configurable conditions and varied actions. Missing for 10: evidence of a rules/conditions engine, multiple trigger types beyond scheduled monitoring, and configurable downstream actions (e.g., webhooks to arbitrary endpoints, multi-step workflows).
- [claimed-docs] “Watch a page, sitemap, or structured dataset on a schedule and receive signed change events.”
ai-native userSchedule recurring jobs or workflows
weight 2 · round to Context.devScrapingBeenone0/10ScrapingBee's evidence pack covers API scraping parameters, JS rendering, proxies, and extraction, but contains no mention of scheduling, recurring jobs, cron-like triggers, or workflow orchestration features.
The docs describe a monitoring feature that watches a page, sitemap, or dataset 'on a schedule' and emits signed change events (context-dev-docs-9), which is a form of recurring job scheduling, but this is scoped only to change-detection, not general recurring crawl/extract/workflow jobs. Missing for 10: evidence of cron-style scheduling for arbitrary crawl/extract jobs, workflow chaining, or a broader job-scheduling API beyond the single 'monitor' feature.
- [claimed-docs] “Watch a page, sitemap, or structured dataset on a schedule and receive signed change events.”
Dev experience — day-to-day developer experience — setup friction, docs, debugging, iteration speedDev experience
Day-to-day developer experience — setup friction, docs, debugging, iteration speed
Collaboration
developerShare scrapers with teammates and manage organizations and role-based permissions
weight 2 · round drawnScrapingBeenone0/10No evidence in the pack addresses team/organization management, sharing scrapers, or role-based permissions; the documentation excerpts focus entirely on API scraping parameters (JS rendering, proxies, extraction rules), not collaboration or account administration features.
Context.devnone0/10No evidence of team/organization features, shared scraper workflows, or role-based permission management beyond restricted API keys, which is a single-key scoping mechanism, not team/org collaboration. Missing for 10: organization/team creation, member invites, role-based access control across users, shared scraper/workflow assets.
- [claimed-docs] “Choose **Restricted** when an integration needs only selected operations; a restricted key with no permissions cannot call the API.”
Deployment flexibility
developerBuild and deploy custom serverless scraping scripts on the platform without managing my own infrastructure
weight 2 · round to ScrapingBeeScrapingBee is a managed scraping API (no infrastructure to manage) and supports JS 'scenarios' for custom page interaction plus extraction rules, which is a lightweight form of custom scraping logic. However, there is no evidence of a true serverless scripting/deployment platform (e.g., custom code upload, scheduled jobs, or a scripting runtime) — the story's 'build and deploy custom scripts' aspect is only partially matched by parameterized API calls. Missing for 10: evidence of a script/job deployment mechanism, scheduling, or custom code execution beyond JS scenario snippets, and independent developer confirmation of this workflow.
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
- [claimed-docs] “If you want to interact with pages you want to scrape before we return your the HTML you can add JavaScript scenario to your API call.”
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
Context.devnone0/10Context.dev exposes a fixed set of hosted scraping endpoints (crawl, extract, screenshot, monitor, parse) accessed via API/CLI/MCP, but there is no evidence of a mechanism for developers to write and deploy their own custom scraping scripts or actors on the platform's infrastructure. This is a fair question for a web-scraping-as-a-service category, so absence of evidence yields 'none' rather than 'na'.
developerDeploy the scraping service via a Docker container for production use
weight 2 · round drawnScrapingBeenone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
developerSelf-host an open-source version of the scraper instead of relying on a hosted cloud service
weight 2 · round drawnScrapingBeenone0/10ScrapingBee is a hosted cloud scraping API with no evidence of an open-source, self-hostable version; community comments explicitly ask about open-sourcing the stack, confirming none exists.
- [community] “Any plans on open sourcing any part of your stack instead of relying on paid services like ScrapingBee? What does your SaaS setup look like?”
- [community] “Have you looked at running something locally instead of paying for ScrapingBee? I'm using Laravel and considering Dusk to retrieve page cont…”
Context.devnone0/10No evidence anywhere in the pack of an open-source or self-hostable version of Context.dev; it is presented exclusively as a hosted cloud API/service with CLI, MCP server, and SDKs pointing to context.dev endpoints. Missing for 10: any open-source repo, self-hosting instructions, Docker image, or license permitting local deployment.
Integrations
developerConnect the scraping API to no-code automation platforms like n8n or Zapier through a prebuilt connector
weight 2 · round drawnScrapingBeenone0/10No evidence of a prebuilt n8n or Zapier connector; the docs cover API parameters, an MCP server, and a CLI, but nothing about no-code automation platform integrations.
Library compatibility
developerBuild scrapers using popular open-source automation libraries like Playwright, Puppeteer, Selenium, or Scrapy
weight 2 · round drawnScrapingBeenone0/10The evidence pack describes ScrapingBee's own API parameters (JS scenario, screenshots, extraction rules, proxies) but contains no mention of official integrations, SDKs, or middleware for Playwright, Puppeteer, Selenium, or Scrapy. No documentation, probe, or community evidence shows developers can plug ScrapingBee into these specific open-source automation libraries.
Migration lock in
developerExport my scraped data and job configurations in a portable format to migrate to another provider without lock-in
weight 3 · round drawnScrapingBeenone0/10No evidence of any export/migration tooling for scraped data or job configs in a portable format; the docs cover API parameters and scraping features but nothing about data portability or provider migration. missing for 10: export format documentation, job/config export mechanism, migration guides or tooling, any mention of avoiding vendor lock-in.
Quickstart
developerPublish my custom scraper to a public marketplace and earn revenue when others use it
weight 1 · round drawnScrapingBeenone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
developerRun a ready-made scraper from a marketplace instead of building one from scratch
weight 2 · round drawnScrapingBeenone0/10Evidence shows only API parameters/docs for building custom scraping requests; there is no marketplace of pre-built, ready-made scrapers a developer could pick and run instead of building their own.
Context.devnone0/10Context.dev's evidence describes a general-purpose scraping/crawling/extraction API, CLI, and MCP server that developers configure themselves, but no marketplace of pre-built, ready-made scrapers for specific sites/use-cases is mentioned anywhere in the docs, community discussion, or probes.
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
developerStart building immediately using a library of ready-made project templates
weight 1 · round drawnScrapingBeenone0/10No evidence of ready-made project templates or scaffolding to jumpstart development; documentation only covers API parameters and usage, not starter templates or boilerplate projects.
Extraction quality — how faithfully content is extracted — structure, fidelity, edge casesExtraction quality
How faithfully content is extracted — structure, fidelity, edge cases
Ai extraction
developerExtract structured data from a page using natural language instructions instead of writing selectors
weight 3 · round drawnScrapingBee's ai_query parameter lets developers specify in natural language the information they want extracted from a webpage, avoiding manual CSS/XPath selectors, as an alternative to the selector-based extract_rules feature. missing for 10: independent/hands-on validation of AI extraction accuracy, and details on structured output schema/reliability beyond the docs blurb.
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
Docs describe an extract endpoint that crawls relevant pages and returns an object matching a JSON Schema with controls for grounding, coverage, and freshness—no CSS/XPath selectors required, just a schema/instructions-driven approach. Missing for 10: no explicit mention of natural-language instruction fields (vs. schema-only), no independent hands-on benchmark of extraction accuracy/quality.
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [claimed-docs] “retrieve brand profiles with logos, colors, descriptions, and social links through the same API.”
developerPass a JSON schema so the API returns structured data matching that schema
weight 2 · round to Context.devScrapingBee offers extract_rules (CSS-selector based structured extraction) and ai_query (AI-driven extraction), which let developers get structured data, but there is no evidence of accepting a formal JSON Schema definition that the API validates/conforms output to — extract_rules is a custom stringified JSON of selectors, not a schema spec. missing for 10: explicit JSON Schema input support, schema validation/conformance guarantee, examples of schema-driven structured output.
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
Docs explicitly describe extracting structured data by supplying a JSON Schema, with the API returning an object matching it, plus controls for grounding, coverage, and freshness; an OpenAPI spec is also available for verification. Missing for 10: independent hands-on confirmation of schema-conformance accuracy and no explicit mention of schema validation/error handling edge cases.
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.context.dev/openapi.json — contains "openapi" key”
ai-native userHave an LLM read a page and decide what structured fields to pull out without pre-written selectors
weight 2 · round drawnScrapingBee has an ai_query parameter that lets an LLM extract requested information from a page without pre-written CSS/XPath selectors, directly matching the story's intent, but this is described only in a single doc line rather than deeply documented with examples of dynamic field discovery. missing for 10: no documentation showing the AI deciding on its own what structured fields/schema to output (vs. a user-specified query), no independent/hands-on evidence of extraction quality or reliability, and no example of full structured JSON field inference without any query guidance.
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
The extract-structured-data guide shows the product accepts a JSON Schema and returns matching structured data with grounding/coverage controls, which fits an LLM-driven extraction without pre-written CSS/XPath selectors. However, the evidence doesn't explicitly describe the underlying mechanism as an LLM 'deciding' fields freely versus schema-guided extraction, and there's no example of open-ended field discovery without a supplied schema. Missing for 10: evidence of schema-less/free-form field discovery, and independent hands-on confirmation of extraction quality without selectors.
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
developerPlug in a local or self-hosted LLM as the extraction backend instead of a cloud-only model
weight 2 · round drawnScrapingBeenone0/10ScrapingBee's AI extraction (ai_query) uses its own cloud-based AI backend with no documented option to plug in a local or self-hosted LLM; evidence shows only a fixed AI extraction parameter, not a configurable backend.
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
Basic scraping
developerScrape a web page with a single API call and get its raw HTML back
weight 3 · round to ScrapingBeeDocs confirm a single API call with just an API key and target URL returns the page's HTML, with straightforward defaults (docs-1) and no complex setup required. Additional options (JS rendering, wait selectors, markdown/extract_rules) show this basic case is well-supported and flexible, though there's no independent hands-on confirmation of raw HTML fidelity. Missing for 10: independent/community verification of raw HTML output quality.
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
- [claimed-docs] “If you want to interact with pages you want to scrape before we return your the HTML you can add JavaScript scenario to your API call.”
- [claimed-docs] “Our headless browsers will wait for the CSS / Xpath selector passed in the parameter before returning the HTML.”
Context.dev's primary scrape endpoints convert pages to Markdown by default (docs-1, docs-2), and raw HTML is only mentioned as an output option for the async batch-crawl job that must be polled for completion (docs-3), not as an immediate single-call response for a single page. This satisfies the general 'scrape a page via API' need but not the specific 'single call → raw HTML' expectation. Missing for 10: documented synchronous single-page endpoint that returns raw HTML directly, independent confirmation of HTML fidelity/quality.
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
Data safety
data-engineerAutomatically detect and filter personally identifiable information out of scraped content before it reaches storage
weight 2 · round drawnScrapingBeenone0/10No evidence of any PII detection, redaction, or filtering feature in ScrapingBee's documentation or capabilities; the product offers extraction rules and AI query tools but nothing about identifying or stripping personal data before storage.
Document extraction
data-engineerExtract text content from PDFs, Word, Excel, and PowerPoint files without hosting them myself
weight 2 · round to Context.devScrapingBeenone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
Docs explicitly describe a 'parse-documents' API that converts PDFs, Office documents, and spreadsheets into Markdown, including OCR recovery for scanned PDFs, delivered as a hosted API (no self-hosting required). Missing for 10: independent/hands-on verification of extraction quality and no explicit mention of PowerPoint file type beyond generic 'Office documents'.
- [claimed-docs] “Convert PDFs, Office documents, spreadsheets, and other files into Markdown. Recover scanned PDF pages with optional OCR.”
Multimodal extraction
ai-native userGet automatic captions for images on a page so a text-only model can reason about visual content
weight 2 · round drawnScrapingBeenone0/10No evidence of an image-captioning or alt-text generation feature; ScrapingBee's AI features (ai_query) extract structured data from page text/HTML, not image captions for visual content, and by default it blocks images entirely. Missing for 10: any documented image captioning/vision-to-text capability, alt-text generation, or multimodal image description output.
- [claimed-docs] “By default, and to speed up requests, ScrapingBee blocks all images and CSS in the scraped page, but to scrape them, use `block_resources=fa…”
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
Context.devnone0/10No evidence of automatic image captioning or alt-text generation for visual content; the product's extraction focuses on Markdown/JSON/screenshots and document parsing, not describing images for text-only models. Missing for 10: any mention of image captioning, vision-to-text description, or alt-text generation feature.
Search integration
developerSearch the web and get full page content from results in a single call instead of just links and snippets
weight 3 · round drawnScrapingBeenone0/10Evidence shows ScrapingBee scrapes a given URL (with JS rendering, markdown output, extract_rules, ai_query) but nothing indicates a single API call that performs a web search and returns full page content for each result — the llms.txt probe mentions 'search' only in passing with no supporting detail. Missing for 10: any documented search endpoint, example combining query+results with full page bodies, or independent confirmation of this workflow.
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
- [probe] “PROBE llms.txt: HTTP 200 at https://www.scrapingbee.com/llms.txt # ScrapingBee Documentation > Official documentation index for ScrapingBee…”
Context.devnone0/10Context.dev's documented capabilities are URL-based (crawl, scrape, extract, sitemap discovery, screenshot, document parsing, monitoring) but no evidence shows a web-search endpoint that returns full page content for search results in one call — 'discover website URLs' only reads a site's own sitemap, not the open web.
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [claimed-docs] “Read a website's public sitemaps and return a filtered URL list without rendering each page.”
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
Selector extraction
developerExtract specific fields from a page using CSS or XPath selector rules
weight 3 · round to ScrapingBeeDocs explicitly document extract_rules for CSS-based field extraction and confirm the headless browser waits on CSS/XPath selectors, directly supporting structured field extraction. missing for 10: no independent/hands-on corroboration of extraction accuracy or XPath-specific examples.
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
- [claimed-docs] “Our headless browsers will wait for the CSS / Xpath selector passed in the parameter before returning the HTML.”
Structured data handling
data-engineerExtract data from very large tables using intelligent chunking so it fits within processing limits
weight 1 · round drawnScrapingBeenone0/10ScrapingBee's evidence covers web scraping features (JS rendering, proxies, extraction rules, AI queries) but nothing addresses handling very large tables, chunking data to fit processing/token limits, or pagination strategies for oversized datasets. Missing for 10: any mention of table extraction, chunking mechanism, size-limit handling, or pagination/splitting of large data outputs.
Js rendering — handling JavaScript-heavy pages — rendering, waiting, dynamic contentJs rendering
Handling JavaScript-heavy pages — rendering, waiting, dynamic content
Headless rendering
developerRender JavaScript-heavy single-page applications and get the fully rendered HTML
weight 3 · round to ScrapingBeeScrapingBee's docs explicitly describe headless-browser rendering of JS-heavy SPAs built with React/Angular/Vue/JQuery, with support for waiting on selectors and running JS scenarios before returning fully rendered HTML. This directly matches the story's core capability, though evidence lacks independent hands-on corroboration of rendering fidelity. Missing for 10: independent/hands-on verification of rendered output quality, benchmarks against specific SPA frameworks.
- [claimed-docs] “This can be useful for scraping a Single Page Application built with frameworks such as React.js, Angular.js, JQuery or Vue.”
- [claimed-docs] “If you want to interact with pages you want to scrape before we return your the HTML you can add JavaScript scenario to your API call.”
- [claimed-docs] “Our headless browsers will wait for the CSS / Xpath selector passed in the parameter before returning the HTML.”
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
Context.dev supports browser actions (click/wait/scroll) before scraping, and screenshot rendering, implying JS execution via a real browser, and crawl/scrape guides return Markdown/HTML output — suggesting rendered SPA content is retrievable. However, there is no explicit statement that scraping fully executes JavaScript-heavy SPAs or waits for hydration/network-idle by default, and no independent/hands-on confirmation of SPA rendering fidelity. missing for 10: explicit documentation confirming full JS/SPA rendering (e.g., wait-for-network-idle, headless browser execution) as default behavior, and independent verification of rendered output correctness for JS-heavy sites.
- [claimed-docs] “Click, wait, or scroll before scraping or extracting a page, then check which interactions succeeded.”
- [claimed-docs] “Render an exact URL or a resolved site page and return a viewport, full-page, or offset PNG capture.”
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
developerHave the API wait for a specific selector to appear before returning the rendered page
weight 2 · round to ScrapingBeeScrapingBee's docs explicitly state headless browsers wait for a CSS/XPath selector before returning HTML, directly matching the story. Missing for 10: independent/hands-on confirmation of this specific wait-for-selector behavior beyond vendor docs, and example code showing the parameter in use.
- [claimed-docs] “Our headless browsers will wait for the CSS / Xpath selector passed in the parameter before returning the HTML.”
- [claimed-docs] “If you want to interact with pages you want to scrape before we return your the HTML you can add JavaScript scenario to your API call.”
Docs describe browser actions supporting 'wait' among click/scroll before scraping or extracting a page, which directly matches waiting for content before returning rendered output, but there's no explicit mention of waiting for a CSS/DOM selector specifically (vs. fixed delays) nor independent confirmation of this behavior. missing for 10: explicit selector-based wait documentation, example showing selector syntax, independent/hands-on verification.
- [claimed-docs] “Click, wait, or scroll before scraping or extracting a page, then check which interactions succeeded.”
Interactive automation
developerAccess a managed remote browser sandbox for interactive, manual browsing workflows
weight 2 · round drawnScrapingBeenone0/10ScrapingBee's documentation describes a headless browser API for automated scraping (JS scenarios, screenshots, extraction rules) but no evidence of an interactive, manual remote-browser sandbox session a developer could drive by hand.
- [claimed-docs] “If you want to interact with pages you want to scrape before we return your the HTML you can add JavaScript scenario to your API call.”
- [claimed-docs] “Our headless browsers will wait for the CSS / Xpath selector passed in the parameter before returning the HTML.”
- [claimed-docs] “If you need to change the dimension of the browser's viewport (window) when scraping the target page you can use the `window_width` and `win…”
developerKeep interacting with an already-scraped page, clicking and filling forms to reach content behind a login wall
weight 2 · round to ScrapingBeeScrapingBee's JS 'scenario' feature lets you script click/fill actions before the page HTML is returned, and session_id lets you reuse the same IP across multiple API calls to preserve login state — enabling a login-wall workflow. However, evidence shows only a stateless-per-request model (scenario executed once, then HTML returned) rather than a persistent, continuously interactive browser session across multiple later calls. Missing for 10: documentation of a true persistent/interactive session object you can repeatedly command, and any hands-on confirmation this pattern reliably defeats login walls.
- [claimed-docs] “If you want to interact with pages you want to scrape before we return your the HTML you can add JavaScript scenario to your API call.”
- [claimed-docs] “Our headless browsers will wait for the CSS / Xpath selector passed in the parameter before returning the HTML.”
- [claimed-docs] “session_id [integer] ("") Route multiple API requests through the same IP address”
Context.dev documents browser actions (click, wait, scroll) that can run before a scrape or extraction, which supports some interactive page manipulation, but there is no evidence of form-filling, typing credentials, or a persistent multi-step session capable of reaching authenticated/login-walled content. Missing for 10: explicit support for filling login forms/typing input, session/cookie persistence across interactions, and any documented login-wall use case or example.
- [claimed-docs] “Click, wait, or scroll before scraping or extracting a page, then check which interactions succeeded.”
developerScript page interactions like clicking, filling inputs, and scrolling before content is returned
weight 3 · round drawnScrapingBee's docs explicitly describe a 'JavaScript scenario' feature to interact with pages (click, fill, scroll, etc.) before HTML is returned, plus wait-for-selector support to ensure content loads after interactions. This directly matches the story of scripting interactions before content is returned, though evidence lacks a full list of supported actions or independent hands-on confirmation. Missing for 10: detailed enumeration of supported interaction commands (click/fill/scroll) beyond generic 'JavaScript scenario' mention, and independent/community validation of this specific feature.
- [claimed-docs] “If you want to interact with pages you want to scrape before we return your the HTML you can add JavaScript scenario to your API call.”
- [claimed-docs] “Our headless browsers will wait for the CSS / Xpath selector passed in the parameter before returning the HTML.”
- [claimed-docs] “This can be useful for scraping a Single Page Application built with frameworks such as React.js, Angular.js, JQuery or Vue.”
Docs explicitly describe a browser-actions capability allowing click, wait, or scroll before scraping/extracting content, with success verification, directly matching the story. Missing for 10: independent/hands-on corroboration of scripted interactions beyond first-party docs, and no detail on filling form inputs specifically.
- [claimed-docs] “Click, wait, or scroll before scraping or extracting a page, then check which interactions succeeded.”
Render configuration
developerControl the browser viewport width and height when rendering a page
weight 1 · round to ScrapingBeeOfficial docs explicitly state window_width and window_height parameters let developers change the browser viewport dimensions when rendering the target page. missing for 10: no independent/hands-on corroboration beyond first-party docs.
- [claimed-docs] “If you need to change the dimension of the browser's viewport (window) when scraping the target page you can use the `window_width` and `win…”
The screenshot guide mentions a 'viewport' capture mode alongside full-page and offset options, implying some viewport-based rendering, but no evidence specifies developer control over exact width/height dimensions. missing for 10: explicit API parameters for setting viewport width and height, documentation confirming custom viewport sizing, and any hands-on confirmation.
- [claimed-docs] “Render an exact URL or a resolved site page and return a viewport, full-page, or offset PNG capture.”
Session persistence
developerPass my own session cookies so the API fetches pages requiring authentication
weight 2 · round drawnScrapingBeenone0/10No evidence pack item mentions passing custom cookies or headers for authenticated sessions; docs cover JS rendering, proxies, extraction, screenshots, but nothing about supplying session cookies for authenticated page fetches.
developerReuse a persistent browser profile with saved cookies and login state across multiple requests
weight 2 · round drawnScrapingBeenone0/10ScrapingBee's docs mention session_id only for routing requests through the same IP address, not for persisting cookies or login state across requests; no evidence of a saved browser profile or session state reuse mechanism.
- [claimed-docs] “session_id [integer] ("") Route multiple API requests through the same IP address”
Context.devnone0/10No evidence of persistent browser profiles, saved cookies, or reusable login/session state across requests; browser-actions doc only covers click/wait/scroll per single request. Missing for 10: any mention of persistent sessions, cookie storage, authentication state reuse, or profile management across multiple API calls.
- [claimed-docs] “Click, wait, or scroll before scraping or extracting a page, then check which interactions succeeded.”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to Context.devScrapingBee is API-first, and the docs show an extensive, feature-rich API surface (JS rendering, screenshots, extraction rules, AI query, proxies, session control) covering essentially all scraping functionality (scrapingbee-docs-1 through 15). However, there is no explicit statement comparing the API's capabilities to what's available in ScrapingBee's dashboard/UI, so full parity can't be confirmed from evidence. Missing for 10: explicit UI-vs-API feature parity documentation, confirmation that dashboard-only tools (e.g. request builder, account settings) have no capabilities absent from the API.
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
- [claimed-docs] “If you want to interact with pages you want to scrape before we return your the HTML you can add JavaScript scenario to your API call.”
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
- [claimed-docs] “screenshot_selector [string] ("") Return a screenshot of a particular area of the page, targeted by a CSS selector”
Context.dev is API-first: the product's core functions (crawl, extract, screenshot, monitor, brand data) are all documented as API endpoints with an OpenAPI spec, and the CLI/MCP/skill installs are just wrappers around that same API, implying no UI-exclusive functionality. missing for 10: explicit confirmation that the web UI itself exposes zero features unavailable via API (e.g., dashboard-only settings) and independent hands-on verification of full parity.
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
- [claimed-docs] “Watch a page, sitemap, or structured dataset on a schedule and receive signed change events.”
- [claimed-docs] “Call Context.dev from your terminal and use JSON responses in scripts or CI.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.context.dev/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.context.dev/install-cli”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnScrapingBee returns scraped content in open formats such as raw HTML, JSON (extract_rules) and Markdown (return_page_markdown), so output data is not locked into a proprietary format. However, there is no evidence of account-level data export, no mention of stored user data portability, and no explicit 'leave anytime with your data' commitment—since it's a stateless scraping API, the 'export and leave' framing only partially applies. Missing for 10: account/usage data export tooling, explicit data-portability statement, independent confirmation of format openness beyond docs.
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
- [claimed-docs] “return_page_markdown [boolean] (false) Return the page content in markdown format”
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
Context.dev's outputs (Markdown, JSON, HTML) are inherently open, portable formats rather than proprietary lock-in formats, and structured extraction lets users get their scraped/monitored data in JSON Schema-conformant form (docs-1, docs-3, docs-4, docs-9). However there is no explicit account-level 'export all your data and leave' feature (e.g., bulk export of saved crawls, monitors, API key configs, or account deletion with data portability) documented anywhere in the evidence. Missing for 10: dedicated account/data export tooling, documentation of account deletion/data portability guarantees, and independent confirmation that historical crawl/monitor data can be bulk-exported.
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
- [claimed-docs] “Watch a page, sitemap, or structured dataset on a schedule and receive signed change events.”
ai-native userRead the product's source under an open license
weight 2 · round drawnScrapingBeenone0/10No evidence ScrapingBee's core product source is available under an open license; it is a closed, paid SaaS API. A community comment even asks whether the vendor plans to open source any part of their stack, implying it currently is not.
- [community] “Any plans on open sourcing any part of your stack instead of relying on paid services like ScrapingBee? What does your SaaS setup look like?”
ai-native userSelf-host the core product
weight 3 · round drawnScrapingBeenone0/10ScrapingBee is a hosted SaaS API; no evidence of any self-hostable core product, on-premise deployment option, or open-source release. Community comment explicitly asks whether ScrapingBee plans to open-source its stack, with no vendor response indicating such an offering exists.
- [community] “Any plans on open sourcing any part of your stack instead of relying on paid services like ScrapingBee? What does your SaaS setup look like?”
- [community] “Have you looked at running something locally instead of paying for ScrapingBee? I'm using Laravel and considering Dusk to retrieve page cont…”
Output formats — stories about output formats in this arenaOutput formats
Stories about output formats in this arena
Content formats
developerReceive scraped content as clean markdown instead of raw HTML
weight 3 · round drawnScrapingBee's docs explicitly offer a `return_page_markdown` parameter to return page content as markdown instead of raw HTML, directly matching the story. Missing for 10: independent/hands-on confirmation of markdown output quality and any community corroboration of this specific feature.
- [claimed-docs] “return_page_markdown [boolean] (false) Return the page content in markdown format”
First-party docs consistently describe scraping/crawling output as Markdown (sync and async crawl endpoints, single-page scrape, document parsing all return Markdown rather than raw HTML), and this is corroborated by a customer case study (SiteGPT) using it to build a knowledge base. Missing for 10: independent hands-on verification of markdown output quality/cleanliness and no explicit sample output shown.
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Convert PDFs, Office documents, spreadsheets, and other files into Markdown. Recover scanned PDF pages with optional OCR.”
- [claimed-docs] “SiteGPT, the AI chatbot platform for customer support, switched from Firecrawl to Context.dev to scrape entire websites and turn them into t…”
developerChoose exactly which output format is returned, such as markdown, HTML, text, or frontmatter
weight 2 · round drawnDocs confirm HTML is the default output and a dedicated `return_page_markdown` parameter lets developers get markdown instead, but there's no documented option for a plain-text-only extraction or a frontmatter output format, and extract_rules/ai_query only allow custom JSON-style extraction, not those specific formats. Missing for 10: explicit plain-text output mode, frontmatter output support, independent confirmation of format switching.
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
- [claimed-docs] “return_page_markdown [boolean] (false) Return the page content in markdown format”
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
Docs show explicit format choice for Markdown (sync/async crawl) and HTML (async crawl), plus JSON output via structured extraction, but no mention of plain 'text' or 'frontmatter' output options anywhere in the docs. missing for 10: explicit text output mode, frontmatter output mode, independent confirmation of format selection working in practice.
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
developerReceive scraped content as structured JSON
weight 3 · round to Context.devScrapingBee offers extract_rules for CSS-selector-based structured data extraction and ai_query for AI-driven extraction, plus return_page_markdown for markdown output, indicating structured output beyond raw HTML. However, there's no explicit documented 'return as JSON' toggle or example showing a full JSON schema response, and no independent/community confirmation of structured JSON output quality. Missing for 10: explicit JSON output examples/schema, independent verification of structured JSON extraction reliability.
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
- [claimed-docs] “return_page_markdown [boolean] (false) Return the page content in markdown format”
Docs explicitly describe extracting structured JSON matching a user-supplied JSON Schema from crawled pages, with controls for grounding, coverage, and freshness, plus an OpenAPI spec confirming API-driven JSON responses and a CLI that returns JSON for scripting/CI. missing for 10: independent hands-on verification of JSON extraction accuracy/quality beyond vendor docs.
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.context.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “Call Context.dev from your terminal and use JSON responses in scripts or CI.”
Llm ready output
ai-native userGet clean LLM-ready text directly instead of dealing with blocking, rendering, and messy HTML myself
weight 3 · round to Context.devScrapingBee offers return_page_markdown to get markdown output plus ai_query for AI-driven extraction and premium proxies/JS rendering to avoid blocking, directly addressing the LLM-ready text need. However, evidence doesn't show a dedicated 'clean text extraction' mode beyond markdown/extract_rules, and no independent benchmarks confirm output quality for LLM consumption. missing for 10: independent validation of markdown/text cleanliness, dedicated boilerplate-removal/reader-mode feature, hands-on confirmation from users of LLM-ready output.
- [claimed-docs] “return_page_markdown [boolean] (false) Return the page content in markdown format”
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
- [claimed-docs] “premium_proxy [boolean] (false) Use premium proxies to bypass difficult to scrape websites”
- [claimed-docs] “Our headless browsers will wait for the CSS / Xpath selector passed in the parameter before returning the HTML.”
Context.dev's core offering is scraping/crawling websites directly into clean Markdown (and JSON) for AI agents, handling rendering, browser actions, and document parsing so the user doesn't deal with raw HTML; this is corroborated by docs and a real-world migration story (SiteGPT switching from Firecrawl). missing for 10: independent hands-on benchmark of output cleanliness/quality versus alternatives, and no detail on how well it strips boilerplate/ads beyond doc claims.
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Click, wait, or scroll before scraping or extracting a page, then check which interactions succeeded.”
- [claimed-docs] “Convert PDFs, Office documents, spreadsheets, and other files into Markdown. Recover scanned PDF pages with optional OCR.”
- [claimed-docs] “SiteGPT, the AI chatbot platform for customer support, switched from Firecrawl to Context.dev to scrape entire websites and turn them into t…”
ai-native userRequest semantically chunked output instead of one large content blob, so it feeds cleanly into a retrieval pipeline
weight 2 · round drawnScrapingBeenone0/10ScrapingBee offers markdown conversion, CSS-based extraction rules, and AI query extraction, but no evidence of semantic/chunked output splitting content into retrieval-ready segments. The docs list output options (HTML, markdown, screenshots, extract_rules) but never mention chunking or segmenting content for RAG pipelines.
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
- [claimed-docs] “return_page_markdown [boolean] (false) Return the page content in markdown format”
Context.devnone0/10Context.dev's docs describe scraping/crawling into full-page Markdown, JSON extraction, and document parsing, but nowhere mention a chunking feature (e.g., configurable chunk size, semantic segmentation, or overlap controls) intended for retrieval pipelines. Output is delivered as whole-page Markdown/HTML/JSON blobs per page, not sub-page semantic chunks.
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Convert PDFs, Office documents, spreadsheets, and other files into Markdown. Recover scanned PDF pages with optional OCR.”
Visual capture
developerCapture a screenshot of a full page or a specific selected area
weight 2 · round to Context.devDocs confirm a `screenshot_selector` parameter for capturing a specific CSS-selected area of a page, directly supporting selected-area screenshots. However, no evidence explicitly documents a full-page screenshot parameter or option, so only half the story is substantiated. Missing for 10: explicit full-page screenshot parameter/documentation, independent/hands-on confirmation of screenshot output quality.
- [claimed-docs] “screenshot_selector [string] ("") Return a screenshot of a particular area of the page, targeted by a CSS selector”
Docs explicitly describe rendering an exact URL or resolved page and returning a viewport, full-page, or offset PNG capture, directly matching the story of full-page or selected-area screenshots. Missing for 10: independent/hands-on corroboration of screenshot quality or selector-based area capture beyond viewport/offset options.
- [claimed-docs] “Render an exact URL or a resolved site page and return a viewport, full-page, or offset PNG capture.”
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
Cost optimization
developerLet the API automatically pick the cheapest configuration that still succeeds
weight 2 · round to ScrapingBeeScrapingBee's docs explicitly describe a `mode=auto` parameter that lets the API pick the cheapest configuration that still succeeds, directly matching the story. This is first-party documented evidence, though there's no independent/hands-on corroboration of its effectiveness. Missing for 10: independent verification that auto mode reliably picks the cheapest successful config in practice.
- [claimed-docs] “mode [string] ("") Let ScrapingBee pick the cheapest configuration that succeeds. Only value is auto”
developerBlock ads on the target page to speed up scraping requests
weight 1 · round to ScrapingBeeOfficial docs explicitly document the `block_ads=true` parameter to prevent ad loading and speed up scraping requests, directly matching the story. Missing for 10: independent/hands-on corroboration of the speed benefit and no third-party benchmark confirming the claim.
- [claimed-docs] “By default, ScrapingBee does not block ads. To avoid scraping them (e.g.,to speed up your request), use `block_ads=true`”
Context.devnone0/10No evidence pack item mentions ad-blocking, resource blocking, or any performance optimization feature to skip ads/media during scraping; the docs cover crawling, extraction, screenshots, and browser actions but never ad-blocking specifically. Missing for 10: any documentation of an ad-block or resource-blocking option, any performance/speed benefit tied to blocking ads.
developerBlock images and CSS resources by default to reduce bandwidth and speed up requests
weight 1 · round to ScrapingBeeOfficial docs explicitly state ScrapingBee blocks all images and CSS by default to speed up requests, with an opt-out via block_resources=false, directly matching the story. Missing for 10: independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “By default, and to speed up requests, ScrapingBee blocks all images and CSS in the scraped page, but to scrape them, use `block_resources=fa…”
ai-native userSet how much reasoning effort an autonomous agent spends on a data-gathering task (low, medium, high)
weight 2 · round drawnScrapingBeenone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
Cost transparency
developerWhether exceeding my plan's monthly credit or request quota triggers overage charges or a hard cutoff
weight 3 · round drawnScrapingBeenone0/10No evidence in the pack addresses billing behavior when exceeding plan credits/requests—no mention of overage charges, hard cutoffs, or quota enforcement policy.
developerWhether failed, blocked, or empty-result requests still consume my billing quota
weight 2 · round to Context.devScrapingBeenone0/10The evidence pack contains no documentation or discussion of billing behavior for failed, blocked, or empty-result requests—no mention of credit refunds, only-charge-on-success policies, or how failed/blocked scrapes affect quota consumption. Community comments discuss cost/pricing generally but not this specific billing mechanic.
Docs explicitly state that in the timeout/return-partial flow, if no usable result exists the request 'fails without a charge,' directly addressing billing behavior on failure. However, there's no broader documentation covering all failure modes (e.g., blocked requests, empty-result extractions, rate-limited calls) confirming whether they also skip billing. Missing for 10: explicit policy for blocked requests, empty JSON extraction results, and general error responses beyond the timeout optimization guide; independent/community confirmation of billing behavior.
- [claimed-docs] “`return-partial` | Return usable completed work with a completion marker. If no usable result exists, fail without a charge.”
developerSet a spending cap or usage alert so proxy/credit consumption doesn't silently blow past my budget
weight 3 · round drawnScrapingBeenone0/10No evidence of any spending cap, usage alert, or budget notification feature in ScrapingBee's docs or community reports; community comments even highlight cost as a pain point without mentioning any budget-control tooling.
- [community] “Using ScrapingBee is expensive; I've brought the cost of a CRM creation down to about 1.5 cents (+3 cents for a custom cover image) by looki…”
Performance tuning
developerTrade off latency against completeness by controlling exactly when content is returned
weight 1 · round drawnScrapingBee lets developers control timing/completeness tradeoffs via wait-for-selector, JS scenarios, block_ads/block_resources flags, and an 'auto' mode that picks the cheapest successful configuration, giving direct levers over latency vs. completeness. However, this is all documented capability with no independent benchmarking or hands-on confirmation of actual latency impact. Missing for 10: independent/hands-on verification of latency-completeness tradeoffs, explicit 'wait' or timeout parameter documentation, and real-world performance data beyond vendor docs.
- [claimed-docs] “If you want to interact with pages you want to scrape before we return your the HTML you can add JavaScript scenario to your API call.”
- [claimed-docs] “Our headless browsers will wait for the CSS / Xpath selector passed in the parameter before returning the HTML.”
- [claimed-docs] “By default, ScrapingBee does not block ads. To avoid scraping them (e.g.,to speed up your request), use `block_ads=true`”
- [claimed-docs] “By default, and to speed up requests, ScrapingBee blocks all images and CSS in the scraped page, but to scrape them, use `block_resources=fa…”
- [claimed-docs] “mode [string] ("") Let ScrapingBee pick the cheapest configuration that succeeds. Only value is auto”
Context.dev offers explicit controls that trade off latency vs completeness: synchronous small crawls (fast, limited to 500 pages) vs async background crawls up to 25,000 pages, plus a 'return-partial' timeout policy that returns usable completed work with a completion marker rather than waiting for full completion. This directly supports controlling when content is returned along a latency/completeness axis, though it's documented only in claimed-docs with no independent hands-on validation of the tradeoff behavior. Missing for 10: independent/community confirmation of the return-partial and sync/async tradeoff working as documented, and more granular mid-request streaming or partial-result controls beyond the two crawl modes and timeout policy.
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “`return-partial` | Return usable completed work with a completion marker. If no usable result exists, fail without a charge.”
Plan scale limits
data-engineerThe maximum concurrent sessions or requests allowed on my pricing tier and the cost to raise that cap
weight 2 · round drawnScrapingBeenone0/10No evidence in the pack specifies concurrent session/request limits per pricing tier or the cost to increase that cap; documentation snippets cover feature parameters (JS scenario, proxies, extraction) but not concurrency caps or upgrade pricing.
Context.devnone0/10Docs mention rate-limit headers exist and per-minute limits apply, but there is no evidence of tier-specific concurrency/session caps or the cost to raise them. Missing for 10: documented tier limits table, concrete numeric caps per plan, and pricing/upgrade path to raise the cap.
- [claimed-docs] “Authenticated API responses expose these headers when a per-minute limit applies”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnScrapingBeenone0/10Evidence covers proxy geolocation for scraping targets (country_code) but no mention of data residency or storage region controls for ScrapingBee's own data handling/storage; no privacy/compliance documentation is present. Missing for 10: any documentation of data storage regions, residency options, or compliance certifications (e.g., EU data hosting).
- [claimed-docs] “country_code [string] ("") Premium proxy geolocation”
Context.devnone0/10No evidence anywhere in the pack mentions data residency, region selection, or storage location options for Context.dev; the product is a web-scraping/data API with no documented control over where data is stored. Missing for 10: any mention of regional hosting, data residency options, or compliance certifications tied to storage location.
ai-native userPrevent my data from being used to train AI models
weight 3 · round drawnScrapingBeenone0/10No evidence in the pack addresses data-use, training-data opt-out, or AI-training privacy policies for ScrapingBee's service; nothing documents a mechanism to prevent scraped/customer data from being used to train AI models.
ai-native userControl data retention and deletion
weight 2 · round drawnScrapingBeenone0/10No evidence in the pack addresses data retention policies, deletion controls, or privacy/data lifecycle management for ScrapingBee; documentation excerpts focus solely on scraping features and API parameters. Missing for 10: any mention of data retention windows, deletion APIs/requests, privacy policy details, or compliance certifications.
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnScrapingBeenone0/10No evidence pack mentions telemetry, usage tracking, or an opt-out mechanism for ScrapingBee's own product usage; documentation excerpts focus solely on scraping API parameters.
Scale reliability — behavior under load — scaling limits, uptime, failure handlingScale reliability
Behavior under load — scaling limits, uptime, failure handling
Ai driven crawling
ai-native userRely on adaptive crawling that automatically stops once enough information has been gathered to answer my query
weight 2 · round drawnScrapingBeenone0/10ScrapingBee's docs describe single-page scraping, AI-based extraction (ai_query), and cost-optimizing 'auto' mode, but there is no evidence of adaptive multi-step crawling that dynamically decides when enough information has been gathered to stop. No crawling/agentic loop or stopping-criteria feature is documented.
- [claimed-docs] “ai_query [string] ("") The information you want to extract from the webpage using AI”
- [claimed-docs] “mode [string] ("") Let ScrapingBee pick the cheapest configuration that succeeds. Only value is auto”
Context.devnone0/10The docs describe crawling with fixed page caps (500 for sync, 25,000 for async) and extraction with 'coverage' controls, but there is no evidence of an adaptive mechanism that halts crawling once sufficient information for a query has been gathered.
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
Batch processing
data-engineerBatch scrape thousands of URLs asynchronously
weight 3 · round to Context.devScrapingBeenone0/10The evidence pack only documents single-URL synchronous scraping API parameters (JS rendering, extraction rules, proxies, screenshots) with no mention of batch job submission, async processing, concurrency limits, or a queue/webhook system for handling thousands of URLs at scale.
Docs explicitly describe an async background crawl job handling up to 25,000 pages with progress tracking and retrieval on completion, plus rate-limit headers and partial-result timeout handling that support reliability at scale. However, this is framed as crawling one site rather than an arbitrary list of thousands of distinct URLs, and there is no independent/hands-on evidence confirming real-world throughput or reliability at that scale. Missing for 10: evidence of scraping an arbitrary batch/list of thousands of URLs (not just one site's crawl), independent benchmarks or user reports validating async batch reliability at scale.
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “`return-partial` | Return usable completed work with a completion marker. If no usable result exists, fail without a charge.”
- [claimed-docs] “Authenticated API responses expose these headers when a per-minute limit applies”
developerApply different crawl configurations to different URL patterns within a single batch job
weight 1 · round drawnScrapingBeenone0/10ScrapingBee's API is per-URL request based with configuration parameters set per call; there is no evidence of a 'batch job' concept or a way to define per-URL-pattern rules within a single job. The docs describe single-page scraping options (JS scenario, extract_rules, proxies, etc.) but nothing about batch jobs with pattern-based configuration.
Context.devnone0/10The docs describe a single batch crawl job (up to 25,000 pages) with one set of settings, but there is no evidence of applying different crawl configurations to different URL patterns within the same job.
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
Concurrency
data-engineerSpin up many concurrent scraping sessions to gather data at scale
weight 3 · round to Context.devScrapingBee is inherently an API you can call many times, and docs mention session_id for routing multiple requests through the same IP, but the evidence pack contains no explicit documentation of concurrency limits, parallel-request quotas, or scaling architecture for high-volume data-engineering workloads. Missing for 10: explicit concurrency/rate-limit specs, documented plan-based concurrent request caps, and independent evidence of successful large-scale concurrent scraping.
- [claimed-docs] “session_id [integer] ("") Route multiple API requests through the same IP address”
- [claimed-docs] “premium_proxy [boolean] (false) Use premium proxies to bypass difficult to scrape websites”
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
Context.dev supports large single crawls (up to 25,000 pages async) and exposes rate-limit headers, implying some capacity for scaled scraping, but there is no explicit documentation of running many concurrent scraping sessions or session-level concurrency controls. Community feedback also raises doubts about scaling to high-volume/high-value scraping due to lack of rotating/residential proxy support. missing for 10: explicit concurrency/session-limit documentation, evidence of parallel job orchestration, and independent benchmarks confirming multi-session scale.
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Authenticated API responses expose these headers when a per-minute limit applies”
- [community] “Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies? AKA unusable for…”
- [community] “Are you using residential proxies? How do you handle websites that don't want to be scraped. EG if I start passing in Linkedin pages what is…”
Crawl compliance
data-engineerConfigure the crawler to respect robots.txt rules and target-site rate limits automatically
weight 2 · round drawnScrapingBeenone0/10No evidence that ScrapingBee offers robots.txt compliance settings or automatic rate-limit throttling per target site; docs cover proxies, JS rendering, extraction, and viewport settings but nothing about robots.txt or rate-limiting configuration.
Context.devnone0/10No documentation describes automatic robots.txt compliance or target-site rate-limiting; the only rate-limit doc (context-dev-docs-18) covers API-caller limits, not crawl politeness. Community evidence (context-dev-comm-4) even states the company relies on a manual opt-out blocklist rather than respecting robots.txt automatically, undercutting the story further.
- [claimed-docs] “Authenticated API responses expose these headers when a per-minute limit applies”
- [community] “\"Websites can opt out of our service, and we respect these requests and add them to our block list.\" I.e: robots.txt already exists and is…”
Fault tolerance
data-engineerResume a crashed deep crawl from a saved checkpoint instead of restarting from scratch
weight 2 · round drawnScrapingBeenone0/10No evidence of any crawl checkpoint/resume feature; ScrapingBee's docs describe single-page API requests, sessions, and proxy parameters but nothing about deep crawl state persistence or resuming crashed crawls.
Context.devnone0/10Evidence shows async batch crawling with progress tracking (up to 25,000 pages) but no mention of checkpointing or resuming a crashed crawl from a saved state; only completed-job retrieval or partial-result return on timeout is documented, not crash recovery/resume.
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “`return-partial` | Return usable completed work with a completion marker. If no usable result exists, fail without a charge.”
Operational transparency
data-engineerCheck a public status page showing uptime history and past incident postmortems before committing to the service
weight 2 · round drawnScrapingBeenone0/10No evidence of a public status page, uptime history, or incident postmortems anywhere in the evidence pack; docs focus on API features and community items discuss cost/alternatives, not reliability transparency.
Scheduling monitoring
data-engineerMonitor target pages for content changes, such as price or listing updates, and get notified as they happen
weight 2 · round to Context.devScrapingBeenone0/10ScrapingBee is an on-demand scraping API (fetch a page, extract data, render JS) with no evidence of scheduled monitoring, change-detection, diffing, or notification/webhook features for tracking content changes over time. The evidence pack only covers single-request scraping parameters, proxies, and rendering options, not continuous monitoring or alerting.
Docs explicitly describe a monitoring feature that watches a page, sitemap, or dataset on a schedule and delivers signed change events, directly matching the story's core ask. However, there's no independent/hands-on corroboration of this feature working in practice, and no detail on notification channels (webhooks, email, etc.) or reliability at scale. Missing for 10: independent evidence of monitoring reliability, details on notification delivery mechanisms/channels, and evidence of scale/performance under continuous monitoring.
- [claimed-docs] “Watch a page, sitemap, or structured dataset on a schedule and receive signed change events.”
data-engineerMonitor job performance, validate data quality, and receive alerts when something fails
weight 2 · round to Context.devScrapingBeenone0/10Evidence covers scraping features (JS rendering, extraction, proxies) but nothing about job monitoring dashboards, data quality validation, or failure alerting mechanisms; community comments focus on cost/alternatives, not reliability tooling.
Context.dev offers async crawl jobs with progress tracking (docs-3), some quality controls like grounding/coverage/freshness for extraction (docs-4), and scheduled change monitoring with signed events (docs-9), which loosely cover job status and alerting. However there is no dedicated job-performance dashboard, no explicit failure-alert/webhook system for scraping jobs, and no formal data-quality validation framework described. Missing for 10: job performance metrics/dashboard, explicit failure alerting (e.g. webhooks on job error), and structured data quality checks beyond extraction fidelity.
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “Crawl relevant pages and return an object that matches your JSON Schema, with controls for grounding, coverage, and freshness.”
- [claimed-docs] “Watch a page, sitemap, or structured dataset on a schedule and receive signed change events.”
- [claimed-docs] “`return-partial` | Return usable completed work with a completion marker. If no usable result exists, fail without a charge.”
- [claimed-docs] “Authenticated API responses expose these headers when a per-minute limit applies”
developerMonitor live system metrics and worker/browser pool status through a real-time dashboard
weight 1 · round drawnScrapingBeenone0/10The evidence pack covers API parameters, docs, and community discussion but contains no mention of a real-time dashboard for monitoring system metrics or worker/browser pool status; ScrapingBee's dashboard (if any) is not documented here.
developerSchedule scraping jobs to run automatically at specific times
weight 2 · round to Context.devScrapingBeenone0/10ScrapingBee's evidence describes only on-demand API scraping (parameters, JS rendering, proxies, extraction) with no mention of a scheduling feature, cron-like triggers, or job scheduler UI. No evidence supports automated, time-based recurring scraping jobs.
Context.dev's monitor-website-changes feature watches a page, sitemap, or dataset "on a schedule" and emits change events, which functions as scheduled recurring scraping, but this is framed narrowly as change-detection rather than a general-purpose cron/scheduler for arbitrary scrape/crawl jobs. Missing for 10: explicit documentation of configurable schedule intervals/cron syntax, ability to schedule full crawl or extract jobs (not just change monitors), and any independent/hands-on confirmation of scheduling reliability.
- [claimed-docs] “Watch a page, sitemap, or structured dataset on a schedule and receive signed change events.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
Site crawling
data-engineerRun a deep crawl using a breadth-first strategy with a configurable maximum page limit
weight 2 · round to Context.devScrapingBeenone0/10ScrapingBee's documented API is per-page scraping (single URL requests with rendering, extraction, proxy options) with no evidence of a crawl orchestration feature supporting breadth-first traversal or a configurable max-page limit for multi-page crawls.
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
- [claimed-docs] “extract_rules [stringified JSON] ("") Data extraction from CSS selectors”
Context.dev documents crawling with configurable maximum page limits (500 for sync, up to 25,000 for async batch crawls), satisfying the page-limit part of the story, but no evidence describes a selectable crawl strategy (e.g., breadth-first vs depth-first) as a configurable parameter. Missing for 10: explicit breadth-first strategy option/documentation, evidence of strategy configurability alongside the page limit.
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
developerCrawl an entire website and get content from all its pages with one request
weight 3 · round to Context.devScrapingBeenone0/10ScrapingBee's API is designed for single-page scraping requests (one URL per call); the evidence shows no crawler feature that follows links across a domain or aggregates content from multiple pages in one request. No mention of a 'crawl' endpoint, sitemap traversal, or multi-page job in a single API call.
- [claimed-docs] “To scrape a web page, you only need two things: Your API key... The encoded web page URL you want to scrape”
- [claimed-docs] “This can be useful for scraping a Single Page Application built with frameworks such as React.js, Angular.js, JQuery or Vue.”
Docs describe a one-request crawl endpoint that returns page Markdown for a site (up to 500 pages synchronously) plus an async option for up to 25,000 pages, and a real customer (SiteGPT) is cited using it to scrape entire websites into a knowledge base. Missing for 10: independent hands-on verification of crawl completeness/accuracy at scale and no third-party benchmark of crawl reliability beyond vendor docs and one customer quote.
- [claimed-docs] “Crawl a small website section and return page Markdown in one response, with a maximum of 500 pages.”
- [claimed-docs] “Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.”
- [claimed-docs] “SiteGPT, the AI chatbot platform for customer support, switched from Firecrawl to Context.dev to scrape entire websites and turn them into t…”
- [claimed-docs] “Scrape websites into Markdown, crawl linked pages, and extract JSON for AI agents and applications.”
developerInstantly discover all URLs on a website without fully crawling it
weight 2 · round to Context.devScrapingBeenone0/10ScrapingBee's evidence covers page scraping, JS rendering, extraction rules, proxies, and AI queries, but nothing describes a sitemap/URL-discovery feature that lists all URLs on a site without crawling each page. No sitemap parsing, URL enumeration, or site-mapping endpoint is documented.
Context.dev has a dedicated URL discovery endpoint that reads a site's public sitemaps and returns a filtered URL list "without rendering each page," explicitly avoiding a full crawl — directly matching the story. Missing for 10: independent/hands-on corroboration of discovery speed or scale beyond vendor docs.
- [claimed-docs] “Read a website's public sitemaps and return a filtered URL list without rendering each page.”
Not comparable on these axes
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · not comparableScrapingBeen/aScrapingBee is a web-scraping API/SaaS product, not an agent or orchestration platform that would itself consume other MCP servers' tools; evidence only shows it exposes its own MCP server (mcp.scrapingbee.com), i.e., it is the tool provider, not a tool consumer. Plugging external MCP servers into ScrapingBee to gain their tools is a category error for this kind of product.
- [probe] “official MCP server documented at https://mcp.scrapingbee.com/”
Context.devn/aContext.dev is a web-scraping/data-extraction API/service that itself exposes an MCP server (context-dev-docs-13, context-dev-probe-3) so that AI clients can call ITS tools — it is not an agentic product that would consume other MCP servers' tools. The 'plug MCP servers in' client-role story is a category error for this kind of product.
- [claimed-docs] “Connect your AI client to Context.dev tools for live web and company data.”
- [probe] “official MCP server documented at https://mcp.context.dev/mcp”
ai-native userVersion, review, and roll back my automations
weight 1 · not comparableScrapingBeen/aScrapingBee is a web scraping API/proxy service, not an automation-building platform with workflows to version or roll back; versioning/review/rollback of automations is a category error for this product type.