Cloudflare Sandboxes vs Runloop

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Runloop
   Cloudflare Sandboxes~partialprobed6/10

   A probe confirms Cloudflare's developer docs expose a working llms.txt (HTTP 200) covering the whole docs site including Sandbox, giving agents a structured entry point. However, a probe for the markdown version of the specific Sandbox get-started page returned 404, showing per-page agent-friendly docs aren't uniformly available. Missing for 10: dedicated llms.txt or .md endpoints scoped to the Sandbox product docs, and no community/hands-on confirmation of agents actually consuming it successfully.

   • [probe] “PROBE llms.txt: HTTP 200 at https://developers.cloudflare.com/llms.txt # Cloudflare Developer Documentation Explore guides and tutorials to…”
   • [probe] “PROBE docs-md: HTTP 404 at https://developers.cloudflare.com/sandbox/get-started/.md”

   proof ↗⚑ flag

   Runloop✓fullprobed9/10

   Direct probe evidence confirms a working llms.txt file (HTTP 200) with a documentation index, plus docs-md endpoints explicitly pointing agents to fetch llms.txt for full documentation, and Cursor rules files further supporting agent-oriented consumption of docs. missing for 10: independent third-party confirmation that agents successfully use these docs in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”
   • [probe] “PROBE docs-md: HTTP 200 at https://docs.runloop.ai/docs/overview/what-is-runloop.md > ## Documentation Index > Fetch the complete documentat…”
   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Runloop
   Cloudflare Sandboxes~partialcommunity6/10

   Cloudflare Sandboxes are SDK/Workers-driven (exec, code execution, git operations, bindings access) which is inherently headless/programmatic and well-suited to CI/CD automation, and docs explicitly mention CI/CD as a target use case. However there's no dedicated CLI, no CI integration examples (GitHub Actions, etc.), and no evidence of auto-cleanup/idle-timeout controls needed for reliable CI runs — community notes explicitly flag the lack of auto-destroy on idle, requiring manual cleanup logic. missing for 10: explicit CI/CD pipeline examples or integrations, a standalone CLI for headless invocation, documented idle-timeout/auto-teardown controls, and independent verification of headless automation at scale.

   • [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
   • [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
   • [claimed-docs] “Clone repositories, manage branches, and automate Git operations.”
   • [claimed-docs] “Access KV, R2, Durable Objects, and other bindings from a sandbox.”
   • [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop provides a CLI (rli), SDKs, an Agents API, GitHub Actions integration for deploying agents, and benchmark jobs runnable via CLI—all supporting headless/CI automation of sandboxes and agents. missing for 10: no independent/hands-on CI case study, and no explicit CI provider-specific guide beyond GitHub Actions example.

   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round drawn
   Cloudflare Sandboxes—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Runloop—none0/10

   Evidence shows Runloop publishes its own MCP server so external agents can call Runloop's tools (runloop-probe-4), but there is no evidence that Runloop itself can act as an MCP client and plug in third-party MCP servers to use their tools. missing for 10: any documentation of an MCP-client integration point, config for adding external MCP servers, or examples of Devboxes/agents consuming external MCP tool servers.

   • [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round to Runloop
   Cloudflare Sandboxes—none0/10

   Cloudflare Sandboxes is an execution/infrastructure SDK (not itself an agent), so an official MCP server for agent connectivity is a fair axis to ask about—but the evidence pack contains no mention of MCP, an MCP server, or any protocol for connecting AI agents beyond raw SDK APIs (exec, files, sessions, tunnels, etc.).

   ⚑ flag

   Runloop✓fullprobed7/10

   Runloop is a sandbox/devbox platform (not itself an agent), so shipping an official MCP server is a valid axis, and the docs explicitly reference one at docs.runloop.ai/docs/tools/ai-tools. This confirms the capability exists, though the evidence pack lacks detail on the specific tools/resources exposed or independent hands-on confirmation. Missing for 10: detailed documentation of MCP server tool surface, independent/community verification of usage.

   • [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to Runloop
   Cloudflare Sandboxes—none0/10

   The evidence pack shows the Sandbox SDK is an API/library used from Workers code (exec, files, sessions, etc.) but nowhere mentions an official CLI tool for AI-native workflows; interaction is entirely via SDK calls or the general Wrangler CLI, not a dedicated Sandbox CLI.

   ⚑ flag

   Runloop✓fullprobed8/10

   Runloop ships an official CLI (`rli`) with both interactive TUI and traditional commands for managing devboxes, blueprints, SSH sessions, and benchmarks, documented directly in first-party docs and confirmed by probe. This is clearly built for AI-native/agentic workflows (blueprint creation, benchmark submission, devbox lifecycle management). Missing for 10: independent/hands-on community corroboration beyond vendor docs, and no changelog/version history showing active CLI maintenance cadence.

   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “From the interactive menu, select a devbox and choose **SSH** to open a secure shell session directly into your devbox. The CLI handles all …”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [probe] “official CLI documented at https://docs.runloop.ai/docs/tools/rl-cli”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round drawn
   Cloudflare Sandboxes✓fullprobed8/10

   Cloudflare publishes extensive, structured API documentation for the Sandbox SDK (exec, file I/O, sessions, code interpreter, ports/tunnels, watch, snapshots, git, Docker, bindings) explicitly marketed for building AI agents that execute code (docs-24, docs-34). The docs are machine-readable (llms.txt, openapi.json probes return 200) and cover the full surface an AI-native developer would need to drive. missing for 10: independent hands-on confirmation that agents successfully integrate via this API in production, and a dedicated REST/OpenAPI spec for the sandbox endpoints themselves (only the SDK is documented, not a standalone HTTP API schema).

   • [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
   • [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
   • [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
   • [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
   • [claimed-docs] “Execute Python, JavaScript, and TypeScript code with support for data visualizations, tables, and rich output formats.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://developers.cloudflare.com/llms.txt # Cloudflare Developer Documentation Explore guides and tutorials to…”
   • [probe] “PROBE openapi: HTTP 200 at https://developers.cloudflare.com/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

   Runloop✓fullprobed8/10

   Runloop documents a full public API surface: Devboxes API, Agents API, SDKs in common languages, CLI (rli), and GitHub Action integration, all callable programmatically for agent-driven workflows. This is strong first-party documentation of a public API, though probes show no discoverable OpenAPI spec, slightly weakening independent verification. Missing for 10: a published OpenAPI/swagger spec and independent third-party corroboration of API usage.

   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”
   • [probe] “official CLI documented at https://docs.runloop.ai/docs/tools/rl-cli”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to Runloop
   Cloudflare Sandboxes—none0/10

   The evidence pack documents sandbox execution, filesystem, networking, and session features but contains no mention of scoped or least-privilege API credential/token issuance for agents accessing the sandbox APIs; in fact, community feedback notes egress control is 'all or nothing' rather than fine-grained, suggesting limited scoping capability overall.

   • [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”

   proof ↗⚑ flag

   Runloop~partialclaimed6/10

   Runloop's Agent Gateways let agents call LLM APIs like Anthropic/OpenAI 'without ever seeing your API keys,' which is a direct least-privilege credentialing mechanism for agents. However, the docs give only a single brief mention with no detail on fine-grained scopes, per-agent permission levels, revocation, or audit logging. Missing for 10: detailed scoping/permission model documentation, credential rotation/revocation controls, and independent verification of the gateway's isolation guarantees.

   • [claimed-docs] “Agent Gateways let your agents call LLM APIs like Anthropic and OpenAI **without ever seeing your API keys**.”
   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Cloudflare Sandboxes
   Cloudflare Sandboxes✓fullclaimed8/10

   Cloudflare publishes an extensive official SDK (TypeScript/JavaScript) with detailed docs covering exec, file I/O, sessions, code interpreter for Python/JS, port exposure, snapshots, and bindings, explicitly positioned for building AI agents that execute code. Community commentary discusses pricing/latency but does not dispute the SDK's functionality or documentation quality. Missing for 10: independent hands-on developer reviews of the SDK's ergonomics/reliability, and confirmation of multi-language SDK parity beyond JS/TS/Python.

   • [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
   • [claimed-docs] “Execute Python and JavaScript code with rich outputs including charts, tables, and formatted data.”
   • [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
   • [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
   • [claimed-docs] “Execute Python, JavaScript, and TypeScript code with support for data visualizations, tables, and rich output formats.”
   • [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”

   proof ↗⚑ flag

   Runloop~partialclaimed6/10

   Runloop explicitly documents official SDKs in common languages for interacting with its API, plus tutorials integrating with SDKs like OpenAI Agents SDK, supporting an AI-native builder workflow. However, evidence lacks specifics on which languages are covered, versioning/maturity, or independent developer corroboration. Missing for 10: enumerated language list/maturity details, independent hands-on confirmation, changelog/versioning evidence.

   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Cloudflare Sandboxes—none0/10

   No evidence anywhere in the pack of a webhook subscription mechanism for sandbox events; the SDK exposes file watching, SSE streams, and callback-based output but nothing about registering external webhook endpoints for event notifications.

   ⚑ flag

   Runloop—none0/10

   No evidence mentions webhooks or event subscription mechanisms; Runloop's docs describe devbox lifecycle events via Axons event streams but nothing about webhook subscriptions for external systems. Missing for 10: any documentation of a webhooks API, subscription/callback URLs, or event delivery mechanism to external endpoints.

   ⚑ flag

Agentic features

1. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Runloop
   Cloudflare Sandboxes~partialcommunity6/10

   Docs show sandboxes can run background processes, maintain persistent sessions/state, and reconnect via stable IDs (docs-43, docs-9, docs-37, docs-32), which supports basic autonomous background execution. However, there's no documented native scheduling/trigger mechanism for kicking off automations, and community feedback notes the lack of auto-idle cleanup means developers must manually manage sandbox lifecycle rather than it running fully autonomously (comm-7). Missing for 10: built-in cron/trigger scheduling, evidence of true unattended long-running automation, and confirmation that background processes persist reliably without manual intervention.

   • [claimed-docs] “Execute commands and manage background processes in the sandbox's isolated container environment.”
   • [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
   • [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”
   • [claimed-docs] “const buildSession = await sandbox.createSession({ id: "build", env: { NODE_ENV: "production" }, cwd: "/build" });”
   • [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”

   proof ↗⚑ flag

   Runloop~partialclaimed7/10

   Runloop's Axons feature gives agents a persistent event stream letting them suspend when idle, wake on demand, recover state, and hand off between agents 'without you building the plumbing' (runloop-docs-17, runloop-docs-18), and devboxes support suspend/resume to preserve state across runs (runloop-docs-27). The GitHub Action for agent deployment (runloop-docs-24) supports triggering agent runs from CI workflows, enabling background automation setups. Missing for 10: no explicit cron/scheduler primitive, no independent/hands-on account of long-running unattended automations actually working in production, and no example of a full always-on autonomous pipeline beyond docs claims.

   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”

   proof ↗⚑ flag

2. ai-native userOperate the product with natural-language commands

   weight 2 · round to Runloop
   Cloudflare Sandboxes—none0/10

   Cloudflare Sandboxes is a programmatic SDK (sandbox.exec(), writeFile(), createSession(), etc.) intended to be called from Worker code, not a product with a natural-language command interface; while it markets itself for building AI agents (docs-24), there is no evidence of a chat/NL layer for operating the sandbox itself. missing for 10: any NL command parsing/interface, evidence of a conversational control layer, examples of natural-language-driven sandbox operations.

   • [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
   • [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
   • [claimed-docs] “Execute commands and manage background processes in the sandbox's isolated container environment.”

   proof ↗⚑ flag

   Runloop~partialprobed6/10

   Runloop exposes an official MCP server and ships Cursor '.mdc' rule files that give AI assistants context on the SDK (devbox lifecycle, file ops, blueprints, snapshots, tunnels), enabling natural-language operation through AI coding assistants rather than a bespoke NL interface. There's no first-party chat/NL command surface of Runloop's own, and no independent evidence of how well the MCP/Cursor integration performs in practice. Missing for 10: dedicated natural-language interface docs, hands-on/independent validation of the MCP or Cursor integration actually working end-to-end.

   • [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”
   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Cloudflare Sandboxes—none0/10

   The evidence pack shows extensive static code snippets across Cloudflare's Sandbox docs (exec, file I/O, sessions, interpreter, etc.) but nothing indicates an interactive, in-browser 'try it now' API reference or runnable playground — the openapi.json probe is for Cloudflare's general API, not a Sandbox-specific interactive explorer. Missing for 10: an in-page runnable code sandbox/playground, an interactive API explorer (e.g., Swagger/Redoc try-it-out), or any embedded live execution environment tied to the docs.

   • [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
   • [claimed-docs] “import pandas as pd data = {'product': ['A', 'B', 'C'], 'sales': [100, 200, 150]} df = pd.DataFrame(data) df['sales'].sum() # Last expressi…”
   • [probe] “PROBE openapi: HTTP 200 at https://developers.cloudflare.com/openapi.json — contains "openapi" key”
   • [probe] “PROBE docs-md: HTTP 404 at https://developers.cloudflare.com/sandbox/get-started/.md”

   proof ↗⚑ flag

   Runloop—none0/10

   No evidence of an interactive API reference with runnable examples; the explicit probe for OpenAPI/swagger specs returned 404s across all candidate paths, and docs mention SDKs/CLI but not a runnable API explorer.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round to Cloudflare Sandboxes
   Cloudflare Sandboxes~partialprobed4/10

   Cloudflare's developer platform exposes a general openapi.json (HTTP 200, containing an 'openapi' key), but there's no evidence this spec specifically documents the Sandbox SDK's methods (exec, writeFile, sessions, etc.), which is a code-level SDK rather than a REST API; a dedicated machine-readable doc for the sandbox pages itself returned 404. Missing for 10: confirmation that the openapi.json actually covers Sandbox-specific endpoints/methods, and a dedicated llms.txt/OpenAPI export scoped to the Sandbox SDK docs.

   • [probe] “PROBE openapi: HTTP 200 at https://developers.cloudflare.com/openapi.json — contains "openapi" key”
   • [probe] “PROBE docs-md: HTTP 404 at https://developers.cloudflare.com/sandbox/get-started/.md”
   • [probe] “PROBE llms.txt: HTTP 200 at https://developers.cloudflare.com/llms.txt # Cloudflare Developer Documentation Explore guides and tutorials to…”

   proof ↗⚑ flag

   Runloop—none0/10

   Runloop offers SDKs and a CLI but there is no evidence of a downloadable OpenAPI/Swagger spec; explicit probes to common OpenAPI paths (openapi.json, swagger.json, etc.) all returned 404, confirming absence rather than just lack of mention.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Runloop
   Cloudflare Sandboxes✓fullcommunity7/10

   Cloudflare Sandboxes are explicitly designed as isolated VM/container environments for running untrusted code, separate from production infra, with sessions, snapshots, and opt-in bindings to access KV/R2/Durable Objects only when explicitly wired in — matching the AI-native 'test without touching production' need well (docs-34, docs-38, docs-50, docs-20, docs-24). Community feedback doesn't contradict the isolation guarantee itself, though it flags coarse outbound network controls and unclear persistence semantics as caveats (cloudflare-sandbox-comm-2, cloudflare-sandbox-comm-4). Missing for 10: independent hands-on validation that no production data leakage occurs in practice, and finer-grained egress/network isolation controls.

   • [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
   • [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
   • [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
   • [claimed-docs] “Access KV, R2, Durable Objects, and other bindings from a sandbox.”
   • [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
   • [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”
   • [community] “There is an open question about how file persistence works. The docs claim they persist the filesystem even when they move the container to …”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop's Devboxes are isolated sandbox execution environments (secure, snapshot/blueprint-based, no production data by default) explicitly designed for testing AI agents, fully separate from any production system. missing for 10: no independent/hands-on corroboration beyond vendor docs, no explicit statement contrasting sandbox vs production data isolation guarantees.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Cloudflare Sandboxes—none0/10

   The evidence pack contains extensive API documentation for the Sandbox SDK's features (exec, files, sessions, snapshots, etc.) but nowhere mentions API versioning schemes, version numbers, or a documented deprecation policy for the SDK/API surface. No evidence of semver commitments, changelogs with deprecation notices, or migration guides for breaking changes.

   ⚑ flag

   Runloop—none0/10

   No evidence of API versioning scheme or a documented deprecation policy; OpenAPI spec probe returned 404s and no changelog/versioning docs are cited.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Runloop
   Cloudflare Sandboxes~partialcommunity5/10

   Cloudflare Sandboxes supports concurrent/parallel operations via a multiplexed connection that 'avoids subrequest limits when making many concurrent operations' (docs-17) and allows multiple isolated sessions per sandbox for parallel workflows (docs-9, docs-18, docs-39), plus getSandbox() to manage many sandbox instances by ID. However there's no documented batch/bulk API primitive for iterating over large item sets, and community feedback flags unclear resource limits and high per-vCPU pricing that would constrain large-scale bulk workloads (comm-3, comm-6, comm-11). Missing for 10: an explicit bulk/batch operation API or example, documented concurrency/resource ceilings, and independent evidence of successful large-scale parallel execution.

   • [claimed-docs] “Multiplexes all SDK calls over a single persistent connection. It avoids subrequest limits when making many concurrent operations.”
   • [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
   • [claimed-docs] “Create additional sessions for separate workflows in the same sandbox”
   • [claimed-docs] “Create additional sessions for separate workflows in the same sandbox:”
   • [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”
   • [community] “Cloudflare Containers (and therefore Sandbox) pricing is way too expensive... vCPU: $51.84 per vCPU (!!!)... These prices are more expensive…”
   • [community] “The pricing with such offerings is the biggest throwoff. This one comes out to be more than $58/month for just 1vCpu and 1GiB RAM when used …”
   • [community] “I am keen to understand how I would limit sandbox resource usage.”

   proof ↗⚑ flag

   Runloop~partialclaimed6/10

   Runloop supports running 10k+ parallel sandboxes, benchmark jobs that provision devboxes/run agents/aggregate outputs 'with a single command,' and snapshots for fan-out across multiple approaches — evidence of bulk/parallel operation across many devboxes. However, there's no explicit documentation of a generic bulk-operation API/CLI verb (e.g., bulk delete, bulk update, batch create) for arbitrary resources beyond devboxes/benchmarks. missing for 10: explicit bulk CRUD operations across resources like snapshots/storage objects/secrets, documented batch API endpoints, and independent evidence of large-scale bulk usage beyond vendor docs.

   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “Run your agent against well-known open source benchmarks like terminal bench 2, AIME, and more.”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round drawn
   Cloudflare Sandboxes~partialclaimed4/10

   The SDK provides a file-system watch capability with an `onEvent` callback (inotify-based) that lets developers trigger custom actions when files change, which is a form of event-driven automation, but this is a narrow, code-level primitive rather than a declarative rules/automation engine. Missing for 10: no declarative rule-definition mechanism, no support for triggering on non-filesystem events (e.g., scheduled/cron, external webhooks, custom event types), and no evidence of a rules/condition-action system beyond raw watcher callbacks.

   • [claimed-docs] “Monitor real-time filesystem changes using native inotify. Build development tools, hot-reload systems, and responsive file processing.”
   • [claimed-docs] “const watcher = await sandbox.watch('/workspace/src', { include: ['*.js', '*.ts'],”
   • [claimed-docs] “const watcher = await sandbox.watch('/workspace/src', { include: ['*.js', '*.ts'], onEvent: (event) => {”

   proof ↗⚑ flag

   Runloop~partialclaimed4/10

   Runloop's Axons give agents a persistent event stream that can suspend on idle, wake on demand, recover state, and hand off between agents, which is a form of automatic event-driven behavior, and the Broker component sits between Axon and agent process to manage this. However, there's no documentation of a general user-facing rules engine where arbitrary conditions map to arbitrary actions across the platform (e.g., webhooks, triggers on Devbox/snapshot events, custom rule definitions). Missing for 10: a documented rules/trigger definition interface, examples of custom event-to-action mappings beyond agent suspend/wake, and evidence of triggers on other platform events (snapshots, storage, secrets).

   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round drawn
   Cloudflare Sandboxes—none0/10

   The evidence describes on-demand sandbox execution, sessions, and lifecycle management, but nowhere mentions cron-style scheduling, recurring triggers, or workflow orchestration for repeated runs; community threads even highlight the lack of automatic idle/cleanup logic, implying no built-in scheduling primitives.

   • [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
   • [claimed-docs] “Create additional sessions for separate workflows in the same sandbox”
   • [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”

   proof ↗⚑ flag

   Runloop—none0/10

   The evidence describes Devboxes, snapshots, benchmarks, and Axons for event-driven suspend/resume, but nothing about scheduling recurring jobs or cron-like recurring workflows. Absence of evidence for this applicable automation capability yields 'none'.

   ⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round drawn
   Cloudflare Sandboxes~partialclaimed4/10

   Sandboxes offer git clone/branch operations for code and R2-based directory snapshot/restore, which give raw building blocks for versioning and rolling back sandbox state, but there is no dedicated 'automation' versioning, review, or rollback UI/workflow — it's up to the developer to wire this together. Missing for 10: no built-in review/diff tooling for automations, no automatic rollback trigger, no evidence of a workflow-specific version history feature, and no independent confirmation these primitives are actually used for automation rollback in practice.

   • [claimed-docs] “Clone repositories, manage branches, and automate Git operations.”
   • [claimed-docs] “Snapshot a sandbox directory to R2 and restore it later.”
   • [claimed-docs] “Create point-in-time snapshots of directories and restore them from R2.”
   • [claimed-docs] “Mount S3-compatible buckets (R2, S3, GCS) as local filesystems for persistent data storage across sandbox lifecycles.”

   proof ↗⚑ flag

   Runloop~partialclaimed4/10

   Runloop's Snapshots explicitly support rolling back Devbox state to a known good point, and Blueprints let you version/reuse startup configurations, giving a rudimentary rollback mechanism for agent environments. However, there is no evidence of a review workflow (diffs, approvals, audit trail) or of 'automations' as a first-class versioned object distinct from raw disk/environment snapshots. Missing for 10: explicit review/diff tooling for automations, first-class automation versioning (not just devbox/environment snapshots), audit/history UI for changes.

   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”

   proof ↗⚑ flag

Agent workloads

1. ai-native userRun coding agents like Claude Code or Codex inside the sandbox following the vendor's own recipe

   weight 2 · round to Runloop
   Cloudflare Sandboxes—none0/10

   The docs describe Sandboxes as ideal for building AI agents and provide generic exec/git/docker primitives, but there is no vendor-published recipe or guide specifically for running Claude Code or Codex inside a Sandbox. missing for 10: an explicit vendor tutorial/recipe naming Claude Code or Codex, any first-party integration guide, and community confirmation that such agents run inside the sandbox.

   • [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
   • [claimed-docs] “Run Docker commands inside a sandbox container.”
   • [claimed-docs] “Clone repositories, manage branches, and automate Git operations.”

   proof ↗⚑ flag

   Runloop~partialclaimed5/10

   Runloop documents running AI coding agents generally inside Devboxes (blueprints, benchmarks, Agents API, OpenAI Agents SDK tutorial) but no specific vendor recipe or tutorial naming Claude Code or Codex is present in the evidence. missing for 10: explicit Claude Code/Codex quickstart or blueprint, named integration guide for these specific agents.

   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [claimed-docs] “Run your agent against well-known open source benchmarks like terminal bench 2, AIME, and more.”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”

   proof ↗⚑ flag

Workloads

1. developerRun a headless browser or full desktop environment inside the sandbox

   weight 2 · round drawn
   Cloudflare Sandboxes—none0/10

   The evidence pack details shell execution, file I/O, sessions, tunnels, snapshots, and code interpretation, but nowhere mentions running a headless browser (e.g., Playwright/Puppeteer) or a full desktop/GUI environment inside the sandbox. This is a reasonable capability to ask of a code-execution sandbox product, but no docs or community evidence confirm it exists.

   ⚑ flag

   Runloop—none0/10

   No evidence Runloop supports running a headless browser or desktop GUI inside its Devbox sandboxes; the only browser-related tutorial explicitly states the architecture avoids this, running Chromium externally via Browserbase and connecting over CDP so 'no Chromium ever runs in the devbox.' Blueprints/docs never mention GUI, X11, VNC, or bundled browser binaries as a supported environment type.

   • [claimed-docs] “the agent runs in a devbox, the browser runs on Browserbase, and the devbox drives it by connecting Playwright over CDP to the session’s con…”

   proof ↗⚑ flag

2. developerAttach GPUs to sandboxed workloads

   weight 2 · round drawn
   Cloudflare Sandboxes—none0/10

   No evidence anywhere in the docs or community discussion mentions GPU support, GPU attachment, or GPU-accelerated workloads for Cloudflare Sandboxes; the product is described purely in terms of CPU-based containers/VMs for code execution.

   ⚑ flag

   Runloop—none0/10

   The evidence pack describes Devboxes, blueprints, snapshots, mounts, storage, and networking features but never mentions GPU support or GPU-attached compute for sandboxed workloads; nothing indicates hardware acceleration is available.

   ⚑ flag

Files

1. developerRead, write, upload, and download files in the sandbox filesystem via the SDK

   weight 2 · round to Cloudflare Sandboxes
   Cloudflare Sandboxes✓fullclaimed9/10

   Docs explicitly cover readFile/writeFile/mkdir with absolute paths, streaming/binary uploads >32MiB via ReadableStream, and directory watching, giving comprehensive first-party coverage of file read/write/upload/download via the SDK. Missing for 10: independent hands-on developer confirmation of file upload/download behavior (community discussion focuses on persistence questions, not concrete failures).

   • [claimed-docs] “await sandbox.writeFile("/workspace/hello.txt", "Hello, Sandbox!"); const file = await sandbox.readFile("/workspace/hello.txt");”
   • [claimed-docs] “This allows binary data and files greater than 32 MiB to be written to the sandbox. It replaces the "base64" encoding option.”
   • [claimed-docs] “the writeFile() method supports passing a ReadableStream as the content parameter. This allows binary data and files greater than 32 MiB”
   • [claimed-docs] “Read, write, and manage files in the sandbox filesystem. All paths are absolute (e.g., `/workspace/app.js`).”
   • [claimed-docs] “the `writeFile()` method supports passing a `ReadableStream` as the `content` parameter.”
   • [claimed-docs] “await sandbox.mkdir('/workspace/project/src', { recursive: true });”

   proof ↗⚑ flag

   Runloop~partialclaimed4/10

   Docs confirm the SDK/CLI supports 'file operations' as part of Devbox management (mentioned alongside lifecycle, command execution, etc. in the Cursor rules integration), implying read/write file capability, but no dedicated documentation page or code sample specifically shows upload/download or read/write file APIs. Missing for 10: explicit SDK method docs for file read/write, and explicit upload/download examples or independent corroboration.

   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”

   proof ↗⚑ flag

Runtimes

1. developerDefine custom sandbox templates or bring my own container image

   weight 2 · round to Runloop
   Cloudflare Sandboxes—none0/10

   The evidence describes running commands, files, sessions, Python/JS execution, tunnels, and Docker-command execution inside a sandbox, but nothing documents defining a custom sandbox template or supplying a custom base container image for the sandbox environment itself. Since bring-your-own-image is a standard, expected axis for a container/sandbox product, the lack of any supporting evidence yields 'none' rather than 'na'.

   • [claimed-docs] “Run Docker commands inside a sandbox container.”
   • [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
   • [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
   • [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop's Blueprints support building custom sandbox images from a Dockerfile (`rli blueprint from-dockerfile`), plus reusable prebuilt blueprints and snapshotting of custom states, directly enabling developers to bring their own container image or define custom templates. Missing for 10: independent/hands-on verification of Dockerfile-based blueprint builds and more detail on supported image registries or base image constraints.

   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”
   • [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”

   proof ↗⚑ flag

2. developerExecute code in multiple language runtimes (Python, JavaScript, and more) and get rich results back

   weight 2 · round to Cloudflare Sandboxes
   Cloudflare Sandboxes✓fullclaimed8/10

   Docs clearly document a code interpreter API executing Python, JavaScript, and TypeScript with rich outputs (charts, tables, dataframes, stateful contexts), plus shell exec with stdout/stderr capture, matching the story directly. Missing for 10: independent/hands-on verification of the rich-output rendering quality and broader language support beyond the three named runtimes.

   • [claimed-docs] “Execute Python and JavaScript code with rich outputs including charts, tables, and formatted data.”
   • [claimed-docs] “Execute Python, JavaScript, and TypeScript code with support for data visualizations, tables, and rich output formats.”
   • [claimed-docs] “import pandas as pd data = {'product': ['A', 'B', 'C'], 'sales': [100, 200, 150]} df = pd.DataFrame(data) df['sales'].sum() # Last expressi…”
   • [claimed-docs] “Contexts maintain state (variables, imports, functions) across executions.”
   • [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
   • [claimed-docs] “you'll create a Worker that can execute Python code and work with files in isolated containers.”

   proof ↗⚑ flag

   Runloop~partialclaimed5/10

   Runloop's Devboxes are described as full-featured secure sandboxed execution environments that can run arbitrary commands (via blueprints, SSH, CLI) which implies multi-language support, but there is no explicit documentation of a code-execution API returning 'rich results' (e.g., structured outputs, images, dataframes) as in dedicated code-interpreter products. missing for 10: explicit multi-language runtime examples (Python/JS/etc.), rich/structured result formatting (e.g., stdout+images+tables), and independent confirmation of these capabilities.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”

   proof ↗⚑ flag

Untrusted code

1. developerExecute untrusted, AI-generated code without risking my own infrastructure

   weight 3 · round to Runloop
   Cloudflare Sandboxes~partialcommunity7/10

   Cloudflare's docs explicitly market Sandboxes for running untrusted, AI-generated code with VM/container-based isolation, file/exec APIs, and strong security boundaries (cloudflare-sandbox-docs-34, -38, -50, -24). However, hands-on community feedback flags real gaps for this exact use case: outbound network access is 'all or nothing' with no fine-grained egress control, and there's no built-in auto-destroy for idle sandboxes, forcing manual cleanup. Missing for 10: granular network egress control, automatic idle sandbox teardown, independent security audit of the VM isolation claims.

   • [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
   • [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
   • [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
   • [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
   • [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”
   • [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”

   proof ↗⚑ flag

   Runloop✓fullprobed8/10

   Runloop's core product is Devboxes, secure sandboxed execution environments purpose-built to run AI agent/code workloads isolated from user infrastructure, with snapshots, tunnels, secrets isolation, and scaling to 10k+ parallel sandboxes. This directly matches the story of executing untrusted AI-generated code without risking one's own infra. Missing for 10: independent/third-party hands-on validation of isolation guarantees, and explicit documentation of the underlying sandboxing/virtualization technology or security boundary details.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Let’s use devbox tunnels to securely access ports on your devbox over a simple url.”
   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”

   proof ↗⚑ flag

2. developerRun arbitrary shell commands and install packages inside the sandbox

   weight 2 · round to Cloudflare Sandboxes
   Cloudflare Sandboxes✓fullclaimed9/10

   Docs explicitly document sandbox.exec() for running arbitrary shell commands with stdout/stderr/exit codes, stdin support, streaming/SSE output, background processes, plus Docker command execution and Git operations inside the sandbox, giving developers full shell/package-install capability. Community threads corroborate real-world usage of the sandbox for running containerized workloads, though none specifically contradict shell/package execution. missing for 10: independent hands-on verification of installing arbitrary packages (e.g., apt/pip) beyond docs examples.

   • [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
   • [claimed-docs] “Pass input via stdin (no shell injection risks)”
   • [claimed-docs] “Execute a command and return a Server-Sent Events stream for real-time processing.”
   • [claimed-docs] “Run Docker commands inside a sandbox container.”
   • [claimed-docs] “Clone repositories, manage branches, and automate Git operations.”
   • [claimed-docs] “stdin - Data to pass to the command's standard input (enables arbitrary input without shell injection risks)”
   • [claimed-docs] “onOutput - Callback for real-time output: (stream: 'stdout' | 'stderr', data: string) => void”
   • [claimed-docs] “Execute commands and manage background processes in the sandbox's isolated container environment.”
   • [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”

   proof ↗⚑ flag

   Runloop✓fullclaimed7/10

   Runloop Devboxes are full Linux execution environments with SSH access via the CLI (runloop-docs-9) and blueprints built from Dockerfiles that install packages/dependencies at startup (runloop-docs-2, runloop-docs-14), implying arbitrary shell command execution and package installation. However, the evidence pack lacks a direct hands-on example of running an explicit shell command or apt/pip install inside a Devbox. missing for 10: explicit example/output of running an arbitrary shell command and installing a package inside a Devbox, independent third-party confirmation.

   • [claimed-docs] “From the interactive menu, select a devbox and choose **SSH** to open a secure shell session directly into your devbox. The CLI handles all …”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”

   proof ↗⚑ flag

Agent lifecycle

1. ai-native userMy agent can provision its own sandbox, execute code, read the results, and tear it down — end to end without a human

   weight 3 · round to Runloop
   Cloudflare Sandboxes~partialcommunity7/10

   Docs clearly show an agent-driven lifecycle: getSandbox() to provision/reconnect, exec()/interpreter code execution with stdout/stderr capture, and readFile to retrieve results — all callable programmatically without a human in the loop. However, there is no documented explicit 'destroy/teardown' API, and community feedback confirms there's no built-in idle-timeout auto-destroy, forcing developers to implement their own cleanup logic. Missing for 10: a documented sandbox.destroy()/terminate call or auto-expiry mechanism, and independent hands-on confirmation of a full provision-execute-teardown loop working end-to-end.

   • [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”
   • [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
   • [claimed-docs] “Execute Python and JavaScript code with rich outputs including charts, tables, and formatted data.”
   • [claimed-docs] “await sandbox.writeFile("/workspace/hello.txt", "Hello, Sandbox!"); const file = await sandbox.readFile("/workspace/hello.txt");”
   • [claimed-docs] “Contexts maintain state (variables, imports, functions) across executions.”
   • [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop's core product is programmatic sandbox (Devbox) provisioning, execution, and teardown via API/SDK/CLI, explicitly designed for AI agents — including an Agents API for creating/managing/deploying agents, blueprints for reusable environments, snapshots for state capture, and suspend/resume lifecycle, all achievable without human intervention (docs-1, docs-13, docs-22, docs-23, docs-2, docs-4, docs-27, docs-12). The quickstart shows a devbox spun up in ~1 minute and CLI/SDK access covers execution and result retrieval end-to-end. missing for 10: independent/hands-on third-party verification of a full agent-driven provision→execute→teardown loop, and a concrete code example showing an agent reading execution results programmatically.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

Isolation

1. platform-engineerRely on a documented hard isolation boundary (microVM or equivalent) between sandboxes and my systems

   weight 3 · round to Cloudflare Sandboxes
   Cloudflare Sandboxes✓fullcommunity7/10

   Cloudflare's own docs explicitly state 'VM-based isolation - Each sandbox runs in its own VM' and that each sandbox 'runs in its own isolated container with a full Linux environment, providing strong security boundaries' for running untrusted code (docs-38, docs-50, docs-34). This gives a documented isolation boundary, though the docs mix 'VM' and 'container' language rather than naming a specific microVM technology, and community discussion raises open questions about network egress granularity rather than disputing the isolation claim itself. Missing for 10: a clearer, consistent technical description of the isolation mechanism (e.g., naming Firecracker or similar) and independent third-party security audit/corroboration.

   • [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
   • [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
   • [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
   • [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”

   proof ↗⚑ flag

   Runloop~partialprobed3/10

   Runloop repeatedly describes Devboxes as "secure sandboxed execution environments" and "isolated Linux sandbox environments," implying some isolation boundary, but the evidence never names or documents the actual isolation mechanism (e.g., microVM, Firecracker, gVisor, container-only) or any architecture diagram/security whitepaper explaining the boundary. missing for 10: explicit documentation of the underlying isolation technology (microVM/hypervisor-level or equivalent), any security architecture doc or third-party audit confirming hard isolation guarantees.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Runloop allows you to run benchmarks at scale in a secure environment.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”

   proof ↗⚑ flag

2. ai-native userGive an agent a sandbox where host secrets and credentials are unreachable by the code it runs

   weight 2 · round drawn
   Cloudflare Sandboxes~partialcommunity6/10

   Cloudflare documents strong process/VM isolation ('VM-based isolation', 'isolated container with a full Linux environment', 'run untrusted code securely in isolated environments') which implies the host's runtime and secrets aren't directly reachable by sandboxed code, and bindings (KV/R2/DO) are explicitly scoped and passed in rather than ambient. However, there is no explicit documentation describing how host secrets/credentials are excluded or scrubbed from the sandbox environment, and community feedback notes only coarse-grained outbound network control ('all or nothing') rather than fine-grained secret/credential access restrictions, leaving open questions about exfiltration vectors. Missing for 10: explicit documentation on secret/credential isolation guarantees, independent security audit or hands-on test confirming host credentials are unreachable, and finer-grained egress/network controls to prevent secret exfiltration.

   • [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
   • [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
   • [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
   • [claimed-docs] “Access KV, R2, Durable Objects, and other bindings from a sandbox.”
   • [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”

   proof ↗⚑ flag

   Runloop~partialclaimed6/10

   Runloop's Devboxes are documented as sandboxed execution environments with encrypted secrets and, notably, Agent Gateways that let agents call LLM APIs 'without ever seeing your API keys' — directly supporting the isolation-of-credentials use case. However, secrets are also stated to be 'automatically made available as environment variables' inside Devboxes, which is somewhat in tension with 'unreachable by code' and isn't clarified as scoped/restricted; there's no independent/hands-on verification of secret unreachability or a security audit. missing for 10: independent verification of secret isolation, clarity on env-var secret exposure vs true unreachability, and details on network/filesystem-level credential isolation guarantees.

   • [claimed-docs] “Agent Gateways let your agents call LLM APIs like Anthropic and OpenAI **without ever seeing your API keys**.”
   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”

   proof ↗⚑ flag

Network policy

1. platform-engineerRestrict or allow the sandbox's network egress with explicit policy

   weight 3 · round to Cloudflare Sandboxes
   Cloudflare Sandboxes!disputedcontradicted3/10

   Docs mention only a mechanism to 'Intercept and handle outbound HTTP from sandboxes using Workers' (docs-23), which is not the same as a declarative allow/deny egress policy, and no docs describe configurable network egress rules. A named community commenter explicitly states outbound network control is 'all or nothing right now' with no fine-grained control available, directly contradicting the story's requirement for explicit egress policy. missing for 10: documented egress allow/deny policy API, first-party confirmation of granular network restriction, and any independent verification that such a policy exists.

   • [claimed-docs] “Intercept and handle outbound HTTP from sandboxes using Workers.”
   • [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”

   proof ↗⚑ flag

   Runloop—none0/10

   No evidence pack item describes network egress controls, firewall rules, or allow/deny policies for Devbox sandboxes; tunnels and secrets features are documented but network egress policy configuration is never mentioned.

   ⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Cloudflare Sandboxes
   Cloudflare Sandboxes✓fullcommunity7/10

   Cloudflare Sandboxes is an API/SDK-first product with no separate management UI described in the evidence; every capability (exec, file I/O, sessions, git, Docker, port exposure, snapshots, code interpreter, filesystem watch, even the browser terminal) is exposed and built entirely through the SDK/API, and the terminal UI itself is just a WebSocket client wired to the same API. Missing for 10: no explicit statement contrasting a Cloudflare dashboard GUI with the API, and no independent/third-party confirmation that no UI-exclusive controls exist (e.g., resource limits raised as unclear in community threads).

   • [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
   • [claimed-docs] “Connect browser-based terminal UIs to sandbox shells via WebSocket, with the xterm.js SandboxAddon for automatic reconnection and resize han…”
   • [claimed-docs] “Run Docker commands inside a sandbox container.”
   • [claimed-docs] “Clone repositories, manage branches, and automate Git operations.”
   • [claimed-docs] “Snapshot a sandbox directory to R2 and restore it later.”
   • [community] “I am keen to understand how I would limit sandbox resource usage.”

   proof ↗⚑ flag

   Runloop~partialprobed6/10

   Runloop is API-first: the CLI (rli) wraps SDKs/API for devboxes, blueprints, snapshots, benchmarks, agents, secrets, etc., suggesting broad API/CLI parity with any dashboard functionality. However there is no explicit statement claiming full UI/API feature parity, no documented OpenAPI spec (404s on all probed paths), and no direct dashboard-vs-API comparison. Missing for 10: explicit parity claim between web dashboard and API, published OpenAPI/reference spec, confirmation that all dashboard-only actions (e.g., billing, org settings) are also API-accessible.

   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   Cloudflare Sandboxes~partialclaimed4/10

   The SDK exposes standard file I/O (readFile/writeFile), S3-compatible bucket mounting (R2/S3/GCS), and snapshot/restore to R2, which are open, non-proprietary formats a user could use to extract their data. However, there is no documented bulk 'export all my data' feature, no account-level portability tooling, and no independent confirmation that a user can fully migrate off the platform with all state intact. missing for 10: explicit data-export/account-portability feature, independent verification of full data extraction, documentation on non-file state (e.g., session/process state) portability.

   • [claimed-docs] “await sandbox.writeFile("/workspace/hello.txt", "Hello, Sandbox!"); const file = await sandbox.readFile("/workspace/hello.txt");”
   • [claimed-docs] “Mount S3-compatible buckets (R2, S3, GCS) as local filesystems for persistent data storage across sandbox lifecycles.”
   • [claimed-docs] “Create point-in-time snapshots of directories and restore them from R2.”
   • [claimed-docs] “Read, write, and manage files in the sandbox filesystem. All paths are absolute (e.g., `/workspace/app.js`).”
   • [claimed-docs] “Snapshot a sandbox directory to R2 and restore it later.”

   proof ↗⚑ flag

   Runloop~partialclaimed4/10

   Runloop offers Snapshots to save Devbox disk state and export/re-create Devboxes, and Dockerfile-based blueprints imply portability of environments, but there is no documented feature for exporting all account/user data (benchmarks, configs, agent history) in open formats or a clear data-export/account-deletion workflow. missing for 10: explicit full account data export tool, open-format guarantees for all data types, data portability/deletion documentation.

   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Cloudflare Sandboxes—none0/10

   No evidence in the pack mentions an open-source license or a public source repository for the Cloudflare Sandbox SDK; all citations are product docs, pricing/feature discussions, or probes, none addressing source availability or licensing.

   ⚑ flag

   Runloop—none0/10

   No evidence indicates Runloop's core source (the sandbox platform, Devboxes, Axons, etc.) is released under an open license; the evidence pack shows only docs, CLI, and SDK references without any license or open-source repository disclosure.

   ⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   Cloudflare Sandboxes—none0/10

   Cloudflare Sandboxes is built entirely on Cloudflare's proprietary Workers/Durable Objects/container infrastructure, and no evidence in the pack mentions any open-source release, self-hosted deployment option, or ability to run the core product outside Cloudflare's platform. Community discussion focuses on pricing, cold-start times, and resource limits, not self-hosting.

   • [claimed-docs] “The SDK supports three transport protocols for communication between the Durable Object and container”
   • [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
   • [community] “The pricing with such offerings is the biggest throwoff. This one comes out to be more than $58/month for just 1vCpu and 1GiB RAM when used …”

   proof ↗⚑ flag

   Runloop—none0/10

   Runloop is presented entirely as a hosted cloud platform (Devboxes, Agents API, benchmarks) with no evidence of an on-premises or self-hostable deployment option, open-source core, or self-hosting documentation.

   ⚑ flag

Latency

1. developerStart sandboxes with documented sub-second-to-few-second cold starts

   weight 3 · round to Runloop
   Cloudflare Sandboxes—none0/10

   No first-party Cloudflare documentation in the evidence pack states specific cold-start timing (sub-second to few-second) for sandboxes; the community threads discuss competitors' or ambiguous Firecracker boot times without confirming Cloudflare's own documented cold-start figures. missing for 10: official docs quantifying cold-start latency, benchmark data, or a first-party performance page addressing startup time.

   • [community] “We rolled out our own that does pretty much the same thing but perhaps more... It does take 1-5 seconds to boot the environment (firecracker…”
   • [community] “1-5 seconds seems high for Firecracker, depending on your requirements. We boot VMs (using Firecracker) at ~20-50ms.”

   proof ↗⚑ flag

   Runloop~partialclaimed6/10

   Runloop's marketing site documents '10GB image startup time in <2s' and a 1-minute quickstart, supporting sub-second-to-few-second cold starts, but this is a single first-party marketing claim without detailed benchmarks, methodology, or independent corroboration. missing for 10: independent/third-party benchmark confirmation, documentation of cold start times across varying image sizes/configurations, and reproducible performance data beyond the single homepage claim.

   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

Scale

1. platform-engineerRun large concurrent fleets of sandboxes with documented concurrency limits

   weight 3 · round to Runloop
   Cloudflare Sandboxes—none0/10

   Docs describe per-sandbox features (sessions, exec, files) and one architectural note that multiplexing avoids subrequest limits, but there is no documentation of concurrency limits, fleet-level scaling guidance, or how many sandboxes/instances can run concurrently. Community comments explicitly flag unanswered questions about resource limits and lack of auto-cleanup, reinforcing the absence of concrete concurrency documentation.

   • [claimed-docs] “Multiplexes all SDK calls over a single persistent connection. It avoids subrequest limits when making many concurrent operations.”
   • [community] “I am keen to understand how I would limit sandbox resource usage.”
   • [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”

   proof ↗⚑ flag

   Runloop~partialclaimed5/10

   Runloop's marketing site claims support for 10k+ parallel sandboxes and sub-2s startup for 10GB images, showing scale is a design goal, but there is no documented concurrency limit, quota tiering, or rate-limit table a platform engineer could plan capacity around. Missing for 10: explicit documented per-account/org concurrency limits, guidance on requesting increases, and independent benchmarks validating the 10k+ claim at sustained load.

   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”

   proof ↗⚑ flag

2. developerKeep a sandbox session running for hours or days for long agent tasks

   weight 2 · round to Runloop
   Cloudflare Sandboxes!disputedcontradicted5/10

   Docs describe stable-ID reconnection (getSandbox) and sessions with persistent state/env, suggesting sandboxes could be resumed across long tasks, but community reports explicitly say containers auto-shutdown after idle periods, that there's no built-in idle-timeout control (forcing manual cleanup), and that persistence semantics during idle states are unclear — directly undercutting reliable multi-hour/day continuous operation for unattended agent tasks. Missing for 10: first-party guidance on running sessions continuously for hours/days without disconnection, clear idle-timeout/keep-alive controls, and independent confirmation of long-duration reliability.

   • [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
   • [claimed-docs] “Create additional sessions for separate workflows in the same sandbox”
   • [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”
   • [claimed-docs] “You can also set a default command timeout for all commands in a session”
   • [community] “There is an open question about how file persistence works. The docs claim they persist the filesystem even when they move the container to …”
   • [community] “Cloudflares docs are written so hard for web dev. Can you host a monolith app that isn't serving http traffic on cloudflare tech like contai…”
   • [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop Devboxes explicitly support long-running persistence via snapshots (save/restore disk state), suspend/resume to pause and continue exactly where left off, and Axons providing a persistent event stream that lets agents suspend when idle and wake on demand/recover state — directly addressing multi-hour/multi-day agent sessions without losing progress. Combined with SSH/tunnel access and mounts/secrets for long tasks, this covers the core story well. Missing for 10: independent/hands-on evidence (e.g., a third-party report) confirming multi-day session stability at scale, and explicit documented max session duration/limits.

   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”

   proof ↗⚑ flag

Pricing

1. developerStart building with a free tier or included credits without talking to sales

   weight 1 · round to Runloop
   Cloudflare Sandboxes—none0/10

   The evidence pack documents extensive product capabilities and self-serve pricing figures (e.g., per-vCPU costs debated in community threads) but contains no mention of a free tier, free trial credits, or explicit self-serve signup flow that lets a developer start without contacting sales. missing for 10: explicit free-tier/credit offer, signup flow documentation, confirmation no sales contact is required.

   • [community] “Cloudflare Containers (and therefore Sandbox) pricing is way too expensive... vCPU: $51.84 per vCPU (!!!)... These prices are more expensive…”
   • [community] “The pricing with such offerings is the biggest throwoff. This one comes out to be more than $58/month for just 1vCpu and 1GiB RAM when used …”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Docs explicitly state Runloop's trial includes $50 free credits with no credit card required, and a quickstart lets a developer spin up a first devbox in about a minute, both without any sales contact. Missing for 10: independent/community confirmation of the free-tier signup experience and clarity on long-term free-tier limits beyond trial credits.

   • [claimed-docs] “It includes **$50 in free credits**, and no credit card is required to sign up.”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

2. platform-engineerPay per second only for the compute a sandbox actually uses

   weight 3 · round drawn
   Cloudflare Sandboxes—none0/10

   No documentation in the evidence pack describes per-second or usage-based billing for sandbox compute; the only pricing-related evidence is community commentary describing flat, expensive per-vCPU pricing (comm-3, comm-6) and the absence of automatic idle shutdown, meaning engineers must build their own cleanup to avoid being billed for idle sandboxes (comm-7). This contradicts rather than supports the story of paying only for actual per-second compute use.

   • [community] “Cloudflare Containers (and therefore Sandbox) pricing is way too expensive... vCPU: $51.84 per vCPU (!!!)... These prices are more expensive…”
   • [community] “The pricing with such offerings is the biggest throwoff. This one comes out to be more than $58/month for just 1vCpu and 1GiB RAM when used …”
   • [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”

   proof ↗⚑ flag

   Runloop—none0/10

   No evidence pack item states an explicit per-second or usage-based billing model; the only tangential mention is that suspend/resume can 'save costs' (runloop-docs-27), which doesn't establish a per-second pricing mechanism.

   ⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   Cloudflare Sandboxes—none0/10

   No evidence in the pack discusses data residency, region selection, or storage location controls for Cloudflare Sandboxes; docs focus on execution, files, sessions, tunnels, and R2 snapshots without any region/residency configuration options mentioned.

   ⚑ flag

   Runloop—none0/10

   No evidence in the pack mentions data residency, region selection, or geographic storage controls anywhere in Runloop's docs; the pack covers sandboxes, blueprints, snapshots, secrets, and tooling but never data-location choice.

   ⚑ flag

2. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Cloudflare Sandboxes—none0/10

   The docs describe file read/write, snapshot/restore to R2, and bucket mounting, but there is no documented mechanism for users to control data retention windows or guarantee deletion of sandbox data. Community evidence explicitly states 'it does not seem to be possible to auto destroy a sandbox after certain amount of idle time... forces whoever is implementing this to do their own cleanup,' and persistence semantics are described as unclear, indicating the product does not provide the retention/deletion controls an AI-native user would need.

   • [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”
   • [community] “There is an open question about how file persistence works. The docs claim they persist the filesystem even when they move the container to …”
   • [claimed-docs] “Create point-in-time snapshots of directories and restore them from R2.”
   • [claimed-docs] “Mount S3-compatible buckets (R2, S3, GCS) as local filesystems for persistent data storage across sandbox lifecycles.”

   proof ↗⚑ flag

   Runloop—none0/10

   The evidence pack covers sandboxing, blueprints, snapshots, secrets encryption, and tooling, but contains no documentation of data retention policies, data deletion controls, or user-facing privacy/compliance settings for Devboxes, snapshots, or storage objects. missing for 10: explicit data retention policy docs, user-initiated deletion/purge APIs, compliance/privacy documentation.

   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Storage Objects provide a way to store and manage files, data, and other resources that can be shared across Devboxes or made publicly avail…”

   proof ↗⚑ flag

3. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Cloudflare Sandboxes—none0/10

   No evidence in the pack addresses telemetry, usage tracking, or opt-out settings for Cloudflare Sandboxes; docs focus purely on execution, filesystem, sessions, and networking features. This is a fair privacy-posture question for a developer platform product, so absence of evidence yields 'none' rather than 'na'.

   ⚑ flag

   Runloop—none0/10

   No evidence in the pack mentions telemetry, analytics opt-out, or usage-tracking controls for Runloop; the docs cover sandboxing, blueprints, snapshots, secrets, etc., but nothing about a privacy/telemetry toggle.

   ⚑ flag

Lifecycle

1. developerSet timeouts so sandboxes shut down automatically and stop billing when idle or done

   weight 2 · round to Runloop
   Cloudflare Sandboxes!disputedcontradicted3/10

   Docs only describe a per-command timeout within sessions (cloudflare-sandbox-docs-19), not an idle-shutdown/auto-destroy mechanism for the whole sandbox tied to billing; one community comment claims the underlying Container platform auto-shuts down after no requests (cloudflare-sandbox-comm-5), but a hands-on developer explicitly states it does not seem possible to auto-destroy a sandbox after an idle period, forcing manual cleanup (cloudflare-sandbox-comm-7). This is a direct contradiction between vendor-adjacent claims and hands-on experience. Missing for 10: explicit first-party docs on idle-timeout auto-shutdown/destroy tied to billing, and independent confirmation it works as described.

   • [claimed-docs] “You can also set a default command timeout for all commands in a session”
   • [community] “Cloudflares docs are written so hard for web dev. Can you host a monolith app that isn't serving http traffic on cloudflare tech like contai…”
   • [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”

   proof ↗⚑ flag

   Runloop~partialclaimed5/10

   Runloop documents suspend/resume to "save state and costs" and Axons that let agents "suspend when idle, wake on demand," indicating some idle-triggered lifecycle management, but there is no explicit documentation of a developer-configurable timeout parameter that auto-shuts-down a Devbox and stops billing after inactivity or completion. missing for 10: explicit timeout/idle-shutdown configuration API or CLI flag, confirmation that suspension halts billing automatically without agent action.

   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”

   proof ↗⚑ flag

2. developerSpin up an isolated sandbox with one API/SDK call and get a live environment in seconds

   weight 3 · round to Runloop
   Cloudflare Sandboxes✓fullcommunity8/10

   Docs show a single call (getSandbox()) creates or reconnects to an isolated VM/container sandbox with immediate access to exec, file, and session APIs (docs-37, docs-38, docs-50), and community discussion corroborates boot times in the 1-5 second range consistent with 'seconds' provisioning (comm-1, comm-8). Missing for 10: an independent benchmark or first-party latency SLA confirming consistent sub-few-second cold starts across load, since community notes it's slower than some Firecracker-based competitors.

   • [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”
   • [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
   • [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
   • [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
   • [community] “We rolled out our own that does pretty much the same thing but perhaps more... It does take 1-5 seconds to boot the environment (firecracker…”
   • [community] “1-5 seconds seems high for Firecracker, depending on your requirements. We boot VMs (using Firecracker) at ~20-50ms.”

   proof ↗⚑ flag

   Runloop✓fullclaimed9/10

   Docs describe Devbox creation via SDK/CLI/API, quickstart claiming ~1 minute to a running devbox, blueprints/prebuilt images for fast startup, and marketing claims of 10k+ parallel sandboxes with <2s startup for a 10GB image, matching the one-call, seconds-to-live-environment story. missing for 10: independent/hands-on benchmark confirming real-world provisioning latency beyond vendor claims.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”

   proof ↗⚑ flag

Preview access

1. developerExpose a port from the sandbox on a public preview URL to reach services running inside

   weight 2 · round to Cloudflare Sandboxes
   Cloudflare Sandboxes✓fullclaimed9/10

   Docs explicitly describe exposing a sandbox port and getting a public preview URL, including zero-config *.trycloudflare.com URLs via sandbox.tunnels.get(port) and Worker-side routing of HTTP/WebSocket requests to the correct sandbox container, directly matching the story. Missing for 10: independent/hands-on confirmation of preview URL reliability beyond first-party docs.

   • [claimed-docs] “Expose services running in the sandbox via preview URLs. Access web servers and APIs from the internet.”
   • [claimed-docs] “Expose services on zero-config `*.trycloudflare.com` URLs via `sandbox.tunnels.get(port)`. Best for quick development and `.workers.dev` dep…”
   • [claimed-docs] “Expose a port and get a preview URL for accessing services running in the sandbox.”
   • [claimed-docs] “Expose services on zero-config `*.trycloudflare.com` URLs via `sandbox.tunnels.get(port)`.”
   • [claimed-docs] “Expose services running in your sandbox via public preview URLs.”
   • [claimed-docs] “Route incoming HTTP and WebSocket requests to the correct sandbox container.”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop explicitly documents devbox tunnels for securely accessing ports on a devbox over a simple URL, directly matching the story's requirement to expose a service port publicly. Missing for 10: independent/hands-on confirmation of tunnel reliability and details on URL persistence/security controls beyond the doc snippet.

   • [claimed-docs] “Let’s use devbox tunnels to securely access ports on your devbox over a simple url.”

   proof ↗⚑ flag

Snapshots

1. developerPause a running sandbox and resume it later with filesystem and memory state intact

   weight 2 · round to Runloop
   Cloudflare Sandboxes~partialcommunity5/10

   Docs confirm directory-level snapshot/restore to R2 ('Create point-in-time snapshots of directories and restore them from R2') and idle-state persistence claims, giving filesystem-level pause/resume support, but there's no documented mechanism for preserving in-memory/process execution state across a pause, and community discussion notes ambiguity about exactly how idle-state persistence works. missing for 10: explicit memory/process-state preservation on pause, clear docs on what 'idle state' persistence actually preserves, independent verification that restored sandboxes resume mid-execution rather than just filesystem contents.

   • [claimed-docs] “Create point-in-time snapshots of directories and restore them from R2.”
   • [claimed-docs] “Snapshot a sandbox directory to R2 and restore it later.”
   • [claimed-docs] “Mount S3-compatible buckets (R2, S3, GCS) as local filesystems for persistent data storage across sandbox lifecycles.”
   • [community] “There is an open question about how file persistence works. The docs claim they persist the filesystem even when they move the container to …”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop explicitly documents 'Suspend and resume' functionality that pauses sandboxes to save state and resume exactly where left off, plus snapshot capability that saves disk state and can recreate Devboxes from that state. This directly matches the story's pause/resume with filesystem+memory state intact. Missing for 10: no independent/hands-on corroboration of memory-state fidelity across suspend/resume, and no detailed technical explanation of how memory state (not just disk) is preserved.

   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”

   proof ↗⚑ flag

2. developerSnapshot a sandbox and later restore or fork new sandboxes from that snapshot

   weight 3 · round to Runloop
   Cloudflare Sandboxes~partialcommunity5/10

   Docs confirm directory-level snapshot/restore via R2 (docs-8, docs-48), but there's no evidence of forking new independent sandbox instances from a snapshot — only restoring a directory's state, and community reports flag ambiguity in how persistence/restore actually behaves (comm-4). missing for 10: explicit 'fork sandbox from snapshot' API, clarity on full sandbox-state (not just directory) restoration, and independent confirmation that restore reliably recreates a working sandbox.

   • [claimed-docs] “Create point-in-time snapshots of directories and restore them from R2.”
   • [claimed-docs] “Snapshot a sandbox directory to R2 and restore it later.”
   • [community] “There is an open question about how file persistence works. The docs claim they persist the filesystem even when they move the container to …”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop's docs explicitly describe snapshotting a Devbox's disk state and creating new Devboxes from that saved state, including fan-out use cases (multiple approaches from one snapshot) and rollback scenarios, directly matching the snapshot/restore/fork story. missing for 10: independent/hands-on corroboration beyond first-party docs, and no explicit code example showing the fork API call in the evidence pack.

   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”

   proof ↗⚑ flag