Vercel Sandbox vs Runloop

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Runloop
   Vercel Sandbox✓fullprobed8/10

   Probes confirm both a working llms.txt at the root domain (vercel.com/llms.txt, HTTP 200) and a markdown-formatted docs endpoint for Sandbox specifically (sandbox.md, HTTP 200), which are exactly the agent-oriented docs formats this story asks for. Missing for 10: no explicit evidence llms.txt itself deep-links to sandbox-specific docs, and no independent/community confirmation of an agent successfully consuming these files.

   • [probe] “PROBE llms.txt: HTTP 200 at https://vercel.com/llms.txt # Vercel > Vercel is a cloud platform for building, deploying, and scaling web appl…”
   • [probe] “PROBE docs-md: HTTP 200 at https://vercel.com/docs/sandbox.md --- title: Vercel Sandbox product: vercel url: /docs/sandbox canonical_url: "h…”
   • [claimed-docs] “Connect to and manage Sandboxes from the dashboard”

   proof ↗⚑ flag

   Runloop✓fullprobed9/10

   Direct probe evidence confirms a working llms.txt file (HTTP 200) with a documentation index, plus docs-md endpoints explicitly pointing agents to fetch llms.txt for full documentation, and Cursor rules files further supporting agent-oriented consumption of docs. missing for 10: independent third-party confirmation that agents successfully use these docs in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”
   • [probe] “PROBE docs-md: HTTP 200 at https://docs.runloop.ai/docs/overview/what-is-runloop.md > ## Documentation Index > Fetch the complete documentat…”
   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round drawn
   Vercel Sandbox✓fullprobed8/10

   Vercel Sandbox is designed for headless/programmatic use via SDKs (@vercel/sandbox) and a CLI, with access tokens explicitly documented for external CI/CD systems and non-Vercel environments, plus OpenAPI-backed REST access confirmed by probe. missing for 10: no explicit third-party/independent CI pipeline case study or GitHub Actions example demonstrating real-world headless automation.

   • [claimed-docs] “Use `@vercel/sandbox` for JavaScript and TypeScript”
   • [claimed-docs] “Use access tokens when `VERCEL_OIDC_TOKEN` is unavailable, such as in external CI/CD systems or non-Vercel environments.”
   • [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
   • [claimed-docs] “For local development, run `vercel link` and `vercel env pull` to get a development token. In production on Vercel, authentication is automa…”
   • [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop provides a CLI (rli), SDKs, an Agents API, GitHub Actions integration for deploying agents, and benchmark jobs runnable via CLI—all supporting headless/CI automation of sandboxes and agents. missing for 10: no independent/hands-on CI case study, and no explicit CI provider-specific guide beyond GitHub Actions example.

   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round drawn
   Vercel Sandbox—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Runloop—none0/10

   Evidence shows Runloop publishes its own MCP server so external agents can call Runloop's tools (runloop-probe-4), but there is no evidence that Runloop itself can act as an MCP client and plug in third-party MCP servers to use their tools. missing for 10: any documentation of an MCP-client integration point, config for adding external MCP servers, or examples of Devboxes/agents consuming external MCP tool servers.

   • [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round to Runloop
   Vercel Sandbox—none0/10

   Vercel Sandbox is an execution/infrastructure product (not itself an agent), so an official MCP server exposing it to agents is a fair, applicable axis; however no evidence in the pack mentions MCP, an MCP server, or any agent-protocol integration—only SDKs, CLI, and REST/OpenAPI docs are described.

   ⚑ flag

   Runloop✓fullprobed7/10

   Runloop is a sandbox/devbox platform (not itself an agent), so shipping an official MCP server is a valid axis, and the docs explicitly reference one at docs.runloop.ai/docs/tools/ai-tools. This confirms the capability exists, though the evidence pack lacks detail on the specific tools/resources exposed or independent hands-on confirmation. Missing for 10: detailed documentation of MCP server tool surface, independent/community verification of usage.

   • [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to Runloop
   Vercel Sandbox✓fullclaimed7/10

   Vercel Sandbox ships an official `sandbox` CLI explicitly positioned for manual testing, agentic workflows, debugging, and one-off operations, with documented commands like `sandbox create --name` and `sandbox run`. missing for 10: independent/hands-on corroboration beyond vendor docs, and no CLI reference/full command list shown.

   • [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
   • [claimed-docs] “Create with an explicit name sandbox create --name my-sandbox”
   • [claimed-docs] “sandbox run echo "Hello Sandbox!"”
   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”

   proof ↗⚑ flag

   Runloop✓fullprobed8/10

   Runloop ships an official CLI (`rli`) with both interactive TUI and traditional commands for managing devboxes, blueprints, SSH sessions, and benchmarks, documented directly in first-party docs and confirmed by probe. This is clearly built for AI-native/agentic workflows (blueprint creation, benchmark submission, devbox lifecycle management). Missing for 10: independent/hands-on community corroboration beyond vendor docs, and no changelog/version history showing active CLI maintenance cadence.

   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “From the interactive menu, select a devbox and choose **SSH** to open a secure shell session directly into your devbox. The CLI handles all …”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [probe] “official CLI documented at https://docs.runloop.ai/docs/tools/rl-cli”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to Vercel Sandbox
   Vercel Sandbox✓fullprobed9/10

   Vercel Sandbox is explicitly built for agentic/AI use cases and exposes a documented SDK (@vercel/sandbox), CLI, and REST API (openapi.json) for programmatic control—creating sandboxes, running commands, streaming logs, managing files, images, and persistence. This directly satisfies driving the product via a documented public API for AI-native workflows. Missing for 10: independent third-party corroboration beyond vendor docs/probe checks.

   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
   • [claimed-docs] “Use `@vercel/sandbox` for JavaScript and TypeScript”
   • [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
   • [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
   • [probe] “PROBE docs-md: HTTP 200 at https://vercel.com/docs/sandbox.md --- title: Vercel Sandbox product: vercel url: /docs/sandbox canonical_url: "h…”
   • [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

   Runloop✓fullprobed8/10

   Runloop documents a full public API surface: Devboxes API, Agents API, SDKs in common languages, CLI (rli), and GitHub Action integration, all callable programmatically for agent-driven workflows. This is strong first-party documentation of a public API, though probes show no discoverable OpenAPI spec, slightly weakening independent verification. Missing for 10: a published OpenAPI/swagger spec and independent third-party corroboration of API usage.

   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”
   • [probe] “official CLI documented at https://docs.runloop.ai/docs/tools/rl-cli”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to Runloop
   Vercel Sandbox—none0/10

   Evidence only describes authentication mechanisms (automatic OIDC token in production, access tokens for CI/CD) but nowhere documents scoped or least-privilege permission models for these credentials — no mention of granular scopes, role restrictions, or per-agent credential limits.

   • [claimed-docs] “In production on Vercel, authentication is automatic.”
   • [claimed-docs] “Use access tokens when `VERCEL_OIDC_TOKEN` is unavailable, such as in external CI/CD systems or non-Vercel environments.”
   • [claimed-docs] “For local development, run `vercel link` and `vercel env pull` to get a development token. In production on Vercel, authentication is automa…”

   proof ↗⚑ flag

   Runloop~partialclaimed6/10

   Runloop's Agent Gateways let agents call LLM APIs like Anthropic/OpenAI 'without ever seeing your API keys,' which is a direct least-privilege credentialing mechanism for agents. However, the docs give only a single brief mention with no detail on fine-grained scopes, per-agent permission levels, revocation, or audit logging. Missing for 10: detailed scoping/permission model documentation, credential rotation/revocation controls, and independent verification of the gateway's isolation guarantees.

   • [claimed-docs] “Agent Gateways let your agents call LLM APIs like Anthropic and OpenAI **without ever seeing your API keys**.”
   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Vercel Sandbox
   Vercel Sandbox✓fullclaimed7/10

   Vercel Sandbox offers an official `@vercel/sandbox` SDK for JS/TS plus a CLI, with docs on authentication, commands, and sandbox lifecycle management — evidence that AI-native builders have a first-party SDK to integrate against. missing for 10: no mention of official SDKs for other languages (e.g., Python), no independent/hands-on corroboration of SDK usage beyond vendor docs.

   • [claimed-docs] “Use `@vercel/sandbox` for JavaScript and TypeScript”
   • [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
   • [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
   • [claimed-docs] “For local development, run `vercel link` and `vercel env pull` to get a development token. In production on Vercel, authentication is automa…”
   • [claimed-docs] “Commands can run in blocking mode (wait for completion) or detached mode (return immediately).”

   proof ↗⚑ flag

   Runloop~partialclaimed6/10

   Runloop explicitly documents official SDKs in common languages for interacting with its API, plus tutorials integrating with SDKs like OpenAI Agents SDK, supporting an AI-native builder workflow. However, evidence lacks specifics on which languages are covered, versioning/maturity, or independent developer corroboration. Missing for 10: enumerated language list/maturity details, independent hands-on confirmation, changelog/versioning evidence.

   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Vercel Sandbox—none0/10

   Evidence covers sandbox creation, execution modes, persistence, and streaming logs from detached commands, but there is no mention of webhooks or event subscription mechanisms for external systems to receive Sandbox events.

   ⚑ flag

   Runloop—none0/10

   No evidence mentions webhooks or event subscription mechanisms; Runloop's docs describe devbox lifecycle events via Axons event streams but nothing about webhook subscriptions for external systems. Missing for 10: any documentation of a webhooks API, subscription/callback URLs, or event delivery mechanism to external endpoints.

   ⚑ flag

Agentic features

1. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Runloop
   Vercel Sandbox~partialclaimed5/10

   Vercel Sandbox supports running agent-generated code, detached/background commands with log streaming, timeouts/extension, and persistence/resume, which enable building autonomous background automations. However, there's no built-in scheduler, trigger system, or native 'automation' orchestration layer — users must build their own scheduling/trigger logic around the SDK/CLI. missing for 10: native scheduling/cron or event-trigger mechanism, independent evidence of long-running unattended automations in production, and explicit 'autonomous background job' framing beyond detached commands.

   • [claimed-docs] “Commands can run in blocking mode (wait for completion) or detached mode (return immediately).”
   • [claimed-docs] “Stream logs from a detached command”
   • [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
   • [claimed-docs] “Persistent sandboxes: Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot m…”
   • [claimed-docs] “Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot management needed.”
   • [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”

   proof ↗⚑ flag

   Runloop~partialclaimed7/10

   Runloop's Axons feature gives agents a persistent event stream letting them suspend when idle, wake on demand, recover state, and hand off between agents 'without you building the plumbing' (runloop-docs-17, runloop-docs-18), and devboxes support suspend/resume to preserve state across runs (runloop-docs-27). The GitHub Action for agent deployment (runloop-docs-24) supports triggering agent runs from CI workflows, enabling background automation setups. Missing for 10: no explicit cron/scheduler primitive, no independent/hands-on account of long-running unattended automations actually working in production, and no example of a full always-on autonomous pipeline beyond docs claims.

   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”

   proof ↗⚑ flag

2. ai-native userOperate the product with natural-language commands

   weight 2 · round to Runloop
   Vercel Sandbox—none0/10

   Vercel Sandbox is designed to run agent-generated code and is explicitly marketed for 'agent workflows,' but the evidence only describes CLI commands (`sandbox create`, `sandbox run`) and SDK calls, not a natural-language interface for operating the product itself. No documentation shows a chat/NL command layer for creating or managing sandboxes. Missing for 10: any NL command parser, chat-driven CLI, or natural-language API for controlling Sandbox operations.

   • [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
   • [claimed-docs] “Create with an explicit name sandbox create --name my-sandbox”
   • [claimed-docs] “sandbox run echo "Hello Sandbox!"”

   proof ↗⚑ flag

   Runloop~partialprobed6/10

   Runloop exposes an official MCP server and ships Cursor '.mdc' rule files that give AI assistants context on the SDK (devbox lifecycle, file ops, blueprints, snapshots, tunnels), enabling natural-language operation through AI coding assistants rather than a bespoke NL interface. There's no first-party chat/NL command surface of Runloop's own, and no independent evidence of how well the MCP/Cursor integration performs in practice. Missing for 10: dedicated natural-language interface docs, hands-on/independent validation of the MCP or Cursor integration actually working end-to-end.

   • [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”
   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Vercel Sandbox—none0/10

   Evidence shows Vercel has an OpenAPI spec and Sandbox docs, but nothing describes an interactive API reference with runnable/live examples for Sandbox itself — no docs playground, no 'try it' console, no runnable code snippets in-page. missing for 10: interactive API reference UI, runnable example execution in docs, evidence of a docs playground feature.

   • [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
   • [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”

   proof ↗⚑ flag

   Runloop—none0/10

   No evidence of an interactive API reference with runnable examples; the explicit probe for OpenAPI/swagger specs returned 404s across all candidate paths, and docs mention SDKs/CLI but not a runnable API explorer.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round to Vercel Sandbox
   Vercel Sandbox✓fullprobed9/10

   A probe confirms Vercel publishes a machine-readable OpenAPI spec at https://vercel.com/openapi.json which validly contains the 'openapi' key, and Sandbox is part of Vercel's documented API/SDK surface. Missing for 10: no explicit confirmation that Sandbox-specific endpoints are included in that spec versus general Vercel API.

   • [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
   • [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”

   proof ↗⚑ flag

   Runloop—none0/10

   Runloop offers SDKs and a CLI but there is no evidence of a downloadable OpenAPI/Swagger spec; explicit probes to common OpenAPI paths (openapi.json, swagger.json, etc.) all returned 404, confirming absence rather than just lack of mention.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round drawn
   Vercel Sandbox✓fullclaimed8/10

   Vercel Sandbox is explicitly designed to run untrusted/agent-generated code in isolated Linux microVMs 'without exposing your production systems,' with per-agent isolation, ephemeral or persistent state, and no shared access to production infrastructure. This directly matches the AI-native story of testing agent code safely away from production data. missing for 10: independent/hands-on evidence of real-world isolation guarantees, and explicit documentation on data-boundary controls preventing production data leakage into sandboxes.

   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
   • [claimed-docs] “Execute untrusted code safely: Run AI agent output, user uploads, or third-party scripts without exposing your production systems.”
   • [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”
   • [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self-contained environment with access to logs, file edit…”
   • [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop's Devboxes are isolated sandbox execution environments (secure, snapshot/blueprint-based, no production data by default) explicitly designed for testing AI agents, fully separate from any production system. missing for 10: no independent/hands-on corroboration beyond vendor docs, no explicit statement contrasting sandbox vs production data isolation guarantees.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Vercel Sandbox—none0/10

   The evidence pack shows SDK/CLI docs, features, and an OpenAPI spec, but no mention of API versioning scheme or a documented deprecation policy for Vercel Sandbox's APIs. Some features are explicitly marked 'beta' (docs-14, docs-44), suggesting instability without a stated deprecation process. missing for 10: explicit API version numbers, a deprecation/sunset policy, changelog entries documenting breaking-change handling.

   ⚑ flag

   Runloop—none0/10

   No evidence of API versioning scheme or a documented deprecation policy; OpenAPI spec probe returned 404s and no changelog/versioning docs are cited.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Runloop
   Vercel Sandbox—none0/10

   Vercel Sandbox docs describe creating/running individual sandboxes, multi-agent isolation, and SDK-driven command execution, but there is no evidence of a bulk/batch operation feature for processing many items at once (e.g., batch job APIs, fan-out orchestration, or bulk file/task processing).

   ⚑ flag

   Runloop~partialclaimed6/10

   Runloop supports running 10k+ parallel sandboxes, benchmark jobs that provision devboxes/run agents/aggregate outputs 'with a single command,' and snapshots for fan-out across multiple approaches — evidence of bulk/parallel operation across many devboxes. However, there's no explicit documentation of a generic bulk-operation API/CLI verb (e.g., bulk delete, bulk update, batch create) for arbitrary resources beyond devboxes/benchmarks. missing for 10: explicit bulk CRUD operations across resources like snapshots/storage objects/secrets, documented batch API endpoints, and independent evidence of large-scale bulk usage beyond vendor docs.

   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “Run your agent against well-known open source benchmarks like terminal bench 2, AIME, and more.”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Runloop
   Vercel Sandbox—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Runloop~partialclaimed4/10

   Runloop's Axons give agents a persistent event stream that can suspend on idle, wake on demand, recover state, and hand off between agents, which is a form of automatic event-driven behavior, and the Broker component sits between Axon and agent process to manage this. However, there's no documentation of a general user-facing rules engine where arbitrary conditions map to arbitrary actions across the platform (e.g., webhooks, triggers on Devbox/snapshot events, custom rule definitions). Missing for 10: a documented rules/trigger definition interface, examples of custom event-to-action mappings beyond agent suspend/wake, and evidence of triggers on other platform events (snapshots, storage, secrets).

   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round drawn
   Vercel Sandbox—none0/10

   Vercel Sandbox provides on-demand ephemeral/persistent execution environments with manual creation, timeouts, and resumption, but no evidence of a scheduler, cron-like trigger, or recurring job orchestration feature; scheduling would need to be built externally (e.g., Vercel Cron) which isn't documented here as part of Sandbox itself.

   • [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
   • [claimed-docs] “The default timeout is 5 minutes. You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.ex…”
   • [claimed-docs] “Or retrieve an existing sandbox by name (resumes if stopped)”

   proof ↗⚑ flag

   Runloop—none0/10

   The evidence describes Devboxes, snapshots, benchmarks, and Axons for event-driven suspend/resume, but nothing about scheduling recurring jobs or cron-like recurring workflows. Absence of evidence for this applicable automation capability yields 'none'.

   ⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round to Runloop
   Vercel Sandbox—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Runloop~partialclaimed4/10

   Runloop's Snapshots explicitly support rolling back Devbox state to a known good point, and Blueprints let you version/reuse startup configurations, giving a rudimentary rollback mechanism for agent environments. However, there is no evidence of a review workflow (diffs, approvals, audit trail) or of 'automations' as a first-class versioned object distinct from raw disk/environment snapshots. Missing for 10: explicit review/diff tooling for automations, first-class automation versioning (not just devbox/environment snapshots), audit/history UI for changes.

   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”

   proof ↗⚑ flag

Agent workloads

1. ai-native userRun coding agents like Claude Code or Codex inside the sandbox following the vendor's own recipe

   weight 2 · round to Vercel Sandbox
   Vercel Sandbox~partialclaimed6/10

   Docs confirm the default 'vercel/sandbox/universal' image ships with coding agents preinstalled and highlight agent workflows (e.g. Claude Code/Codex-style use), but the evidence pack lacks an explicit vendor 'recipe' walkthrough naming Claude Code or Codex step-by-step. missing for 10: a dedicated vendor tutorial/recipe explicitly running Claude Code or Codex inside the sandbox, and independent hands-on confirmation of this exact workflow.

   • [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need. The default image is `vercel/sandbox/universa…”
   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs.”
   • [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”
   • [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”

   proof ↗⚑ flag

   Runloop~partialclaimed5/10

   Runloop documents running AI coding agents generally inside Devboxes (blueprints, benchmarks, Agents API, OpenAI Agents SDK tutorial) but no specific vendor recipe or tutorial naming Claude Code or Codex is present in the evidence. missing for 10: explicit Claude Code/Codex quickstart or blueprint, named integration guide for these specific agents.

   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [claimed-docs] “Run your agent against well-known open source benchmarks like terminal bench 2, AIME, and more.”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”

   proof ↗⚑ flag

Workloads

1. developerRun a headless browser or full desktop environment inside the sandbox

   weight 2 · round drawn
   Vercel Sandbox—none0/10

   Evidence describes Linux microVMs, custom OCI images, and general code execution, but nothing mentions headless browser support, display/VNC, GUI desktop environments, or browser automation tooling. missing for 10: any mention of headless browser (e.g., Chromium/Playwright) support, GPU/display capabilities, or desktop environment provisioning.

   ⚑ flag

   Runloop—none0/10

   No evidence Runloop supports running a headless browser or desktop GUI inside its Devbox sandboxes; the only browser-related tutorial explicitly states the architecture avoids this, running Chromium externally via Browserbase and connecting over CDP so 'no Chromium ever runs in the devbox.' Blueprints/docs never mention GUI, X11, VNC, or bundled browser binaries as a supported environment type.

   • [claimed-docs] “the agent runs in a devbox, the browser runs on Browserbase, and the devbox drives it by connecting Playwright over CDP to the session’s con…”

   proof ↗⚑ flag

2. developerAttach GPUs to sandboxed workloads

   weight 2 · round drawn
   Vercel Sandbox—none0/10

   No evidence anywhere in the docs mentions GPU support or GPU attachment for Vercel Sandbox; sandboxes are described only as CPU-based Linux microVMs with system-privileged process support, storage/drives, and networking. missing for 10: any mention of GPU hardware, GPU-enabled images, or GPU pricing/quota.

   • [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need.”
   • [claimed-docs] “System-privileged processes: Run workloads that need system-level privileges, such as container runtimes like Docker, VPN clients, and FUSE …”
   • [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”

   proof ↗⚑ flag

   Runloop—none0/10

   The evidence pack describes Devboxes, blueprints, snapshots, mounts, storage, and networking features but never mentions GPU support or GPU-attached compute for sandboxed workloads; nothing indicates hardware acceleration is available.

   ⚑ flag

Files

1. developerRead, write, upload, and download files in the sandbox filesystem via the SDK

   weight 2 · round to Vercel Sandbox
   Vercel Sandbox~partialclaimed6/10

   Docs confirm the SDK provides a programmatic interface to manage files (mount remote storage, drives, filesystem persistence via snapshots) and general file access is implied through 'file edits' and FUSE-mounted storage, but no explicit documentation of discrete SDK methods like sandbox.writeFiles(), readFile(), or upload/download APIs is present. missing for 10: explicit SDK method documentation/examples for read, write, upload, and download file operations, and independent/hands-on confirmation of these specific API calls.

   • [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
   • [claimed-docs] “Mount an external object store such as Amazon S3 with a FUSE driver, and read and write remote files through the sandbox filesystem.”
   • [claimed-docs] “Drives (beta): Attach persistent filesystem storage to sandboxes and reuse data across sandbox runs.”
   • [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self-contained environment with access to logs, file edit…”
   • [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”

   proof ↗⚑ flag

   Runloop~partialclaimed4/10

   Docs confirm the SDK/CLI supports 'file operations' as part of Devbox management (mentioned alongside lifecycle, command execution, etc. in the Cursor rules integration), implying read/write file capability, but no dedicated documentation page or code sample specifically shows upload/download or read/write file APIs. Missing for 10: explicit SDK method docs for file read/write, and explicit upload/download examples or independent corroboration.

   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”

   proof ↗⚑ flag

Runtimes

1. developerDefine custom sandbox templates or bring my own container image

   weight 2 · round to Vercel Sandbox
   Vercel Sandbox✓fullclaimed9/10

   Docs explicitly support custom sandbox templates and BYO container images: users can build custom OCI images stored in Vercel Container Registry, start sandboxes from managed, shared/public, or custom images, and Docker-image-based workflows are directly supported. Also supports snapshotting for reusable custom environments. missing for 10: independent/hands-on corroboration of custom image builds beyond first-party docs.

   • [claimed-docs] “Build your system packages and tooling into an image, and share it across projects and teams.”
   • [claimed-docs] “If you already use Docker images to define your environment, store the image in Vercel Container Registry (VCR) and create the sandbox with …”
   • [claimed-docs] “Start sandboxes from Vercel's Managed Images, a shared or public image, or your own OCI images stored in Vercel Container Registry.”
   • [claimed-docs] “Managed and custom images: Start sandboxes from Vercel's Managed Images, a shared or public image, or your own OCI images stored in Vercel C…”
   • [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need. The default image is `vercel/sandbox/universa…”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop's Blueprints support building custom sandbox images from a Dockerfile (`rli blueprint from-dockerfile`), plus reusable prebuilt blueprints and snapshotting of custom states, directly enabling developers to bring their own container image or define custom templates. Missing for 10: independent/hands-on verification of Dockerfile-based blueprint builds and more detail on supported image registries or base image constraints.

   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”
   • [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”

   proof ↗⚑ flag

2. developerExecute code in multiple language runtimes (Python, JavaScript, and more) and get rich results back

   weight 2 · round to Vercel Sandbox
   Vercel Sandbox✓fullclaimed7/10

   Docs confirm sandboxes run Linux images with Python 3.14 and Node.js LTS pre-installed by default, plus support for any other Linux distro via custom OCI images, enabling multi-language execution (JS/TS, Python, and more) (docs-5, docs-8, docs-34, docs-30). Execution results are surfaced via logs, streamed output, and file access (docs-19, docs-20, docs-26), but there's no explicit mention of structured/rich result formats (e.g., images, dataframes, plots) beyond logs and files. Missing for 10: explicit support for rich/structured output types (charts, tables, media) and independent hands-on confirmation of multi-language fidelity.

   • [claimed-docs] “Use `@vercel/sandbox` for JavaScript and TypeScript”
   • [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need.”
   • [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need. The default image is `vercel/sandbox/universa…”
   • [claimed-docs] “Start sandboxes from Vercel's Managed Images, a shared or public image, or your own OCI images stored in Vercel Container Registry.”
   • [claimed-docs] “Commands can run in blocking mode (wait for completion) or detached mode (return immediately).”
   • [claimed-docs] “Stream logs from a detached command”
   • [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self-contained environment with access to logs, file edit…”

   proof ↗⚑ flag

   Runloop~partialclaimed5/10

   Runloop's Devboxes are described as full-featured secure sandboxed execution environments that can run arbitrary commands (via blueprints, SSH, CLI) which implies multi-language support, but there is no explicit documentation of a code-execution API returning 'rich results' (e.g., structured outputs, images, dataframes) as in dedicated code-interpreter products. missing for 10: explicit multi-language runtime examples (Python/JS/etc.), rich/structured result formatting (e.g., stdout+images+tables), and independent confirmation of these capabilities.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”

   proof ↗⚑ flag

Untrusted code

1. developerExecute untrusted, AI-generated code without risking my own infrastructure

   weight 3 · round to Vercel Sandbox
   Vercel Sandbox✓fullclaimed9/10

   Vercel Sandbox's docs directly describe running untrusted or AI agent-generated code in isolated Linux microVMs without exposing production systems, with per-agent isolation, configurable timeouts, and managed images—precisely matching the story. Missing for 10: independent/hands-on verification beyond first-party docs.

   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
   • [claimed-docs] “Execute untrusted code safely: Run AI agent output, user uploads, or third-party scripts without exposing your production systems.”
   • [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”
   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs.”
   • [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self-contained environment with access to logs, file edit…”

   proof ↗⚑ flag

   Runloop✓fullprobed8/10

   Runloop's core product is Devboxes, secure sandboxed execution environments purpose-built to run AI agent/code workloads isolated from user infrastructure, with snapshots, tunnels, secrets isolation, and scaling to 10k+ parallel sandboxes. This directly matches the story of executing untrusted AI-generated code without risking one's own infra. Missing for 10: independent/third-party hands-on validation of isolation guarantees, and explicit documentation of the underlying sandboxing/virtualization technology or security boundary details.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Let’s use devbox tunnels to securely access ports on your devbox over a simple url.”
   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”

   proof ↗⚑ flag

2. developerRun arbitrary shell commands and install packages inside the sandbox

   weight 2 · round to Vercel Sandbox
   Vercel Sandbox✓fullclaimed8/10

   Docs clearly show running arbitrary commands (blocking/detached modes, `sandbox run echo ...`, CLI and SDK) inside Linux microVMs, with custom images and full Linux distros allowing package installs (e.g., Ubuntu/Arch, custom OCI images, system-privileged processes). missing for 10: no independent/hands-on report confirming package installation workflows beyond first-party docs.

   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
   • [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need.”
   • [claimed-docs] “System-privileged processes: Run workloads that need system-level privileges, such as container runtimes like Docker, VPN clients, and FUSE …”
   • [claimed-docs] “Commands can run in blocking mode (wait for completion) or detached mode (return immediately).”
   • [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
   • [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need. The default image is `vercel/sandbox/universa…”
   • [claimed-docs] “sandbox run echo "Hello Sandbox!"”
   • [claimed-docs] “Start sandboxes from Vercel's Managed Images, a shared or public image, or your own OCI images stored in Vercel Container Registry.”

   proof ↗⚑ flag

   Runloop✓fullclaimed7/10

   Runloop Devboxes are full Linux execution environments with SSH access via the CLI (runloop-docs-9) and blueprints built from Dockerfiles that install packages/dependencies at startup (runloop-docs-2, runloop-docs-14), implying arbitrary shell command execution and package installation. However, the evidence pack lacks a direct hands-on example of running an explicit shell command or apt/pip install inside a Devbox. missing for 10: explicit example/output of running an arbitrary shell command and installing a package inside a Devbox, independent third-party confirmation.

   • [claimed-docs] “From the interactive menu, select a devbox and choose **SSH** to open a secure shell session directly into your devbox. The CLI handles all …”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”

   proof ↗⚑ flag

Agent lifecycle

1. ai-native userMy agent can provision its own sandbox, execute code, read the results, and tear it down — end to end without a human

   weight 3 · round drawn
   Vercel Sandbox✓fullclaimed8/10

   Vercel Sandbox provides a full programmatic SDK/CLI to create sandboxes (with names, custom images, regions), run blocking or detached commands, stream logs/output, and stop/tear down sandboxes, with API-based authentication (OIDC/tokens) suitable for unattended agent use — no human-in-the-loop required. missing for 10: no independent/hands-on third-party report confirming an agent autonomously completing the full provision-execute-teardown loop, and no explicit example showing an agent reading command output/results programmatically end-to-end.

   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
   • [claimed-docs] “Execute untrusted code safely: Run AI agent output, user uploads, or third-party scripts without exposing your production systems.”
   • [claimed-docs] “Commands can run in blocking mode (wait for completion) or detached mode (return immediately).”
   • [claimed-docs] “Stream logs from a detached command”
   • [claimed-docs] “You can also stop sandboxes from the Vercel Dashboard by navigating to Observability > Sandboxes and clicking Stop Sandbox.”
   • [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
   • [claimed-docs] “For local development, run `vercel link` and `vercel env pull` to get a development token. In production on Vercel, authentication is automa…”
   • [claimed-docs] “Create with an explicit name sandbox create --name my-sandbox”
   • [claimed-docs] “sandbox run echo "Hello Sandbox!"”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop's core product is programmatic sandbox (Devbox) provisioning, execution, and teardown via API/SDK/CLI, explicitly designed for AI agents — including an Agents API for creating/managing/deploying agents, blueprints for reusable environments, snapshots for state capture, and suspend/resume lifecycle, all achievable without human intervention (docs-1, docs-13, docs-22, docs-23, docs-2, docs-4, docs-27, docs-12). The quickstart shows a devbox spun up in ~1 minute and CLI/SDK access covers execution and result retrieval end-to-end. missing for 10: independent/hands-on third-party verification of a full agent-driven provision→execute→teardown loop, and a concrete code example showing an agent reading execution results programmatically.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

Isolation

1. platform-engineerRely on a documented hard isolation boundary (microVM or equivalent) between sandboxes and my systems

   weight 3 · round to Vercel Sandbox
   Vercel Sandbox~partialclaimed6/10

   Vercel's first-party docs repeatedly and explicitly claim microVM-based isolation for running untrusted/agent-generated code without exposing production systems, plus additional isolation features (per-agent Linux users, system-privileged process support). However, all evidence is vendor documentation with no independent security audit, third-party penetration test, or technical deep-dive into the underlying microVM technology (e.g. Firecracker) confirming the boundary in practice. missing for 10: independent/third-party verification of the isolation boundary, technical detail on the hypervisor/microVM implementation, and any hands-on security assessment.

   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
   • [claimed-docs] “Execute untrusted code safely: Run AI agent output, user uploads, or third-party scripts without exposing your production systems.”
   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs.”
   • [claimed-docs] “System-privileged processes: Run workloads that need system-level privileges, such as container runtimes like Docker, VPN clients, and FUSE …”
   • [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”

   proof ↗⚑ flag

   Runloop~partialprobed3/10

   Runloop repeatedly describes Devboxes as "secure sandboxed execution environments" and "isolated Linux sandbox environments," implying some isolation boundary, but the evidence never names or documents the actual isolation mechanism (e.g., microVM, Firecracker, gVisor, container-only) or any architecture diagram/security whitepaper explaining the boundary. missing for 10: explicit documentation of the underlying isolation technology (microVM/hypervisor-level or equivalent), any security architecture doc or third-party audit confirming hard isolation guarantees.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Runloop allows you to run benchmarks at scale in a secure environment.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”

   proof ↗⚑ flag

2. ai-native userGive an agent a sandbox where host secrets and credentials are unreachable by the code it runs

   weight 2 · round to Vercel Sandbox
   Vercel Sandbox✓fullclaimed8/10

   Vercel Sandbox docs explicitly describe running agent-generated/untrusted code in isolated Linux microVMs 'without exposing your production systems,' and support per-agent Linux user isolation, which directly implies host secrets/credentials are not reachable by sandboxed code. missing for 10: explicit documentation of how host secrets/env vars are excluded or scoped from the sandbox environment, and independent/hands-on verification of this isolation guarantee.

   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
   • [claimed-docs] “Execute untrusted code safely: Run AI agent output, user uploads, or third-party scripts without exposing your production systems.”
   • [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”
   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs.”

   proof ↗⚑ flag

   Runloop~partialclaimed6/10

   Runloop's Devboxes are documented as sandboxed execution environments with encrypted secrets and, notably, Agent Gateways that let agents call LLM APIs 'without ever seeing your API keys' — directly supporting the isolation-of-credentials use case. However, secrets are also stated to be 'automatically made available as environment variables' inside Devboxes, which is somewhat in tension with 'unreachable by code' and isn't clarified as scoped/restricted; there's no independent/hands-on verification of secret unreachability or a security audit. missing for 10: independent verification of secret isolation, clarity on env-var secret exposure vs true unreachability, and details on network/filesystem-level credential isolation guarantees.

   • [claimed-docs] “Agent Gateways let your agents call LLM APIs like Anthropic and OpenAI **without ever seeing your API keys**.”
   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”

   proof ↗⚑ flag

Network policy

1. platform-engineerRestrict or allow the sandbox's network egress with explicit policy

   weight 3 · round drawn
   Vercel Sandbox—none0/10

   No evidence in the pack mentions network egress controls, firewall rules, or outbound traffic policy for Vercel Sandbox—coverage focuses on isolation via microVMs, filesystem, images, and multi-agent user separation but never explicit egress allow/deny configuration.

   ⚑ flag

   Runloop—none0/10

   No evidence pack item describes network egress controls, firewall rules, or allow/deny policies for Devbox sandboxes; tunnels and secrets features are documented but network egress policy configuration is never mentioned.

   ⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Vercel Sandbox
   Vercel Sandbox✓fullprobed7/10

   Vercel Sandbox is fundamentally API/SDK-driven (creation, commands, snapshots, tags, drives, custom images all documented via SDK/CLI), and the dashboard only offers a subset (viewing/stopping sandboxes) that mirrors API capability rather than adding unique UI-only features, and an OpenAPI spec exists confirming programmatic access. Missing for 10: an explicit vendor statement confirming full UI/API parity, and independent confirmation that every dashboard action (e.g., tag/drive management) is scriptable via API with no UI-exclusive features.

   • [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
   • [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
   • [claimed-docs] “You can also stop sandboxes from the Vercel Dashboard by navigating to Observability > Sandboxes and clicking Stop Sandbox.”
   • [claimed-docs] “Connect to and manage Sandboxes from the dashboard”
   • [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

   Runloop~partialprobed6/10

   Runloop is API-first: the CLI (rli) wraps SDKs/API for devboxes, blueprints, snapshots, benchmarks, agents, secrets, etc., suggesting broad API/CLI parity with any dashboard functionality. However there is no explicit statement claiming full UI/API feature parity, no documented OpenAPI spec (404s on all probed paths), and no direct dashboard-vs-API comparison. Missing for 10: explicit parity claim between web dashboard and API, published OpenAPI/reference spec, confirmation that all dashboard-only actions (e.g., billing, org settings) are also API-accessible.

   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to Runloop
   Vercel Sandbox—none0/10

   Vercel Sandbox stores state via drives, snapshots, and container images, so data portability is a fair question, but no evidence describes an explicit data-export feature or guarantee of open, exportable formats for that stored state. Missing for 10: any documented export/download-all-data capability, explicit open-format guarantees for drives/snapshots, or migration tooling to leave the platform.

   • [claimed-docs] “Drives (beta): Attach persistent filesystem storage to sandboxes and reuse data across sandbox runs.”
   • [claimed-docs] “Snapshotting: Save the state of a running sandbox to resume later. Skip dependency installation on subsequent runs.”
   • [claimed-docs] “If you already use Docker images to define your environment, store the image in Vercel Container Registry (VCR) and create the sandbox with …”

   proof ↗⚑ flag

   Runloop~partialclaimed4/10

   Runloop offers Snapshots to save Devbox disk state and export/re-create Devboxes, and Dockerfile-based blueprints imply portability of environments, but there is no documented feature for exporting all account/user data (benchmarks, configs, agent history) in open formats or a clear data-export/account-deletion workflow. missing for 10: explicit full account data export tool, open-format guarantees for all data types, data portability/deletion documentation.

   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Vercel Sandbox—none0/10

   No evidence that Vercel Sandbox's source code is published under an open license; the evidence pack only covers docs, features, pricing, and API surface, with no mention of a public repo or license.

   ⚑ flag

   Runloop—none0/10

   No evidence indicates Runloop's core source (the sandbox platform, Devboxes, Axons, etc.) is released under an open license; the evidence pack shows only docs, CLI, and SDK references without any license or open-source repository disclosure.

   ⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   Vercel Sandbox—none0/10

   Vercel Sandbox is a fully managed, hosted service running on Vercel's own microVM infrastructure (iad1/sfo1/cle1/cdg1 regions); there is no evidence of any self-hostable core, open-source runtime, or on-prem deployment option — it is inherently tied to Vercel's cloud.

   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
   • [claimed-docs] “Vercel Sandbox is available in the `iad1` (default), `sfo1`, `cle1`, and `cdg1` regions. You can choose the region per sandbox or set a proj…”
   • [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”

   proof ↗⚑ flag

   Runloop—none0/10

   Runloop is presented entirely as a hosted cloud platform (Devboxes, Agents API, benchmarks) with no evidence of an on-premises or self-hostable deployment option, open-source core, or self-hosting documentation.

   ⚑ flag

Latency

1. developerStart sandboxes with documented sub-second-to-few-second cold starts

   weight 3 · round to Runloop
   Vercel Sandbox—none0/10

   The docs describe microVM isolation, snapshotting for faster restarts, and persistence, but no evidence gives concrete cold-start latency figures (e.g., sub-second or few-second boot times) for starting a fresh sandbox. Missing for 10: documented benchmark numbers or explicit cold-start time claims, any independent performance measurement.

   • [claimed-docs] “Using a snapshot is much faster than creating from scratch because it avoids reinstalling dependencies and repeating setup steps.”
   • [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”

   proof ↗⚑ flag

   Runloop~partialclaimed6/10

   Runloop's marketing site documents '10GB image startup time in <2s' and a 1-minute quickstart, supporting sub-second-to-few-second cold starts, but this is a single first-party marketing claim without detailed benchmarks, methodology, or independent corroboration. missing for 10: independent/third-party benchmark confirmation, documentation of cold start times across varying image sizes/configurations, and reproducible performance data beyond the single homepage claim.

   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

Scale

1. platform-engineerRun large concurrent fleets of sandboxes with documented concurrency limits

   weight 3 · round to Runloop
   Vercel Sandbox~partialclaimed3/10

   Docs mention that Sandbox usage is metered and that quotas apply per plan, plus region availability and spend management for cost control, but no concrete documented concurrency limits (e.g., max simultaneous sandboxes) are shown in the evidence. missing for 10: explicit concurrency limit numbers, guidance on running large fleets of sandboxes concurrently, and any performance/scale benchmarks.

   • [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”
   • [claimed-docs] “Vercel Sandbox is available in the `iad1` (default), `sfo1`, `cle1`, and `cdg1` regions. You can choose the region per sandbox or set a proj…”
   • [claimed-docs] “To control costs, configure Spend Management to receive alerts or pause projects when you reach a specified amount.”

   proof ↗⚑ flag

   Runloop~partialclaimed5/10

   Runloop's marketing site claims support for 10k+ parallel sandboxes and sub-2s startup for 10GB images, showing scale is a design goal, but there is no documented concurrency limit, quota tiering, or rate-limit table a platform engineer could plan capacity around. Missing for 10: explicit documented per-account/org concurrency limits, guidance on requesting increases, and independent benchmarks validating the 10k+ claim at sustained load.

   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”

   proof ↗⚑ flag

2. developerKeep a sandbox session running for hours or days for long agent tasks

   weight 2 · round to Runloop
   Vercel Sandbox~partialclaimed6/10

   Docs show sandboxes have a default 5-minute timeout that can be explicitly extended via `timeout` option or `sandbox.extendTimeout()`, and persistent sandboxes auto-snapshot on stop so you can quickly resume a long agent task across sessions rather than losing state. However, there's no evidence of an unbounded or multi-day continuous session—developers must actively manage/extend timeouts or rely on stop/resume snapshotting rather than a single always-on session. missing for 10: documented maximum extendable duration or explicit support for a single continuous multi-hour/day session without developer-managed timeout extensions, independent corroboration of long-running stability.

   • [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
   • [claimed-docs] “The default timeout is 5 minutes. You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.ex…”
   • [claimed-docs] “Persistent sandboxes: Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot m…”
   • [claimed-docs] “Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot management needed.”
   • [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”
   • [claimed-docs] “Using a snapshot is much faster than creating from scratch because it avoids reinstalling dependencies and repeating setup steps.”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop Devboxes explicitly support long-running persistence via snapshots (save/restore disk state), suspend/resume to pause and continue exactly where left off, and Axons providing a persistent event stream that lets agents suspend when idle and wake on demand/recover state — directly addressing multi-hour/multi-day agent sessions without losing progress. Combined with SSH/tunnel access and mounts/secrets for long tasks, this covers the core story well. Missing for 10: independent/hands-on evidence (e.g., a third-party report) confirming multi-day session stability at scale, and explicit documented max session duration/limits.

   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”

   proof ↗⚑ flag

Pricing

1. developerStart building with a free tier or included credits without talking to sales

   weight 1 · round to Runloop
   Vercel Sandbox—none0/10

   The evidence pack mentions billing, plans, and quotas (e.g., vercel-sandbox-docs-42, docs-24) but never explicitly states a free tier or included credits usable without contacting sales. Missing for 10: explicit free-tier/plan pricing details, evidence of self-serve signup without sales contact.

   • [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”
   • [claimed-docs] “To control costs, configure Spend Management to receive alerts or pause projects when you reach a specified amount.”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Docs explicitly state Runloop's trial includes $50 free credits with no credit card required, and a quickstart lets a developer spin up a first devbox in about a minute, both without any sales contact. Missing for 10: independent/community confirmation of the free-tier signup experience and clarity on long-term free-tier limits beyond trial credits.

   • [claimed-docs] “It includes **$50 in free credits**, and no credit card is required to sign up.”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

2. platform-engineerPay per second only for the compute a sandbox actually uses

   weight 3 · round to Vercel Sandbox
   Vercel Sandbox~partialclaimed4/10

   Docs confirm Sandbox usage is metered on a pricing page with configurable timeouts and spend-management alerts to control costs, implying granular, usage-based billing, but no evidence explicitly states billing is charged 'per second' of compute used. missing for 10: explicit per-second billing granularity documentation, real-world billing statement/invoice examples showing sub-minute metering.

   • [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”
   • [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
   • [claimed-docs] “The default timeout is 5 minutes. You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.ex…”
   • [claimed-docs] “To control costs, configure Spend Management to receive alerts or pause projects when you reach a specified amount.”

   proof ↗⚑ flag

   Runloop—none0/10

   No evidence pack item states an explicit per-second or usage-based billing model; the only tangential mention is that suspend/resume can 'save costs' (runloop-docs-27), which doesn't establish a per-second pricing mechanism.

   ⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round to Vercel Sandbox
   Vercel Sandbox~partialclaimed5/10

   Docs confirm region selection (iad1, sfo1, cle1, cdg1) can be set per sandbox or as a project default, giving users some control over where sandboxes execute, and one changelog note claims availability 'in all regions.' However, there is no explicit data residency/compliance guarantee, no mention of where persistent Drives/snapshots/data-at-rest are stored, and no discussion of data sovereignty controls beyond compute region choice. missing for 10: explicit data-at-rest residency guarantees, compliance/certification details tied to region choice, documentation on storage location for Drives/snapshots, and independent confirmation of region enforcement.

   • [claimed-docs] “Vercel Sandbox is available in the `iad1` (default), `sfo1`, `cle1`, and `cdg1` regions. You can choose the region per sandbox or set a proj…”
   • [claimed-docs] “Vercel Sandbox is now available in all regions”

   proof ↗⚑ flag

   Runloop—none0/10

   No evidence in the pack mentions data residency, region selection, or geographic storage controls anywhere in Runloop's docs; the pack covers sandboxes, blueprints, snapshots, secrets, and tooling but never data-location choice.

   ⚑ flag

2. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Vercel Sandbox—none0/10

   Docs describe sandbox lifecycle (timeouts, stopping, persistence, snapshots) but there is no explicit statement about data retention policies, deletion guarantees, or privacy controls for stored sandbox data/drives. missing for 10: explicit data retention policy, explicit deletion/purge controls, privacy/compliance documentation for stored data.

   • [claimed-docs] “You can also stop sandboxes from the Vercel Dashboard by navigating to Observability > Sandboxes and clicking Stop Sandbox.”
   • [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
   • [claimed-docs] “Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot management needed.”
   • [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”

   proof ↗⚑ flag

   Runloop—none0/10

   The evidence pack covers sandboxing, blueprints, snapshots, secrets encryption, and tooling, but contains no documentation of data retention policies, data deletion controls, or user-facing privacy/compliance settings for Devboxes, snapshots, or storage objects. missing for 10: explicit data retention policy docs, user-initiated deletion/purge APIs, compliance/privacy documentation.

   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Storage Objects provide a way to store and manage files, data, and other resources that can be shared across Devboxes or made publicly avail…”

   proof ↗⚑ flag

3. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Vercel Sandbox—none0/10

   No evidence in the pack addresses telemetry/usage-tracking opt-out for Vercel Sandbox; documentation covers execution, isolation, images, and pricing but nothing about telemetry controls.

   ⚑ flag

   Runloop—none0/10

   No evidence in the pack mentions telemetry, analytics opt-out, or usage-tracking controls for Runloop; the docs cover sandboxing, blueprints, snapshots, secrets, etc., but nothing about a privacy/telemetry toggle.

   ⚑ flag

Lifecycle

1. developerSet timeouts so sandboxes shut down automatically and stop billing when idle or done

   weight 2 · round to Vercel Sandbox
   Vercel Sandbox✓fullclaimed8/10

   Docs explicitly state sandboxes have a default 5-minute timeout, configurable via the `timeout` option at creation, extendable via `sandbox.extendTimeout()`, and can be manually stopped from the dashboard, directly supporting automatic shutdown to stop billing when idle or done. Missing for 10: independent/hands-on confirmation that billing actually stops immediately upon timeout, and documentation of any maximum timeout limits or edge cases.

   • [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
   • [claimed-docs] “The default timeout is 5 minutes. You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.ex…”
   • [claimed-docs] “You can also stop sandboxes from the Vercel Dashboard by navigating to Observability > Sandboxes and clicking Stop Sandbox.”
   • [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”

   proof ↗⚑ flag

   Runloop~partialclaimed5/10

   Runloop documents suspend/resume to "save state and costs" and Axons that let agents "suspend when idle, wake on demand," indicating some idle-triggered lifecycle management, but there is no explicit documentation of a developer-configurable timeout parameter that auto-shuts-down a Devbox and stops billing after inactivity or completion. missing for 10: explicit timeout/idle-shutdown configuration API or CLI flag, confirmation that suspension halts billing automatically without agent action.

   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”

   proof ↗⚑ flag

2. developerSpin up an isolated sandbox with one API/SDK call and get a live environment in seconds

   weight 3 · round to Runloop
   Vercel Sandbox✓fullclaimed8/10

   Docs show a clear SDK/CLI path (`@vercel/sandbox`, `sandbox create`/`sandbox run`) to spin up isolated Linux microVMs, with fast startup via managed images or snapshots that skip dependency installation, matching the 'one call, live environment in seconds' story. Missing for 10: independent/hands-on timing benchmarks confirming 'seconds' latency and third-party corroboration beyond first-party docs.

   • [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
   • [claimed-docs] “Use `@vercel/sandbox` for JavaScript and TypeScript”
   • [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
   • [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
   • [claimed-docs] “sandbox run echo "Hello Sandbox!"”
   • [claimed-docs] “Using a snapshot is much faster than creating from scratch because it avoids reinstalling dependencies and repeating setup steps.”
   • [claimed-docs] “Start sandboxes from Vercel's Managed Images, a shared or public image, or your own OCI images stored in Vercel Container Registry.”

   proof ↗⚑ flag

   Runloop✓fullclaimed9/10

   Docs describe Devbox creation via SDK/CLI/API, quickstart claiming ~1 minute to a running devbox, blueprints/prebuilt images for fast startup, and marketing claims of 10k+ parallel sandboxes with <2s startup for a 10GB image, matching the one-call, seconds-to-live-environment story. missing for 10: independent/hands-on benchmark confirming real-world provisioning latency beyond vendor claims.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”

   proof ↗⚑ flag

Preview access

1. developerExpose a port from the sandbox on a public preview URL to reach services running inside

   weight 2 · round to Runloop
   Vercel Sandbox~partialclaimed5/10

   Docs mention running dev servers with 'live previews' and testing agent-generated code with access to 'live previews', implying some port-exposure/public-URL capability, but no evidence pack item describes the actual API/CLI mechanism (e.g., a domain/getUrl method, port argument, or URL format) for exposing a sandboxed port publicly. missing for 10: explicit documentation of the port-exposure API or CLI command, example of the resulting public URL, and confirmation of access control/authentication on that URL.

   • [claimed-docs] “Run development servers: Spin up and test applications with live previews.”
   • [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self-contained environment with access to logs, file edit…”
   • [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop explicitly documents devbox tunnels for securely accessing ports on a devbox over a simple URL, directly matching the story's requirement to expose a service port publicly. Missing for 10: independent/hands-on confirmation of tunnel reliability and details on URL persistence/security controls beyond the doc snippet.

   • [claimed-docs] “Let’s use devbox tunnels to securely access ports on your devbox over a simple url.”

   proof ↗⚑ flag

Snapshots

1. developerPause a running sandbox and resume it later with filesystem and memory state intact

   weight 2 · round drawn
   Vercel Sandbox✓fullclaimed8/10

   Docs explicitly describe persistent sandboxes that auto-save filesystem state on stop and resume where left off by default, with configuration preserved across sessions, plus explicit snapshotting to skip dependency reinstallation and resuming an existing sandbox by name. This directly matches the pause/resume-with-state story via first-party documentation. Missing for 10: independent hands-on verification that memory/process state (not just filesystem) is preserved across pause/resume, and no third-party corroboration beyond vendor docs.

   • [claimed-docs] “Persistent sandboxes: Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot m…”
   • [claimed-docs] “Snapshotting: Save the state of a running sandbox to resume later. Skip dependency installation on subsequent runs.”
   • [claimed-docs] “Or retrieve an existing sandbox by name (resumes if stopped)”
   • [claimed-docs] “Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot management needed.”
   • [claimed-docs] “Using a snapshot is much faster than creating from scratch because it avoids reinstalling dependencies and repeating setup steps.”
   • [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop explicitly documents 'Suspend and resume' functionality that pauses sandboxes to save state and resume exactly where left off, plus snapshot capability that saves disk state and can recreate Devboxes from that state. This directly matches the story's pause/resume with filesystem+memory state intact. Missing for 10: no independent/hands-on corroboration of memory-state fidelity across suspend/resume, and no detailed technical explanation of how memory state (not just disk) is preserved.

   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”

   proof ↗⚑ flag

2. developerSnapshot a sandbox and later restore or fork new sandboxes from that snapshot

   weight 3 · round to Runloop
   Vercel Sandbox~partialclaimed6/10

   Vercel Sandbox docs clearly describe automatic snapshotting of a sandbox's filesystem/state on stop and restoring it on resume, with named sandboxes resumable and snapshots speeding up subsequent runs by skipping setup. However, the evidence only shows resuming the *same* sandbox from its own snapshot, not explicitly forking multiple independent new sandboxes from one saved snapshot. Missing for 10: explicit support/documentation for forking multiple new sandboxes off a single snapshot, and any hands-on/independent confirmation of this fork workflow.

   • [claimed-docs] “Snapshotting: Save the state of a running sandbox to resume later. Skip dependency installation on subsequent runs.”
   • [claimed-docs] “Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot management needed.”
   • [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”
   • [claimed-docs] “Or retrieve an existing sandbox by name (resumes if stopped)”
   • [claimed-docs] “Using a snapshot is much faster than creating from scratch because it avoids reinstalling dependencies and repeating setup steps.”

   proof ↗⚑ flag

   Runloop✓fullclaimed8/10

   Runloop's docs explicitly describe snapshotting a Devbox's disk state and creating new Devboxes from that saved state, including fan-out use cases (multiple approaches from one snapshot) and rollback scenarios, directly matching the snapshot/restore/fork story. missing for 10: independent/hands-on corroboration beyond first-party docs, and no explicit code example showing the fork API call in the evidence pack.

   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”

   proof ↗⚑ flag