Agent Sandboxes & Code Execution Arena
Modal Sandboxes vs Vercel Sandbox
Vercel Sandbox wins · 11–16 (21 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round drawnModal has a working llms.txt (HTTP 200) with a documentation summary, and docs pages are also served as markdown (.md suffix) confirming agent-oriented doc access. missing for 10: no independent confirmation that agents actually consume this llms.txt successfully in practice, and no dedicated agent-docs index beyond the generic llms.txt.
- [probe] “PROBE llms.txt: HTTP 200 at https://modal.com/llms.txt # Modal Documentation > Modal is a platform for running AI workloads in the cloud wi…”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Probes confirm both a working llms.txt at the root domain (vercel.com/llms.txt, HTTP 200) and a markdown-formatted docs endpoint for Sandbox specifically (sandbox.md, HTTP 200), which are exactly the agent-oriented docs formats this story asks for. Missing for 10: no explicit evidence llms.txt itself deep-links to sandbox-specific docs, and no independent/community confirmation of an agent successfully consuming these files.
- [probe] “PROBE llms.txt: HTTP 200 at https://vercel.com/llms.txt # Vercel > Vercel is a cloud platform for building, deploying, and scaling web appl…”
- [probe] “PROBE docs-md: HTTP 200 at https://vercel.com/docs/sandbox.md --- title: Vercel Sandbox product: vercel url: /docs/sandbox canonical_url: "h…”
- [claimed-docs] “Connect to and manage Sandboxes from the dashboard”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round drawnModal Sandboxes is built for programmatic/headless use via Python, JS/Go SDKs and a CLI, with sandboxes created and controlled entirely through code (modal-docs-2, modal-docs-19, modal-docs-20, modal-gh-1). Community evidence confirms real-world CI/automation usage — a developer built GitHub Actions self-hosted runners on Modal Sandboxes explicitly for scalable, pay-per-use automation (modal-comm-1), and another wired gRPC control for an agent runtime (modal-comm-2). Missing for 10: no first-party CI/CD integration guide (e.g., official GitHub Actions/Jenkins docs) or dedicated 'headless CI' documentation section beyond community projects.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [github] “The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [probe] “official CLI documented at https://modal.com/docs/reference/cli”
Vercel Sandbox is designed for headless/programmatic use via SDKs (@vercel/sandbox) and a CLI, with access tokens explicitly documented for external CI/CD systems and non-Vercel environments, plus OpenAPI-backed REST access confirmed by probe. missing for 10: no explicit third-party/independent CI pipeline case study or GitHub Actions example demonstrating real-world headless automation.
- [claimed-docs] “Use `@vercel/sandbox` for JavaScript and TypeScript”
- [claimed-docs] “Use access tokens when `VERCEL_OIDC_TOKEN` is unavailable, such as in external CI/CD systems or non-Vercel environments.”
- [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
- [claimed-docs] “For local development, run `vercel link` and `vercel env pull` to get a development token. In production on Vercel, authentication is automa…”
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round drawnModal Sandboxesnone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
ai-native userConnect an agent via an official MCP server
weight 3 · round drawnModal Sandboxesnone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
Vercel Sandboxnone0/10Vercel Sandbox is an execution/infrastructure product (not itself an agent), so an official MCP server exposing it to agents is a fair, applicable axis; however no evidence in the pack mentions MCP, an MCP server, or any agent-protocol integration—only SDKs, CLI, and REST/OpenAPI docs are described.
ai-native userUse an official CLI
weight 2 · round to Modal SandboxesModal ships an official CLI (documented at modal.com/docs/reference/cli) supporting sandbox-related operations like running commands/shells and managing/connecting to containers, confirmed by probe access to the CLI reference page. This CLI is core to Modal's agentic/AI-native workflows (creating and interacting with sandboxes for untrusted/agent code). missing for 10: independent hands-on user reports specifically praising/using the CLI (vs. SDK), and more detail on CLI coverage of full sandbox lifecycle (snapshots, networking) beyond the two documented subcommands.
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “Manage and connect to running containers.”
- [probe] “official CLI documented at https://modal.com/docs/reference/cli”
- [claimed-docs] “Running commands”
Vercel Sandbox ships an official `sandbox` CLI explicitly positioned for manual testing, agentic workflows, debugging, and one-off operations, with documented commands like `sandbox create --name` and `sandbox run`. missing for 10: independent/hands-on corroboration beyond vendor docs, and no CLI reference/full command list shown.
- [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
- [claimed-docs] “Create with an explicit name sandbox create --name my-sandbox”
- [claimed-docs] “sandbox run echo "Hello Sandbox!"”
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
ai-native userDrive the product through a documented public API
weight 3 · round to Vercel SandboxModal exposes documented Python, JS, and Go SDKs plus a CLI for programmatically creating, controlling, and interacting with Sandboxes (exec, filesystem, snapshots, networking, logs), and community developers confirm driving it via gRPC/SDK for agent runtimes. missing for 10: no public OpenAPI/REST spec (probe found only 404s), so the 'documented public API' is SDK/CLI-based rather than a formal API contract, and there's no independent audit of API stability/versioning.
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [github] “The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “Manage and connect to running containers.”
- [claimed-docs] “We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
- [probe] “PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, …”
Vercel Sandbox is explicitly built for agentic/AI use cases and exposes a documented SDK (@vercel/sandbox), CLI, and REST API (openapi.json) for programmatic control—creating sandboxes, running commands, streaming logs, managing files, images, and persistence. This directly satisfies driving the product via a documented public API for AI-native workflows. Missing for 10: independent third-party corroboration beyond vendor docs/probe checks.
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
- [claimed-docs] “Use `@vercel/sandbox` for JavaScript and TypeScript”
- [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
- [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
- [probe] “PROBE docs-md: HTTP 200 at https://vercel.com/docs/sandbox.md --- title: Vercel Sandbox product: vercel url: /docs/sandbox canonical_url: "h…”
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round drawnModal Sandboxesnone0/10The evidence describes secure sandboxed execution environments and general platform features, but there is no concrete evidence of a mechanism for issuing scoped/least-privilege API credentials specifically to agents—only a vague marketing line about 'credentials already in place' (modal-docs-3) with no documentation of credential scoping, token permissions, or per-agent access control. Missing for 10: any documented API/credential-scoping mechanism, permission granularity docs, or examples of issuing least-privilege tokens to agents.
- [claimed-docs] “Autonomous agents with the right tools, context, and credentials already in place — running securely in a full, isolated dev environment.”
Vercel Sandboxnone0/10Evidence only describes authentication mechanisms (automatic OIDC token in production, access tokens for CI/CD) but nowhere documents scoped or least-privilege permission models for these credentials — no mention of granular scopes, role restrictions, or per-agent credential limits.
- [claimed-docs] “In production on Vercel, authentication is automatic.”
- [claimed-docs] “Use access tokens when `VERCEL_OIDC_TOKEN` is unavailable, such as in external CI/CD systems or non-Vercel environments.”
- [claimed-docs] “For local development, run `vercel link` and `vercel env pull` to get a development token. In production on Vercel, authentication is automa…”
ai-native userBuild against official SDKs
weight 2 · round to Modal SandboxesModal ships official Python SDK plus JS/Go SDKs for Sandboxes with documented APIs (exec, filesystem, snapshots, networking, logs) and community developers confirm building against them (GitHub Actions runners, gRPC-based JS agent runtime). missing for 10: independent third-party SDK reference/benchmarks beyond Modal's own docs and one community mention, and no evidence of broader language SDK coverage (e.g. Rust, Java).
- [github] “The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [claimed-docs] “We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
Vercel Sandbox offers an official `@vercel/sandbox` SDK for JS/TS plus a CLI, with docs on authentication, commands, and sandbox lifecycle management — evidence that AI-native builders have a first-party SDK to integrate against. missing for 10: no mention of official SDKs for other languages (e.g., Python), no independent/hands-on corroboration of SDK usage beyond vendor docs.
- [claimed-docs] “Use `@vercel/sandbox` for JavaScript and TypeScript”
- [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
- [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
- [claimed-docs] “For local development, run `vercel link` and `vercel env pull` to get a development token. In production on Vercel, authentication is automa…”
- [claimed-docs] “Commands can run in blocking mode (wait for completion) or detached mode (return immediately).”
ai-native userSubscribe to events via webhooks
weight 2 · round drawnModal Sandboxesnone0/10No evidence anywhere in the pack of a webhook subscription mechanism or event-driven notification system for Modal Sandboxes; the docs focus on programmatic spin-up, execution, logging, and filesystem access, not outbound event/webhook subscriptions.
Agentic features
ai-native userSet up automations that run autonomously in the background
weight 2 · round to Modal SandboxesModal Sandboxes explicitly supports running autonomous agent code in isolated environments at scale (docs-3, docs-4, docs-22), and community evidence shows real-world use for background automation (self-hosted CI runners on modal-comm-1, agent runtimes on modal-comm-2). However, the evidence pack lacks explicit documentation of scheduling/cron triggers or persistent background job orchestration—sandboxes are framed as on-demand execution environments rather than a native 'set it and forget it' automation scheduler. Missing for 10: native scheduling/cron trigger docs, first-party persistent orchestration for long-running background automations.
- [claimed-docs] “Autonomous agents with the right tools, context, and credentials already in place — running securely in a full, isolated dev environment.”
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds.”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
Vercel Sandbox supports running agent-generated code, detached/background commands with log streaming, timeouts/extension, and persistence/resume, which enable building autonomous background automations. However, there's no built-in scheduler, trigger system, or native 'automation' orchestration layer — users must build their own scheduling/trigger logic around the SDK/CLI. missing for 10: native scheduling/cron or event-trigger mechanism, independent evidence of long-running unattended automations in production, and explicit 'autonomous background job' framing beyond detached commands.
- [claimed-docs] “Commands can run in blocking mode (wait for completion) or detached mode (return immediately).”
- [claimed-docs] “Stream logs from a detached command”
- [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
- [claimed-docs] “Persistent sandboxes: Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot m…”
- [claimed-docs] “Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot management needed.”
- [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”
ai-native userOperate the product with natural-language commands
weight 2 · round drawnModal Sandboxesnone0/10Modal Sandboxes is controlled exclusively via SDK (Python/JS/Go) calls and a traditional CLI (modal run, modal shell, etc.); there is no evidence of a natural-language command interface or chat-driven control layer for operating sandboxes. Evidence shows programmatic/API-first design (modal-docs-2, modal-docs-15, modal-docs-19) rather than any NL-command capability.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [probe] “official CLI documented at https://modal.com/docs/reference/cli”
Vercel Sandboxnone0/10Vercel Sandbox is designed to run agent-generated code and is explicitly marketed for 'agent workflows,' but the evidence only describes CLI commands (`sandbox create`, `sandbox run`) and SDK calls, not a natural-language interface for operating the product itself. No documentation shows a chat/NL command layer for creating or managing sandboxes. Missing for 10: any NL command parser, chat-driven CLI, or natural-language API for controlling Sandbox operations.
- [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
- [claimed-docs] “Create with an explicit name sandbox create --name my-sandbox”
- [claimed-docs] “sandbox run echo "Hello Sandbox!"”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnModal Sandboxesnone0/10Evidence shows standard docs pages, CLI reference, and SDK code snippets, but no interactive API reference or runnable-example playground; the openapi probe returned 404s and no docs mention an in-browser code runner or live API explorer.
- [probe] “PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, …”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
Vercel Sandboxnone0/10Evidence shows Vercel has an OpenAPI spec and Sandbox docs, but nothing describes an interactive API reference with runnable/live examples for Sandbox itself — no docs playground, no 'try it' console, no runnable code snippets in-page. missing for 10: interactive API reference UI, runnable example execution in docs, evidence of a docs playground feature.
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
- [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round to Vercel SandboxModal Sandboxesnone0/10The probe explicitly found all candidate OpenAPI/swagger endpoints return 404, and no other evidence pack item mentions a machine-readable API spec being published; Modal's public interface is SDK/CLI-based rather than a REST API with a spec.
- [probe] “PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, …”
A probe confirms Vercel publishes a machine-readable OpenAPI spec at https://vercel.com/openapi.json which validly contains the 'openapi' key, and Sandbox is part of Vercel's documented API/SDK surface. Missing for 10: no explicit confirmation that Sandbox-specific endpoints are included in that spec versus general Vercel API.
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
- [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to Vercel SandboxModal Sandboxes are explicitly designed as ephemeral, isolated containers for running untrusted or agent code, with filesystem, networking/security, and snapshot controls documented (modal-docs-2, modal-docs-8, modal-docs-9, modal-docs-10, modal-probe-2), and community examples show real usage isolating workloads from production infra (modal-comm-1, modal-comm-2). This satisfies the 'test without touching production' need since each sandbox is a fresh, isolated environment rather than shared prod state. Missing for 10: explicit documentation or case study framing isolation specifically in terms of 'production data' separation (e.g., no prod DB/staging parity guidance), and no independent audit confirming true isolation guarantees.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Filesystem Access”
- [claimed-docs] “Snapshots”
- [claimed-docs] “Networking and security”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Vercel Sandbox is explicitly designed to run untrusted/agent-generated code in isolated Linux microVMs 'without exposing your production systems,' with per-agent isolation, ephemeral or persistent state, and no shared access to production infrastructure. This directly matches the AI-native story of testing agent code safely away from production data. missing for 10: independent/hands-on evidence of real-world isolation guarantees, and explicit documentation on data-boundary controls preventing production data leakage into sandboxes.
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
- [claimed-docs] “Execute untrusted code safely: Run AI agent output, user uploads, or third-party scripts without exposing your production systems.”
- [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”
- [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self-contained environment with access to logs, file edit…”
- [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnModal Sandboxesnone0/10Evidence shows release notes and beta/alpha labels for some sandbox features, but no documentation of API versioning scheme or a deprecation policy; OpenAPI spec probes also 404. missing for 10: explicit versioning scheme documentation, deprecation policy page, changelog with deprecation notices, stability guarantees for GA vs beta APIs.
- [claimed-docs] “VM Sandboxes Beta”
- [claimed-docs] “Sidecars Alpha”
- [claimed-docs] “We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
- [claimed-docs] “it's now possible to opt into a more performant Sandbox backend, which affords substantially higher creation rates and concurrency”
- [probe] “PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, …”
Vercel Sandboxnone0/10The evidence pack shows SDK/CLI docs, features, and an OpenAPI spec, but no mention of API versioning scheme or a documented deprecation policy for Vercel Sandbox's APIs. Some features are explicitly marked 'beta' (docs-14, docs-44), suggesting instability without a stated deprecation process. missing for 10: explicit API version numbers, a deprecation/sunset policy, changelog entries documenting breaking-change handling.
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to Modal SandboxesModal supports spinning up hundreds of thousands of concurrent sandboxes/rollout environments programmatically, which is strong evidence of bulk/parallel scaling (modal-docs-4, modal-docs-14, modal-docs-5), and this is corroborated by community builds using it for scaled runner fleets (modal-comm-1). However, this is infrastructure-level bulk provisioning, not a documented 'bulk operations' API/pattern for operating across many existing items (e.g., batch file ops, batch snapshot management, or a dedicated batch API) — missing for 10: a first-party batch/bulk-operation API or workflow example (vs just scaling raw sandbox count), independent benchmarks confirming claimed concurrency at scale, and documentation of bulk management operations (e.g., bulk terminate/list/update) rather than just creation.
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds.”
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds. Fast enough to keep your GPU inference resources saturated acro…”
- [claimed-docs] “H100s, A100s, A10Gs available on demand. Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved ca…”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [claimed-docs] “Integrated logging and full visibility into every function, sandbox, and container.”
Vercel Sandboxnone0/10Vercel Sandbox docs describe creating/running individual sandboxes, multi-agent isolation, and SDK-driven command execution, but there is no evidence of a bulk/batch operation feature for processing many items at once (e.g., batch job APIs, fan-out orchestration, or bulk file/task processing).
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round drawnModal Sandboxesnone0/10The evidence describes Modal Sandboxes as a programmatic, on-demand execution environment for untrusted/agent code, but nothing in the pack shows a rules/trigger engine that fires actions automatically on external events (e.g., webhooks, event-driven policies). Absent such evidence, this applicable automation-depth axis is unmet.
ai-native userSchedule recurring jobs or workflows
weight 2 · round drawnModal Sandboxesnone0/10Evidence covers ephemeral sandbox creation, execution, networking, snapshots, and SDKs, but there is no mention of cron-like scheduling, recurring triggers, or persistent workflow orchestration for Modal Sandboxes. missing for 10: any scheduling/cron API, recurring job trigger mechanism, or workflow orchestration documentation tied to Sandboxes.
- [claimed-docs] “VM Sandboxes Beta”
- [claimed-docs] “Running commands”
- [claimed-docs] “Snapshots”
Vercel Sandboxnone0/10Vercel Sandbox provides on-demand ephemeral/persistent execution environments with manual creation, timeouts, and resumption, but no evidence of a scheduler, cron-like trigger, or recurring job orchestration feature; scheduling would need to be built externally (e.g., Vercel Cron) which isn't documented here as part of Sandbox itself.
- [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
- [claimed-docs] “The default timeout is 5 minutes. You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.ex…”
- [claimed-docs] “Or retrieve an existing sandbox by name (resumes if stopped)”
ai-native userVersion, review, and roll back my automations
weight 1 · round drawnModal Sandboxesnone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
Capabilities hardware — stories about capabilities hardware in this arenaCapabilities hardware
Stories about capabilities hardware in this arena
Agent workloads
ai-native userRun coding agents like Claude Code or Codex inside the sandbox following the vendor's own recipe
weight 2 · round to Vercel SandboxModal Sandboxesnone0/10Modal's docs show general support for running 'agents' and 'untrusted code' in Sandboxes (modal-docs-3, modal-docs-22, modal-docs-19), but no evidence pack item mentions a vendor recipe, tutorial, or example specifically for running Claude Code or Codex inside a Sandbox.
Docs confirm the default 'vercel/sandbox/universal' image ships with coding agents preinstalled and highlight agent workflows (e.g. Claude Code/Codex-style use), but the evidence pack lacks an explicit vendor 'recipe' walkthrough naming Claude Code or Codex step-by-step. missing for 10: a dedicated vendor tutorial/recipe explicitly running Claude Code or Codex inside the sandbox, and independent hands-on confirmation of this exact workflow.
- [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need. The default image is `vercel/sandbox/universa…”
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs.”
- [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”
- [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
Workloads
developerRun a headless browser or full desktop environment inside the sandbox
weight 2 · round drawnModal Sandboxesnone0/10Evidence describes Modal Sandboxes as isolated containers/VMs for running arbitrary code, agents, filesystem access, networking, and command execution, but nowhere mentions headless browser or full desktop/GUI environment support (e.g., Xvfb, VNC, browser automation tooling). No docs or community reports reference browser or desktop environments running inside sandboxes.
Vercel Sandboxnone0/10Evidence describes Linux microVMs, custom OCI images, and general code execution, but nothing mentions headless browser support, display/VNC, GUI desktop environments, or browser automation tooling. missing for 10: any mention of headless browser (e.g., Chromium/Playwright) support, GPU/display capabilities, or desktop environment provisioning.
developerAttach GPUs to sandboxed workloads
weight 2 · round to Modal SandboxesModal docs explicitly state GPUs (H100s, A100s, A10Gs) can be attached to any sandbox on demand with per-second billing, and community evidence (HN runner project) confirms real-world GPU-backed sandbox usage. missing for 10: independent hands-on benchmark of GPU attach performance/reliability, and explicit code example showing GPU parameter in sandbox creation API.
- [claimed-docs] “H100s, A100s, A10Gs available on demand. Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved ca…”
- [claimed-docs] “Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved capacity.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
Vercel Sandboxnone0/10No evidence anywhere in the docs mentions GPU support or GPU attachment for Vercel Sandbox; sandboxes are described only as CPU-based Linux microVMs with system-privileged process support, storage/drives, and networking. missing for 10: any mention of GPU hardware, GPU-enabled images, or GPU pricing/quota.
- [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need.”
- [claimed-docs] “System-privileged processes: Run workloads that need system-level privileges, such as container runtimes like Docker, VPN clients, and FUSE …”
- [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”
Code execution — stories about code execution in this arenaCode execution
Stories about code execution in this arena
Files
developerRead, write, upload, and download files in the sandbox filesystem via the SDK
weight 2 · round to Modal SandboxesModal explicitly documents a dedicated 'Filesystem Access' guide for sandboxes (modal-docs-8) alongside SDK examples showing file read/write via mounted volumes and exec commands (modal-docs-20, modal-docs-21), confirming read/write/upload/download-style operations through the SDK. missing for 10: no independent/hands-on community confirmation specifically of file upload/download methods (only exec/volume examples), and no explicit code snippet showing a dedicated upload/download API method distinct from volume mounting.
- [claimed-docs] “Filesystem Access”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Docs confirm the SDK provides a programmatic interface to manage files (mount remote storage, drives, filesystem persistence via snapshots) and general file access is implied through 'file edits' and FUSE-mounted storage, but no explicit documentation of discrete SDK methods like sandbox.writeFiles(), readFile(), or upload/download APIs is present. missing for 10: explicit SDK method documentation/examples for read, write, upload, and download file operations, and independent/hands-on confirmation of these specific API calls.
- [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
- [claimed-docs] “Mount an external object store such as Amazon S3 with a FUSE driver, and read and write remote files through the sandbox filesystem.”
- [claimed-docs] “Drives (beta): Attach persistent filesystem storage to sandboxes and reuse data across sandbox runs.”
- [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self-contained environment with access to logs, file edit…”
- [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”
Runtimes
developerDefine custom sandbox templates or bring my own container image
weight 2 · round to Vercel SandboxModal explicitly supports creating Sandboxes with custom Images ('Custom images, any dependency' and 'Create a Sandbox with the specified Image, and mount a Volume'), and community projects (GitHub Actions runners, MCP-JS) confirm developers configuring custom sandbox environments in practice. Missing for 10: explicit documentation of importing arbitrary Docker/OCI registry images or a dedicated 'bring your own container' guide beyond the Image API reference.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
Docs explicitly support custom sandbox templates and BYO container images: users can build custom OCI images stored in Vercel Container Registry, start sandboxes from managed, shared/public, or custom images, and Docker-image-based workflows are directly supported. Also supports snapshotting for reusable custom environments. missing for 10: independent/hands-on corroboration of custom image builds beyond first-party docs.
- [claimed-docs] “Build your system packages and tooling into an image, and share it across projects and teams.”
- [claimed-docs] “If you already use Docker images to define your environment, store the image in Vercel Container Registry (VCR) and create the sandbox with …”
- [claimed-docs] “Start sandboxes from Vercel's Managed Images, a shared or public image, or your own OCI images stored in Vercel Container Registry.”
- [claimed-docs] “Managed and custom images: Start sandboxes from Vercel's Managed Images, a shared or public image, or your own OCI images stored in Vercel C…”
- [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need. The default image is `vercel/sandbox/universa…”
developerExecute code in multiple language runtimes (Python, JavaScript, and more) and get rich results back
weight 2 · round to Vercel SandboxModal Sandboxes let you run arbitrary commands/code in isolated containers with custom images (any dependency, any language stack), including SDK examples for JS/Go exec and Python examples of running agent code (modal-docs-2, modal-docs-13, modal-docs-19-21). However, evidence never shows explicit multi-language runtime presets (Python vs JS vs others) or structured/rich result payloads (e.g., images, dataframes, mime-typed outputs) — outputs shown are plain stdout/exec results. missing for 10: explicit built-in multi-language runtime templates, documented rich/structured result formats beyond raw stdout.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Running commands”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
Docs confirm sandboxes run Linux images with Python 3.14 and Node.js LTS pre-installed by default, plus support for any other Linux distro via custom OCI images, enabling multi-language execution (JS/TS, Python, and more) (docs-5, docs-8, docs-34, docs-30). Execution results are surfaced via logs, streamed output, and file access (docs-19, docs-20, docs-26), but there's no explicit mention of structured/rich result formats (e.g., images, dataframes, plots) beyond logs and files. Missing for 10: explicit support for rich/structured output types (charts, tables, media) and independent hands-on confirmation of multi-language fidelity.
- [claimed-docs] “Use `@vercel/sandbox` for JavaScript and TypeScript”
- [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need.”
- [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need. The default image is `vercel/sandbox/universa…”
- [claimed-docs] “Start sandboxes from Vercel's Managed Images, a shared or public image, or your own OCI images stored in Vercel Container Registry.”
- [claimed-docs] “Commands can run in blocking mode (wait for completion) or detached mode (return immediately).”
- [claimed-docs] “Stream logs from a detached command”
- [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self-contained environment with access to logs, file edit…”
Untrusted code
developerExecute untrusted, AI-generated code without risking my own infrastructure
weight 3 · round to Vercel SandboxModal Sandboxes is explicitly built for running untrusted/AI-generated code in isolated, ephemeral containers with networking/filesystem controls, resource limits, and logging, and community examples show real-world use (self-hosted CI runners, agent runtimes) confirming the isolation model works in practice. Missing for 10: independent security audit or adversarial escape-testing evidence, and clearer documentation of resource/network isolation guarantees beyond marketing copy.
- [claimed-docs] “Programmatically scale secure, ephemeral environments for running untrusted code.”
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Networking and security”
- [claimed-docs] “VM Sandboxes Beta”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
Vercel Sandbox's docs directly describe running untrusted or AI agent-generated code in isolated Linux microVMs without exposing production systems, with per-agent isolation, configurable timeouts, and managed images—precisely matching the story. Missing for 10: independent/hands-on verification beyond first-party docs.
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
- [claimed-docs] “Execute untrusted code safely: Run AI agent output, user uploads, or third-party scripts without exposing your production systems.”
- [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs.”
- [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self-contained environment with access to logs, file edit…”
developerRun arbitrary shell commands and install packages inside the sandbox
weight 2 · round to Modal SandboxesModal Sandboxes documentation explicitly covers running arbitrary shell commands (sb.exec, CLI 'run a command or interactive shell inside a Modal container'), custom images with any dependency installation, filesystem access, and community reports (HN self-hosted runners, JS agent runtime) corroborate real-world use for arbitrary command execution and untrusted code. Missing for 10: no independent benchmark of package installation limits or explicit documented apt/pip install examples beyond generic 'any dependency' claim.
- [claimed-docs] “Running commands”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Docs clearly show running arbitrary commands (blocking/detached modes, `sandbox run echo ...`, CLI and SDK) inside Linux microVMs, with custom images and full Linux distros allowing package installs (e.g., Ubuntu/Arch, custom OCI images, system-privileged processes). missing for 10: no independent/hands-on report confirming package installation workflows beyond first-party docs.
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
- [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need.”
- [claimed-docs] “System-privileged processes: Run workloads that need system-level privileges, such as container runtimes like Docker, VPN clients, and FUSE …”
- [claimed-docs] “Commands can run in blocking mode (wait for completion) or detached mode (return immediately).”
- [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
- [claimed-docs] “Sandboxes run Linux images, with Ubuntu, Arch Linux, or any other Linux distribution you need. The default image is `vercel/sandbox/universa…”
- [claimed-docs] “sandbox run echo "Hello Sandbox!"”
- [claimed-docs] “Start sandboxes from Vercel's Managed Images, a shared or public image, or your own OCI images stored in Vercel Container Registry.”
Dev experience — day-to-day developer experience — setup friction, docs, debugging, iteration speedDev experience
Day-to-day developer experience — setup friction, docs, debugging, iteration speed
Agent lifecycle
ai-native userMy agent can provision its own sandbox, execute code, read the results, and tear it down — end to end without a human
weight 3 · round drawnModal Sandboxes are explicitly designed for programmatic, API-driven provisioning, code execution, log/result retrieval, and teardown, with SDK examples (Python/JS/Go) showing sandbox creation, exec, file access, and logs, plus a documented agent-code-execution example. missing for 10: independent hands-on report showing a fully autonomous agent loop (provision->execute->read->teardown) with no human intervention, and clearer documentation of teardown/termination API specifics.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Autonomous agents with the right tools, context, and credentials already in place — running securely in a full, isolated dev environment.”
- [claimed-docs] “Running commands”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
- [claimed-docs] “We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
Vercel Sandbox provides a full programmatic SDK/CLI to create sandboxes (with names, custom images, regions), run blocking or detached commands, stream logs/output, and stop/tear down sandboxes, with API-based authentication (OIDC/tokens) suitable for unattended agent use — no human-in-the-loop required. missing for 10: no independent/hands-on third-party report confirming an agent autonomously completing the full provision-execute-teardown loop, and no explicit example showing an agent reading command output/results programmatically end-to-end.
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
- [claimed-docs] “Execute untrusted code safely: Run AI agent output, user uploads, or third-party scripts without exposing your production systems.”
- [claimed-docs] “Commands can run in blocking mode (wait for completion) or detached mode (return immediately).”
- [claimed-docs] “Stream logs from a detached command”
- [claimed-docs] “You can also stop sandboxes from the Vercel Dashboard by navigating to Observability > Sandboxes and clicking Stop Sandbox.”
- [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
- [claimed-docs] “For local development, run `vercel link` and `vercel env pull` to get a development token. In production on Vercel, authentication is automa…”
- [claimed-docs] “Create with an explicit name sandbox create --name my-sandbox”
- [claimed-docs] “sandbox run echo "Hello Sandbox!"”
Isolation security — stories about isolation security in this arenaIsolation security
Stories about isolation security in this arena
Isolation
platform-engineerRely on a documented hard isolation boundary (microVM or equivalent) between sandboxes and my systems
weight 3 · round to Vercel SandboxModal repeatedly describes Sandboxes as 'secure,' 'isolated' containers for untrusted code, and references a 'VM Sandboxes Beta' page implying microVM-based isolation, but no evidence pack item actually documents the underlying isolation technology (e.g., Firecracker/gVisor specifics, threat model, or security boundary guarantees) that a platform engineer would need to rely on. Missing for 10: explicit technical documentation of the isolation mechanism (hypervisor/kernel boundary), any third-party security audit or independent verification of the boundary's robustness.
- [claimed-docs] “VM Sandboxes Beta”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
Vercel's first-party docs repeatedly and explicitly claim microVM-based isolation for running untrusted/agent-generated code without exposing production systems, plus additional isolation features (per-agent Linux users, system-privileged process support). However, all evidence is vendor documentation with no independent security audit, third-party penetration test, or technical deep-dive into the underlying microVM technology (e.g. Firecracker) confirming the boundary in practice. missing for 10: independent/third-party verification of the isolation boundary, technical detail on the hypervisor/microVM implementation, and any hands-on security assessment.
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
- [claimed-docs] “Execute untrusted code safely: Run AI agent output, user uploads, or third-party scripts without exposing your production systems.”
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs.”
- [claimed-docs] “System-privileged processes: Run workloads that need system-level privileges, such as container runtimes like Docker, VPN clients, and FUSE …”
- [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”
ai-native userGive an agent a sandbox where host secrets and credentials are unreachable by the code it runs
weight 2 · round to Vercel SandboxModal Sandboxes are documented as isolated, ephemeral containers for running untrusted/agent code with dedicated networking-and-security guides, supporting the general isolation story (modal-docs-1, modal-docs-2, modal-docs-10, modal-probe-2, modal-docs-19, modal-docs-22). However, no evidence explicitly states that host secrets/credentials are unreachable by default or describes the secrets-injection model that would confirm this — one doc even implies credentials are provisioned into the agent environment (modal-docs-3), which cuts against a strong 'unreachable unless granted' guarantee. Missing for 10: explicit secrets-isolation documentation, details of the opt-in secrets mechanism, and independent/hands-on confirmation that host credentials cannot leak into sandboxed code.
- [claimed-docs] “Programmatically scale secure, ephemeral environments for running untrusted code.”
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Autonomous agents with the right tools, context, and credentials already in place — running securely in a full, isolated dev environment.”
- [claimed-docs] “Networking and security”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Vercel Sandbox docs explicitly describe running agent-generated/untrusted code in isolated Linux microVMs 'without exposing your production systems,' and support per-agent Linux user isolation, which directly implies host secrets/credentials are not reachable by sandboxed code. missing for 10: explicit documentation of how host secrets/env vars are excluded or scoped from the sandbox environment, and independent/hands-on verification of this isolation guarantee.
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
- [claimed-docs] “Execute untrusted code safely: Run AI agent output, user uploads, or third-party scripts without exposing your production systems.”
- [claimed-docs] “Multi-agent isolation: Give each AI agent its own Linux user with a private home directory, and share files between agents with groups.”
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs.”
Network policy
platform-engineerRestrict or allow the sandbox's network egress with explicit policy
weight 3 · round to Modal SandboxesModal has a dedicated 'Networking and security' guide for Sandboxes, confirming the platform documents network controls for sandboxes, but the evidence pack only shows the page title without excerpts detailing how egress allow/deny policies are configured. missing for 10: actual policy syntax/examples (e.g. allowlist/blocklist configuration), confirmation of default-deny behavior, independent/hands-on verification that egress restrictions work as documented.
- [claimed-docs] “Networking and security”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to Vercel SandboxModal Sandboxes is fundamentally API/SDK-driven (Python, JS/Go) with a full CLI for running commands and managing containers, and docs describe programmatic control over image building, execution, filesystem, networking, snapshots, and logs — suggesting the API is the primary and most complete interface. However, there's no explicit documentation stating full UI/API parity or listing any UI-exclusive features, and no OpenAPI spec is discoverable (probe found 404s), making it hard to confirm completeness of a public REST API surface beyond the SDKs/CLI. Missing for 10: explicit parity statement between dashboard UI and API/CLI capabilities, and a public API reference/spec confirming no UI-only actions.
- [claimed-docs] “Filesystem Access”
- [claimed-docs] “Snapshots”
- [claimed-docs] “Networking and security”
- [claimed-docs] “Running commands”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “Manage and connect to running containers.”
- [github] “The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [probe] “PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, …”
- [probe] “official CLI documented at https://modal.com/docs/reference/cli”
Vercel Sandbox is fundamentally API/SDK-driven (creation, commands, snapshots, tags, drives, custom images all documented via SDK/CLI), and the dashboard only offers a subset (viewing/stopping sandboxes) that mirrors API capability rather than adding unique UI-only features, and an OpenAPI spec exists confirming programmatic access. Missing for 10: an explicit vendor statement confirming full UI/API parity, and independent confirmation that every dashboard action (e.g., tag/drive management) is scriptable via API with no UI-exclusive features.
- [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
- [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
- [claimed-docs] “You can also stop sandboxes from the Vercel Dashboard by navigating to Observability > Sandboxes and clicking Stop Sandbox.”
- [claimed-docs] “Connect to and manage Sandboxes from the dashboard”
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnModal Sandboxesnone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
Vercel Sandboxnone0/10Vercel Sandbox stores state via drives, snapshots, and container images, so data portability is a fair question, but no evidence describes an explicit data-export feature or guarantee of open, exportable formats for that stored state. Missing for 10: any documented export/download-all-data capability, explicit open-format guarantees for drives/snapshots, or migration tooling to leave the platform.
- [claimed-docs] “Drives (beta): Attach persistent filesystem storage to sandboxes and reuse data across sandbox runs.”
- [claimed-docs] “Snapshotting: Save the state of a running sandbox to resume later. Skip dependency installation on subsequent runs.”
- [claimed-docs] “If you already use Docker images to define your environment, store the image in Vercel Container Registry (VCR) and create the sandbox with …”
ai-native userRead the product's source under an open license
weight 2 · round drawnModal Sandboxesnone0/10Evidence only shows a GitHub repo for the client SDK (modal-client) used to invoke Modal Sandboxes, not the underlying sandbox/runtime platform itself, and no license terms are mentioned. There is no evidence that Modal Sandboxes' core product source is published under an open license.
- [github] “The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
ai-native userSelf-host the core product
weight 3 · round drawnModal Sandboxesnone0/10Modal is a fully-managed cloud platform for running Sandboxes; there is no evidence of any self-hostable/on-prem version of the core product, and everything described (pay-per-second billing, GPU access, hosted CLI) implies a proprietary SaaS deployment model only.
Vercel Sandboxnone0/10Vercel Sandbox is a fully managed, hosted service running on Vercel's own microVM infrastructure (iad1/sfo1/cle1/cdg1 regions); there is no evidence of any self-hostable core, open-source runtime, or on-prem deployment option — it is inherently tied to Vercel's cloud.
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
- [claimed-docs] “Vercel Sandbox is available in the `iad1` (default), `sfo1`, `cle1`, and `cdg1` regions. You can choose the region per sandbox or set a proj…”
- [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”
Performance scale — stories about performance scale in this arenaPerformance scale
Stories about performance scale in this arena
Latency
developerStart sandboxes with documented sub-second-to-few-second cold starts
weight 3 · round to Modal SandboxesEvidence shows general speed/scale claims ("spin up hundreds of thousands of concurrent rollout environments in seconds," faster backend for higher creation rates) but no explicit documented cold-start latency benchmark (e.g., sub-second or few-second startup time) for sandboxes specifically. Missing for 10: a concrete cold-start time benchmark/number, independent performance verification, and comparison across sandbox sizes/images.
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds.”
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds. Fast enough to keep your GPU inference resources saturated acro…”
- [claimed-docs] “it's now possible to opt into a more performant Sandbox backend, which affords substantially higher creation rates and concurrency”
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
Vercel Sandboxnone0/10The docs describe microVM isolation, snapshotting for faster restarts, and persistence, but no evidence gives concrete cold-start latency figures (e.g., sub-second or few-second boot times) for starting a fresh sandbox. Missing for 10: documented benchmark numbers or explicit cold-start time claims, any independent performance measurement.
- [claimed-docs] “Using a snapshot is much faster than creating from scratch because it avoids reinstalling dependencies and repeating setup steps.”
- [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”
Scale
platform-engineerRun large concurrent fleets of sandboxes with documented concurrency limits
weight 3 · round to Modal SandboxesModal markets very high concurrency ('hundreds of thousands of concurrent rollout environments', 'scale to thousands of concurrent runs') and even documents a backend upgrade for higher creation rates/concurrency, plus community reports of building scalable runner fleets on Sandboxes, showing the capability is real and used at scale. However there is no evidence of documented hard limits, quotas, or per-account/per-plan concurrency ceilings that a platform engineer could plan capacity against. Missing for 10: explicit documented concurrency/rate limits or quotas, benchmark/load-test data, and any per-tier scaling caps.
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds.”
- [claimed-docs] “H100s, A100s, A10Gs available on demand. Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved ca…”
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds. Fast enough to keep your GPU inference resources saturated acro…”
- [claimed-docs] “Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved capacity.”
- [claimed-docs] “it's now possible to opt into a more performant Sandbox backend, which affords substantially higher creation rates and concurrency”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
Docs mention that Sandbox usage is metered and that quotas apply per plan, plus region availability and spend management for cost control, but no concrete documented concurrency limits (e.g., max simultaneous sandboxes) are shown in the evidence. missing for 10: explicit concurrency limit numbers, guidance on running large fleets of sandboxes concurrently, and any performance/scale benchmarks.
- [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”
- [claimed-docs] “Vercel Sandbox is available in the `iad1` (default), `sfo1`, `cle1`, and `cdg1` regions. You can choose the region per sandbox or set a proj…”
- [claimed-docs] “To control costs, configure Spend Management to receive alerts or pause projects when you reach a specified amount.”
developerKeep a sandbox session running for hours or days for long agent tasks
weight 2 · round to Vercel SandboxModal docs describe sandboxes as isolated, scalable execution environments with snapshotting, filesystem, and networking support, and a community project (self-hosted GitHub Actions runners) demonstrates sandboxes being kept alive for extended, real workloads. However, no evidence pack item states explicit maximum session duration, timeout configuration, or guarantees for multi-day persistence, so long-running-session support is inferred rather than directly documented. Missing for 10: explicit docs on configurable/no timeout limits, guidance for multi-day sandbox lifecycles, and independent confirmation of hours/days-long sessions beyond one community anecdote.
- [claimed-docs] “Snapshots”
- [claimed-docs] “VM Sandboxes Beta”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [claimed-docs] “Integrated logging and full visibility into every function, sandbox, and container.”
- [claimed-docs] “We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
Docs show sandboxes have a default 5-minute timeout that can be explicitly extended via `timeout` option or `sandbox.extendTimeout()`, and persistent sandboxes auto-snapshot on stop so you can quickly resume a long agent task across sessions rather than losing state. However, there's no evidence of an unbounded or multi-day continuous session—developers must actively manage/extend timeouts or rely on stop/resume snapshotting rather than a single always-on session. missing for 10: documented maximum extendable duration or explicit support for a single continuous multi-hour/day session without developer-managed timeout extensions, independent corroboration of long-running stability.
- [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
- [claimed-docs] “The default timeout is 5 minutes. You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.ex…”
- [claimed-docs] “Persistent sandboxes: Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot m…”
- [claimed-docs] “Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot management needed.”
- [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”
- [claimed-docs] “Using a snapshot is much faster than creating from scratch because it avoids reinstalling dependencies and repeating setup steps.”
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
Pricing
developerStart building with a free tier or included credits without talking to sales
weight 1 · round drawnModal Sandboxesnone0/10The evidence pack includes pricing-related docs (pay-per-use, burst capacity) but no mention of a free tier, free credits, or self-serve signup without contacting sales; nothing confirms developers can start building without sales contact.
Vercel Sandboxnone0/10The evidence pack mentions billing, plans, and quotas (e.g., vercel-sandbox-docs-42, docs-24) but never explicitly states a free tier or included credits usable without contacting sales. Missing for 10: explicit free-tier/plan pricing details, evidence of self-serve signup without sales contact.
- [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”
- [claimed-docs] “To control costs, configure Spend Management to receive alerts or pause projects when you reach a specified amount.”
platform-engineerPay per second only for the compute a sandbox actually uses
weight 3 · round to Modal SandboxesModal's pricing docs and marketing explicitly state per-second billing with no reserved capacity ('pay by the second with no reserved capacity', 'only pay for what you use... burst up to what you need without over-allocating'), directly matching the story, and a community user corroborates paying only for resources actually consumed when building GPU runners on Sandboxes. Missing for 10: independent verification of exact billing granularity/metering mechanics and no first-party pricing breakdown showing per-second line items in the evidence pack.
- [claimed-docs] “Only pay for what you use. Burst up to what you need without over-allocating CPU or memory in advance.”
- [claimed-docs] “Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved capacity.”
- [claimed-docs] “Burst up to what you need without over-allocating CPU or memory in advance.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
Docs confirm Sandbox usage is metered on a pricing page with configurable timeouts and spend-management alerts to control costs, implying granular, usage-based billing, but no evidence explicitly states billing is charged 'per second' of compute used. missing for 10: explicit per-second billing granularity documentation, real-world billing statement/invoice examples showing sub-minute metering.
- [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”
- [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
- [claimed-docs] “The default timeout is 5 minutes. You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.ex…”
- [claimed-docs] “To control costs, configure Spend Management to receive alerts or pause projects when you reach a specified amount.”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round to Vercel SandboxModal Sandboxesnone0/10No evidence pack item mentions data residency, region selection, or geographic data storage controls for Modal Sandboxes; the pack covers execution, scaling, GPUs, filesystem, and networking but nothing about choosing storage region/residency.
Docs confirm region selection (iad1, sfo1, cle1, cdg1) can be set per sandbox or as a project default, giving users some control over where sandboxes execute, and one changelog note claims availability 'in all regions.' However, there is no explicit data residency/compliance guarantee, no mention of where persistent Drives/snapshots/data-at-rest are stored, and no discussion of data sovereignty controls beyond compute region choice. missing for 10: explicit data-at-rest residency guarantees, compliance/certification details tied to region choice, documentation on storage location for Drives/snapshots, and independent confirmation of region enforcement.
- [claimed-docs] “Vercel Sandbox is available in the `iad1` (default), `sfo1`, `cle1`, and `cdg1` regions. You can choose the region per sandbox or set a proj…”
- [claimed-docs] “Vercel Sandbox is now available in all regions”
ai-native userControl data retention and deletion
weight 2 · round drawnModal Sandboxesnone0/10Evidence covers sandbox ephemerality, filesystem access, snapshots, and networking/security, but nothing describes data retention policies, deletion controls, or compliance/data-handling settings that a user could configure. missing for 10: explicit data retention policy documentation, user-controllable deletion/purge mechanisms, compliance certifications (e.g., SOC2/GDPR data handling), retention period configuration.
Vercel Sandboxnone0/10Docs describe sandbox lifecycle (timeouts, stopping, persistence, snapshots) but there is no explicit statement about data retention policies, deletion guarantees, or privacy controls for stored sandbox data/drives. missing for 10: explicit data retention policy, explicit deletion/purge controls, privacy/compliance documentation for stored data.
- [claimed-docs] “You can also stop sandboxes from the Vercel Dashboard by navigating to Observability > Sandboxes and clicking Stop Sandbox.”
- [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
- [claimed-docs] “Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot management needed.”
- [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnModal Sandboxesnone0/10No evidence in the pack mentions telemetry, usage tracking, analytics collection, or an opt-out mechanism for Modal Sandboxes; the docs cover sandboxing, GPUs, filesystem, and logging but not data-collection/privacy controls.
Provisioning lifecycle — creating, updating, and tearing down resources across their lifecycleProvisioning lifecycle
Creating, updating, and tearing down resources across their lifecycle
Lifecycle
developerSet timeouts so sandboxes shut down automatically and stop billing when idle or done
weight 2 · round to Vercel SandboxModal Sandboxesnone0/10The evidence pack only mentions general pay-per-second billing and bursty scaling, but never documents an explicit sandbox timeout or idle-shutdown parameter/config. No citation confirms configurable timeouts that auto-terminate sandboxes when idle or complete.
Docs explicitly state sandboxes have a default 5-minute timeout, configurable via the `timeout` option at creation, extendable via `sandbox.extendTimeout()`, and can be manually stopped from the dashboard, directly supporting automatic shutdown to stop billing when idle or done. Missing for 10: independent/hands-on confirmation that billing actually stops immediately upon timeout, and documentation of any maximum timeout limits or edge cases.
- [claimed-docs] “You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.extendTimeout()`.”
- [claimed-docs] “The default timeout is 5 minutes. You can configure this using the `timeout` option when creating a sandbox, and extend it using `sandbox.ex…”
- [claimed-docs] “You can also stop sandboxes from the Vercel Dashboard by navigating to Observability > Sandboxes and clicking Stop Sandbox.”
- [claimed-docs] “Vercel Sandbox usage is metered across several dimensions. This page explains how billing works for each plan, what quotas apply, and how to…”
developerSpin up an isolated sandbox with one API/SDK call and get a live environment in seconds
weight 3 · round drawnDocs and SDK references (Python, JS/Go) show a single call (e.g. modal.Sandbox.create) spins up an isolated, ephemeral container/VM in seconds, with fast creation rates highlighted (hundreds of thousands of concurrent environments in seconds, faster backend for higher creation rates). Community builders (HN posts) independently confirm using Modal Sandboxes programmatically for real workloads like CI runners and agent runtimes. Missing for 10: independent benchmark of actual cold-start latency and a public OpenAPI spec (probe found 404s), so exact 'seconds' claim isn't third-party verified.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds.”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “it's now possible to opt into a more performant Sandbox backend, which affords substantially higher creation rates and concurrency”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Docs show a clear SDK/CLI path (`@vercel/sandbox`, `sandbox create`/`sandbox run`) to spin up isolated Linux microVMs, with fast startup via managed images or snapshots that skip dependency installation, matching the 'one call, live environment in seconds' story. Missing for 10: independent/hands-on timing benchmarks confirming 'seconds' latency and third-party corroboration beyond first-party docs.
- [claimed-docs] “Run untrusted or agent-generated code in isolated Linux microVMs. Use Vercel Sandbox for agent workflows, debugging, and one-off commands.”
- [claimed-docs] “Use `@vercel/sandbox` for JavaScript and TypeScript”
- [claimed-docs] “Use the `sandbox` CLI for manual testing, agentic workflows, debugging, and one-off operations”
- [claimed-docs] “The Sandbox SDKs are the recommended way to integrate Vercel Sandbox into your applications. They provide a programmatic interface to create…”
- [claimed-docs] “sandbox run echo "Hello Sandbox!"”
- [claimed-docs] “Using a snapshot is much faster than creating from scratch because it avoids reinstalling dependencies and repeating setup steps.”
- [claimed-docs] “Start sandboxes from Vercel's Managed Images, a shared or public image, or your own OCI images stored in Vercel Container Registry.”
Preview access
developerExpose a port from the sandbox on a public preview URL to reach services running inside
weight 2 · round to Vercel SandboxThe docs pack includes a 'Networking and security' guide for sandboxes (modal-docs-10) which is the likely location for tunnel/port-exposure features, but no evidence text actually describes exposing a port to a public preview URL, tunnel URLs, or reachability details. missing for 10: explicit documentation or example showing sandbox.tunnels()/port forwarding, a public URL format, and independent confirmation of accessing services inside a sandbox via that URL.
- [claimed-docs] “Networking and security”
- [claimed-docs] “VM Sandboxes Beta”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Docs mention running dev servers with 'live previews' and testing agent-generated code with access to 'live previews', implying some port-exposure/public-URL capability, but no evidence pack item describes the actual API/CLI mechanism (e.g., a domain/getUrl method, port argument, or URL format) for exposing a sandboxed port publicly. missing for 10: explicit documentation of the port-exposure API or CLI command, example of the resulting public URL, and confirmation of access control/authentication on that URL.
- [claimed-docs] “Run development servers: Spin up and test applications with live previews.”
- [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self-contained environment with access to logs, file edit…”
- [claimed-docs] “Test in isolation: Preview how user-submitted or agent-generated code behaves in a self”
Snapshot persistence — stories about snapshot persistence in this arenaSnapshot persistence
Stories about snapshot persistence in this arena
Snapshots
developerPause a running sandbox and resume it later with filesystem and memory state intact
weight 2 · round to Vercel SandboxModal has a dedicated 'Snapshots' doc for sandboxes suggesting some pause/resume-with-state capability, but the evidence pack only gives the page title with no detail on what state (filesystem/memory) is preserved or how resume works. Missing for 10: documented mechanics of snapshot/resume (filesystem+memory fidelity), explicit confirmation memory state is preserved, independent/hands-on validation of the feature working as described.
- [claimed-docs] “Snapshots”
- [claimed-docs] “Filesystem Access”
Docs explicitly describe persistent sandboxes that auto-save filesystem state on stop and resume where left off by default, with configuration preserved across sessions, plus explicit snapshotting to skip dependency reinstallation and resuming an existing sandbox by name. This directly matches the pause/resume-with-state story via first-party documentation. Missing for 10: independent hands-on verification that memory/process state (not just filesystem) is preserved across pause/resume, and no third-party corroboration beyond vendor docs.
- [claimed-docs] “Persistent sandboxes: Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot m…”
- [claimed-docs] “Snapshotting: Save the state of a running sandbox to resume later. Skip dependency installation on subsequent runs.”
- [claimed-docs] “Or retrieve an existing sandbox by name (resumes if stopped)”
- [claimed-docs] “Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot management needed.”
- [claimed-docs] “Using a snapshot is much faster than creating from scratch because it avoids reinstalling dependencies and repeating setup steps.”
- [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”
developerSnapshot a sandbox and later restore or fork new sandboxes from that snapshot
weight 3 · round to Vercel SandboxModal explicitly documents a 'Sandbox Snapshots' feature (modal-docs-9), confirming snapshot capability exists, but the evidence pack gives no detail on the actual snapshot/restore/fork API mechanics, no confirmation that new independent sandboxes can be forked from a single snapshot, and no community/hands-on corroboration of this specific workflow. missing for 10: documented API details on restoring/forking multiple sandboxes from one snapshot, independent/hands-on confirmation of snapshot-restore-fork workflow, and clarity on any beta/experimental limitations.
- [claimed-docs] “Snapshots”
- [claimed-docs] “VM Sandboxes Beta”
Vercel Sandbox docs clearly describe automatic snapshotting of a sandbox's filesystem/state on stop and restoring it on resume, with named sandboxes resumable and snapshots speeding up subsequent runs by skipping setup. However, the evidence only shows resuming the *same* sandbox from its own snapshot, not explicitly forking multiple independent new sandboxes from one saved snapshot. Missing for 10: explicit support/documentation for forking multiple new sandboxes off a single snapshot, and any hands-on/independent confirmation of this fork workflow.
- [claimed-docs] “Snapshotting: Save the state of a running sandbox to resume later. Skip dependency installation on subsequent runs.”
- [claimed-docs] “Sandboxes that auto-save state on stop and resume where you left off. Persistence is the default. No manual snapshot management needed.”
- [claimed-docs] “when a sandbox stops, the SDK automatically snapshots its filesystem, and the sandbox configuration is preserved across sessions, so both ar…”
- [claimed-docs] “Or retrieve an existing sandbox by name (resumes if stopped)”
- [claimed-docs] “Using a snapshot is much faster than creating from scratch because it avoids reinstalling dependencies and repeating setup steps.”
Not comparable on these axes
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · not comparableModal Sandboxesn/aModal Sandboxes is compute/sandbox infrastructure for running untrusted or agent code, not a data product that ingests user data and surfaces AI-generated insights/suggestions from it — this axis is a category error for this product type.
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · not comparableModal Sandboxesn/aModal Sandboxes is an infrastructure platform for running untrusted/agent code, not itself an AI assistant or agent product; it provides no built-in assistant to delegate tasks to — this is a wrong-axis question for this product category.
ai-native userPrevent my data from being used to train AI models
weight 3 · not comparableModal Sandboxesn/aModal Sandboxes is an infrastructure/execution platform for running untrusted code, not an AI model or AI product with a training-data policy; there's no concept of 'training on user data' applicable to a sandbox execution service. This axis is a category error for this product type.
Vercel Sandboxn/aVercel Sandbox is a code-execution/microVM infrastructure product, not an AI model or AI-training data product; 'preventing data from being used to train AI models' is a data/privacy-policy axis for AI model providers, not for a sandbox execution service. This story is a category error for this product type.