Agent Sandboxes & Code Execution Arena
Modal Sandboxes vs Cloudflare Sandboxes
Modal Sandboxes wins · 13–12 (23 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to Modal SandboxesModal has a working llms.txt (HTTP 200) with a documentation summary, and docs pages are also served as markdown (.md suffix) confirming agent-oriented doc access. missing for 10: no independent confirmation that agents actually consume this llms.txt successfully in practice, and no dedicated agent-docs index beyond the generic llms.txt.
- [probe] “PROBE llms.txt: HTTP 200 at https://modal.com/llms.txt # Modal Documentation > Modal is a platform for running AI workloads in the cloud wi…”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
A probe confirms Cloudflare's developer docs expose a working llms.txt (HTTP 200) covering the whole docs site including Sandbox, giving agents a structured entry point. However, a probe for the markdown version of the specific Sandbox get-started page returned 404, showing per-page agent-friendly docs aren't uniformly available. Missing for 10: dedicated llms.txt or .md endpoints scoped to the Sandbox product docs, and no community/hands-on confirmation of agents actually consuming it successfully.
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to Modal SandboxesModal Sandboxes is built for programmatic/headless use via Python, JS/Go SDKs and a CLI, with sandboxes created and controlled entirely through code (modal-docs-2, modal-docs-19, modal-docs-20, modal-gh-1). Community evidence confirms real-world CI/automation usage — a developer built GitHub Actions self-hosted runners on Modal Sandboxes explicitly for scalable, pay-per-use automation (modal-comm-1), and another wired gRPC control for an agent runtime (modal-comm-2). Missing for 10: no first-party CI/CD integration guide (e.g., official GitHub Actions/Jenkins docs) or dedicated 'headless CI' documentation section beyond community projects.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [github] “The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [probe] “official CLI documented at https://modal.com/docs/reference/cli”
Cloudflare Sandboxes are SDK/Workers-driven (exec, code execution, git operations, bindings access) which is inherently headless/programmatic and well-suited to CI/CD automation, and docs explicitly mention CI/CD as a target use case. However there's no dedicated CLI, no CI integration examples (GitHub Actions, etc.), and no evidence of auto-cleanup/idle-timeout controls needed for reliable CI runs — community notes explicitly flag the lack of auto-destroy on idle, requiring manual cleanup logic. missing for 10: explicit CI/CD pipeline examples or integrations, a standalone CLI for headless invocation, documented idle-timeout/auto-teardown controls, and independent verification of headless automation at scale.
- [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
- [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
- [claimed-docs] “Clone repositories, manage branches, and automate Git operations.”
- [claimed-docs] “Access KV, R2, Durable Objects, and other bindings from a sandbox.”
- [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round drawnModal Sandboxesnone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
ai-native userConnect an agent via an official MCP server
weight 3 · round drawnModal Sandboxesnone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
Cloudflare Sandboxesnone0/10Cloudflare Sandboxes is an execution/infrastructure SDK (not itself an agent), so an official MCP server for agent connectivity is a fair axis to ask about—but the evidence pack contains no mention of MCP, an MCP server, or any protocol for connecting AI agents beyond raw SDK APIs (exec, files, sessions, tunnels, etc.).
ai-native userUse an official CLI
weight 2 · round to Modal SandboxesModal ships an official CLI (documented at modal.com/docs/reference/cli) supporting sandbox-related operations like running commands/shells and managing/connecting to containers, confirmed by probe access to the CLI reference page. This CLI is core to Modal's agentic/AI-native workflows (creating and interacting with sandboxes for untrusted/agent code). missing for 10: independent hands-on user reports specifically praising/using the CLI (vs. SDK), and more detail on CLI coverage of full sandbox lifecycle (snapshots, networking) beyond the two documented subcommands.
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “Manage and connect to running containers.”
- [probe] “official CLI documented at https://modal.com/docs/reference/cli”
- [claimed-docs] “Running commands”
Cloudflare Sandboxesnone0/10The evidence pack shows the Sandbox SDK is an API/library used from Workers code (exec, files, sessions, etc.) but nowhere mentions an official CLI tool for AI-native workflows; interaction is entirely via SDK calls or the general Wrangler CLI, not a dedicated Sandbox CLI.
ai-native userDrive the product through a documented public API
weight 3 · round drawnModal exposes documented Python, JS, and Go SDKs plus a CLI for programmatically creating, controlling, and interacting with Sandboxes (exec, filesystem, snapshots, networking, logs), and community developers confirm driving it via gRPC/SDK for agent runtimes. missing for 10: no public OpenAPI/REST spec (probe found only 404s), so the 'documented public API' is SDK/CLI-based rather than a formal API contract, and there's no independent audit of API stability/versioning.
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [github] “The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “Manage and connect to running containers.”
- [claimed-docs] “We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
- [probe] “PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, …”
Cloudflare publishes extensive, structured API documentation for the Sandbox SDK (exec, file I/O, sessions, code interpreter, ports/tunnels, watch, snapshots, git, Docker, bindings) explicitly marketed for building AI agents that execute code (docs-24, docs-34). The docs are machine-readable (llms.txt, openapi.json probes return 200) and cover the full surface an AI-native developer would need to drive. missing for 10: independent hands-on confirmation that agents successfully integrate via this API in production, and a dedicated REST/OpenAPI spec for the sandbox endpoints themselves (only the SDK is documented, not a standalone HTTP API schema).
- [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
- [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
- [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
- [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
- [claimed-docs] “Execute Python, JavaScript, and TypeScript code with support for data visualizations, tables, and rich output formats.”
- [probe] “PROBE llms.txt: HTTP 200 at https://developers.cloudflare.com/llms.txt # Cloudflare Developer Documentation Explore guides and tutorials to…”
- [probe] “PROBE openapi: HTTP 200 at https://developers.cloudflare.com/openapi.json — contains "openapi" key”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round drawnModal Sandboxesnone0/10The evidence describes secure sandboxed execution environments and general platform features, but there is no concrete evidence of a mechanism for issuing scoped/least-privilege API credentials specifically to agents—only a vague marketing line about 'credentials already in place' (modal-docs-3) with no documentation of credential scoping, token permissions, or per-agent access control. Missing for 10: any documented API/credential-scoping mechanism, permission granularity docs, or examples of issuing least-privilege tokens to agents.
- [claimed-docs] “Autonomous agents with the right tools, context, and credentials already in place — running securely in a full, isolated dev environment.”
Cloudflare Sandboxesnone0/10The evidence pack documents sandbox execution, filesystem, networking, and session features but contains no mention of scoped or least-privilege API credential/token issuance for agents accessing the sandbox APIs; in fact, community feedback notes egress control is 'all or nothing' rather than fine-grained, suggesting limited scoping capability overall.
- [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”
ai-native userBuild against official SDKs
weight 2 · round drawnModal ships official Python SDK plus JS/Go SDKs for Sandboxes with documented APIs (exec, filesystem, snapshots, networking, logs) and community developers confirm building against them (GitHub Actions runners, gRPC-based JS agent runtime). missing for 10: independent third-party SDK reference/benchmarks beyond Modal's own docs and one community mention, and no evidence of broader language SDK coverage (e.g. Rust, Java).
- [github] “The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [claimed-docs] “We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
Cloudflare publishes an extensive official SDK (TypeScript/JavaScript) with detailed docs covering exec, file I/O, sessions, code interpreter for Python/JS, port exposure, snapshots, and bindings, explicitly positioned for building AI agents that execute code. Community commentary discusses pricing/latency but does not dispute the SDK's functionality or documentation quality. Missing for 10: independent hands-on developer reviews of the SDK's ergonomics/reliability, and confirmation of multi-language SDK parity beyond JS/TS/Python.
- [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
- [claimed-docs] “Execute Python and JavaScript code with rich outputs including charts, tables, and formatted data.”
- [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
- [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
- [claimed-docs] “Execute Python, JavaScript, and TypeScript code with support for data visualizations, tables, and rich output formats.”
- [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”
ai-native userSubscribe to events via webhooks
weight 2 · round drawnModal Sandboxesnone0/10No evidence anywhere in the pack of a webhook subscription mechanism or event-driven notification system for Modal Sandboxes; the docs focus on programmatic spin-up, execution, logging, and filesystem access, not outbound event/webhook subscriptions.
Agentic features
ai-native userSet up automations that run autonomously in the background
weight 2 · round drawnModal Sandboxes explicitly supports running autonomous agent code in isolated environments at scale (docs-3, docs-4, docs-22), and community evidence shows real-world use for background automation (self-hosted CI runners on modal-comm-1, agent runtimes on modal-comm-2). However, the evidence pack lacks explicit documentation of scheduling/cron triggers or persistent background job orchestration—sandboxes are framed as on-demand execution environments rather than a native 'set it and forget it' automation scheduler. Missing for 10: native scheduling/cron trigger docs, first-party persistent orchestration for long-running background automations.
- [claimed-docs] “Autonomous agents with the right tools, context, and credentials already in place — running securely in a full, isolated dev environment.”
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds.”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
Docs show sandboxes can run background processes, maintain persistent sessions/state, and reconnect via stable IDs (docs-43, docs-9, docs-37, docs-32), which supports basic autonomous background execution. However, there's no documented native scheduling/trigger mechanism for kicking off automations, and community feedback notes the lack of auto-idle cleanup means developers must manually manage sandbox lifecycle rather than it running fully autonomously (comm-7). Missing for 10: built-in cron/trigger scheduling, evidence of true unattended long-running automation, and confirmation that background processes persist reliably without manual intervention.
- [claimed-docs] “Execute commands and manage background processes in the sandbox's isolated container environment.”
- [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
- [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”
- [claimed-docs] “const buildSession = await sandbox.createSession({ id: "build", env: { NODE_ENV: "production" }, cwd: "/build" });”
- [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”
ai-native userOperate the product with natural-language commands
weight 2 · round drawnModal Sandboxesnone0/10Modal Sandboxes is controlled exclusively via SDK (Python/JS/Go) calls and a traditional CLI (modal run, modal shell, etc.); there is no evidence of a natural-language command interface or chat-driven control layer for operating sandboxes. Evidence shows programmatic/API-first design (modal-docs-2, modal-docs-15, modal-docs-19) rather than any NL-command capability.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [probe] “official CLI documented at https://modal.com/docs/reference/cli”
Cloudflare Sandboxesnone0/10Cloudflare Sandboxes is a programmatic SDK (sandbox.exec(), writeFile(), createSession(), etc.) intended to be called from Worker code, not a product with a natural-language command interface; while it markets itself for building AI agents (docs-24), there is no evidence of a chat/NL layer for operating the sandbox itself. missing for 10: any NL command parsing/interface, evidence of a conversational control layer, examples of natural-language-driven sandbox operations.
- [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
- [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
- [claimed-docs] “Execute commands and manage background processes in the sandbox's isolated container environment.”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnModal Sandboxesnone0/10Evidence shows standard docs pages, CLI reference, and SDK code snippets, but no interactive API reference or runnable-example playground; the openapi probe returned 404s and no docs mention an in-browser code runner or live API explorer.
- [probe] “PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, …”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
Cloudflare Sandboxesnone0/10The evidence pack shows extensive static code snippets across Cloudflare's Sandbox docs (exec, file I/O, sessions, interpreter, etc.) but nothing indicates an interactive, in-browser 'try it now' API reference or runnable playground — the openapi.json probe is for Cloudflare's general API, not a Sandbox-specific interactive explorer. Missing for 10: an in-page runnable code sandbox/playground, an interactive API explorer (e.g., Swagger/Redoc try-it-out), or any embedded live execution environment tied to the docs.
- [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
- [claimed-docs] “import pandas as pd data = {'product': ['A', 'B', 'C'], 'sales': [100, 200, 150]} df = pd.DataFrame(data) df['sales'].sum() # Last expressi…”
- [probe] “PROBE openapi: HTTP 200 at https://developers.cloudflare.com/openapi.json — contains "openapi" key”
- [probe] “PROBE docs-md: HTTP 404 at https://developers.cloudflare.com/sandbox/get-started/.md”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round to Cloudflare SandboxesModal Sandboxesnone0/10The probe explicitly found all candidate OpenAPI/swagger endpoints return 404, and no other evidence pack item mentions a machine-readable API spec being published; Modal's public interface is SDK/CLI-based rather than a REST API with a spec.
- [probe] “PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, …”
Cloudflare's developer platform exposes a general openapi.json (HTTP 200, containing an 'openapi' key), but there's no evidence this spec specifically documents the Sandbox SDK's methods (exec, writeFile, sessions, etc.), which is a code-level SDK rather than a REST API; a dedicated machine-readable doc for the sandbox pages itself returned 404. Missing for 10: confirmation that the openapi.json actually covers Sandbox-specific endpoints/methods, and a dedicated llms.txt/OpenAPI export scoped to the Sandbox SDK docs.
- [probe] “PROBE openapi: HTTP 200 at https://developers.cloudflare.com/openapi.json — contains "openapi" key”
- [probe] “PROBE docs-md: HTTP 404 at https://developers.cloudflare.com/sandbox/get-started/.md”
- [probe] “PROBE llms.txt: HTTP 200 at https://developers.cloudflare.com/llms.txt # Cloudflare Developer Documentation Explore guides and tutorials to…”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round drawnModal Sandboxes are explicitly designed as ephemeral, isolated containers for running untrusted or agent code, with filesystem, networking/security, and snapshot controls documented (modal-docs-2, modal-docs-8, modal-docs-9, modal-docs-10, modal-probe-2), and community examples show real usage isolating workloads from production infra (modal-comm-1, modal-comm-2). This satisfies the 'test without touching production' need since each sandbox is a fresh, isolated environment rather than shared prod state. Missing for 10: explicit documentation or case study framing isolation specifically in terms of 'production data' separation (e.g., no prod DB/staging parity guidance), and no independent audit confirming true isolation guarantees.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Filesystem Access”
- [claimed-docs] “Snapshots”
- [claimed-docs] “Networking and security”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Cloudflare Sandboxes are explicitly designed as isolated VM/container environments for running untrusted code, separate from production infra, with sessions, snapshots, and opt-in bindings to access KV/R2/Durable Objects only when explicitly wired in — matching the AI-native 'test without touching production' need well (docs-34, docs-38, docs-50, docs-20, docs-24). Community feedback doesn't contradict the isolation guarantee itself, though it flags coarse outbound network controls and unclear persistence semantics as caveats (cloudflare-sandbox-comm-2, cloudflare-sandbox-comm-4). Missing for 10: independent hands-on validation that no production data leakage occurs in practice, and finer-grained egress/network isolation controls.
- [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
- [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
- [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
- [claimed-docs] “Access KV, R2, Durable Objects, and other bindings from a sandbox.”
- [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
- [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”
- [community] “There is an open question about how file persistence works. The docs claim they persist the filesystem even when they move the container to …”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnModal Sandboxesnone0/10Evidence shows release notes and beta/alpha labels for some sandbox features, but no documentation of API versioning scheme or a deprecation policy; OpenAPI spec probes also 404. missing for 10: explicit versioning scheme documentation, deprecation policy page, changelog with deprecation notices, stability guarantees for GA vs beta APIs.
- [claimed-docs] “VM Sandboxes Beta”
- [claimed-docs] “Sidecars Alpha”
- [claimed-docs] “We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
- [claimed-docs] “it's now possible to opt into a more performant Sandbox backend, which affords substantially higher creation rates and concurrency”
- [probe] “PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, …”
Cloudflare Sandboxesnone0/10The evidence pack contains extensive API documentation for the Sandbox SDK's features (exec, files, sessions, snapshots, etc.) but nowhere mentions API versioning schemes, version numbers, or a documented deprecation policy for the SDK/API surface. No evidence of semver commitments, changelogs with deprecation notices, or migration guides for breaking changes.
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to Modal SandboxesModal supports spinning up hundreds of thousands of concurrent sandboxes/rollout environments programmatically, which is strong evidence of bulk/parallel scaling (modal-docs-4, modal-docs-14, modal-docs-5), and this is corroborated by community builds using it for scaled runner fleets (modal-comm-1). However, this is infrastructure-level bulk provisioning, not a documented 'bulk operations' API/pattern for operating across many existing items (e.g., batch file ops, batch snapshot management, or a dedicated batch API) — missing for 10: a first-party batch/bulk-operation API or workflow example (vs just scaling raw sandbox count), independent benchmarks confirming claimed concurrency at scale, and documentation of bulk management operations (e.g., bulk terminate/list/update) rather than just creation.
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds.”
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds. Fast enough to keep your GPU inference resources saturated acro…”
- [claimed-docs] “H100s, A100s, A10Gs available on demand. Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved ca…”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [claimed-docs] “Integrated logging and full visibility into every function, sandbox, and container.”
Cloudflare Sandboxes supports concurrent/parallel operations via a multiplexed connection that 'avoids subrequest limits when making many concurrent operations' (docs-17) and allows multiple isolated sessions per sandbox for parallel workflows (docs-9, docs-18, docs-39), plus getSandbox() to manage many sandbox instances by ID. However there's no documented batch/bulk API primitive for iterating over large item sets, and community feedback flags unclear resource limits and high per-vCPU pricing that would constrain large-scale bulk workloads (comm-3, comm-6, comm-11). Missing for 10: an explicit bulk/batch operation API or example, documented concurrency/resource ceilings, and independent evidence of successful large-scale parallel execution.
- [claimed-docs] “Multiplexes all SDK calls over a single persistent connection. It avoids subrequest limits when making many concurrent operations.”
- [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
- [claimed-docs] “Create additional sessions for separate workflows in the same sandbox”
- [claimed-docs] “Create additional sessions for separate workflows in the same sandbox:”
- [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”
- [community] “Cloudflare Containers (and therefore Sandbox) pricing is way too expensive... vCPU: $51.84 per vCPU (!!!)... These prices are more expensive…”
- [community] “The pricing with such offerings is the biggest throwoff. This one comes out to be more than $58/month for just 1vCpu and 1GiB RAM when used …”
- [community] “I am keen to understand how I would limit sandbox resource usage.”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to Cloudflare SandboxesModal Sandboxesnone0/10The evidence describes Modal Sandboxes as a programmatic, on-demand execution environment for untrusted/agent code, but nothing in the pack shows a rules/trigger engine that fires actions automatically on external events (e.g., webhooks, event-driven policies). Absent such evidence, this applicable automation-depth axis is unmet.
The SDK provides a file-system watch capability with an `onEvent` callback (inotify-based) that lets developers trigger custom actions when files change, which is a form of event-driven automation, but this is a narrow, code-level primitive rather than a declarative rules/automation engine. Missing for 10: no declarative rule-definition mechanism, no support for triggering on non-filesystem events (e.g., scheduled/cron, external webhooks, custom event types), and no evidence of a rules/condition-action system beyond raw watcher callbacks.
- [claimed-docs] “Monitor real-time filesystem changes using native inotify. Build development tools, hot-reload systems, and responsive file processing.”
- [claimed-docs] “const watcher = await sandbox.watch('/workspace/src', { include: ['*.js', '*.ts'],”
- [claimed-docs] “const watcher = await sandbox.watch('/workspace/src', { include: ['*.js', '*.ts'], onEvent: (event) => {”
ai-native userSchedule recurring jobs or workflows
weight 2 · round drawnModal Sandboxesnone0/10Evidence covers ephemeral sandbox creation, execution, networking, snapshots, and SDKs, but there is no mention of cron-like scheduling, recurring triggers, or persistent workflow orchestration for Modal Sandboxes. missing for 10: any scheduling/cron API, recurring job trigger mechanism, or workflow orchestration documentation tied to Sandboxes.
- [claimed-docs] “VM Sandboxes Beta”
- [claimed-docs] “Running commands”
- [claimed-docs] “Snapshots”
Cloudflare Sandboxesnone0/10The evidence describes on-demand sandbox execution, sessions, and lifecycle management, but nowhere mentions cron-style scheduling, recurring triggers, or workflow orchestration for repeated runs; community threads even highlight the lack of automatic idle/cleanup logic, implying no built-in scheduling primitives.
- [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
- [claimed-docs] “Create additional sessions for separate workflows in the same sandbox”
- [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”
ai-native userVersion, review, and roll back my automations
weight 1 · round to Cloudflare SandboxesModal Sandboxesnone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
Sandboxes offer git clone/branch operations for code and R2-based directory snapshot/restore, which give raw building blocks for versioning and rolling back sandbox state, but there is no dedicated 'automation' versioning, review, or rollback UI/workflow — it's up to the developer to wire this together. Missing for 10: no built-in review/diff tooling for automations, no automatic rollback trigger, no evidence of a workflow-specific version history feature, and no independent confirmation these primitives are actually used for automation rollback in practice.
- [claimed-docs] “Clone repositories, manage branches, and automate Git operations.”
- [claimed-docs] “Snapshot a sandbox directory to R2 and restore it later.”
- [claimed-docs] “Create point-in-time snapshots of directories and restore them from R2.”
- [claimed-docs] “Mount S3-compatible buckets (R2, S3, GCS) as local filesystems for persistent data storage across sandbox lifecycles.”
Capabilities hardware — stories about capabilities hardware in this arenaCapabilities hardware
Stories about capabilities hardware in this arena
Agent workloads
ai-native userRun coding agents like Claude Code or Codex inside the sandbox following the vendor's own recipe
weight 2 · round drawnModal Sandboxesnone0/10Modal's docs show general support for running 'agents' and 'untrusted code' in Sandboxes (modal-docs-3, modal-docs-22, modal-docs-19), but no evidence pack item mentions a vendor recipe, tutorial, or example specifically for running Claude Code or Codex inside a Sandbox.
Cloudflare Sandboxesnone0/10The docs describe Sandboxes as ideal for building AI agents and provide generic exec/git/docker primitives, but there is no vendor-published recipe or guide specifically for running Claude Code or Codex inside a Sandbox. missing for 10: an explicit vendor tutorial/recipe naming Claude Code or Codex, any first-party integration guide, and community confirmation that such agents run inside the sandbox.
- [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
- [claimed-docs] “Run Docker commands inside a sandbox container.”
- [claimed-docs] “Clone repositories, manage branches, and automate Git operations.”
Workloads
developerRun a headless browser or full desktop environment inside the sandbox
weight 2 · round drawnModal Sandboxesnone0/10Evidence describes Modal Sandboxes as isolated containers/VMs for running arbitrary code, agents, filesystem access, networking, and command execution, but nowhere mentions headless browser or full desktop/GUI environment support (e.g., Xvfb, VNC, browser automation tooling). No docs or community reports reference browser or desktop environments running inside sandboxes.
Cloudflare Sandboxesnone0/10The evidence pack details shell execution, file I/O, sessions, tunnels, snapshots, and code interpretation, but nowhere mentions running a headless browser (e.g., Playwright/Puppeteer) or a full desktop/GUI environment inside the sandbox. This is a reasonable capability to ask of a code-execution sandbox product, but no docs or community evidence confirm it exists.
developerAttach GPUs to sandboxed workloads
weight 2 · round to Modal SandboxesModal docs explicitly state GPUs (H100s, A100s, A10Gs) can be attached to any sandbox on demand with per-second billing, and community evidence (HN runner project) confirms real-world GPU-backed sandbox usage. missing for 10: independent hands-on benchmark of GPU attach performance/reliability, and explicit code example showing GPU parameter in sandbox creation API.
- [claimed-docs] “H100s, A100s, A10Gs available on demand. Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved ca…”
- [claimed-docs] “Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved capacity.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
Code execution — stories about code execution in this arenaCode execution
Stories about code execution in this arena
Files
developerRead, write, upload, and download files in the sandbox filesystem via the SDK
weight 2 · round to Cloudflare SandboxesModal explicitly documents a dedicated 'Filesystem Access' guide for sandboxes (modal-docs-8) alongside SDK examples showing file read/write via mounted volumes and exec commands (modal-docs-20, modal-docs-21), confirming read/write/upload/download-style operations through the SDK. missing for 10: no independent/hands-on community confirmation specifically of file upload/download methods (only exec/volume examples), and no explicit code snippet showing a dedicated upload/download API method distinct from volume mounting.
- [claimed-docs] “Filesystem Access”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Docs explicitly cover readFile/writeFile/mkdir with absolute paths, streaming/binary uploads >32MiB via ReadableStream, and directory watching, giving comprehensive first-party coverage of file read/write/upload/download via the SDK. Missing for 10: independent hands-on developer confirmation of file upload/download behavior (community discussion focuses on persistence questions, not concrete failures).
- [claimed-docs] “await sandbox.writeFile("/workspace/hello.txt", "Hello, Sandbox!"); const file = await sandbox.readFile("/workspace/hello.txt");”
- [claimed-docs] “This allows binary data and files greater than 32 MiB to be written to the sandbox. It replaces the "base64" encoding option.”
- [claimed-docs] “the writeFile() method supports passing a ReadableStream as the content parameter. This allows binary data and files greater than 32 MiB”
- [claimed-docs] “Read, write, and manage files in the sandbox filesystem. All paths are absolute (e.g., `/workspace/app.js`).”
- [claimed-docs] “the `writeFile()` method supports passing a `ReadableStream` as the `content` parameter.”
- [claimed-docs] “await sandbox.mkdir('/workspace/project/src', { recursive: true });”
Runtimes
developerDefine custom sandbox templates or bring my own container image
weight 2 · round to Modal SandboxesModal explicitly supports creating Sandboxes with custom Images ('Custom images, any dependency' and 'Create a Sandbox with the specified Image, and mount a Volume'), and community projects (GitHub Actions runners, MCP-JS) confirm developers configuring custom sandbox environments in practice. Missing for 10: explicit documentation of importing arbitrary Docker/OCI registry images or a dedicated 'bring your own container' guide beyond the Image API reference.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
Cloudflare Sandboxesnone0/10The evidence describes running commands, files, sessions, Python/JS execution, tunnels, and Docker-command execution inside a sandbox, but nothing documents defining a custom sandbox template or supplying a custom base container image for the sandbox environment itself. Since bring-your-own-image is a standard, expected axis for a container/sandbox product, the lack of any supporting evidence yields 'none' rather than 'na'.
- [claimed-docs] “Run Docker commands inside a sandbox container.”
- [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
- [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
- [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
developerExecute code in multiple language runtimes (Python, JavaScript, and more) and get rich results back
weight 2 · round to Cloudflare SandboxesModal Sandboxes let you run arbitrary commands/code in isolated containers with custom images (any dependency, any language stack), including SDK examples for JS/Go exec and Python examples of running agent code (modal-docs-2, modal-docs-13, modal-docs-19-21). However, evidence never shows explicit multi-language runtime presets (Python vs JS vs others) or structured/rich result payloads (e.g., images, dataframes, mime-typed outputs) — outputs shown are plain stdout/exec results. missing for 10: explicit built-in multi-language runtime templates, documented rich/structured result formats beyond raw stdout.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Running commands”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
Docs clearly document a code interpreter API executing Python, JavaScript, and TypeScript with rich outputs (charts, tables, dataframes, stateful contexts), plus shell exec with stdout/stderr capture, matching the story directly. Missing for 10: independent/hands-on verification of the rich-output rendering quality and broader language support beyond the three named runtimes.
- [claimed-docs] “Execute Python and JavaScript code with rich outputs including charts, tables, and formatted data.”
- [claimed-docs] “Execute Python, JavaScript, and TypeScript code with support for data visualizations, tables, and rich output formats.”
- [claimed-docs] “import pandas as pd data = {'product': ['A', 'B', 'C'], 'sales': [100, 200, 150]} df = pd.DataFrame(data) df['sales'].sum() # Last expressi…”
- [claimed-docs] “Contexts maintain state (variables, imports, functions) across executions.”
- [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
- [claimed-docs] “you'll create a Worker that can execute Python code and work with files in isolated containers.”
Untrusted code
developerExecute untrusted, AI-generated code without risking my own infrastructure
weight 3 · round to Modal SandboxesModal Sandboxes is explicitly built for running untrusted/AI-generated code in isolated, ephemeral containers with networking/filesystem controls, resource limits, and logging, and community examples show real-world use (self-hosted CI runners, agent runtimes) confirming the isolation model works in practice. Missing for 10: independent security audit or adversarial escape-testing evidence, and clearer documentation of resource/network isolation guarantees beyond marketing copy.
- [claimed-docs] “Programmatically scale secure, ephemeral environments for running untrusted code.”
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Networking and security”
- [claimed-docs] “VM Sandboxes Beta”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
Cloudflare's docs explicitly market Sandboxes for running untrusted, AI-generated code with VM/container-based isolation, file/exec APIs, and strong security boundaries (cloudflare-sandbox-docs-34, -38, -50, -24). However, hands-on community feedback flags real gaps for this exact use case: outbound network access is 'all or nothing' with no fine-grained egress control, and there's no built-in auto-destroy for idle sandboxes, forcing manual cleanup. Missing for 10: granular network egress control, automatic idle sandbox teardown, independent security audit of the VM isolation claims.
- [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
- [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
- [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
- [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
- [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”
- [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”
developerRun arbitrary shell commands and install packages inside the sandbox
weight 2 · round drawnModal Sandboxes documentation explicitly covers running arbitrary shell commands (sb.exec, CLI 'run a command or interactive shell inside a Modal container'), custom images with any dependency installation, filesystem access, and community reports (HN self-hosted runners, JS agent runtime) corroborate real-world use for arbitrary command execution and untrusted code. Missing for 10: no independent benchmark of package installation limits or explicit documented apt/pip install examples beyond generic 'any dependency' claim.
- [claimed-docs] “Running commands”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Docs explicitly document sandbox.exec() for running arbitrary shell commands with stdout/stderr/exit codes, stdin support, streaming/SSE output, background processes, plus Docker command execution and Git operations inside the sandbox, giving developers full shell/package-install capability. Community threads corroborate real-world usage of the sandbox for running containerized workloads, though none specifically contradict shell/package execution. missing for 10: independent hands-on verification of installing arbitrary packages (e.g., apt/pip) beyond docs examples.
- [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
- [claimed-docs] “Pass input via stdin (no shell injection risks)”
- [claimed-docs] “Execute a command and return a Server-Sent Events stream for real-time processing.”
- [claimed-docs] “Run Docker commands inside a sandbox container.”
- [claimed-docs] “Clone repositories, manage branches, and automate Git operations.”
- [claimed-docs] “stdin - Data to pass to the command's standard input (enables arbitrary input without shell injection risks)”
- [claimed-docs] “onOutput - Callback for real-time output: (stream: 'stdout' | 'stderr', data: string) => void”
- [claimed-docs] “Execute commands and manage background processes in the sandbox's isolated container environment.”
- [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
Dev experience — day-to-day developer experience — setup friction, docs, debugging, iteration speedDev experience
Day-to-day developer experience — setup friction, docs, debugging, iteration speed
Agent lifecycle
ai-native userMy agent can provision its own sandbox, execute code, read the results, and tear it down — end to end without a human
weight 3 · round to Modal SandboxesModal Sandboxes are explicitly designed for programmatic, API-driven provisioning, code execution, log/result retrieval, and teardown, with SDK examples (Python/JS/Go) showing sandbox creation, exec, file access, and logs, plus a documented agent-code-execution example. missing for 10: independent hands-on report showing a fully autonomous agent loop (provision->execute->read->teardown) with no human intervention, and clearer documentation of teardown/termination API specifics.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Autonomous agents with the right tools, context, and credentials already in place — running securely in a full, isolated dev environment.”
- [claimed-docs] “Running commands”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
- [claimed-docs] “We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
Docs clearly show an agent-driven lifecycle: getSandbox() to provision/reconnect, exec()/interpreter code execution with stdout/stderr capture, and readFile to retrieve results — all callable programmatically without a human in the loop. However, there is no documented explicit 'destroy/teardown' API, and community feedback confirms there's no built-in idle-timeout auto-destroy, forcing developers to implement their own cleanup logic. Missing for 10: a documented sandbox.destroy()/terminate call or auto-expiry mechanism, and independent hands-on confirmation of a full provision-execute-teardown loop working end-to-end.
- [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”
- [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
- [claimed-docs] “Execute Python and JavaScript code with rich outputs including charts, tables, and formatted data.”
- [claimed-docs] “await sandbox.writeFile("/workspace/hello.txt", "Hello, Sandbox!"); const file = await sandbox.readFile("/workspace/hello.txt");”
- [claimed-docs] “Contexts maintain state (variables, imports, functions) across executions.”
- [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”
Isolation security — stories about isolation security in this arenaIsolation security
Stories about isolation security in this arena
Isolation
platform-engineerRely on a documented hard isolation boundary (microVM or equivalent) between sandboxes and my systems
weight 3 · round to Cloudflare SandboxesModal repeatedly describes Sandboxes as 'secure,' 'isolated' containers for untrusted code, and references a 'VM Sandboxes Beta' page implying microVM-based isolation, but no evidence pack item actually documents the underlying isolation technology (e.g., Firecracker/gVisor specifics, threat model, or security boundary guarantees) that a platform engineer would need to rely on. Missing for 10: explicit technical documentation of the isolation mechanism (hypervisor/kernel boundary), any third-party security audit or independent verification of the boundary's robustness.
- [claimed-docs] “VM Sandboxes Beta”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
Cloudflare's own docs explicitly state 'VM-based isolation - Each sandbox runs in its own VM' and that each sandbox 'runs in its own isolated container with a full Linux environment, providing strong security boundaries' for running untrusted code (docs-38, docs-50, docs-34). This gives a documented isolation boundary, though the docs mix 'VM' and 'container' language rather than naming a specific microVM technology, and community discussion raises open questions about network egress granularity rather than disputing the isolation claim itself. Missing for 10: a clearer, consistent technical description of the isolation mechanism (e.g., naming Firecracker or similar) and independent third-party security audit/corroboration.
- [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
- [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
- [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
- [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”
ai-native userGive an agent a sandbox where host secrets and credentials are unreachable by the code it runs
weight 2 · round to Cloudflare SandboxesModal Sandboxes are documented as isolated, ephemeral containers for running untrusted/agent code with dedicated networking-and-security guides, supporting the general isolation story (modal-docs-1, modal-docs-2, modal-docs-10, modal-probe-2, modal-docs-19, modal-docs-22). However, no evidence explicitly states that host secrets/credentials are unreachable by default or describes the secrets-injection model that would confirm this — one doc even implies credentials are provisioned into the agent environment (modal-docs-3), which cuts against a strong 'unreachable unless granted' guarantee. Missing for 10: explicit secrets-isolation documentation, details of the opt-in secrets mechanism, and independent/hands-on confirmation that host credentials cannot leak into sandboxed code.
- [claimed-docs] “Programmatically scale secure, ephemeral environments for running untrusted code.”
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Autonomous agents with the right tools, context, and credentials already in place — running securely in a full, isolated dev environment.”
- [claimed-docs] “Networking and security”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Cloudflare documents strong process/VM isolation ('VM-based isolation', 'isolated container with a full Linux environment', 'run untrusted code securely in isolated environments') which implies the host's runtime and secrets aren't directly reachable by sandboxed code, and bindings (KV/R2/DO) are explicitly scoped and passed in rather than ambient. However, there is no explicit documentation describing how host secrets/credentials are excluded or scrubbed from the sandbox environment, and community feedback notes only coarse-grained outbound network control ('all or nothing') rather than fine-grained secret/credential access restrictions, leaving open questions about exfiltration vectors. Missing for 10: explicit documentation on secret/credential isolation guarantees, independent security audit or hands-on test confirming host credentials are unreachable, and finer-grained egress/network controls to prevent secret exfiltration.
- [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
- [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
- [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
- [claimed-docs] “Access KV, R2, Durable Objects, and other bindings from a sandbox.”
- [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”
Network policy
platform-engineerRestrict or allow the sandbox's network egress with explicit policy
weight 3 · round to Modal SandboxesModal has a dedicated 'Networking and security' guide for Sandboxes, confirming the platform documents network controls for sandboxes, but the evidence pack only shows the page title without excerpts detailing how egress allow/deny policies are configured. missing for 10: actual policy syntax/examples (e.g. allowlist/blocklist configuration), confirmation of default-deny behavior, independent/hands-on verification that egress restrictions work as documented.
- [claimed-docs] “Networking and security”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Cloudflare Sandboxesdisputedcontradicted3/10Docs mention only a mechanism to 'Intercept and handle outbound HTTP from sandboxes using Workers' (docs-23), which is not the same as a declarative allow/deny egress policy, and no docs describe configurable network egress rules. A named community commenter explicitly states outbound network control is 'all or nothing right now' with no fine-grained control available, directly contradicting the story's requirement for explicit egress policy. missing for 10: documented egress allow/deny policy API, first-party confirmation of granular network restriction, and any independent verification that such a policy exists.
- [claimed-docs] “Intercept and handle outbound HTTP from sandboxes using Workers.”
- [community] “Looks like there's one feature missing from this that I care about: I'd like more finely grained control over what outbound internet connect…”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to Cloudflare SandboxesModal Sandboxes is fundamentally API/SDK-driven (Python, JS/Go) with a full CLI for running commands and managing containers, and docs describe programmatic control over image building, execution, filesystem, networking, snapshots, and logs — suggesting the API is the primary and most complete interface. However, there's no explicit documentation stating full UI/API parity or listing any UI-exclusive features, and no OpenAPI spec is discoverable (probe found 404s), making it hard to confirm completeness of a public REST API surface beyond the SDKs/CLI. Missing for 10: explicit parity statement between dashboard UI and API/CLI capabilities, and a public API reference/spec confirming no UI-only actions.
- [claimed-docs] “Filesystem Access”
- [claimed-docs] “Snapshots”
- [claimed-docs] “Networking and security”
- [claimed-docs] “Running commands”
- [claimed-docs] “Run a command or interactive shell inside a Modal container.”
- [claimed-docs] “Manage and connect to running containers.”
- [github] “The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [probe] “PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, …”
- [probe] “official CLI documented at https://modal.com/docs/reference/cli”
Cloudflare Sandboxes is an API/SDK-first product with no separate management UI described in the evidence; every capability (exec, file I/O, sessions, git, Docker, port exposure, snapshots, code interpreter, filesystem watch, even the browser terminal) is exposed and built entirely through the SDK/API, and the terminal UI itself is just a WebSocket client wired to the same API. Missing for 10: no explicit statement contrasting a Cloudflare dashboard GUI with the API, and no independent/third-party confirmation that no UI-exclusive controls exist (e.g., resource limits raised as unclear in community threads).
- [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
- [claimed-docs] “Connect browser-based terminal UIs to sandbox shells via WebSocket, with the xterm.js SandboxAddon for automatic reconnection and resize han…”
- [claimed-docs] “Run Docker commands inside a sandbox container.”
- [claimed-docs] “Clone repositories, manage branches, and automate Git operations.”
- [claimed-docs] “Snapshot a sandbox directory to R2 and restore it later.”
- [community] “I am keen to understand how I would limit sandbox resource usage.”
ai-native userExport all of my data in open formats and leave
weight 3 · round to Cloudflare SandboxesModal Sandboxesnone0/10The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)
The SDK exposes standard file I/O (readFile/writeFile), S3-compatible bucket mounting (R2/S3/GCS), and snapshot/restore to R2, which are open, non-proprietary formats a user could use to extract their data. However, there is no documented bulk 'export all my data' feature, no account-level portability tooling, and no independent confirmation that a user can fully migrate off the platform with all state intact. missing for 10: explicit data-export/account-portability feature, independent verification of full data extraction, documentation on non-file state (e.g., session/process state) portability.
- [claimed-docs] “await sandbox.writeFile("/workspace/hello.txt", "Hello, Sandbox!"); const file = await sandbox.readFile("/workspace/hello.txt");”
- [claimed-docs] “Mount S3-compatible buckets (R2, S3, GCS) as local filesystems for persistent data storage across sandbox lifecycles.”
- [claimed-docs] “Create point-in-time snapshots of directories and restore them from R2.”
- [claimed-docs] “Read, write, and manage files in the sandbox filesystem. All paths are absolute (e.g., `/workspace/app.js`).”
- [claimed-docs] “Snapshot a sandbox directory to R2 and restore it later.”
ai-native userRead the product's source under an open license
weight 2 · round drawnModal Sandboxesnone0/10Evidence only shows a GitHub repo for the client SDK (modal-client) used to invoke Modal Sandboxes, not the underlying sandbox/runtime platform itself, and no license terms are mentioned. There is no evidence that Modal Sandboxes' core product source is published under an open license.
- [github] “The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
ai-native userSelf-host the core product
weight 3 · round drawnModal Sandboxesnone0/10Modal is a fully-managed cloud platform for running Sandboxes; there is no evidence of any self-hostable/on-prem version of the core product, and everything described (pay-per-second billing, GPU access, hosted CLI) implies a proprietary SaaS deployment model only.
Cloudflare Sandboxesnone0/10Cloudflare Sandboxes is built entirely on Cloudflare's proprietary Workers/Durable Objects/container infrastructure, and no evidence in the pack mentions any open-source release, self-hosted deployment option, or ability to run the core product outside Cloudflare's platform. Community discussion focuses on pricing, cold-start times, and resource limits, not self-hosting.
- [claimed-docs] “The SDK supports three transport protocols for communication between the Durable Object and container”
- [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
- [community] “The pricing with such offerings is the biggest throwoff. This one comes out to be more than $58/month for just 1vCpu and 1GiB RAM when used …”
Performance scale — stories about performance scale in this arenaPerformance scale
Stories about performance scale in this arena
Latency
developerStart sandboxes with documented sub-second-to-few-second cold starts
weight 3 · round to Modal SandboxesEvidence shows general speed/scale claims ("spin up hundreds of thousands of concurrent rollout environments in seconds," faster backend for higher creation rates) but no explicit documented cold-start latency benchmark (e.g., sub-second or few-second startup time) for sandboxes specifically. Missing for 10: a concrete cold-start time benchmark/number, independent performance verification, and comparison across sandbox sizes/images.
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds.”
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds. Fast enough to keep your GPU inference resources saturated acro…”
- [claimed-docs] “it's now possible to opt into a more performant Sandbox backend, which affords substantially higher creation rates and concurrency”
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
Cloudflare Sandboxesnone0/10No first-party Cloudflare documentation in the evidence pack states specific cold-start timing (sub-second to few-second) for sandboxes; the community threads discuss competitors' or ambiguous Firecracker boot times without confirming Cloudflare's own documented cold-start figures. missing for 10: official docs quantifying cold-start latency, benchmark data, or a first-party performance page addressing startup time.
- [community] “We rolled out our own that does pretty much the same thing but perhaps more... It does take 1-5 seconds to boot the environment (firecracker…”
- [community] “1-5 seconds seems high for Firecracker, depending on your requirements. We boot VMs (using Firecracker) at ~20-50ms.”
Scale
platform-engineerRun large concurrent fleets of sandboxes with documented concurrency limits
weight 3 · round to Modal SandboxesModal markets very high concurrency ('hundreds of thousands of concurrent rollout environments', 'scale to thousands of concurrent runs') and even documents a backend upgrade for higher creation rates/concurrency, plus community reports of building scalable runner fleets on Sandboxes, showing the capability is real and used at scale. However there is no evidence of documented hard limits, quotas, or per-account/per-plan concurrency ceilings that a platform engineer could plan capacity against. Missing for 10: explicit documented concurrency/rate limits or quotas, benchmark/load-test data, and any per-tier scaling caps.
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds.”
- [claimed-docs] “H100s, A100s, A10Gs available on demand. Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved ca…”
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds. Fast enough to keep your GPU inference resources saturated acro…”
- [claimed-docs] “Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved capacity.”
- [claimed-docs] “it's now possible to opt into a more performant Sandbox backend, which affords substantially higher creation rates and concurrency”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
Cloudflare Sandboxesnone0/10Docs describe per-sandbox features (sessions, exec, files) and one architectural note that multiplexing avoids subrequest limits, but there is no documentation of concurrency limits, fleet-level scaling guidance, or how many sandboxes/instances can run concurrently. Community comments explicitly flag unanswered questions about resource limits and lack of auto-cleanup, reinforcing the absence of concrete concurrency documentation.
- [claimed-docs] “Multiplexes all SDK calls over a single persistent connection. It avoids subrequest limits when making many concurrent operations.”
- [community] “I am keen to understand how I would limit sandbox resource usage.”
- [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”
developerKeep a sandbox session running for hours or days for long agent tasks
weight 2 · round to Modal SandboxesModal docs describe sandboxes as isolated, scalable execution environments with snapshotting, filesystem, and networking support, and a community project (self-hosted GitHub Actions runners) demonstrates sandboxes being kept alive for extended, real workloads. However, no evidence pack item states explicit maximum session duration, timeout configuration, or guarantees for multi-day persistence, so long-running-session support is inferred rather than directly documented. Missing for 10: explicit docs on configurable/no timeout limits, guidance for multi-day sandbox lifecycles, and independent confirmation of hours/days-long sessions beyond one community anecdote.
- [claimed-docs] “Snapshots”
- [claimed-docs] “VM Sandboxes Beta”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [claimed-docs] “Integrated logging and full visibility into every function, sandbox, and container.”
- [claimed-docs] “We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
Cloudflare Sandboxesdisputedcontradicted5/10Docs describe stable-ID reconnection (getSandbox) and sessions with persistent state/env, suggesting sandboxes could be resumed across long tasks, but community reports explicitly say containers auto-shutdown after idle periods, that there's no built-in idle-timeout control (forcing manual cleanup), and that persistence semantics during idle states are unclear — directly undercutting reliable multi-hour/day continuous operation for unattended agent tasks. Missing for 10: first-party guidance on running sessions continuously for hours/days without disconnection, clear idle-timeout/keep-alive controls, and independent confirmation of long-duration reliability.
- [claimed-docs] “Create isolated execution contexts within a sandbox. Each session maintains its own shell state, environment variables, and working director…”
- [claimed-docs] “Create additional sessions for separate workflows in the same sandbox”
- [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”
- [claimed-docs] “You can also set a default command timeout for all commands in a session”
- [community] “There is an open question about how file persistence works. The docs claim they persist the filesystem even when they move the container to …”
- [community] “Cloudflares docs are written so hard for web dev. Can you host a monolith app that isn't serving http traffic on cloudflare tech like contai…”
- [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
Pricing
developerStart building with a free tier or included credits without talking to sales
weight 1 · round drawnModal Sandboxesnone0/10The evidence pack includes pricing-related docs (pay-per-use, burst capacity) but no mention of a free tier, free credits, or self-serve signup without contacting sales; nothing confirms developers can start building without sales contact.
Cloudflare Sandboxesnone0/10The evidence pack documents extensive product capabilities and self-serve pricing figures (e.g., per-vCPU costs debated in community threads) but contains no mention of a free tier, free trial credits, or explicit self-serve signup flow that lets a developer start without contacting sales. missing for 10: explicit free-tier/credit offer, signup flow documentation, confirmation no sales contact is required.
- [community] “Cloudflare Containers (and therefore Sandbox) pricing is way too expensive... vCPU: $51.84 per vCPU (!!!)... These prices are more expensive…”
- [community] “The pricing with such offerings is the biggest throwoff. This one comes out to be more than $58/month for just 1vCpu and 1GiB RAM when used …”
platform-engineerPay per second only for the compute a sandbox actually uses
weight 3 · round to Modal SandboxesModal's pricing docs and marketing explicitly state per-second billing with no reserved capacity ('pay by the second with no reserved capacity', 'only pay for what you use... burst up to what you need without over-allocating'), directly matching the story, and a community user corroborates paying only for resources actually consumed when building GPU runners on Sandboxes. Missing for 10: independent verification of exact billing granularity/metering mechanics and no first-party pricing breakdown showing per-second line items in the evidence pack.
- [claimed-docs] “Only pay for what you use. Burst up to what you need without over-allocating CPU or memory in advance.”
- [claimed-docs] “Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved capacity.”
- [claimed-docs] “Burst up to what you need without over-allocating CPU or memory in advance.”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
Cloudflare Sandboxesnone0/10No documentation in the evidence pack describes per-second or usage-based billing for sandbox compute; the only pricing-related evidence is community commentary describing flat, expensive per-vCPU pricing (comm-3, comm-6) and the absence of automatic idle shutdown, meaning engineers must build their own cleanup to avoid being billed for idle sandboxes (comm-7). This contradicts rather than supports the story of paying only for actual per-second compute use.
- [community] “Cloudflare Containers (and therefore Sandbox) pricing is way too expensive... vCPU: $51.84 per vCPU (!!!)... These prices are more expensive…”
- [community] “The pricing with such offerings is the biggest throwoff. This one comes out to be more than $58/month for just 1vCpu and 1GiB RAM when used …”
- [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnModal Sandboxesnone0/10No evidence pack item mentions data residency, region selection, or geographic data storage controls for Modal Sandboxes; the pack covers execution, scaling, GPUs, filesystem, and networking but nothing about choosing storage region/residency.
ai-native userControl data retention and deletion
weight 2 · round drawnModal Sandboxesnone0/10Evidence covers sandbox ephemerality, filesystem access, snapshots, and networking/security, but nothing describes data retention policies, deletion controls, or compliance/data-handling settings that a user could configure. missing for 10: explicit data retention policy documentation, user-controllable deletion/purge mechanisms, compliance certifications (e.g., SOC2/GDPR data handling), retention period configuration.
Cloudflare Sandboxesnone0/10The docs describe file read/write, snapshot/restore to R2, and bucket mounting, but there is no documented mechanism for users to control data retention windows or guarantee deletion of sandbox data. Community evidence explicitly states 'it does not seem to be possible to auto destroy a sandbox after certain amount of idle time... forces whoever is implementing this to do their own cleanup,' and persistence semantics are described as unclear, indicating the product does not provide the retention/deletion controls an AI-native user would need.
- [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”
- [community] “There is an open question about how file persistence works. The docs claim they persist the filesystem even when they move the container to …”
- [claimed-docs] “Create point-in-time snapshots of directories and restore them from R2.”
- [claimed-docs] “Mount S3-compatible buckets (R2, S3, GCS) as local filesystems for persistent data storage across sandbox lifecycles.”
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnModal Sandboxesnone0/10No evidence in the pack mentions telemetry, usage tracking, analytics collection, or an opt-out mechanism for Modal Sandboxes; the docs cover sandboxing, GPUs, filesystem, and logging but not data-collection/privacy controls.
Cloudflare Sandboxesnone0/10No evidence in the pack addresses telemetry, usage tracking, or opt-out settings for Cloudflare Sandboxes; docs focus purely on execution, filesystem, sessions, and networking features. This is a fair privacy-posture question for a developer platform product, so absence of evidence yields 'none' rather than 'na'.
Provisioning lifecycle — creating, updating, and tearing down resources across their lifecycleProvisioning lifecycle
Creating, updating, and tearing down resources across their lifecycle
Lifecycle
developerSet timeouts so sandboxes shut down automatically and stop billing when idle or done
weight 2 · round to Cloudflare SandboxesModal Sandboxesnone0/10The evidence pack only mentions general pay-per-second billing and bursty scaling, but never documents an explicit sandbox timeout or idle-shutdown parameter/config. No citation confirms configurable timeouts that auto-terminate sandboxes when idle or complete.
Cloudflare Sandboxesdisputedcontradicted3/10Docs only describe a per-command timeout within sessions (cloudflare-sandbox-docs-19), not an idle-shutdown/auto-destroy mechanism for the whole sandbox tied to billing; one community comment claims the underlying Container platform auto-shuts down after no requests (cloudflare-sandbox-comm-5), but a hands-on developer explicitly states it does not seem possible to auto-destroy a sandbox after an idle period, forcing manual cleanup (cloudflare-sandbox-comm-7). This is a direct contradiction between vendor-adjacent claims and hands-on experience. Missing for 10: explicit first-party docs on idle-timeout auto-shutdown/destroy tied to billing, and independent confirmation it works as described.
- [claimed-docs] “You can also set a default command timeout for all commands in a session”
- [community] “Cloudflares docs are written so hard for web dev. Can you host a monolith app that isn't serving http traffic on cloudflare tech like contai…”
- [community] “I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces…”
developerSpin up an isolated sandbox with one API/SDK call and get a live environment in seconds
weight 3 · round drawnDocs and SDK references (Python, JS/Go) show a single call (e.g. modal.Sandbox.create) spins up an isolated, ephemeral container/VM in seconds, with fast creation rates highlighted (hundreds of thousands of concurrent environments in seconds, faster backend for higher creation rates). Community builders (HN posts) independently confirm using Modal Sandboxes programmatically for real workloads like CI runners and agent runtimes. Missing for 10: independent benchmark of actual cold-start latency and a public OpenAPI spec (probe found 404s), so exact 'seconds' claim isn't third-party verified.
- [claimed-docs] “Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products dem…”
- [claimed-docs] “Spin up hundreds of thousands of concurrent rollout environments in seconds.”
- [claimed-docs] “Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
- [claimed-docs] “Create a Sandbox with the specified Image, and mount a Volume”
- [claimed-docs] “it's now possible to opt into a more performant Sandbox backend, which affords substantially higher creation rates and concurrency”
- [community] “Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runne…”
- [community] “Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runt…”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Docs show a single call (getSandbox()) creates or reconnects to an isolated VM/container sandbox with immediate access to exec, file, and session APIs (docs-37, docs-38, docs-50), and community discussion corroborates boot times in the 1-5 second range consistent with 'seconds' provisioning (comm-1, comm-8). Missing for 10: an independent benchmark or first-party latency SLA confirming consistent sub-few-second cold starts across load, since community notes it's slower than some Firecracker-based competitors.
- [claimed-docs] “getSandbox() - Gets or creates a sandbox instance by ID. Use a stable ID to reconnect to the same sandbox.”
- [claimed-docs] “VM-based isolation - Each sandbox runs in its own VM”
- [claimed-docs] “Each sandbox runs in its own isolated container with a full Linux environment, providing strong security boundaries while maintaining perfor…”
- [claimed-docs] “sandbox.exec() - Execute shell commands in the sandbox and capture stdout, stderr, and exit codes.”
- [community] “We rolled out our own that does pretty much the same thing but perhaps more... It does take 1-5 seconds to boot the environment (firecracker…”
- [community] “1-5 seconds seems high for Firecracker, depending on your requirements. We boot VMs (using Firecracker) at ~20-50ms.”
Preview access
developerExpose a port from the sandbox on a public preview URL to reach services running inside
weight 2 · round to Cloudflare SandboxesThe docs pack includes a 'Networking and security' guide for sandboxes (modal-docs-10) which is the likely location for tunnel/port-exposure features, but no evidence text actually describes exposing a port to a public preview URL, tunnel URLs, or reachability details. missing for 10: explicit documentation or example showing sandbox.tunnels()/port forwarding, a public URL format, and independent confirmation of accessing services inside a sandbox via that URL.
- [claimed-docs] “Networking and security”
- [claimed-docs] “VM Sandboxes Beta”
- [probe] “PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure conta…”
Docs explicitly describe exposing a sandbox port and getting a public preview URL, including zero-config *.trycloudflare.com URLs via sandbox.tunnels.get(port) and Worker-side routing of HTTP/WebSocket requests to the correct sandbox container, directly matching the story. Missing for 10: independent/hands-on confirmation of preview URL reliability beyond first-party docs.
- [claimed-docs] “Expose services running in the sandbox via preview URLs. Access web servers and APIs from the internet.”
- [claimed-docs] “Expose services on zero-config `*.trycloudflare.com` URLs via `sandbox.tunnels.get(port)`. Best for quick development and `.workers.dev` dep…”
- [claimed-docs] “Expose a port and get a preview URL for accessing services running in the sandbox.”
- [claimed-docs] “Expose services on zero-config `*.trycloudflare.com` URLs via `sandbox.tunnels.get(port)`.”
- [claimed-docs] “Expose services running in your sandbox via public preview URLs.”
- [claimed-docs] “Route incoming HTTP and WebSocket requests to the correct sandbox container.”
Snapshot persistence — stories about snapshot persistence in this arenaSnapshot persistence
Stories about snapshot persistence in this arena
Snapshots
developerPause a running sandbox and resume it later with filesystem and memory state intact
weight 2 · round to Cloudflare SandboxesModal has a dedicated 'Snapshots' doc for sandboxes suggesting some pause/resume-with-state capability, but the evidence pack only gives the page title with no detail on what state (filesystem/memory) is preserved or how resume works. Missing for 10: documented mechanics of snapshot/resume (filesystem+memory fidelity), explicit confirmation memory state is preserved, independent/hands-on validation of the feature working as described.
- [claimed-docs] “Snapshots”
- [claimed-docs] “Filesystem Access”
Docs confirm directory-level snapshot/restore to R2 ('Create point-in-time snapshots of directories and restore them from R2') and idle-state persistence claims, giving filesystem-level pause/resume support, but there's no documented mechanism for preserving in-memory/process execution state across a pause, and community discussion notes ambiguity about exactly how idle-state persistence works. missing for 10: explicit memory/process-state preservation on pause, clear docs on what 'idle state' persistence actually preserves, independent verification that restored sandboxes resume mid-execution rather than just filesystem contents.
- [claimed-docs] “Create point-in-time snapshots of directories and restore them from R2.”
- [claimed-docs] “Snapshot a sandbox directory to R2 and restore it later.”
- [claimed-docs] “Mount S3-compatible buckets (R2, S3, GCS) as local filesystems for persistent data storage across sandbox lifecycles.”
- [community] “There is an open question about how file persistence works. The docs claim they persist the filesystem even when they move the container to …”
developerSnapshot a sandbox and later restore or fork new sandboxes from that snapshot
weight 3 · round drawnModal explicitly documents a 'Sandbox Snapshots' feature (modal-docs-9), confirming snapshot capability exists, but the evidence pack gives no detail on the actual snapshot/restore/fork API mechanics, no confirmation that new independent sandboxes can be forked from a single snapshot, and no community/hands-on corroboration of this specific workflow. missing for 10: documented API details on restoring/forking multiple sandboxes from one snapshot, independent/hands-on confirmation of snapshot-restore-fork workflow, and clarity on any beta/experimental limitations.
- [claimed-docs] “Snapshots”
- [claimed-docs] “VM Sandboxes Beta”
Docs confirm directory-level snapshot/restore via R2 (docs-8, docs-48), but there's no evidence of forking new independent sandbox instances from a snapshot — only restoring a directory's state, and community reports flag ambiguity in how persistence/restore actually behaves (comm-4). missing for 10: explicit 'fork sandbox from snapshot' API, clarity on full sandbox-state (not just directory) restoration, and independent confirmation that restore reliably recreates a working sandbox.
- [claimed-docs] “Create point-in-time snapshots of directories and restore them from R2.”
- [claimed-docs] “Snapshot a sandbox directory to R2 and restore it later.”
- [community] “There is an open question about how file persistence works. The docs claim they persist the filesystem even when they move the container to …”
Not comparable on these axes
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · not comparableModal Sandboxesn/aModal Sandboxes is compute/sandbox infrastructure for running untrusted or agent code, not a data product that ingests user data and surfaces AI-generated insights/suggestions from it — this axis is a category error for this product type.
Cloudflare Sandboxesn/aCloudflare Sandboxes is a code-execution/infrastructure primitive (VM-isolated containers, file I/O, command exec) for building agents or apps, not a product with its own dataset or built-in AI insight/suggestion features over user data. The evidence pack shows only execution/session/file/tunnel APIs — no AI-generated insights or analysis output is a first-party product feature, making this axis a category mismatch rather than a gap.
- [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
- [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · not comparableModal Sandboxesn/aModal Sandboxes is an infrastructure platform for running untrusted/agent code, not itself an AI assistant or agent product; it provides no built-in assistant to delegate tasks to — this is a wrong-axis question for this product category.
Cloudflare Sandboxesn/aCloudflare Sandboxes is an infrastructure/execution-environment product (isolated containers/VMs for running code, files, commands) used by developers to build agents, not a product that itself ships a built-in AI assistant UI for end users to delegate tasks to. The 'delegate tasks to a built-in AI assistant' story is a category error for this kind of infra tool.
- [claimed-docs] “Sandboxes are ideal for building AI agents that need to execute code, interactive development environments, data analysis platforms, CI/CD s…”
- [claimed-docs] “The Sandbox SDK enables you to run untrusted code securely in isolated environments.”
ai-native userPrevent my data from being used to train AI models
weight 3 · not comparableModal Sandboxesn/aModal Sandboxes is an infrastructure/execution platform for running untrusted code, not an AI model or AI product with a training-data policy; there's no concept of 'training on user data' applicable to a sandbox execution service. This axis is a category error for this product type.
Cloudflare Sandboxesn/aCloudflare Sandboxes is a developer infrastructure product for running isolated code execution environments, not a consumer-facing AI assistant/chat product with a data-training opt-out policy; the evidence covers execution, filesystem, networking, and architecture features but nothing about AI model training data usage or opt-out controls. This axis is a category error for this product type.