Agent Sandboxes & Code Execution Arena
Runloop vs Maritime
Runloop wins · 20–9 (20 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round drawnDirect probe evidence confirms a working llms.txt file (HTTP 200) with a documentation index, plus docs-md endpoints explicitly pointing agents to fetch llms.txt for full documentation, and Cursor rules files further supporting agent-oriented consumption of docs. missing for 10: independent third-party confirmation that agents successfully use these docs in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.runloop.ai/docs/overview/what-is-runloop.md > ## Documentation Index > Fetch the complete documentat…”
- [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”
Maritime hosts a live llms.txt (confirmed by a 200 probe) plus a dedicated CLI llms.txt and an ai-agents docs page explicitly designed for agents (Claude Code, Cursor, etc.) to consume, authenticate, and act on — directly matching the story. Missing for 10: no independent third-party confirmation that agents actually navigate these docs successfully in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://maritime.sh/llms.txt # Maritime > Maritime is a cloud platform for deploying, hosting, and managing AI …”
- [claimed-docs] “Templates are cloneable agent packages; publishers can share them via link or the public library.”
- [claimed-docs] “Pass --json on EVERY command. Success: one JSON value on stdout, stderr empty.”
- [claimed-docs] “The Maritime CLI is built to be operated by other AI agents, not just humans: machine-readable output on every command, stable exit codes, a…”
- [claimed-docs] “Paste one prompt into Claude, Cursor, or any assistant and it can install the CLI, authenticate, and deploy agents for you.”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round drawnRunloop provides a CLI (rli), SDKs, an Agents API, GitHub Actions integration for deploying agents, and benchmark jobs runnable via CLI—all supporting headless/CI automation of sandboxes and agents. missing for 10: no independent/hands-on CI case study, and no explicit CI provider-specific guide beyond GitHub Actions example.
- [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
- [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
- [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
- [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
- [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”
Maritime provides a CLI with --json flag on every command, machine-readable output, stable exit codes, and env-var (non-interactive) authentication via MARITIME_TOKEN, explicitly designed for automation/CI use and control by other AI agents rather than humans, plus a RESTful API and idempotent provisioning for scripted workflows. This directly supports headless/CI operation across CLI and SDK/API paths. Missing for 10: no explicit CI pipeline example (e.g., GitHub Actions/GitLab CI snippet) or independent third-party confirmation of CI usage.
- [claimed-docs] “Pass --json on EVERY command. Success: one JSON value on stdout, stderr empty.”
- [claimed-docs] “The Maritime CLI is built to be operated by other AI agents, not just humans: machine-readable output on every command, stable exit codes, a…”
- [claimed-docs] “Authenticate with an env var, never interactively. Mint a long-lived key (`mk_…`) with `maritime keys create` and export it as `MARITIME_TOK…”
- [claimed-docs] “Mint a long-lived key (`mk_…`) with `maritime keys create` and export it as `MARITIME_TOKEN`. Keys never expire; browser-login tokens do.”
- [claimed-docs] “RESTful API for managing agents programmatically.”
- [claimed-docs] “provision is idempotent on it: it returns the existing agent if there is one, else creates a new one. Safe to call on every sign-in.”
- [probe] “official CLI documented at https://maritime.sh/docs/cli”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round drawnRunloopnone0/10Evidence shows Runloop publishes its own MCP server so external agents can call Runloop's tools (runloop-probe-4), but there is no evidence that Runloop itself can act as an MCP client and plug in third-party MCP servers to use their tools. missing for 10: any documentation of an MCP-client integration point, config for adding external MCP servers, or examples of Devboxes/agents consuming external MCP tool servers.
- [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”
Maritimenone0/10Maritime is a platform for deploying and hosting agents built with frameworks (CrewAI, LangGraph, etc.), but the evidence pack shows no mention of MCP server integration or a mechanism for agents to consume external MCP tool servers. Nothing describes plugging in MCP servers to extend tool use.
ai-native userConnect an agent via an official MCP server
weight 3 · round to RunloopRunloop is a sandbox/devbox platform (not itself an agent), so shipping an official MCP server is a valid axis, and the docs explicitly reference one at docs.runloop.ai/docs/tools/ai-tools. This confirms the capability exists, though the evidence pack lacks detail on the specific tools/resources exposed or independent hands-on confirmation. Missing for 10: detailed documentation of MCP server tool surface, independent/community verification of usage.
- [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”
Maritimenone0/10Maritime is a platform for deploying/hosting agents (not itself an agent), so an official MCP server is a fair axis to expect, but no evidence pack item mentions MCP, an MCP server, or MCP-compatible endpoints — only REST API, CLI, SDKs, and webhooks are documented.
- [claimed-docs] “RESTful API for managing agents programmatically.”
- [claimed-docs] “The Maritime CLI lets you deploy and manage agents from your terminal.”
- [claimed-docs] “Mint a personal API key with `maritime keys create` and send it as a Bearer token, the same token the CLI itself uses.”
ai-native userUse an official CLI
weight 2 · round to MaritimeRunloop ships an official CLI (`rli`) with both interactive TUI and traditional commands for managing devboxes, blueprints, SSH sessions, and benchmarks, documented directly in first-party docs and confirmed by probe. This is clearly built for AI-native/agentic workflows (blueprint creation, benchmark submission, devbox lifecycle management). Missing for 10: independent/hands-on community corroboration beyond vendor docs, and no changelog/version history showing active CLI maintenance cadence.
- [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
- [claimed-docs] “From the interactive menu, select a devbox and choose **SSH** to open a secure shell session directly into your devbox. The CLI handles all …”
- [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
- [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
- [probe] “official CLI documented at https://docs.runloop.ai/docs/tools/rl-cli”
Maritime ships an official npm-installable CLI explicitly designed for AI-native use: machine-readable/--json output on every command, stable exit codes, a self-describing contract, and env-var (non-interactive) authentication via MARITIME_TOKEN, all documented at /docs/cli and /docs/ai-agents. It also supports agent-driven install/auth/deploy flows via prompted assistants. Missing for 10: independent/hands-on verification beyond vendor docs.
- [claimed-docs] “The Maritime CLI lets you deploy and manage agents from your terminal.”
- [claimed-docs] “npm install -g maritime-cli”
- [claimed-docs] “The Maritime CLI is built to be operated by other AI agents, not just humans: machine-readable output on every command, stable exit codes, a…”
- [claimed-docs] “Authenticate with an env var, never interactively. Mint a long-lived key (`mk_…`) with `maritime keys create` and export it as `MARITIME_TOK…”
- [claimed-docs] “Pass --json on EVERY command. Success: one JSON value on stdout, stderr empty.”
- [claimed-docs] “Paste one prompt into Claude, Cursor, or any assistant and it can install the CLI, authenticate, and deploy agents for you.”
- [probe] “official CLI documented at https://maritime.sh/docs/cli”
ai-native userDrive the product through a documented public API
weight 3 · round drawnRunloop documents a full public API surface: Devboxes API, Agents API, SDKs in common languages, CLI (rli), and GitHub Action integration, all callable programmatically for agent-driven workflows. This is strong first-party documentation of a public API, though probes show no discoverable OpenAPI spec, slightly weakening independent verification. Missing for 10: a published OpenAPI/swagger spec and independent third-party corroboration of API usage.
- [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
- [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
- [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
- [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”
- [probe] “official CLI documented at https://docs.runloop.ai/docs/tools/rl-cli”
Maritime documents a full public API (REST endpoints, TypeScript/Python SDKs, idempotent provisioning, webhooks, scoped bearer-token auth) plus a CLI explicitly built for AI agents with machine-readable JSON output and stable exit codes, directly enabling AI-native driving of the platform. missing for 10: a discoverable machine-readable spec (openapi.json/swagger.json returned 404) and independent third-party corroboration of the API's stability.
- [claimed-docs] “RESTful API for managing agents programmatically.”
- [claimed-docs] “Mint a personal API key with `maritime keys create` and send it as a Bearer token, the same token the CLI itself uses.”
- [claimed-docs] “Tag each agent with your own id via `externalId`. `provision` is **idempotent** on it: it returns the existing agent if there is one, else c…”
- [claimed-docs] “provision is idempotent on it: it returns the existing agent if there is one, else creates a new one. Safe to call on every sign-in.”
- [claimed-docs] “Get notified when a customer's agent changes state, instead of polling. Subscribe a URL, and Maritime POSTs a signed JSON body on every life…”
- [claimed-docs] “The Maritime CLI is built to be operated by other AI agents, not just humans: machine-readable output on every command, stable exit codes, a…”
- [claimed-docs] “Authenticate with an env var, never interactively. Mint a long-lived key (`mk_…`) with `maritime keys create` and export it as `MARITIME_TOK…”
- [claimed-docs] “Pass --json on EVERY command. Success: one JSON value on stdout, stderr empty.”
- [probe] “PROBE openapi: all candidate paths 404 (https://maritime.sh/openapi.json, https://maritime.sh/swagger.json, https://maritime.sh/api/openapi.…”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to MaritimeRunloop's Agent Gateways let agents call LLM APIs like Anthropic/OpenAI 'without ever seeing your API keys,' which is a direct least-privilege credentialing mechanism for agents. However, the docs give only a single brief mention with no detail on fine-grained scopes, per-agent permission levels, revocation, or audit logging. Missing for 10: detailed scoping/permission model documentation, credential rotation/revocation controls, and independent verification of the gateway's isolation guarantees.
- [claimed-docs] “Agent Gateways let your agents call LLM APIs like Anthropic and OpenAI **without ever seeing your API keys**.”
- [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
Maritime documents scoped API keys explicitly: every key carries scopes, users should hand subsystems the narrowest key they need, and a dedicated example shows a Computers-only key being refused on all other endpoints — a concrete least-privilege mechanism. This pairs with CLI/env-based key minting (mk_ tokens) suited to agent automation. Missing for 10: an exhaustive scope catalog/reference and independent (non-vendor) verification of scope enforcement in practice.
- [claimed-docs] “Hand a subsystem the narrowest key it needs. A leaked key can then only do what its scope allows.”
- [claimed-docs] “To hand out a key that reaches computers and nothing else, create a **Computers key** in the Computers section of Settings, API keys”
- [claimed-docs] “Every key carries scopes. Hand a subsystem the _narrowest_ key it needs. A leaked key can then only do what its scope allows.”
- [claimed-docs] “a key whose only scope is `computers` is refused on every other Maritime endpoint”
- [claimed-docs] “Hand a subsystem the _narrowest_ key it needs. A leaked key can then only do what its scope allows.”
- [claimed-docs] “Authenticate with an env var, never interactively. Mint a long-lived key (`mk_…`) with `maritime keys create` and export it as `MARITIME_TOK…”
- [claimed-docs] “Mint a long-lived key (`mk_…`) with `maritime keys create` and export it as `MARITIME_TOKEN`. Keys never expire; browser-login tokens do.”
ai-native userBuild against official SDKs
weight 2 · round to MaritimeRunloop explicitly documents official SDKs in common languages for interacting with its API, plus tutorials integrating with SDKs like OpenAI Agents SDK, supporting an AI-native builder workflow. However, evidence lacks specifics on which languages are covered, versioning/maturity, or independent developer corroboration. Missing for 10: enumerated language list/maturity details, independent hands-on confirmation, changelog/versioning evidence.
- [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
- [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
- [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”
Maritime documents official TypeScript and Python SDKs with idempotent agent provisioning, scoped authentication, webhooks, messaging, and a Files API, plus an npm-installable CLI and RESTful API, all clearly aimed at building AI-native integrations. missing for 10: no public OpenAPI/swagger spec found (404s) and no independent/hands-on corroboration beyond first-party docs.
- [claimed-docs] “Tag each agent with your own id via `externalId`. `provision` is **idempotent** on it: it returns the existing agent if there is one, else c…”
- [claimed-docs] “Mint a personal API key with `maritime keys create` and send it as a Bearer token, the same token the CLI itself uses.”
- [claimed-docs] “Give every one of your users their own agent with the TypeScript or Python SDK.”
- [claimed-docs] “Send a message and wait for the reply. Sleeping agents wake automatically.”
- [claimed-docs] “RESTful API for managing agents programmatically.”
- [claimed-docs] “The Maritime Files API: eight endpoints and both SDKs to browse, upload, download, and edit an agent's disk over HTTP, even while the agent …”
- [claimed-docs] “provision is idempotent on it: it returns the existing agent if there is one, else creates a new one. Safe to call on every sign-in.”
- [probe] “PROBE openapi: all candidate paths 404 (https://maritime.sh/openapi.json, https://maritime.sh/swagger.json, https://maritime.sh/api/openapi.…”
ai-native userSubscribe to events via webhooks
weight 2 · round to MaritimeRunloopnone0/10No evidence mentions webhooks or event subscription mechanisms; Runloop's docs describe devbox lifecycle events via Axons event streams but nothing about webhook subscriptions for external systems. Missing for 10: any documentation of a webhooks API, subscription/callback URLs, or event delivery mechanism to external endpoints.
Docs explicitly describe subscribing a URL to receive signed JSON POSTs on every agent lifecycle event, replacing polling, which directly matches the story. missing for 10: independent/hands-on confirmation of webhook delivery reliability, payload schema details, and signature verification specifics.
- [claimed-docs] “Get notified when a customer's agent changes state, instead of polling. Subscribe a URL, and Maritime POSTs a signed JSON body on every life…”
- [claimed-docs] “Hand a subsystem the narrowest key it needs. A leaked key can then only do what its scope allows.”
Agentic features
ai-native userSet up automations that run autonomously in the background
weight 2 · round to MaritimeRunloop's Axons feature gives agents a persistent event stream letting them suspend when idle, wake on demand, recover state, and hand off between agents 'without you building the plumbing' (runloop-docs-17, runloop-docs-18), and devboxes support suspend/resume to preserve state across runs (runloop-docs-27). The GitHub Action for agent deployment (runloop-docs-24) supports triggering agent runs from CI workflows, enabling background automation setups. Missing for 10: no explicit cron/scheduler primitive, no independent/hands-on account of long-running unattended automations actually working in production, and no example of a full always-on autonomous pipeline beyond docs claims.
- [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
- [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”
- [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
- [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”
Maritime provides cron triggers with standard five-field expressions that wake agents on schedule and deliver prompts, plus agents can self-schedule their own wake triggers (e.g., 'send me a digest every weekday at 9'), and agents run persistently on always-on micro-VMs with sleep/wake economics, enabling true background autonomous automation. Missing for 10: independent/hands-on verification of trigger reliability and no mention of monitoring/alerting on automation failures.
- [claimed-docs] “A cron trigger takes a standard five-field expression, wakes the agent on schedule, and delivers its `prompt` as a chat message on every fir…”
- [claimed-docs] “Ask the agent (OpenClaw-family and Hermes templates): tell it "send me a digest every weekday at 9" and the platform syncs the schedule it s…”
- [claimed-docs] “tell it "send me a digest every weekday at 9" and the platform syncs the schedule it sets for itself into real wake triggers”
- [claimed-docs] “Always-on is a flat add-on for the rare agent that must react in milliseconds around the clock.”
- [claimed-docs] “Send a message and wait for the reply. Sleeping agents wake automatically.”
- [claimed-docs] “Maritime gives it what a laptop session can't: its own always-on micro-VM in the cloud, persistent storage, sleep/wake economics, and every …”
ai-native userOperate the product with natural-language commands
weight 2 · round drawnRunloop exposes an official MCP server and ships Cursor '.mdc' rule files that give AI assistants context on the SDK (devbox lifecycle, file ops, blueprints, snapshots, tunnels), enabling natural-language operation through AI coding assistants rather than a bespoke NL interface. There's no first-party chat/NL command surface of Runloop's own, and no independent evidence of how well the MCP/Cursor integration performs in practice. Missing for 10: dedicated natural-language interface docs, hands-on/independent validation of the MCP or Cursor integration actually working end-to-end.
- [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”
- [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”
- [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
Maritime documents natural-language operation at the agent level (chat with agents, cron-triggered prompts set via natural language like 'send me a digest every weekday at 9') and CLI/API designed to be driven by AI agents via prompts (paste a prompt into Claude/Cursor to install and operate). However, the core Maritime platform itself (dashboard, agent management) is not shown to be operable purely via natural-language commands by the end user — the NL interface is mediated through the CLI/API being agent-friendly rather than a native NL command surface for platform operations. missing for 10: evidence of a native NL interface for platform-level operations (not just agent chat/prompts), independent/hands-on confirmation of this workflow succeeding.
- [claimed-docs] “Ask the agent (OpenClaw-family and Hermes templates): tell it "send me a digest every weekday at 9" and the platform syncs the schedule it s…”
- [claimed-docs] “Paste one prompt into Claude, Cursor, or any assistant and it can install the CLI, authenticate, and deploy agents for you.”
- [claimed-docs] “tell it "send me a digest every weekday at 9" and the platform syncs the schedule it sets for itself into real wake triggers”
- [claimed-docs] “The Maritime CLI is built to be operated by other AI agents, not just humans: machine-readable output on every command, stable exit codes, a…”
- [claimed-docs] “point Claude Code or your Maritime agent at a provider's runbook page and it can inventory the old provider with read-only credentials, recr…”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnRunloopnone0/10No evidence of an interactive API reference with runnable examples; the explicit probe for OpenAPI/swagger specs returned 404s across all candidate paths, and docs mention SDKs/CLI but not a runnable API explorer.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”
Maritimenone0/10Docs describe a RESTful API and CLI but there's no evidence of an interactive API reference (e.g., Swagger/OpenAPI explorer) with runnable examples; probes explicitly show no openapi.json/swagger.json spec found at any candidate path. missing for 10: interactive API console/playground, OpenAPI spec, runnable code examples in docs.
- [claimed-docs] “RESTful API for managing agents programmatically.”
- [claimed-docs] “Mint a personal API key with `maritime keys create` and send it as a Bearer token, the same token the CLI itself uses.”
- [probe] “PROBE openapi: all candidate paths 404 (https://maritime.sh/openapi.json, https://maritime.sh/swagger.json, https://maritime.sh/api/openapi.…”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round drawnRunloopnone0/10Runloop offers SDKs and a CLI but there is no evidence of a downloadable OpenAPI/Swagger spec; explicit probes to common OpenAPI paths (openapi.json, swagger.json, etc.) all returned 404, confirming absence rather than just lack of mention.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”
- [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
Maritimenone0/10Maritime documents a RESTful API and CLI with --json output, but there is no evidence of a downloadable machine-readable spec (OpenAPI/Swagger); active probes for openapi.json, swagger.json, and similar paths all returned 404.
- [probe] “PROBE openapi: all candidate paths 404 (https://maritime.sh/openapi.json, https://maritime.sh/swagger.json, https://maritime.sh/api/openapi.…”
- [claimed-docs] “RESTful API for managing agents programmatically.”
- [claimed-docs] “Mint a personal API key with `maritime keys create` and send it as a Bearer token, the same token the CLI itself uses.”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to RunloopRunloop's Devboxes are isolated sandbox execution environments (secure, snapshot/blueprint-based, no production data by default) explicitly designed for testing AI agents, fully separate from any production system. missing for 10: no independent/hands-on corroboration beyond vendor docs, no explicit statement contrasting sandbox vs production data isolation guarantees.
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
- [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
- [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
- [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
- [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”
Maritimenone0/10Maritime's docs cover deployment, keys/scopes, templates, triggers, and agent lifecycle, but nothing describes a sandbox/staging environment isolated from production data for testing purposes. Missing for 10: any mention of a sandbox/test mode, staging environment, or data isolation guarantees between test and production agents.
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnRunloopnone0/10No evidence of API versioning scheme or a documented deprecation policy; OpenAPI spec probe returned 404s and no changelog/versioning docs are cited.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”
Maritimenone0/10No evidence of API versioning scheme (e.g., v1/v2 paths) or any documented deprecation policy; OpenAPI spec probes all 404, and no docs mention version support windows or breaking-change policy.
- [probe] “PROBE openapi: all candidate paths 404 (https://maritime.sh/openapi.json, https://maritime.sh/swagger.json, https://maritime.sh/api/openapi.…”
- [claimed-docs] “RESTful API for managing agents programmatically.”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to RunloopRunloop supports running 10k+ parallel sandboxes, benchmark jobs that provision devboxes/run agents/aggregate outputs 'with a single command,' and snapshots for fan-out across multiple approaches — evidence of bulk/parallel operation across many devboxes. However, there's no explicit documentation of a generic bulk-operation API/CLI verb (e.g., bulk delete, bulk update, batch create) for arbitrary resources beyond devboxes/benchmarks. missing for 10: explicit bulk CRUD operations across resources like snapshots/storage objects/secrets, documented batch API endpoints, and independent evidence of large-scale bulk usage beyond vendor docs.
- [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
- [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
- [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
- [claimed-docs] “Run your agent against well-known open source benchmarks like terminal bench 2, AIME, and more.”
There is explicit support for bulk import of config/env vars (maritime-docs-11) and a scriptable CLI/API with JSON output (maritime-docs-9, maritime-docs-36) that an AI-native user could loop over to perform bulk actions, but there is no documented batch/bulk endpoint for operating on many agents or resources at once. missing for 10: a dedicated bulk-operation API/CLI command (e.g. bulk create/delete/update agents), evidence of scaling to many items reliably, and independent confirmation of bulk workflows in practice.
- [claimed-docs] “Bulk import from a .env file (or stdin), then hot-reload”
- [claimed-docs] “Pass --json on EVERY command. Success: one JSON value on stdout, stderr empty.”
- [claimed-docs] “RESTful API for managing agents programmatically.”
- [claimed-docs] “The Maritime CLI is built to be operated by other AI agents, not just humans: machine-readable output on every command, stable exit codes, a…”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to MaritimeRunloop's Axons give agents a persistent event stream that can suspend on idle, wake on demand, recover state, and hand off between agents, which is a form of automatic event-driven behavior, and the Broker component sits between Axon and agent process to manage this. However, there's no documentation of a general user-facing rules engine where arbitrary conditions map to arbitrary actions across the platform (e.g., webhooks, triggers on Devbox/snapshot events, custom rule definitions). Missing for 10: a documented rules/trigger definition interface, examples of custom event-to-action mappings beyond agent suspend/wake, and evidence of triggers on other platform events (snapshots, storage, secrets).
- [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
- [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”
Maritime supports triggers (cron with five-field expressions, natural-language schedule setting, webhooks for lifecycle events) that wake agents and fire actions automatically, which covers rule-based automation on events. However, this is event/schedule-based triggering rather than a general-purpose conditional rules engine (e.g., no evidence of arbitrary if/then logic across multiple event types or complex conditions). missing for 10: evidence of a general conditional rules/policy engine beyond cron and webhook triggers, and independent/hands-on confirmation of trigger reliability.
- [claimed-docs] “A cron trigger takes a standard five-field expression, wakes the agent on schedule, and delivers its `prompt` as a chat message on every fir…”
- [claimed-docs] “Ask the agent (OpenClaw-family and Hermes templates): tell it "send me a digest every weekday at 9" and the platform syncs the schedule it s…”
- [claimed-docs] “tell it "send me a digest every weekday at 9" and the platform syncs the schedule it sets for itself into real wake triggers”
- [claimed-docs] “Get notified when a customer's agent changes state, instead of polling. Subscribe a URL, and Maritime POSTs a signed JSON body on every life…”
- [claimed-docs] “Always-on is a flat add-on for the rare agent that must react in milliseconds around the clock.”
ai-native userSchedule recurring jobs or workflows
weight 2 · round to MaritimeRunloopnone0/10The evidence describes Devboxes, snapshots, benchmarks, and Axons for event-driven suspend/resume, but nothing about scheduling recurring jobs or cron-like recurring workflows. Absence of evidence for this applicable automation capability yields 'none'.
Maritime explicitly supports cron triggers with standard five-field expressions that wake agents and deliver prompts on schedule, plus a natural-language flow where an agent sets its own recurring schedule ('send me a digest every weekday at 9') that syncs into real wake triggers. This directly covers recurring job/workflow scheduling for AI-native users. Missing for 10: independent/hands-on verification beyond vendor docs, and no detail on managing/editing/cancelling existing scheduled triggers via CLI/API.
- [claimed-docs] “A cron trigger takes a standard five-field expression, wakes the agent on schedule, and delivers its `prompt` as a chat message on every fir…”
- [claimed-docs] “Ask the agent (OpenClaw-family and Hermes templates): tell it "send me a digest every weekday at 9" and the platform syncs the schedule it s…”
- [claimed-docs] “tell it "send me a digest every weekday at 9" and the platform syncs the schedule it sets for itself into real wake triggers”
- [claimed-docs] “Always-on is a flat add-on for the rare agent that must react in milliseconds around the clock.”
ai-native userVersion, review, and roll back my automations
weight 1 · round to RunloopRunloop's Snapshots explicitly support rolling back Devbox state to a known good point, and Blueprints let you version/reuse startup configurations, giving a rudimentary rollback mechanism for agent environments. However, there is no evidence of a review workflow (diffs, approvals, audit trail) or of 'automations' as a first-class versioned object distinct from raw disk/environment snapshots. Missing for 10: explicit review/diff tooling for automations, first-class automation versioning (not just devbox/environment snapshots), audit/history UI for changes.
- [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
- [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
- [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
Capabilities hardware — stories about capabilities hardware in this arenaCapabilities hardware
Stories about capabilities hardware in this arena
Agent workloads
ai-native userRun coding agents like Claude Code or Codex inside the sandbox following the vendor's own recipe
weight 2 · round to MaritimeRunloop documents running AI coding agents generally inside Devboxes (blueprints, benchmarks, Agents API, OpenAI Agents SDK tutorial) but no specific vendor recipe or tutorial naming Claude Code or Codex is present in the evidence. missing for 10: explicit Claude Code/Codex quickstart or blueprint, named integration guide for these specific agents.
- [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
- [claimed-docs] “Run your agent against well-known open source benchmarks like terminal bench 2, AIME, and more.”
- [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
- [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
- [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
- [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”
Maritime documents dedicated template recipes for running Claude Code and Codex inside its sandboxed micro-VMs (persistent storage, sleep/wake, metered LLM access by default, chat channels), directly matching the vendor's own recipe framing for these named coding agents. missing for 10: independent/hands-on verification beyond vendor docs, and no direct evidence of a step-by-step 'follow this exact recipe' walkthrough output for a live run.
- [claimed-docs] “Maritime gives it what a laptop session can't: its own always-on micro-VM in the cloud, persistent storage, sleep/wake economics, and every …”
- [claimed-docs] “No key is required: the agent uses Maritime's metered LLM access by default.”
- [claimed-docs] “Every ready-made template is a real agent framework, pre-configured and one image pull from live. Pick one, name it, deploy.”
- [claimed-docs] “point Claude Code or your Maritime agent at a provider's runbook page and it can inventory the old provider with read-only credentials, recr…”
- [claimed-docs] “Paste one prompt into Claude, Cursor, or any assistant and it can install the CLI, authenticate, and deploy agents for you.”
Workloads
developerRun a headless browser or full desktop environment inside the sandbox
weight 2 · round to MaritimeRunloopnone0/10No evidence Runloop supports running a headless browser or desktop GUI inside its Devbox sandboxes; the only browser-related tutorial explicitly states the architecture avoids this, running Chromium externally via Browserbase and connecting over CDP so 'no Chromium ever runs in the devbox.' Blueprints/docs never mention GUI, X11, VNC, or bundled browser binaries as a supported environment type.
- [claimed-docs] “the agent runs in a devbox, the browser runs on Browserbase, and the devbox drives it by connecting Playwright over CDP to the session’s con…”
Maritime docs explicitly describe 'Computers': a persistent Linux desktop (XFCE, Chromium, LibreOffice) per user driven by a model, and the changelog confirms agents get a real Linux desktop with screenshot/mouse/keyboard control and live takeover — this covers full desktop environments including a browser (Chromium) inside the sandbox. missing for 10: no independent/hands-on corroboration of performance or headless-only browser mode, and no detail on resource limits or concurrency.
- [claimed-docs] “A persistent Linux desktop (XFCE, Chromium, LibreOffice) for each of _your_ end users, driven by _your_ model.”
- [claimed-docs] “Agents are getting a real Linux desktop they can drive: screenshot, mouse, keyboard, and a live view you can take over.”
- [claimed-docs] “To hand out a key that reaches computers and nothing else, create a **Computers key** in the Computers section of Settings, API keys”
developerAttach GPUs to sandboxed workloads
weight 2 · round drawnRunloopnone0/10The evidence pack describes Devboxes, blueprints, snapshots, mounts, storage, and networking features but never mentions GPU support or GPU-attached compute for sandboxed workloads; nothing indicates hardware acceleration is available.
Maritimenone0/10No evidence anywhere in the pack mentions GPUs, GPU-accelerated instances, or hardware acceleration for sandboxed agent workloads; all compute descriptions reference micro-VMs, sleep/wake, and CPU-based containers only. Missing for 10: any mention of GPU attachment, GPU instance types, or hardware accelerator support.
- [claimed-docs] “Maritime gives it what a laptop session can't: its own always-on micro-VM in the cloud, persistent storage, sleep/wake economics, and every …”
- [claimed-docs] “Any Docker container becomes an agent: three endpoints get you sleep/wake, channels, and triggers.”
Code execution — stories about code execution in this arenaCode execution
Stories about code execution in this arena
Files
developerRead, write, upload, and download files in the sandbox filesystem via the SDK
weight 2 · round to RunloopDocs confirm the SDK/CLI supports 'file operations' as part of Devbox management (mentioned alongside lifecycle, command execution, etc. in the Cursor rules integration), implying read/write file capability, but no dedicated documentation page or code sample specifically shows upload/download or read/write file APIs. Missing for 10: explicit SDK method docs for file read/write, and explicit upload/download examples or independent corroboration.
- [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
- [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
Maritimenone0/10No evidence pack item mentions SDK-based file read/write/upload/download operations against a sandbox filesystem; the docs cover agent provisioning, webhooks, auth scoping, triggers, computers (desktop), CLI, and migration but nothing about filesystem file I/O via SDK. Missing for full: no filesystem API/SDK method docs, no upload/download endpoint references, no code samples showing file operations.
Runtimes
developerDefine custom sandbox templates or bring my own container image
weight 2 · round to RunloopRunloop's Blueprints support building custom sandbox images from a Dockerfile (`rli blueprint from-dockerfile`), plus reusable prebuilt blueprints and snapshotting of custom states, directly enabling developers to bring their own container image or define custom templates. Missing for 10: independent/hands-on verification of Dockerfile-based blueprint builds and more detail on supported image registries or base image constraints.
- [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
- [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”
- [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
- [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
- [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
Docs confirm bring-your-own-image support via a documented three-endpoint contract that works with any Docker image (CrewAI, LangGraph, AutoGen, custom scripts), plus a build log and template system for packaging custom agents. However, there's no detailed spec of the contract itself, no example custom Dockerfile, and no independent/hands-on confirmation of successfully running a fully custom image end-to-end. missing for 10: full technical spec of the three-endpoint contract, a worked example of a custom image deployment, independent verification of custom image support beyond docs claims.
- [claimed-docs] “CrewAI, LangGraph, AutoGen, your own Python script: any Docker image that speaks a three-endpoint contract gets the whole platform.”
- [claimed-docs] “Templates are cloneable agent packages; publishers can share them via link or the public library.”
- [claimed-docs] “The agent page streams the build log live under **Overview → Builds**.”
- [claimed-docs] “Every ready-made template is a real agent framework, pre-configured and one image pull from live. Pick one, name it, deploy.”
developerExecute code in multiple language runtimes (Python, JavaScript, and more) and get rich results back
weight 2 · round to RunloopRunloop's Devboxes are described as full-featured secure sandboxed execution environments that can run arbitrary commands (via blueprints, SSH, CLI) which implies multi-language support, but there is no explicit documentation of a code-execution API returning 'rich results' (e.g., structured outputs, images, dataframes) as in dedicated code-interpreter products. missing for 10: explicit multi-language runtime examples (Python/JS/etc.), rich/structured result formatting (e.g., stdout+images+tables), and independent confirmation of these capabilities.
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
- [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
- [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
- [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
Maritimenone0/10Maritime's evidence describes deploying persistent Docker-based agents (any Docker image with sleep/wake/channels/triggers), a Files API, and a Linux desktop 'Computers' feature, but nothing describes a code-execution/sandbox API that runs snippets in multiple language runtimes (Python, JS, etc.) and returns structured/rich results (e.g., stdout, images, dataframes) the way a code-interpreter tool would.
- [claimed-docs] “Any Docker container becomes an agent: three endpoints get you sleep/wake, channels, and triggers.”
- [claimed-docs] “A persistent Linux desktop (XFCE, Chromium, LibreOffice) for each of _your_ end users, driven by _your_ model.”
- [claimed-docs] “The Maritime Files API: eight endpoints and both SDKs to browse, upload, download, and edit an agent's disk over HTTP, even while the agent …”
- [claimed-docs] “CrewAI, LangGraph, AutoGen, your own Python script: any Docker image that speaks a three-endpoint contract gets the whole platform.”
Untrusted code
developerExecute untrusted, AI-generated code without risking my own infrastructure
weight 3 · round to RunloopRunloop's core product is Devboxes, secure sandboxed execution environments purpose-built to run AI agent/code workloads isolated from user infrastructure, with snapshots, tunnels, secrets isolation, and scaling to 10k+ parallel sandboxes. This directly matches the story of executing untrusted AI-generated code without risking one's own infra. Missing for 10: independent/third-party hands-on validation of isolation guarantees, and explicit documentation of the underlying sandboxing/virtualization technology or security boundary details.
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
- [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
- [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
- [claimed-docs] “Let’s use devbox tunnels to securely access ports on your devbox over a simple url.”
- [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”
Maritime hosts agents as Docker images on 'serverless infrastructure' (maritime-docs-1, maritime-probe-1) and provides persistent isolated Linux desktops per end user (maritime-docs-5), which implies agent code runs off the developer's own machines/infra. However, there is no explicit documentation of sandboxing guarantees, resource/network isolation, or security boundaries specifically for running untrusted or AI-generated code safely. Missing for 10: explicit sandbox/isolation security model, resource-limit guarantees, and any independent verification that untrusted code cannot escape or affect other tenants/infra.
- [claimed-docs] “CrewAI, LangGraph, AutoGen, your own Python script: any Docker image that speaks a three-endpoint contract gets the whole platform.”
- [claimed-docs] “A persistent Linux desktop (XFCE, Chromium, LibreOffice) for each of _your_ end users, driven by _your_ model.”
- [probe] “PROBE llms.txt: HTTP 200 at https://maritime.sh/llms.txt # Maritime > Maritime is a cloud platform for deploying, hosting, and managing AI …”
developerRun arbitrary shell commands and install packages inside the sandbox
weight 2 · round to RunloopRunloop Devboxes are full Linux execution environments with SSH access via the CLI (runloop-docs-9) and blueprints built from Dockerfiles that install packages/dependencies at startup (runloop-docs-2, runloop-docs-14), implying arbitrary shell command execution and package installation. However, the evidence pack lacks a direct hands-on example of running an explicit shell command or apt/pip install inside a Devbox. missing for 10: explicit example/output of running an arbitrary shell command and installing a package inside a Devbox, independent third-party confirmation.
- [claimed-docs] “From the interactive menu, select a devbox and choose **SSH** to open a secure shell session directly into your devbox. The CLI handles all …”
- [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
- [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
Maritimenone0/10Maritime's docs describe deploying agents via Docker images, templates, and a custom framework contract, but there is no evidence of an interactive shell/terminal access or arbitrary package installation inside a running sandbox for a developer to use directly. The 'Computers' feature offers a persistent desktop for end-users, not a shell for arbitrary command execution by the developer.
Dev experience — day-to-day developer experience — setup friction, docs, debugging, iteration speedDev experience
Day-to-day developer experience — setup friction, docs, debugging, iteration speed
Agent lifecycle
ai-native userMy agent can provision its own sandbox, execute code, read the results, and tear it down — end to end without a human
weight 3 · round to RunloopRunloop's core product is programmatic sandbox (Devbox) provisioning, execution, and teardown via API/SDK/CLI, explicitly designed for AI agents — including an Agents API for creating/managing/deploying agents, blueprints for reusable environments, snapshots for state capture, and suspend/resume lifecycle, all achievable without human intervention (docs-1, docs-13, docs-22, docs-23, docs-2, docs-4, docs-27, docs-12). The quickstart shows a devbox spun up in ~1 minute and CLI/SDK access covers execution and result retrieval end-to-end. missing for 10: independent/hands-on third-party verification of a full agent-driven provision→execute→teardown loop, and a concrete code example showing an agent reading execution results programmatically.
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
- [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
- [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
- [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
- [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
- [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
- [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
- [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”
Maritime's CLI and SDK let an agent provision (idempotent) itself, run in a sandboxed micro-VM/container, and be managed via authenticated API/CLI calls with JSON output designed for machine consumption, and agents can be torn down/put to sleep automatically when idle. However, the evidence is docs-only with no hands-on trace of an agent actually reading back code-execution results or explicit 'delete/destroy' teardown call, and no independent verification of the full autonomous loop. missing for 10: concrete evidence of an agent reading execution output programmatically, an explicit teardown/delete API example, and independent/hands-on confirmation of the full provision→execute→read→teardown loop.
- [claimed-docs] “Tag each agent with your own id via `externalId`. `provision` is **idempotent** on it: it returns the existing agent if there is one, else c…”
- [claimed-docs] “Pass --json on EVERY command. Success: one JSON value on stdout, stderr empty.”
- [claimed-docs] “The Maritime CLI is built to be operated by other AI agents, not just humans: machine-readable output on every command, stable exit codes, a…”
- [claimed-docs] “Authenticate with an env var, never interactively. Mint a long-lived key (`mk_…`) with `maritime keys create` and export it as `MARITIME_TOK…”
- [claimed-docs] “Send a message and wait for the reply. Sleeping agents wake automatically.”
- [claimed-docs] “provision is idempotent on it: it returns the existing agent if there is one, else creates a new one. Safe to call on every sign-in.”
- [claimed-docs] “Any Docker container becomes an agent: three endpoints get you sleep/wake, channels, and triggers.”
Isolation security — stories about isolation security in this arenaIsolation security
Stories about isolation security in this arena
Isolation
platform-engineerRely on a documented hard isolation boundary (microVM or equivalent) between sandboxes and my systems
weight 3 · round to RunloopRunloop repeatedly describes Devboxes as "secure sandboxed execution environments" and "isolated Linux sandbox environments," implying some isolation boundary, but the evidence never names or documents the actual isolation mechanism (e.g., microVM, Firecracker, gVisor, container-only) or any architecture diagram/security whitepaper explaining the boundary. missing for 10: explicit documentation of the underlying isolation technology (microVM/hypervisor-level or equivalent), any security architecture doc or third-party audit confirming hard isolation guarantees.
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
- [claimed-docs] “Runloop allows you to run benchmarks at scale in a secure environment.”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”
Maritimenone0/10No evidence pack item documents a specific isolation mechanism (microVM, gVisor, container hardening, VM boundary) between sandboxes and host systems; docs mention Docker images, serverless infra, and scoped API keys but not the underlying execution isolation boundary. Missing for 10: any explicit statement of sandbox isolation technology, security model documentation, or third-party audit/confirmation of hard isolation.
ai-native userGive an agent a sandbox where host secrets and credentials are unreachable by the code it runs
weight 2 · round drawnRunloop's Devboxes are documented as sandboxed execution environments with encrypted secrets and, notably, Agent Gateways that let agents call LLM APIs 'without ever seeing your API keys' — directly supporting the isolation-of-credentials use case. However, secrets are also stated to be 'automatically made available as environment variables' inside Devboxes, which is somewhat in tension with 'unreachable by code' and isn't clarified as scoped/restricted; there's no independent/hands-on verification of secret unreachability or a security audit. missing for 10: independent verification of secret isolation, clarity on env-var secret exposure vs true unreachability, and details on network/filesystem-level credential isolation guarantees.
- [claimed-docs] “Agent Gateways let your agents call LLM APIs like Anthropic and OpenAI **without ever seeing your API keys**.”
- [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
Maritime's docs describe per-agent micro-VM isolation with secrets 'injected only into that agent's VM' and encrypted at rest, plus scoped API keys so a leaked key 'can only do what its scope allows,' which together imply host-level credentials aren't reachable by agent code. However, there's no explicit documentation of sandbox mechanics like blocking cloud metadata endpoints, filesystem isolation from the host, or independent security testing confirming secrets are truly unreachable. missing for 10: explicit host-credential isolation guarantees (e.g., no metadata service access, filesystem boundary details), independent/hands-on security verification.
- [claimed-docs] “Variables are **secret by default**: stored encrypted at rest, injected only into that agent's VM, and always masked in the UI.”
- [claimed-docs] “Hand a subsystem the narrowest key it needs. A leaked key can then only do what its scope allows.”
- [claimed-docs] “To hand out a key that reaches computers and nothing else, create a **Computers key** in the Computers section of Settings, API keys”
- [claimed-docs] “a key whose only scope is `computers` is refused on every other Maritime endpoint”
- [claimed-docs] “Every key carries scopes. Hand a subsystem the _narrowest_ key it needs. A leaked key can then only do what its scope allows.”
Network policy
platform-engineerRestrict or allow the sandbox's network egress with explicit policy
weight 3 · round drawnRunloopnone0/10No evidence pack item describes network egress controls, firewall rules, or allow/deny policies for Devbox sandboxes; tunnels and secrets features are documented but network egress policy configuration is never mentioned.
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round drawnRunloop is API-first: the CLI (rli) wraps SDKs/API for devboxes, blueprints, snapshots, benchmarks, agents, secrets, etc., suggesting broad API/CLI parity with any dashboard functionality. However there is no explicit statement claiming full UI/API feature parity, no documented OpenAPI spec (404s on all probed paths), and no direct dashboard-vs-API comparison. Missing for 10: explicit parity claim between web dashboard and API, published OpenAPI/reference spec, confirmation that all dashboard-only actions (e.g., billing, org settings) are also API-accessible.
- [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
- [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
- [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
- [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”
Docs show a broad API/SDK/CLI surface (provisioning, keys, webhooks, triggers, computers, templates, builds) and CLI --json mode implying the CLI itself is just an API wrapper, suggesting an API-first architecture. However there's no explicit statement of full UI/API parity and no discoverable OpenAPI spec (probes return 404s for all schema locations), so completeness of API coverage relative to every UI feature can't be confirmed. missing for 10: explicit parity statement, published OpenAPI/API reference, independent confirmation that every UI action has an API equivalent.
- [claimed-docs] “Mint a personal API key with `maritime keys create` and send it as a Bearer token, the same token the CLI itself uses.”
- [claimed-docs] “Pass --json on EVERY command. Success: one JSON value on stdout, stderr empty.”
- [claimed-docs] “Tag each agent with your own id via `externalId`. `provision` is **idempotent** on it: it returns the existing agent if there is one, else c…”
- [claimed-docs] “Hand a subsystem the narrowest key it needs. A leaked key can then only do what its scope allows.”
- [claimed-docs] “A persistent Linux desktop (XFCE, Chromium, LibreOffice) for each of _your_ end users, driven by _your_ model.”
- [claimed-docs] “The agent page streams the build log live under **Overview → Builds**.”
- [claimed-docs] “Templates are cloneable agent packages; publishers can share them via link or the public library.”
- [probe] “PROBE openapi: all candidate paths 404 (https://maritime.sh/openapi.json, https://maritime.sh/swagger.json, https://maritime.sh/api/openapi.…”
ai-native userExport all of my data in open formats and leave
weight 3 · round to RunloopRunloop offers Snapshots to save Devbox disk state and export/re-create Devboxes, and Dockerfile-based blueprints imply portability of environments, but there is no documented feature for exporting all account/user data (benchmarks, configs, agent history) in open formats or a clear data-export/account-deletion workflow. missing for 10: explicit full account data export tool, open-format guarantees for all data types, data portability/deletion documentation.
- [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
- [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
- [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”
Maritimenone0/10Evidence shows import/migration INTO Maritime (from AWS/Heroku/VPS, .env bulk import) and a CLI with --json output, but nothing documents exporting agent configs, data, secrets, or state OUT in open formats to leave the platform. missing for 10: an export command or documented data-portability feature, open-format export of agent state/configs, any migration-out guide.
- [claimed-docs] “Bulk import from a .env file (or stdin), then hot-reload”
- [claimed-docs] “Migrate from AWS, Heroku, or a VPS”
- [claimed-docs] “Pass --json on EVERY command. Success: one JSON value on stdout, stderr empty.”
ai-native userRead the product's source under an open license
weight 2 · round drawnRunloopnone0/10No evidence indicates Runloop's core source (the sandbox platform, Devboxes, Axons, etc.) is released under an open license; the evidence pack shows only docs, CLI, and SDK references without any license or open-source repository disclosure.
Maritimenone0/10Maritime is a closed commercial SaaS platform; there is no evidence of an open-source license or public source repository anywhere in the docs or probes. missing for 10: an open-source license, a public source code repository, any mention of licensing terms for the platform's own code.
ai-native userSelf-host the core product
weight 3 · round drawnRunloopnone0/10Runloop is presented entirely as a hosted cloud platform (Devboxes, Agents API, benchmarks) with no evidence of an on-premises or self-hostable deployment option, open-source core, or self-hosting documentation.
Maritimenone0/10Maritime is explicitly described as a cloud platform with agents running on serverless infrastructure, and no documentation offers a self-hosted deployment option for the core platform itself (Docker images are for agent frameworks running atop Maritime's cloud, not for self-hosting Maritime).
- [probe] “PROBE llms.txt: HTTP 200 at https://maritime.sh/llms.txt # Maritime > Maritime is a cloud platform for deploying, hosting, and managing AI …”
- [claimed-docs] “Migrate from AWS, Heroku, or a VPS”
Performance scale — stories about performance scale in this arenaPerformance scale
Stories about performance scale in this arena
Latency
developerStart sandboxes with documented sub-second-to-few-second cold starts
weight 3 · round to RunloopRunloop's marketing site documents '10GB image startup time in <2s' and a 1-minute quickstart, supporting sub-second-to-few-second cold starts, but this is a single first-party marketing claim without detailed benchmarks, methodology, or independent corroboration. missing for 10: independent/third-party benchmark confirmation, documentation of cold start times across varying image sizes/configurations, and reproducible performance data beyond the single homepage claim.
- [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
- [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”
Maritimenone0/10Maritime's docs describe sleep/wake behavior (e.g., 'Sleeping agents wake automatically', 'puts it to sleep when idle, and wakes it on the next visit') and an 'always-on' add-on for millisecond reaction, but no evidence anywhere documents actual cold-start latency numbers (sub-second to few-second) for waking a sandboxed agent from sleep.
- [claimed-docs] “Send a message and wait for the reply. Sleeping agents wake automatically.”
- [claimed-docs] “Any repo with a Dockerfile can be served on a public, no-login HTTPS URL. Maritime builds it, runs it serverlessly, puts it to sleep when id…”
- [claimed-docs] “Always-on is a flat add-on for the rare agent that must react in milliseconds around the clock.”
Scale
platform-engineerRun large concurrent fleets of sandboxes with documented concurrency limits
weight 3 · round to RunloopRunloop's marketing site claims support for 10k+ parallel sandboxes and sub-2s startup for 10GB images, showing scale is a design goal, but there is no documented concurrency limit, quota tiering, or rate-limit table a platform engineer could plan capacity around. Missing for 10: explicit documented per-account/org concurrency limits, guidance on requesting increases, and independent benchmarks validating the 10k+ claim at sustained load.
- [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
developerKeep a sandbox session running for hours or days for long agent tasks
weight 2 · round to RunloopRunloop Devboxes explicitly support long-running persistence via snapshots (save/restore disk state), suspend/resume to pause and continue exactly where left off, and Axons providing a persistent event stream that lets agents suspend when idle and wake on demand/recover state — directly addressing multi-hour/multi-day agent sessions without losing progress. Combined with SSH/tunnel access and mounts/secrets for long tasks, this covers the core story well. Missing for 10: independent/hands-on evidence (e.g., a third-party report) confirming multi-day session stability at scale, and explicit documented max session duration/limits.
- [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
- [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
- [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
- [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
- [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”
Maritime advertises serverless infra by default with an 'always-on' add-on for agents that must run continuously, and a persistent Linux desktop model for end users, suggesting long-lived sessions are possible but only via a paid add-on rather than a default guarantee for hours/days-long sandbox sessions. missing for 10: explicit documentation of session/session-timeout limits, confirmation that a sandbox session can run unattended for hours or days without the always-on add-on, and independent verification of long-running task stability.
- [claimed-docs] “Always-on is a flat add-on for the rare agent that must react in milliseconds around the clock.”
- [claimed-docs] “A persistent Linux desktop (XFCE, Chromium, LibreOffice) for each of _your_ end users, driven by _your_ model.”
- [probe] “PROBE llms.txt: HTTP 200 at https://maritime.sh/llms.txt # Maritime > Maritime is a cloud platform for deploying, hosting, and managing AI …”
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
Pricing
developerStart building with a free tier or included credits without talking to sales
weight 1 · round drawnDocs explicitly state Runloop's trial includes $50 free credits with no credit card required, and a quickstart lets a developer spin up a first devbox in about a minute, both without any sales contact. Missing for 10: independent/community confirmation of the free-tier signup experience and clarity on long-term free-tier limits beyond trial credits.
- [claimed-docs] “It includes **$50 in free credits**, and no credit card is required to sign up.”
- [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”
Docs explicitly state the free plan includes 3 agents with no card required, and quickstart/CLI docs show a self-serve path (npm install, deploy in under 5 minutes) with no sales contact implied anywhere. missing for 10: independent/third-party confirmation of the free tier's real-world limits and no hands-on verification of the signup flow.
- [claimed-docs] “the free plan includes 3 agents, no card required”
- [claimed-docs] “Deploy your first agent and talk to it, in under 5 minutes.”
- [claimed-docs] “npm install -g maritime-cli”
- [claimed-docs] “One subscription, a set number of agents, flat prices for extras. No meters, no hourly rates: how much your agents run never changes the bil…”
platform-engineerPay per second only for the compute a sandbox actually uses
weight 3 · round drawnRunloopnone0/10No evidence pack item states an explicit per-second or usage-based billing model; the only tangential mention is that suspend/resume can 'save costs' (runloop-docs-27), which doesn't establish a per-second pricing mechanism.
Maritimenone0/10Maritime's own billing docs explicitly state the opposite of per-second usage billing: 'One subscription, a set number of agents, flat prices for extras. No meters, no hourly rates: how much your agents run never changes the bill' and 'Always-on is a flat add-on.' There is no evidence of any per-second or usage-metered compute pricing for sandboxes.
- [claimed-docs] “One subscription, a set number of agents, flat prices for extras. No meters, no hourly rates: how much your agents run never changes the bil…”
- [claimed-docs] “Always-on is a flat add-on for the rare agent that must react in milliseconds around the clock.”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnRunloopnone0/10No evidence in the pack mentions data residency, region selection, or geographic storage controls anywhere in Runloop's docs; the pack covers sandboxes, blueprints, snapshots, secrets, and tooling but never data-location choice.
ai-native userPrevent my data from being used to train AI models
weight 3 · round drawnRunloopnone0/10No evidence anywhere in the pack addresses AI training data usage, opt-out policies, or data-retention commitments for model training; the docs focus on sandbox infrastructure (Devboxes, secrets, gateways) but never mention training-data policy.
ai-native userControl data retention and deletion
weight 2 · round drawnRunloopnone0/10The evidence pack covers sandboxing, blueprints, snapshots, secrets encryption, and tooling, but contains no documentation of data retention policies, data deletion controls, or user-facing privacy/compliance settings for Devboxes, snapshots, or storage objects. missing for 10: explicit data retention policy docs, user-initiated deletion/purge APIs, compliance/privacy documentation.
- [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
- [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
- [claimed-docs] “Storage Objects provide a way to store and manage files, data, and other resources that can be shared across Devboxes or made publicly avail…”
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnRunloopnone0/10No evidence in the pack mentions telemetry, analytics opt-out, or usage-tracking controls for Runloop; the docs cover sandboxing, blueprints, snapshots, secrets, etc., but nothing about a privacy/telemetry toggle.
Provisioning lifecycle — creating, updating, and tearing down resources across their lifecycleProvisioning lifecycle
Creating, updating, and tearing down resources across their lifecycle
Lifecycle
developerSet timeouts so sandboxes shut down automatically and stop billing when idle or done
weight 2 · round to RunloopRunloop documents suspend/resume to "save state and costs" and Axons that let agents "suspend when idle, wake on demand," indicating some idle-triggered lifecycle management, but there is no explicit documentation of a developer-configurable timeout parameter that auto-shuts-down a Devbox and stops billing after inactivity or completion. missing for 10: explicit timeout/idle-shutdown configuration API or CLI flag, confirmation that suspension halts billing automatically without agent action.
- [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
- [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
- [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”
Maritimenone0/10The docs describe Maritime as serverless agent hosting and mention an 'always-on' paid add-on (implying default agents are not always running), but no evidence describes a developer-configurable idle timeout or automatic shutdown-to-stop-billing mechanism. Missing for 10: any documented timeout setting, idle-detection policy, or billing-pause behavior tied to inactivity.
- [claimed-docs] “Always-on is a flat add-on for the rare agent that must react in milliseconds around the clock.”
developerSpin up an isolated sandbox with one API/SDK call and get a live environment in seconds
weight 3 · round to RunloopDocs describe Devbox creation via SDK/CLI/API, quickstart claiming ~1 minute to a running devbox, blueprints/prebuilt images for fast startup, and marketing claims of 10k+ parallel sandboxes with <2s startup for a 10GB image, matching the one-call, seconds-to-live-environment story. missing for 10: independent/hands-on benchmark confirming real-world provisioning latency beyond vendor claims.
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
- [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
- [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
- [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”
- [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
- [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”
Maritime's SDK exposes an idempotent `provision` call that creates/returns a live agent environment, and docs claim full agent deployment (image pull → live, chat-ready) in under 5 minutes, with CLI/API equivalents (`maritime keys create`, RESTful API). This matches the 'one call, live environment fast' story well, though the fastest documented time is minutes not literal seconds and there's no independent/hands-on latency benchmark or public OpenAPI spec confirming the exact call shape. Missing for 10: independent timing/benchmark evidence corroborating 'seconds', and a verifiable OpenAPI/API schema (probe shows openapi.json 404).
- [claimed-docs] “Tag each agent with your own id via `externalId`. `provision` is **idempotent** on it: it returns the existing agent if there is one, else c…”
- [claimed-docs] “provision is idempotent on it: it returns the existing agent if there is one, else creates a new one. Safe to call on every sign-in.”
- [claimed-docs] “Deploy your first agent and talk to it, in under 5 minutes.”
- [claimed-docs] “Every ready-made template is a real agent framework, pre-configured and one image pull from live. Pick one, name it, deploy.”
- [claimed-docs] “RESTful API for managing agents programmatically.”
- [claimed-docs] “The Maritime CLI lets you deploy and manage agents from your terminal.”
- [probe] “PROBE openapi: all candidate paths 404 (https://maritime.sh/openapi.json, https://maritime.sh/swagger.json, https://maritime.sh/api/openapi.…”
Preview access
developerExpose a port from the sandbox on a public preview URL to reach services running inside
weight 2 · round to RunloopRunloop explicitly documents devbox tunnels for securely accessing ports on a devbox over a simple URL, directly matching the story's requirement to expose a service port publicly. Missing for 10: independent/hands-on confirmation of tunnel reliability and details on URL persistence/security controls beyond the doc snippet.
- [claimed-docs] “Let’s use devbox tunnels to securely access ports on your devbox over a simple url.”
Maritime's docs describe deploying any Dockerfile-based repo to a public, no-login HTTPS URL that reaches the running container's service, matching the core story of exposing an internal service via a public preview URL (maritime-docs-29, maritime-docs-47). However, the docs never explicitly describe port-selection mechanics (e.g., multiple ports, custom port binding) and all evidence is first-party with no independent/hands-on corroboration. missing for 10: explicit port-selection/multi-port docs, independent verification that the public URL reaches an arbitrary internal port.
- [claimed-docs] “Any repo with a Dockerfile can be served on a public, no-login HTTPS URL. Maritime builds it, runs it serverlessly, puts it to sleep when id…”
- [claimed-docs] “Any repo with a Dockerfile can be served on a public, no-login HTTPS URL.”
Snapshot persistence — stories about snapshot persistence in this arenaSnapshot persistence
Stories about snapshot persistence in this arena
Snapshots
developerPause a running sandbox and resume it later with filesystem and memory state intact
weight 2 · round to RunloopRunloop explicitly documents 'Suspend and resume' functionality that pauses sandboxes to save state and resume exactly where left off, plus snapshot capability that saves disk state and can recreate Devboxes from that state. This directly matches the story's pause/resume with filesystem+memory state intact. Missing for 10: no independent/hands-on corroboration of memory-state fidelity across suspend/resume, and no detailed technical explanation of how memory state (not just disk) is preserved.
- [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
- [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
- [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
developerSnapshot a sandbox and later restore or fork new sandboxes from that snapshot
weight 3 · round to RunloopRunloop's docs explicitly describe snapshotting a Devbox's disk state and creating new Devboxes from that saved state, including fan-out use cases (multiple approaches from one snapshot) and rollback scenarios, directly matching the snapshot/restore/fork story. missing for 10: independent/hands-on corroboration beyond first-party docs, and no explicit code example showing the fork API call in the evidence pack.
- [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
- [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
Not comparable on these axes
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · not comparableRunloopn/aRunloop is sandbox/infrastructure tooling for running AI coding agents (Devboxes, blueprints, benchmarks, agent APIs) — it is not a data platform that ingests a user's data and surfaces AI-generated insights or suggestions. This story is a category error for this product type.
Maritimenone0/10Maritime's evidence describes agent deployment, hosting, webhooks, CLI/SDK tooling, and templates, but nothing shows the platform itself surfacing AI-generated insights or suggestions from a user's own data within the product UI — the docs describe running agents, not an analytics/insights layer on user data.
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · not comparableRunloopn/aRunloop is a sandbox/infrastructure platform for running and orchestrating AI coding agents (Devboxes, Agents API, benchmarks), not an end-user product with its own embedded AI assistant to delegate tasks to. The evidence describes tools for building/hosting agents (and integrating with external assistants like Cursor), not a built-in assistant persona within Runloop itself.
Maritimenone0/10Maritime is presented as infrastructure for hosting/deploying user-built AI agents, not as a product with its own built-in assistant for task delegation. Evidence shows external assistants (Claude, Cursor) can drive the Maritime CLI on the user's behalf, but nothing indicates Maritime itself ships an embedded assistant the user delegates tasks to inside the product.
- [claimed-docs] “Paste one prompt into Claude, Cursor, or any assistant and it can install the CLI, authenticate, and deploy agents for you.”
- [claimed-docs] “The Maritime CLI is built to be operated by other AI agents, not just humans: machine-readable output on every command, stable exit codes, a…”