Runloop vs Blaxel

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round drawn
   Runloop✓fullprobed9/10

   Direct probe evidence confirms a working llms.txt file (HTTP 200) with a documentation index, plus docs-md endpoints explicitly pointing agents to fetch llms.txt for full documentation, and Cursor rules files further supporting agent-oriented consumption of docs. missing for 10: independent third-party confirmation that agents successfully use these docs in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”
   • [probe] “PROBE docs-md: HTTP 200 at https://docs.runloop.ai/docs/overview/what-is-runloop.md > ## Documentation Index > Fetch the complete documentat…”
   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”

   proof ↗⚑ flag

   Blaxel✓fullprobed9/10

   A probe directly confirms llms.txt is live and returns HTTP 200 with a structured documentation summary at docs.blaxel.ai/llms.txt, and Blaxel also ships agent-oriented skills/MCP docs for agents to consume. Missing for 10: no independent third-party confirmation that agents actually consume/parse this llms.txt successfully in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.blaxel.ai/llms.txt # Documentation > Blaxel documentation: cloud sandboxes, AI agent hosting, MCP …”
   • [claimed-docs] “Install the Blaxel skill to give your agent the ability to deploy AI agents, create sandboxes for code execution, host MCP servers, run batc…”
   • [claimed-docs] “The open source Blaxel migration skill gives agents the knowledge they need to migrate sandbox code from other providers to Blaxel.”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Runloop
   Runloop✓fullclaimed8/10

   Runloop provides a CLI (rli), SDKs, an Agents API, GitHub Actions integration for deploying agents, and benchmark jobs runnable via CLI—all supporting headless/CI automation of sandboxes and agents. missing for 10: no independent/hands-on CI case study, and no explicit CI provider-specific guide beyond GitHub Actions example.

   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”

   proof ↗⚑ flag

   Blaxel~partialprobed6/10

   Blaxel documents CLI with API keys usable in CI, and explicitly points to an official GitHub Action for CI pipeline integration, plus batch Jobs for automation-style workloads. However, there's no evidence of a full headless CI walkthrough, no independent corroboration, and no explicit CI examples for deploying/testing agents beyond the brief GitHub Action mention. Missing for 10: detailed CI workflow examples/docs, independent hands-on confirmation of CI usage, broader automation guidance beyond batch jobs.

   • [claimed-docs] “API keys can be used in the Blaxel APIs, CLI and SDK.”
   • [claimed-docs] “The easiest way to integrate the Blaxel CLI into a CI pipeline is with the official Blaxel GitHub Action.”
   • [claimed-docs] “Jobs allow you to run many AI tasks in parallel using batch processing.”
   • [probe] “official CLI documented at https://docs.blaxel.ai/cli-reference/introduction”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to Blaxel
   Runloop—none0/10

   Evidence shows Runloop publishes its own MCP server so external agents can call Runloop's tools (runloop-probe-4), but there is no evidence that Runloop itself can act as an MCP client and plug in third-party MCP servers to use their tools. missing for 10: any documentation of an MCP-client integration point, config for adding external MCP servers, or examples of Devboxes/agents consuming external MCP tool servers.

   • [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”

   proof ↗⚑ flag

   Blaxel~partialprobed5/10

   Blaxel documents hosting MCP servers (docs-11) and exposing sandboxes as MCP servers (docs-5), and its skill can 'host MCP servers' for agents (docs-1), suggesting agents built on Blaxel can be wired to MCP tool servers. However there is no explicit documentation of connecting arbitrary third-party/external MCP servers into an agent so it can call their tools as a client — the evidence is server-hosting/exposure oriented rather than consumption-oriented. Missing for 10: explicit client-side MCP integration docs, examples of an agent consuming an external MCP server's tools, and independent confirmation this works in practice.

   • [claimed-docs] “Install the Blaxel skill to give your agent the ability to deploy AI agents, create sandboxes for code execution, host MCP servers, run batc…”
   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [claimed-docs] “MCP Server Hosting is a serverless computing service that allows you to host remote MCP servers without having to manage infrastructure.”
   • [probe] “official MCP server documented at https://docs.blaxel.ai/skills-mcp”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round drawn
   Runloop✓fullprobed7/10

   Runloop is a sandbox/devbox platform (not itself an agent), so shipping an official MCP server is a valid axis, and the docs explicitly reference one at docs.runloop.ai/docs/tools/ai-tools. This confirms the capability exists, though the evidence pack lacks detail on the specific tools/resources exposed or independent hands-on confirmation. Missing for 10: detailed documentation of MCP server tool surface, independent/community verification of usage.

   • [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”

   proof ↗⚑ flag

   Blaxel✓fullprobed7/10

   Blaxel documents that every sandbox is exposed via an official MCP server enabling agents to operate it via tool calls, and it offers a serverless 'MCP Server Hosting' service for hosting remote MCP servers, with a dedicated docs page confirmed by probe. This is a platform (not itself an agent), so the connect-via-official-MCP-server axis clearly applies and is met by first-party evidence. missing for 10: independent/hands-on confirmation of connecting an external agent to these MCP servers, and more detail on the protocol/setup beyond the docs summary.

   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [claimed-docs] “MCP Server Hosting is a serverless computing service that allows you to host remote MCP servers without having to manage infrastructure.”
   • [probe] “official MCP server documented at https://docs.blaxel.ai/skills-mcp”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to Runloop
   Runloop✓fullprobed8/10

   Runloop ships an official CLI (`rli`) with both interactive TUI and traditional commands for managing devboxes, blueprints, SSH sessions, and benchmarks, documented directly in first-party docs and confirmed by probe. This is clearly built for AI-native/agentic workflows (blueprint creation, benchmark submission, devbox lifecycle management). Missing for 10: independent/hands-on community corroboration beyond vendor docs, and no changelog/version history showing active CLI maintenance cadence.

   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “From the interactive menu, select a devbox and choose **SSH** to open a secure shell session directly into your devbox. The CLI handles all …”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [probe] “official CLI documented at https://docs.runloop.ai/docs/tools/rl-cli”

   proof ↗⚑ flag

   Blaxel✓fullprobed7/10

   Blaxel has an official CLI documented at a dedicated CLI reference, integrable into CI via a GitHub Action, and used for API key auth alongside SDK/API — consistent with AI-native, agentic workflows (deploying agents, sandboxes, MCP servers, jobs). Missing for 10: no independent/hands-on corroboration of CLI usage, no detailed command list or examples in the evidence pack beyond the reference page's existence.

   • [probe] “official CLI documented at https://docs.blaxel.ai/cli-reference/introduction”
   • [claimed-docs] “API keys can be used in the Blaxel APIs, CLI and SDK.”
   • [claimed-docs] “The easiest way to integrate the Blaxel CLI into a CI pipeline is with the official Blaxel GitHub Action.”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to Runloop
   Runloop✓fullprobed8/10

   Runloop documents a full public API surface: Devboxes API, Agents API, SDKs in common languages, CLI (rli), and GitHub Action integration, all callable programmatically for agent-driven workflows. This is strong first-party documentation of a public API, though probes show no discoverable OpenAPI spec, slightly weakening independent verification. Missing for 10: a published OpenAPI/swagger spec and independent third-party corroboration of API usage.

   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”
   • [probe] “official CLI documented at https://docs.runloop.ai/docs/tools/rl-cli”

   proof ↗⚑ flag

   Blaxel~partialprobed5/10

   Docs confirm a public API exists and that API keys can drive it (used across CLI, SDK, and API) and CLI reference documentation is available, but a dedicated public API reference/OpenAPI spec could not be located (all openapi.json/swagger.json paths returned 404), so the story is only partially evidenced. missing for 10: a discoverable OpenAPI/API reference doc, hands-on example of calling the API directly, independent confirmation of API completeness.

   • [claimed-docs] “API keys can be used in the Blaxel APIs, CLI and SDK.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.blaxel.ai/openapi.json, https://docs.blaxel.ai/swagger.json, https://docs.blaxel.ai/api…”
   • [probe] “official CLI documented at https://docs.blaxel.ai/cli-reference/introduction”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to Runloop
   Runloop~partialclaimed6/10

   Runloop's Agent Gateways let agents call LLM APIs like Anthropic/OpenAI 'without ever seeing your API keys,' which is a direct least-privilege credentialing mechanism for agents. However, the docs give only a single brief mention with no detail on fine-grained scopes, per-agent permission levels, revocation, or audit logging. Missing for 10: detailed scoping/permission model documentation, credential rotation/revocation controls, and independent verification of the gateway's isolation guarantees.

   • [claimed-docs] “Agent Gateways let your agents call LLM APIs like Anthropic and OpenAI **without ever seeing your API keys**.”
   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”

   proof ↗⚑ flag

   Blaxel—none0/10

   Evidence shows Blaxel issues generic API keys usable across its APIs, CLI, and SDK (blaxel-docs-12), and preview URLs can require an access token (blaxel-docs-8), but there is no documentation of scoping, role-based permissions, or least-privilege credential configuration for agents specifically.

   • [claimed-docs] “API keys can be used in the Blaxel APIs, CLI and SDK.”
   • [claimed-docs] “When you create a private preview URL a token is required to access the URL.”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Runloop
   Runloop~partialclaimed6/10

   Runloop explicitly documents official SDKs in common languages for interacting with its API, plus tutorials integrating with SDKs like OpenAI Agents SDK, supporting an AI-native builder workflow. However, evidence lacks specifics on which languages are covered, versioning/maturity, or independent developer corroboration. Missing for 10: enumerated language list/maturity details, independent hands-on confirmation, changelog/versioning evidence.

   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”

   proof ↗⚑ flag

   Blaxel~partialclaimed3/10

   Only indirect evidence exists that an SDK exists (API keys usable in "Blaxel APIs, CLI and SDK"), but there is no dedicated SDK documentation, language coverage, install instructions, or examples in the pack. missing for 10: dedicated SDK docs/reference, supported languages, code examples, independent corroboration of SDK usage.

   • [claimed-docs] “API keys can be used in the Blaxel APIs, CLI and SDK.”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Runloop—none0/10

   No evidence mentions webhooks or event subscription mechanisms; Runloop's docs describe devbox lifecycle events via Axons event streams but nothing about webhook subscriptions for external systems. Missing for 10: any documentation of a webhooks API, subscription/callback URLs, or event delivery mechanism to external endpoints.

   ⚑ flag

   Blaxel—none0/10

   No evidence in the pack mentions webhooks, event subscriptions, or push notifications from Blaxel; documentation covers sandboxes, MCP hosting, jobs, and CLI/API but nothing about webhook subscription mechanisms.

   ⚑ flag

Agentic features

1. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Runloop
   Runloop~partialclaimed7/10

   Runloop's Axons feature gives agents a persistent event stream letting them suspend when idle, wake on demand, recover state, and hand off between agents 'without you building the plumbing' (runloop-docs-17, runloop-docs-18), and devboxes support suspend/resume to preserve state across runs (runloop-docs-27). The GitHub Action for agent deployment (runloop-docs-24) supports triggering agent runs from CI workflows, enabling background automation setups. Missing for 10: no explicit cron/scheduler primitive, no independent/hands-on account of long-running unattended automations actually working in production, and no example of a full always-on autonomous pipeline beyond docs claims.

   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”

   proof ↗⚑ flag

   Blaxel~partialclaimed5/10

   Blaxel supports background-running infrastructure like Jobs for batch processing, scale-to-zero agents/sandboxes, and MCP-server hosting, which can enable autonomous background workloads, but there's no explicit documentation of a scheduling/trigger/automation-workflow feature (e.g., cron-like triggers or event-based automation) for setting up self-running automations. missing for 10: explicit scheduler/trigger mechanism for autonomous automations, independent/hands-on evidence of autonomous background execution, and clear docs describing 'automation' as a first-class concept rather than inferred from batch jobs and hosting.

   • [claimed-docs] “Jobs allow you to run many AI tasks in parallel using batch processing.”
   • [claimed-docs] “MCP Server Hosting is a serverless computing service that allows you to host remote MCP servers without having to manage infrastructure.”
   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”
   • [claimed-docs] “Agent Drive is a distributed filesystem that can be mounted to multiple sandboxes or agents at any time, including while they are already ru…”

   proof ↗⚑ flag

2. ai-native userOperate the product with natural-language commands

   weight 2 · round drawn
   Runloop~partialprobed6/10

   Runloop exposes an official MCP server and ships Cursor '.mdc' rule files that give AI assistants context on the SDK (devbox lifecycle, file ops, blueprints, snapshots, tunnels), enabling natural-language operation through AI coding assistants rather than a bespoke NL interface. There's no first-party chat/NL command surface of Runloop's own, and no independent evidence of how well the MCP/Cursor integration performs in practice. Missing for 10: dedicated natural-language interface docs, hands-on/independent validation of the MCP or Cursor integration actually working end-to-end.

   • [probe] “official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”
   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”

   proof ↗⚑ flag

   Blaxel~partialprobed6/10

   Blaxel offers an installable 'skill' that lets an agent deploy agents, create sandboxes, host MCP servers, and run jobs via simple prompts with zero code, directly supporting natural-language operation. However, this is scoped to agent-orchestration skills rather than a full natural-language interface across the entire product surface (e.g., CLI/dashboard operations still require explicit commands). Missing for 10: independent/hands-on verification of the skill's natural-language reliability, and evidence that all major product functions (not just deploy/sandbox/mcp/jobs) are controllable via natural language.

   • [claimed-docs] “Install the Blaxel skill to give your agent the ability to deploy AI agents, create sandboxes for code execution, host MCP servers, run batc…”
   • [claimed-docs] “The open source Blaxel migration skill gives agents the knowledge they need to migrate sandbox code from other providers to Blaxel.”
   • [probe] “official MCP server documented at https://docs.blaxel.ai/skills-mcp”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Runloop—none0/10

   No evidence of an interactive API reference with runnable examples; the explicit probe for OpenAPI/swagger specs returned 404s across all candidate paths, and docs mention SDKs/CLI but not a runnable API explorer.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”

   proof ↗⚑ flag

   Blaxel—none0/10

   The probe explicitly found no OpenAPI/Swagger spec at any candidate path (404s), and no evidence exists of an interactive API reference with runnable examples—only static markdown docs and CLI/skills references are documented.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.blaxel.ai/openapi.json, https://docs.blaxel.ai/swagger.json, https://docs.blaxel.ai/api…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   Runloop—none0/10

   Runloop offers SDKs and a CLI but there is no evidence of a downloadable OpenAPI/Swagger spec; explicit probes to common OpenAPI paths (openapi.json, swagger.json, etc.) all returned 404, confirming absence rather than just lack of mention.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”

   proof ↗⚑ flag

   Blaxel—none0/10

   The probe explicitly checked common OpenAPI spec locations and found all returned 404, and no documentation item references a downloadable machine-readable API spec; only an llms.txt for documentation is present.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.blaxel.ai/openapi.json, https://docs.blaxel.ai/swagger.json, https://docs.blaxel.ai/api…”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.blaxel.ai/llms.txt # Documentation > Blaxel documentation: cloud sandboxes, AI agent hosting, MCP …”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Runloop
   Runloop✓fullclaimed8/10

   Runloop's Devboxes are isolated sandbox execution environments (secure, snapshot/blueprint-based, no production data by default) explicitly designed for testing AI agents, fully separate from any production system. missing for 10: no independent/hands-on corroboration beyond vendor docs, no explicit statement contrasting sandbox vs production data isolation guarantees.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

   Blaxel✓fullclaimed7/10

   Blaxel's core product is isolated sandbox environments for code execution, separate from production infra, with dedicated MCP-controlled sandbox instances, custom templates, and preview URLs, clearly enabling safe testing without touching production data. missing for 10: no explicit documentation contrasting sandbox vs production data isolation guarantees, and no independent/hands-on verification of data isolation beyond vendor docs.

   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”
   • [claimed-docs] “Agent Drive is a distributed filesystem that can be mounted to multiple sandboxes or agents at any time, including while they are already ru…”
   • [claimed-docs] “When you create a private preview URL a token is required to access the URL.”
   • [claimed-docs] “Install the Blaxel skill to give your agent the ability to deploy AI agents, create sandboxes for code execution, host MCP servers, run batc…”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Runloop—none0/10

   No evidence of API versioning scheme or a documented deprecation policy; OpenAPI spec probe returned 404s and no changelog/versioning docs are cited.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”

   proof ↗⚑ flag

   Blaxel—none0/10

   No evidence of API versioning scheme or a documented deprecation policy; the OpenAPI spec probe even returned 404s, and no docs mention version history or deprecation timelines.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.blaxel.ai/openapi.json, https://docs.blaxel.ai/swagger.json, https://docs.blaxel.ai/api…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round drawn
   Runloop~partialclaimed6/10

   Runloop supports running 10k+ parallel sandboxes, benchmark jobs that provision devboxes/run agents/aggregate outputs 'with a single command,' and snapshots for fan-out across multiple approaches — evidence of bulk/parallel operation across many devboxes. However, there's no explicit documentation of a generic bulk-operation API/CLI verb (e.g., bulk delete, bulk update, batch create) for arbitrary resources beyond devboxes/benchmarks. missing for 10: explicit bulk CRUD operations across resources like snapshots/storage objects/secrets, documented batch API endpoints, and independent evidence of large-scale bulk usage beyond vendor docs.

   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “Run your agent against well-known open source benchmarks like terminal bench 2, AIME, and more.”

   proof ↗⚑ flag

   Blaxel~partialclaimed6/10

   Blaxel's Jobs feature explicitly supports running many AI tasks in parallel via batch processing, and the Blaxel skill can invoke batch jobs via simple prompts, showing bulk-operation capability for AI-native users. However, the evidence gives no detail on limits, monitoring, or how a user would trigger operations across a large item set (e.g., bulk API endpoints), nor any independent/hands-on validation of batch performance. Missing for 10: concrete batch API/CLI examples, scale limits, and independent corroboration of bulk job execution.

   • [claimed-docs] “Jobs allow you to run many AI tasks in parallel using batch processing.”
   • [claimed-docs] “Install the Blaxel skill to give your agent the ability to deploy AI agents, create sandboxes for code execution, host MCP servers, run batc…”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Runloop
   Runloop~partialclaimed4/10

   Runloop's Axons give agents a persistent event stream that can suspend on idle, wake on demand, recover state, and hand off between agents, which is a form of automatic event-driven behavior, and the Broker component sits between Axon and agent process to manage this. However, there's no documentation of a general user-facing rules engine where arbitrary conditions map to arbitrary actions across the platform (e.g., webhooks, triggers on Devbox/snapshot events, custom rule definitions). Missing for 10: a documented rules/trigger definition interface, examples of custom event-to-action mappings beyond agent suspend/wake, and evidence of triggers on other platform events (snapshots, storage, secrets).

   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”

   proof ↗⚑ flag

   Blaxel—none0/10

   Blaxel's evidence covers deployment, sandboxes, MCP hosting, batch jobs, and scale-to-zero infrastructure, but nothing describes a rules/trigger engine where users define conditions that automatically fire actions on events (e.g., webhooks, event-based automation rules). Since this is a plausible capability for an agent-hosting platform, absence of evidence means 'none' rather than 'na'.

   ⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round drawn
   Runloop—none0/10

   The evidence describes Devboxes, snapshots, benchmarks, and Axons for event-driven suspend/resume, but nothing about scheduling recurring jobs or cron-like recurring workflows. Absence of evidence for this applicable automation capability yields 'none'.

   ⚑ flag

   Blaxel—none0/10

   Blaxel documents one-off batch 'Jobs' for parallel AI task processing (blaxel-docs-10) but no evidence shows support for recurring/scheduled triggers, cron-like scheduling, or workflow orchestration over time. missing for 10: scheduling/cron mechanism, recurring trigger configuration, workflow automation docs.

   • [claimed-docs] “Jobs allow you to run many AI tasks in parallel using batch processing.”

   proof ↗⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round to Runloop
   Runloop~partialclaimed4/10

   Runloop's Snapshots explicitly support rolling back Devbox state to a known good point, and Blueprints let you version/reuse startup configurations, giving a rudimentary rollback mechanism for agent environments. However, there is no evidence of a review workflow (diffs, approvals, audit trail) or of 'automations' as a first-class versioned object distinct from raw disk/environment snapshots. Missing for 10: explicit review/diff tooling for automations, first-class automation versioning (not just devbox/environment snapshots), audit/history UI for changes.

   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”

   proof ↗⚑ flag

   Blaxel—none0/10

   No evidence of versioning, review workflows, or rollback capabilities for automations/agents/jobs; documentation covers deployment, sandboxes, scaling, and infra regions but nothing about tracking automation history or reverting changes.

   ⚑ flag

Agent workloads

1. ai-native userRun coding agents like Claude Code or Codex inside the sandbox following the vendor's own recipe

   weight 2 · round to Runloop
   Runloop~partialclaimed5/10

   Runloop documents running AI coding agents generally inside Devboxes (blueprints, benchmarks, Agents API, OpenAI Agents SDK tutorial) but no specific vendor recipe or tutorial naming Claude Code or Codex is present in the evidence. missing for 10: explicit Claude Code/Codex quickstart or blueprint, named integration guide for these specific agents.

   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [claimed-docs] “Run your agent against well-known open source benchmarks like terminal bench 2, AIME, and more.”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”

   proof ↗⚑ flag

   Blaxel—none0/10

   The evidence describes Blaxel sandboxes, MCP exposure, skills, and job/batch features, but nowhere mentions a documented recipe or guide for running Claude Code or Codex specifically inside a Blaxel sandbox. Missing for 10: any vendor recipe/tutorial naming Claude Code or Codex, or a hands-on report of doing so.

   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”
   • [claimed-docs] “Install the Blaxel skill to give your agent the ability to deploy AI agents, create sandboxes for code execution, host MCP servers, run batc…”

   proof ↗⚑ flag

Workloads

1. developerRun a headless browser or full desktop environment inside the sandbox

   weight 2 · round drawn
   Runloop—none0/10

   No evidence Runloop supports running a headless browser or desktop GUI inside its Devbox sandboxes; the only browser-related tutorial explicitly states the architecture avoids this, running Chromium externally via Browserbase and connecting over CDP so 'no Chromium ever runs in the devbox.' Blueprints/docs never mention GUI, X11, VNC, or bundled browser binaries as a supported environment type.

   • [claimed-docs] “the agent runs in a devbox, the browser runs on Browserbase, and the devbox drives it by connecting Playwright over CDP to the session’s con…”

   proof ↗⚑ flag

   Blaxel—none0/10

   Evidence describes sandboxes for code execution, MCP server exposure, filesystem mounting, and batch jobs, but no mention of headless browser support or full desktop/GUI environment capability inside a sandbox.

   ⚑ flag

2. developerAttach GPUs to sandboxed workloads

   weight 2 · round drawn
   Runloop—none0/10

   The evidence pack describes Devboxes, blueprints, snapshots, mounts, storage, and networking features but never mentions GPU support or GPU-attached compute for sandboxed workloads; nothing indicates hardware acceleration is available.

   ⚑ flag

   Blaxel—none0/10

   No evidence pack item mentions GPU support, GPU instance types, or hardware acceleration for sandboxes; documentation covers scaling, preview URLs, filesystems, jobs, and MCP hosting but never GPU attachment. missing for 10: any mention of GPU-enabled sandbox instances, GPU pricing/config, or GPU-specific workload examples.

   ⚑ flag

Files

1. developerRead, write, upload, and download files in the sandbox filesystem via the SDK

   weight 2 · round to Runloop
   Runloop~partialclaimed4/10

   Docs confirm the SDK/CLI supports 'file operations' as part of Devbox management (mentioned alongside lifecycle, command execution, etc. in the Cursor rules integration), implying read/write file capability, but no dedicated documentation page or code sample specifically shows upload/download or read/write file APIs. Missing for 10: explicit SDK method docs for file read/write, and explicit upload/download examples or independent corroboration.

   • [claimed-docs] “We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operati…”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”

   proof ↗⚑ flag

   Blaxel—none0/10

   The evidence describes sandboxes, MCP-based tool operation, and a separate 'Agent Drive' distributed filesystem, but nothing documents SDK methods for reading, writing, uploading, or downloading files within a sandbox. Missing for 10: SDK file I/O API reference, upload/download examples, hands-on confirmation of file operations in sandbox.

   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [claimed-docs] “Agent Drive is a distributed filesystem that can be mounted to multiple sandboxes or agents at any time, including while they are already ru…”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”

   proof ↗⚑ flag

Runtimes

1. developerDefine custom sandbox templates or bring my own container image

   weight 2 · round to Runloop
   Runloop✓fullclaimed8/10

   Runloop's Blueprints support building custom sandbox images from a Dockerfile (`rli blueprint from-dockerfile`), plus reusable prebuilt blueprints and snapshotting of custom states, directly enabling developers to bring their own container image or define custom templates. Missing for 10: independent/hands-on verification of Dockerfile-based blueprint builds and more detail on supported image registries or base image constraints.

   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”
   • [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”

   proof ↗⚑ flag

   Blaxel~partialclaimed6/10

   Docs explicitly reference 'Sandbox images' allowing creation of customized & reusable sandbox environments (blaxel-docs-7), which is direct evidence of custom template/image support. However, no evidence details how to bring an arbitrary Dockerfile/OCI image, base image customization workflow, or CLI/SDK commands for building/pushing custom images. missing for 10: concrete docs/examples on bring-your-own-container-image workflow, supported base image formats, and CLI commands for building/publishing custom sandbox templates.

   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”

   proof ↗⚑ flag

2. developerExecute code in multiple language runtimes (Python, JavaScript, and more) and get rich results back

   weight 2 · round to Runloop
   Runloop~partialclaimed5/10

   Runloop's Devboxes are described as full-featured secure sandboxed execution environments that can run arbitrary commands (via blueprints, SSH, CLI) which implies multi-language support, but there is no explicit documentation of a code-execution API returning 'rich results' (e.g., structured outputs, images, dataframes) as in dedicated code-interpreter products. missing for 10: explicit multi-language runtime examples (Python/JS/etc.), rich/structured result formatting (e.g., stdout+images+tables), and independent confirmation of these capabilities.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”

   proof ↗⚑ flag

   Blaxel~partialclaimed3/10

   Blaxel's docs confirm sandboxes exist for code execution (blaxel-docs-1, blaxel-docs-7) and can be customized via templates, but there is no explicit evidence of specific language runtime support (Python, JavaScript, etc.) or of structured/rich result output formats from executed code. Missing for 10: explicit list of supported language runtimes, documentation of output/result format (e.g., stdout, images, rich objects), and any hands-on demonstration of multi-language execution.

   • [claimed-docs] “Install the Blaxel skill to give your agent the ability to deploy AI agents, create sandboxes for code execution, host MCP servers, run batc…”
   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”

   proof ↗⚑ flag

Untrusted code

1. developerExecute untrusted, AI-generated code without risking my own infrastructure

   weight 3 · round to Runloop
   Runloop✓fullprobed8/10

   Runloop's core product is Devboxes, secure sandboxed execution environments purpose-built to run AI agent/code workloads isolated from user infrastructure, with snapshots, tunnels, secrets isolation, and scaling to 10k+ parallel sandboxes. This directly matches the story of executing untrusted AI-generated code without risking one's own infra. Missing for 10: independent/third-party hands-on validation of isolation guarantees, and explicit documentation of the underlying sandboxing/virtualization technology or security boundary details.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Let’s use devbox tunnels to securely access ports on your devbox over a simple url.”
   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”

   proof ↗⚑ flag

   Blaxel✓fullclaimed7/10

   Blaxel provides sandboxes explicitly for isolated code execution, exposed via MCP tool calls, with scale-to-zero/resume infrastructure meaning execution happens off the developer's own infra, plus custom sandbox images and private preview URLs with tokens for security. Missing for 10: independent/hands-on evidence of isolation guarantees (e.g., security audits, escape testing) and explicit detail on resource/network sandboxing limits beyond docs claims.

   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”
   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”
   • [claimed-docs] “When you create a private preview URL a token is required to access the URL.”
   • [claimed-docs] “Install the Blaxel skill to give your agent the ability to deploy AI agents, create sandboxes for code execution, host MCP servers, run batc…”

   proof ↗⚑ flag

2. developerRun arbitrary shell commands and install packages inside the sandbox

   weight 2 · round to Runloop
   Runloop✓fullclaimed7/10

   Runloop Devboxes are full Linux execution environments with SSH access via the CLI (runloop-docs-9) and blueprints built from Dockerfiles that install packages/dependencies at startup (runloop-docs-2, runloop-docs-14), implying arbitrary shell command execution and package installation. However, the evidence pack lacks a direct hands-on example of running an explicit shell command or apt/pip install inside a Devbox. missing for 10: explicit example/output of running an arbitrary shell command and installing a package inside a Devbox, independent third-party confirmation.

   • [claimed-docs] “From the interactive menu, select a devbox and choose **SSH** to open a secure shell session directly into your devbox. The CLI handles all …”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”

   proof ↗⚑ flag

   Blaxel~partialclaimed5/10

   Blaxel's sandboxes are documented as code-execution environments exposed via MCP tool calls, and customizable sandbox images imply support for installing packages/dependencies, but no evidence explicitly shows arbitrary shell command execution or package installation examples/APIs. missing for 10: explicit shell/exec command documentation, package manager install examples, hands-on demonstration of arbitrary command execution.

   • [claimed-docs] “Install the Blaxel skill to give your agent the ability to deploy AI agents, create sandboxes for code execution, host MCP servers, run batc…”
   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”

   proof ↗⚑ flag

Agent lifecycle

1. ai-native userMy agent can provision its own sandbox, execute code, read the results, and tear it down — end to end without a human

   weight 3 · round drawn
   Runloop✓fullclaimed8/10

   Runloop's core product is programmatic sandbox (Devbox) provisioning, execution, and teardown via API/SDK/CLI, explicitly designed for AI agents — including an Agents API for creating/managing/deploying agents, blueprints for reusable environments, snapshots for state capture, and suspend/resume lifecycle, all achievable without human intervention (docs-1, docs-13, docs-22, docs-23, docs-2, docs-4, docs-27, docs-12). The quickstart shows a devbox spun up in ~1 minute and CLI/SDK access covers execution and result retrieval end-to-end. missing for 10: independent/hands-on third-party verification of a full agent-driven provision→execute→teardown loop, and a concrete code example showing an agent reading execution results programmatically.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

   Blaxel✓fullclaimed8/10

   Blaxel docs describe agents provisioning sandboxes via a skill/API with zero code, exposing each sandbox as an MCP server so agents can execute code via tool calls, and scale-to-zero/teardown behavior after inactivity — covering provision, execute, read results, and teardown end-to-end without human intervention. missing for 10: independent hands-on verification of a full agentic loop and explicit documentation of an agent reading back execution results programmatically.

   • [claimed-docs] “Install the Blaxel skill to give your agent the ability to deploy AI agents, create sandboxes for code execution, host MCP servers, run batc…”
   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”

   proof ↗⚑ flag

Isolation

1. platform-engineerRely on a documented hard isolation boundary (microVM or equivalent) between sandboxes and my systems

   weight 3 · round to Runloop
   Runloop~partialprobed3/10

   Runloop repeatedly describes Devboxes as "secure sandboxed execution environments" and "isolated Linux sandbox environments," implying some isolation boundary, but the evidence never names or documents the actual isolation mechanism (e.g., microVM, Firecracker, gVisor, container-only) or any architecture diagram/security whitepaper explaining the boundary. missing for 10: explicit documentation of the underlying isolation technology (microVM/hypervisor-level or equivalent), any security architecture doc or third-party audit confirming hard isolation guarantees.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “Runloop allows you to run benchmarks at scale in a secure environment.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox en…”

   proof ↗⚑ flag

   Blaxel—none0/10

   The evidence pack describes Blaxel's sandbox features (scale-to-zero, MCP exposure, preview URLs, templates) but never documents the underlying isolation mechanism (e.g., microVM, gVisor, container hardening) or any explicit security boundary guarantee between sandboxes and host systems. Missing for 10: any mention of microVM/hypervisor-based isolation, a security/architecture whitepaper describing the isolation boundary, or third-party/independent verification of sandbox isolation.

   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”
   • [claimed-docs] “When you create a private preview URL a token is required to access the URL.”

   proof ↗⚑ flag

2. ai-native userGive an agent a sandbox where host secrets and credentials are unreachable by the code it runs

   weight 2 · round to Runloop
   Runloop~partialclaimed6/10

   Runloop's Devboxes are documented as sandboxed execution environments with encrypted secrets and, notably, Agent Gateways that let agents call LLM APIs 'without ever seeing your API keys' — directly supporting the isolation-of-credentials use case. However, secrets are also stated to be 'automatically made available as environment variables' inside Devboxes, which is somewhat in tension with 'unreachable by code' and isn't clarified as scoped/restricted; there's no independent/hands-on verification of secret unreachability or a security audit. missing for 10: independent verification of secret isolation, clarity on env-var secret exposure vs true unreachability, and details on network/filesystem-level credential isolation guarantees.

   • [claimed-docs] “Agent Gateways let your agents call LLM APIs like Anthropic and OpenAI **without ever seeing your API keys**.”
   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”

   proof ↗⚑ flag

   Blaxel~partialclaimed5/10

   Blaxel provides isolated sandboxes for code execution with MCP-based control and access tokens for API/CLI/SDK auth, implying separation between host credentials and sandbox execution, but there's no explicit documentation stating host secrets/credentials are unreachable by sandbox code or describing secret-injection isolation guarantees. missing for 10: explicit secret-isolation architecture docs, third-party security audit or penetration-test evidence, and confirmation that host credentials cannot leak into sandbox runtime.

   • [claimed-docs] “Install the Blaxel skill to give your agent the ability to deploy AI agents, create sandboxes for code execution, host MCP servers, run batc…”
   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”
   • [claimed-docs] “API keys can be used in the Blaxel APIs, CLI and SDK.”

   proof ↗⚑ flag

Network policy

1. platform-engineerRestrict or allow the sandbox's network egress with explicit policy

   weight 3 · round drawn
   Runloop—none0/10

   No evidence pack item describes network egress controls, firewall rules, or allow/deny policies for Devbox sandboxes; tunnels and secrets features are documented but network egress policy configuration is never mentioned.

   ⚑ flag

   Blaxel—none0/10

   No evidence describes any network egress control, firewall, or allow/deny-list policy for sandboxes—docs cover scaling, preview URLs, filesystem mounting, and regions but nothing about network access restriction. missing for 10: any mention of egress policy configuration, allow/deny lists, network isolation controls, or firewall rules for sandboxes.

   ⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Runloop
   Runloop~partialprobed6/10

   Runloop is API-first: the CLI (rli) wraps SDKs/API for devboxes, blueprints, snapshots, benchmarks, agents, secrets, etc., suggesting broad API/CLI parity with any dashboard functionality. However there is no explicit statement claiming full UI/API feature parity, no documented OpenAPI spec (404s on all probed paths), and no direct dashboard-vs-API comparison. Missing for 10: explicit parity claim between web dashboard and API, published OpenAPI/reference spec, confirmation that all dashboard-only actions (e.g., billing, org settings) are also API-accessible.

   • [claimed-docs] “The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
   • [claimed-docs] “The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregatin…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/…”

   proof ↗⚑ flag

   Blaxel~partialprobed5/10

   Blaxel offers API keys usable across API, CLI, and SDK, and documents CLI/SDK for core resources (sandboxes, agents, jobs, MCP servers), suggesting broad programmatic control. However, no OpenAPI/spec was found (all candidate paths 404), and there is no explicit documentation claiming full UI/API feature parity. Missing for 10: a public OpenAPI reference, an explicit parity statement, and independent confirmation that all UI actions are exposed via API.

   • [claimed-docs] “API keys can be used in the Blaxel APIs, CLI and SDK.”
   • [probe] “official CLI documented at https://docs.blaxel.ai/cli-reference/introduction”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.blaxel.ai/openapi.json, https://docs.blaxel.ai/swagger.json, https://docs.blaxel.ai/api…”
   • [claimed-docs] “The easiest way to integrate the Blaxel CLI into a CI pipeline is with the official Blaxel GitHub Action.”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to Runloop
   Runloop~partialclaimed4/10

   Runloop offers Snapshots to save Devbox disk state and export/re-create Devboxes, and Dockerfile-based blueprints imply portability of environments, but there is no documented feature for exporting all account/user data (benchmarks, configs, agent history) in open formats or a clear data-export/account-deletion workflow. missing for 10: explicit full account data export tool, open-format guarantees for all data types, data portability/deletion documentation.

   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “rli blueprint from-dockerfile --name my-agent-env”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”

   proof ↗⚑ flag

   Blaxel—none0/10

   No evidence pack items address data export or portability in open formats, or any means to leave the platform with one's data; the closest is a migration skill for importing sandbox code from other providers, not exporting data out of Blaxel.

   ⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Runloop—none0/10

   No evidence indicates Runloop's core source (the sandbox platform, Devboxes, Axons, etc.) is released under an open license; the evidence pack shows only docs, CLI, and SDK references without any license or open-source repository disclosure.

   ⚑ flag

   Blaxel—none0/10

   No evidence indicates Blaxel's core source code is published under an open license; only one skill (migration skill) is mentioned as open source, but the product itself (platform/CLI/SDK) has no cited open-license repository or licensing statement.

   • [claimed-docs] “The open source Blaxel migration skill gives agents the knowledge they need to migrate sandbox code from other providers to Blaxel.”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   Runloop—none0/10

   Runloop is presented entirely as a hosted cloud platform (Devboxes, Agents API, benchmarks) with no evidence of an on-premises or self-hostable deployment option, open-source core, or self-hosting documentation.

   ⚑ flag

   Blaxel—none0/10

   Blaxel is presented entirely as a serverless/managed cloud platform (scale-to-zero, regions, hosted MCP servers, hosted sandboxes) with no mention of a self-hostable core product, on-prem deployment, or open-source server component that a user could run themselves; only a migration skill and CLI/SDK for interacting with the hosted service are documented.

   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”
   • [claimed-docs] “you can optionally pin an agent or MCP server to a specific region by setting the region field in your blaxel.toml”
   • [claimed-docs] “MCP Server Hosting is a serverless computing service that allows you to host remote MCP servers without having to manage infrastructure.”
   • [claimed-docs] “When deploying a resource on Blaxel, you sometimes have the option to choose the deployment region.”

   proof ↗⚑ flag

Latency

1. developerStart sandboxes with documented sub-second-to-few-second cold starts

   weight 3 · round drawn
   Runloop~partialclaimed6/10

   Runloop's marketing site documents '10GB image startup time in <2s' and a 1-minute quickstart, supporting sub-second-to-few-second cold starts, but this is a single first-party marketing claim without detailed benchmarks, methodology, or independent corroboration. missing for 10: independent/third-party benchmark confirmation, documentation of cold start times across varying image sizes/configurations, and reproducible performance data beyond the single homepage claim.

   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

   Blaxel~partialclaimed6/10

   Docs claim resume from standby under 25ms even after weeks of inactivity, implying sub-second warm start, and scale-to-zero after 5s inactivity, which is documented performance behavior specific to sandboxes. However, this is vendor-claimed only with no independent benchmark or hands-on cold-start timing (first-boot from image) reported, and no numbers for true 'cold start' (first creation) vs resume-from-standby. Missing for 10: independent benchmarks/hands-on cold-start measurements, explicit distinction between first-time cold start and standby resume times.

   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”

   proof ↗⚑ flag

Scale

1. platform-engineerRun large concurrent fleets of sandboxes with documented concurrency limits

   weight 3 · round to Runloop
   Runloop~partialclaimed5/10

   Runloop's marketing site claims support for 10k+ parallel sandboxes and sub-2s startup for 10GB images, showing scale is a design goal, but there is no documented concurrency limit, quota tiering, or rate-limit table a platform engineer could plan capacity around. Missing for 10: explicit documented per-account/org concurrency limits, guidance on requesting increases, and independent benchmarks validating the 10k+ claim at sustained load.

   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”

   proof ↗⚑ flag

   Blaxel~partialclaimed3/10

   Docs show sandboxes can run in parallel (Jobs batch processing, scale-to-zero/resume, regional pinning) suggesting fleet-scale operation, but no evidence anywhere states explicit concurrency limits, quotas, or maximum fleet sizes a platform-engineer could plan against. missing for 10: documented per-account/per-region concurrency limits, quota API or dashboard, load-test or benchmark data showing fleet scale in practice.

   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”
   • [claimed-docs] “Jobs allow you to run many AI tasks in parallel using batch processing.”
   • [claimed-docs] “you can optionally pin an agent or MCP server to a specific region by setting the region field in your blaxel.toml”
   • [claimed-docs] “Agent Drive is a distributed filesystem that can be mounted to multiple sandboxes or agents at any time, including while they are already ru…”

   proof ↗⚑ flag

2. developerKeep a sandbox session running for hours or days for long agent tasks

   weight 2 · round to Runloop
   Runloop✓fullclaimed8/10

   Runloop Devboxes explicitly support long-running persistence via snapshots (save/restore disk state), suspend/resume to pause and continue exactly where left off, and Axons providing a persistent event stream that lets agents suspend when idle and wake on demand/recover state — directly addressing multi-hour/multi-day agent sessions without losing progress. Combined with SSH/tunnel access and mounts/secrets for long tasks, this covers the core story well. Missing for 10: independent/hands-on evidence (e.g., a third-party report) confirming multi-day session stability at scale, and explicit documented max session duration/limits.

   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”

   proof ↗⚑ flag

   Blaxel~partialclaimed5/10

   Docs describe scale-to-zero after 5s inactivity with sub-25ms resume even after weeks, implying persistent state across long periods, but this describes a serverless suspend/resume model rather than an explicit 'keep running continuously for hours/days' capability, and there is no documentation of session duration limits or long-running execution during active agent work. missing for 10: explicit session/timeout limits, evidence of continuous multi-hour/day execution without suspension, independent confirmation of long-running task support.

   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”

   proof ↗⚑ flag

Pricing

1. developerStart building with a free tier or included credits without talking to sales

   weight 1 · round to Runloop
   Runloop✓fullclaimed8/10

   Docs explicitly state Runloop's trial includes $50 free credits with no credit card required, and a quickstart lets a developer spin up a first devbox in about a minute, both without any sales contact. Missing for 10: independent/community confirmation of the free-tier signup experience and clarity on long-term free-tier limits beyond trial credits.

   • [claimed-docs] “It includes **$50 in free credits**, and no credit card is required to sign up.”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”

   proof ↗⚑ flag

   Blaxel—none0/10

   No evidence pack items mention pricing, free tier, included credits, or self-service signup without sales contact; all evidence covers technical features (sandboxes, MCP, jobs, regions).

   ⚑ flag

2. platform-engineerPay per second only for the compute a sandbox actually uses

   weight 3 · round to Blaxel
   Runloop—none0/10

   No evidence pack item states an explicit per-second or usage-based billing model; the only tangential mention is that suspend/resume can 'save costs' (runloop-docs-27), which doesn't establish a per-second pricing mechanism.

   ⚑ flag

   Blaxel~partialclaimed3/10

   Docs describe scale-to-zero after 5s of inactivity and fast resume (blaxel-docs-2), implying sandboxes only consume compute while active, which is suggestive of fine-grained usage-based billing, but no evidence explicitly states a per-second pricing/billing model or shows pricing pages confirming this granularity. missing for 10: explicit pricing/billing documentation stating per-second metering, evidence of actual invoicing granularity, independent confirmation of cost savings from scale-to-zero.

   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round to Blaxel
   Runloop—none0/10

   No evidence in the pack mentions data residency, region selection, or geographic storage controls anywhere in Runloop's docs; the pack covers sandboxes, blueprints, snapshots, secrets, and tooling but never data-location choice.

   ⚑ flag

   Blaxel~partialclaimed6/10

   Docs confirm users can pin agents/MCP servers to a specific region via blaxel.toml and that deployment region choice is sometimes available, which directly supports data residency control. Missing for 10: a full list of available regions, compliance/certification details (e.g., GDPR, SOC2 residency guarantees), and independent confirmation that this pinning is enforced for all data at rest (not just compute location).

   • [claimed-docs] “you can optionally pin an agent or MCP server to a specific region by setting the region field in your blaxel.toml”
   • [claimed-docs] “When deploying a resource on Blaxel, you sometimes have the option to choose the deployment region.”

   proof ↗⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Runloop—none0/10

   No evidence anywhere in the pack addresses AI training data usage, opt-out policies, or data-retention commitments for model training; the docs focus on sandbox infrastructure (Devboxes, secrets, gateways) but never mention training-data policy.

   ⚑ flag

   Blaxel—none0/10

   No evidence pack item addresses data usage for AI model training, opt-out policies, or privacy commitments regarding training data; this is an infrastructure/agent-hosting product where such a policy could exist but is undocumented here. Missing for 10: any privacy policy or ToS statement on training data usage, opt-out mechanism, or data retention controls.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Runloop—none0/10

   The evidence pack covers sandboxing, blueprints, snapshots, secrets encryption, and tooling, but contains no documentation of data retention policies, data deletion controls, or user-facing privacy/compliance settings for Devboxes, snapshots, or storage objects. missing for 10: explicit data retention policy docs, user-initiated deletion/purge APIs, compliance/privacy documentation.

   • [claimed-docs] “Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Storage Objects provide a way to store and manage files, data, and other resources that can be shared across Devboxes or made publicly avail…”

   proof ↗⚑ flag

   Blaxel—none0/10

   No evidence in the pack addresses data retention policies, deletion controls, or data lifecycle management for user data/sandboxes; the docs focus on infrastructure, scaling, and access tokens rather than privacy/retention controls.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Runloop—none0/10

   No evidence in the pack mentions telemetry, analytics opt-out, or usage-tracking controls for Runloop; the docs cover sandboxing, blueprints, snapshots, secrets, etc., but nothing about a privacy/telemetry toggle.

   ⚑ flag

   Blaxel—none0/10

   No evidence in the pack addresses telemetry, usage tracking, or any opt-out mechanism; the docs cover deployment, sandboxes, MCP hosting, and security tokens but nothing about data collection settings. missing for 10: any documentation or setting related to telemetry collection or opt-out, privacy policy mention, or usage-tracking configuration.

   ⚑ flag

Lifecycle

1. developerSet timeouts so sandboxes shut down automatically and stop billing when idle or done

   weight 2 · round to Blaxel
   Runloop~partialclaimed5/10

   Runloop documents suspend/resume to "save state and costs" and Axons that let agents "suspend when idle, wake on demand," indicating some idle-triggered lifecycle management, but there is no explicit documentation of a developer-configurable timeout parameter that auto-shuts-down a Devbox and stops billing after inactivity or completion. missing for 10: explicit timeout/idle-shutdown configuration API or CLI flag, confirmation that suspension halts billing automatically without agent action.

   • [claimed-docs] “They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — wi…”
   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Broker sits between an Axon and an agent process running inside a Devbox.”

   proof ↗⚑ flag

   Blaxel~partialclaimed6/10

   Docs confirm automatic scale-to-zero after 5s of inactivity with fast resume, which directly supports idle shutdown and implies billing stops when inactive (blaxel-docs-2). However, there is no evidence of a developer-configurable timeout parameter (e.g., custom idle duration) or explicit confirmation that billing halts during scale-to-zero, and no independent/hands-on corroboration of this behavior. Missing for 10: explicit configurable timeout setting, billing-stop confirmation, independent verification.

   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”

   proof ↗⚑ flag

2. developerSpin up an isolated sandbox with one API/SDK call and get a live environment in seconds

   weight 3 · round to Runloop
   Runloop✓fullclaimed9/10

   Docs describe Devbox creation via SDK/CLI/API, quickstart claiming ~1 minute to a running devbox, blueprints/prebuilt images for fast startup, and marketing claims of 10k+ parallel sandboxes with <2s startup for a 10GB image, matching the one-call, seconds-to-live-environment story. missing for 10: independent/hands-on benchmark confirming real-world provisioning latency beyond vendor claims.

   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes.”
   • [claimed-docs] “Run 10k+ parallel sandboxes 10GB image startup time in <2s”
   • [claimed-docs] “Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for…”
   • [claimed-docs] “This tutorial gets you up and running with your first devbox in about 1 minute.”
   • [claimed-docs] “Runloop provides SDKs in common languages to interact with the Runloop API.”
   • [claimed-docs] “Runloop provides optimized public prebuilt blueprints for common environments.”

   proof ↗⚑ flag

   Blaxel~partialprobed6/10

   Docs confirm sandboxes are a core primitive with fast resume (<25ms from standby), customizable/reusable images, API/CLI/SDK access via API keys, and MCP-server exposure for tool control, all pointing to programmatic creation. However, there's no explicit example or evidence of a single one-call SDK/API invocation creating a sandbox with a concrete latency claim for cold creation (only resume-from-standby timing is given), and no independent/hands-on confirmation of 'seconds to live environment' for a fresh (non-cached) sandbox. missing for 10: explicit one-call code sample creating a sandbox, documented cold-start provisioning latency, independent verification of speed claims.

   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”
   • [claimed-docs] “API keys can be used in the Blaxel APIs, CLI and SDK.”
   • [claimed-docs] “Every sandbox is exposed via an MCP server that allows agents to operate it using tool calls.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.blaxel.ai/llms.txt # Documentation > Blaxel documentation: cloud sandboxes, AI agent hosting, MCP …”

   proof ↗⚑ flag

Preview access

1. developerExpose a port from the sandbox on a public preview URL to reach services running inside

   weight 2 · round drawn
   Runloop✓fullclaimed8/10

   Runloop explicitly documents devbox tunnels for securely accessing ports on a devbox over a simple URL, directly matching the story's requirement to expose a service port publicly. Missing for 10: independent/hands-on confirmation of tunnel reliability and details on URL persistence/security controls beyond the doc snippet.

   • [claimed-docs] “Let’s use devbox tunnels to securely access ports on your devbox over a simple url.”

   proof ↗⚑ flag

   Blaxel✓fullclaimed8/10

   Docs confirm sandboxes support preview URLs, including custom domains and private/token-protected URLs, directly enabling exposing a port to reach services running inside the sandbox via a public URL. Missing for 10: no explicit hands-on example showing the port-to-URL mapping mechanics or independent third-party confirmation.

   • [claimed-docs] “Preview URLs with your own custom domain.”
   • [claimed-docs] “When you create a private preview URL a token is required to access the URL.”

   proof ↗⚑ flag

Snapshots

1. developerPause a running sandbox and resume it later with filesystem and memory state intact

   weight 2 · round to Runloop
   Runloop✓fullclaimed8/10

   Runloop explicitly documents 'Suspend and resume' functionality that pauses sandboxes to save state and resume exactly where left off, plus snapshot capability that saves disk state and can recreate Devboxes from that state. This directly matches the story's pause/resume with filesystem+memory state intact. Missing for 10: no independent/hands-on corroboration of memory-state fidelity across suspend/resume, and no detailed technical explanation of how memory state (not just disk) is preserved.

   • [claimed-docs] “**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”

   proof ↗⚑ flag

   Blaxel~partialclaimed5/10

   Docs mention automatic scale-to-zero after inactivity with resume from standby in under 25ms 'even after weeks' (blaxel-docs-2), implying state is preserved across pause/resume cycles, and Agent Drive provides a persistent distributed filesystem mountable to sandboxes (blaxel-docs-9). However, there is no explicit developer-triggered pause/resume API, and no confirmation that in-memory process state (not just filesystem) is preserved. missing for 10: explicit manual pause/resume command, explicit statement that memory/process state (not just filesystem) survives resume, independent verification of the claim.

   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”
   • [claimed-docs] “Agent Drive is a distributed filesystem that can be mounted to multiple sandboxes or agents at any time, including while they are already ru…”

   proof ↗⚑ flag

2. developerSnapshot a sandbox and later restore or fork new sandboxes from that snapshot

   weight 3 · round to Runloop
   Runloop✓fullclaimed8/10

   Runloop's docs explicitly describe snapshotting a Devbox's disk state and creating new Devboxes from that saved state, including fan-out use cases (multiple approaches from one snapshot) and rollback scenarios, directly matching the snapshot/restore/fork story. missing for 10: independent/hands-on corroboration beyond first-party docs, and no explicit code example showing the fork API call in the evidence pack.

   • [claimed-docs] “Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
   • [claimed-docs] “Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-…”

   proof ↗⚑ flag

   Blaxel—none0/10

   The evidence covers sandbox creation, scale-to-zero/resume, preview URLs, templates, and filesystem mounting, but nowhere mentions snapshotting a sandbox's state or forking new sandboxes from a saved snapshot. Sandbox images/templates ([blaxel-docs-7]) are pre-built environments, not runtime snapshots of a live sandbox's state.

   • [claimed-docs] “Automatic scale-to-zero after 5s inactivity, resume from standby under 25ms even after weeks.”
   • [claimed-docs] “Sandbox images allow you to create customized & reusable sandbox environments.”
   • [claimed-docs] “Agent Drive is a distributed filesystem that can be mounted to multiple sandboxes or agents at any time, including while they are already ru…”

   proof ↗⚑ flag