How Runloop’s scores are calculated

Point an agent at llms.txt or agent-oriented docsweight 2

2 (weight) × 9 (quality) × 1.0 (full) = 18.0 of 20 max

• [probe] https://docs.runloop.ai/llms.txt“PROBE llms.txt: HTTP 200 at https://docs.runloop.ai/llms.txt # Runloop > Runloop is the best platform for running isolated Linux sandbox environments (Devboxes) for AI coding agent”
• [probe] https://docs.runloop.ai/docs/overview/what-is-runloop.md“PROBE docs-md: HTTP 200 at https://docs.runloop.ai/docs/overview/what-is-runloop.md > ## Documentation Index > Fetch the complete documentation index at: https://docs.runloop.ai/llms.txt > Use this file t”
• [claimed-docs] https://docs.runloop.ai/docs/tools/cursor-files“We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operations, command execution, blueprints, snapshots, tunnels, and more.”

Run the product headlessly / in CI for automationweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

• [claimed-docs] https://docs.runloop.ai/docs/benchmarks/overview“Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregating outputs. Compare multiple agents side-by-side with a single command.”
• [claimed-docs] https://docs.runloop.ai/docs/tools/rl-cli“The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/agents/using-agents-api“The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
• [claimed-docs] https://docs.runloop.ai/docs/tools/sdks“Runloop provides SDKs in common languages to interact with the Runloop API.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/agents/deploying-with-github-actions“Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”

Plug MCP servers into this product so it can use their toolsweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

• [probe] https://docs.runloop.ai/docs/tools/ai-tools“official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”

Connect an agent via an official MCP serverweight 3

3 (weight) × 7 (quality) × 1.0 (full) = 21.0 of 30 max

• [probe] https://docs.runloop.ai/docs/tools/ai-tools“official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”

Use an official CLIweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

• [claimed-docs] https://docs.runloop.ai/docs/tools/rl-cli“The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
• [claimed-docs] https://docs.runloop.ai/docs/tools/rl-cli“From the interactive menu, select a devbox and choose **SSH** to open a secure shell session directly into your devbox. The CLI handles all the SSH key setup and connection details automatically.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/blueprints/overview“rli blueprint from-dockerfile --name my-agent-env”
• [claimed-docs] https://docs.runloop.ai/docs/benchmarks/overview“Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregating outputs. Compare multiple agents side-by-side with a single command.”
• [probe] https://docs.runloop.ai/docs/tools/rl-cli“official CLI documented at https://docs.runloop.ai/docs/tools/rl-cli”

Drive the product through a documented public APIweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

• [claimed-docs] https://docs.runloop.ai/docs/devboxes/agents/using-agents-api“The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
• [claimed-docs] https://docs.runloop.ai/docs/tools/sdks“Runloop provides SDKs in common languages to interact with the Runloop API.”
• [claimed-docs] https://docs.runloop.ai/docs/tools/rl-cli“The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/agents/deploying-with-github-actions“Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”
• [probe] https://docs.runloop.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/api/openapi.json, https://docs.runloop.ai/.well-known/openapi.json)”
• [probe] https://docs.runloop.ai/docs/tools/rl-cli“official CLI documented at https://docs.runloop.ai/docs/tools/rl-cli”

Issue scoped/least-privilege API credentials for an agentweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

• [claimed-docs] https://docs.runloop.ai/docs/devboxes/ai-gateways“Agent Gateways let your agents call LLM APIs like Anthropic and OpenAI **without ever seeing your API keys**.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/configuration/account-secrets“Secrets are encrypted at rest and automatically made available as environment variables in your Devboxes.”

Build against official SDKsweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

• [claimed-docs] https://docs.runloop.ai/docs/tools/sdks“Runloop provides SDKs in common languages to interact with the Runloop API.”
• [claimed-docs] https://docs.runloop.ai/docs/tutorials/openai-agentssdk-runloop“**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
• [claimed-docs] https://docs.runloop.ai/docs/tools/cursor-files“We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operations, command execution, blueprints, snapshots, tunnels, and more.”

Subscribe to events via webhooksweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Explore an interactive API reference with runnable examplesweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

• [probe] https://docs.runloop.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/api/openapi.json, https://docs.runloop.ai/.well-known/openapi.json)”

Download a machine-readable API spec (OpenAPI or equivalent)weight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

• [probe] https://docs.runloop.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/api/openapi.json, https://docs.runloop.ai/.well-known/openapi.json)”
• [claimed-docs] https://docs.runloop.ai/docs/tools/sdks“Runloop provides SDKs in common languages to interact with the Runloop API.”

Test against a sandbox environment without touching production dataweight 1

1 (weight) × 8 (quality) × 1.0 (full) = 8.0 of 10 max

• [claimed-docs] https://docs.runloop.ai/docs/devboxes/overview“Runloop provides secure sandboxed execution environments called Devboxes.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/overview“Runloop provides secure sandboxed execution environments called Devboxes. Runloop Devboxes provide a full-featured execution environment for your AI agents.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/snapshots“Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/snapshots“Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-out and attempt multiple approaches to a code change.”
• [claimed-docs] https://runloop.ai“Run 10k+ parallel sandboxes 10GB image startup time in <2s”
• [claimed-docs] https://docs.runloop.ai/docs/tutorials/quickstart“This tutorial gets you up and running with your first devbox in about 1 minute.”

Rely on versioned APIs with a documented deprecation policyweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

• [probe] https://docs.runloop.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/api/openapi.json, https://docs.runloop.ai/.well-known/openapi.json)”

Do everything through the API that I can do in the UIweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

• [claimed-docs] https://docs.runloop.ai/docs/tools/rl-cli“The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/agents/using-agents-api“The Agents API allows you to create, manage, and deploy AI agents on the Runloop platform.”
• [claimed-docs] https://docs.runloop.ai/docs/tools/sdks“Runloop provides SDKs in common languages to interact with the Runloop API.”
• [claimed-docs] https://docs.runloop.ai/docs/benchmarks/overview“Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregating outputs. Compare multiple agents side-by-side with a single command.”
• [probe] https://docs.runloop.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.runloop.ai/openapi.json, https://docs.runloop.ai/swagger.json, https://docs.runloop.ai/api/openapi.json, https://docs.runloop.ai/.well-known/openapi.json)”

Export all of my data in open formats and leaveweight 3

3 (weight) × 4 (quality) × 0.6 (partial) = 7.2 of 30 max

• [claimed-docs] https://docs.runloop.ai/docs/devboxes/snapshots“Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/blueprints/overview“rli blueprint from-dockerfile --name my-agent-env”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/blueprints/overview“Runloop provides optimized public prebuilt blueprints for common environments.”

Read the product's source under an open licenseweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Self-host the core productweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Get AI-generated insights and suggestions from my data inside the productweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Set up automations that run autonomously in the backgroundweight 2

2 (weight) × 7 (quality) × 0.6 (partial) = 8.4 of 20 max

• [claimed-docs] https://docs.runloop.ai/docs/axons/overview“They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — without you building the plumbing.”
• [claimed-docs] https://docs.runloop.ai/docs/axons/broker“Broker sits between an Axon and an agent process running inside a Devbox.”
• [claimed-docs] https://docs.runloop.ai/docs/tutorials/openai-agentssdk-runloop“**Suspend and resume**: Pause sandboxes to save state and costs, then resume exactly where you left off”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/agents/deploying-with-github-actions“Runloop’s [deploy-agent GitHub Action] automates agent deployment directly from your GitHub workflows.”

Delegate tasks to a built-in AI assistant inside the productweight 3

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Operate the product with natural-language commandsweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

• [probe] https://docs.runloop.ai/docs/tools/ai-tools“official MCP server documented at https://docs.runloop.ai/docs/tools/ai-tools”
• [claimed-docs] https://docs.runloop.ai/docs/tools/cursor-files“We provide [Cursor rules] files (`.mdc`) that give your AI assistant context about the Runloop SDK — covering devbox lifecycle, file operations, command execution, blueprints, snapshots, tunnels, and more.”
• [claimed-docs] https://docs.runloop.ai/docs/tools/rl-cli“The Runloop CLI (`rli`) provides both an interactive terminal UI and traditional CLI commands for managing your Runloop resources.”

Perform bulk operations across many items at onceweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

• [claimed-docs] https://runloop.ai“Run 10k+ parallel sandboxes 10GB image startup time in <2s”
• [claimed-docs] https://docs.runloop.ai/docs/benchmarks/overview“Submit a benchmark job via the CLI and let Runloop handle everything: provisioning devboxes, running agents, scoring results, and aggregating outputs. Compare multiple agents side-by-side with a single command.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/snapshots“Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-out and attempt multiple approaches to a code change.”
• [claimed-docs] https://docs.runloop.ai/docs/benchmarks/overview“Run your agent against well-known open source benchmarks like terminal bench 2, AIME, and more.”

Define rules that trigger actions automatically on eventsweight 3

3 (weight) × 4 (quality) × 0.6 (partial) = 7.2 of 30 max

• [claimed-docs] https://docs.runloop.ai/docs/axons/overview“They give agents a persistent event stream that lets them suspend when idle, wake on demand, recover state, and hand off between agents — without you building the plumbing.”
• [claimed-docs] https://docs.runloop.ai/docs/axons/broker“Broker sits between an Axon and an agent process running inside a Devbox.”

Schedule recurring jobs or workflowsweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Version, review, and roll back my automationsweight 1

1 (weight) × 4 (quality) × 0.6 (partial) = 2.4 of 10 max

• [claimed-docs] https://docs.runloop.ai/docs/devboxes/snapshots“Snapshots can be used to: Improve build times by snapshotting a populated build cache. Roll back to a known good point in time. Perform fan-out and attempt multiple approaches to a code change.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/blueprints/overview“Blueprints allow you to optimize your workflows by specifying these startup actions once, then reuse them across multiple Devboxes.”
• [claimed-docs] https://docs.runloop.ai/docs/devboxes/snapshots“Snapshots can be used to save the current disk state of a Devbox, and to create new Devboxes from a previously saved state.”