Skip to content
ProductArena

How Modal Sandboxes’s scores are calculated

The full audit trail, recomputed from the verdict data at build time through the same code that produced the leaderboard: verdict × quality × story weight per cell, cells sum to dimension scores, dimensions blend into the PA Score. Every number on the product page is reproducible from this page alone; for why the formula looks like this, see the methodology.

verdict factors: full ×1.0 · partial ×0.6 · disputed ×0.3 · none ×0.0 · n/a excluded from both sides · cell points = weight × quality × factor · cell max = weight × 10

PA Score20/100

Agent-ready 41.9 × 0.30 = 12.57

API quality 10.0 × 0.20 = 2.00

Openness 7.2 × 0.20 = 1.44

Built-in AI 18.0 × 0.15 = 2.70

Automation 9.0 × 0.15 = 1.35

(12.57 + 2.00 + 1.44 + 2.70 + 1.35) ÷ (0.30 + 0.20 + 0.20 + 0.15 + 0.15) = 20.06 ÷ 1.00 = 20.1

Scores are stored to 1 decimal; the product page’s pills round to whole numbers for display. Each dimension below shows the stories, verdicts, and cited evidence behind its number.

Agent-ready41.9/100×0.30 of the PA blend

Outside-in: can YOUR agent reach and drive this product — API, MCP, CLI, headless runs, agent docs.

Point an agent at llms.txt or agent-oriented docsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [probe] https://modal.com/llms.txt“PROBE llms.txt: HTTP 200 at https://modal.com/llms.txt # Modal Documentation > Modal is a platform for running AI workloads in the cloud with minimal > configuration. Key use”
  • [probe] https://modal.com/docs/guide/sandboxes.md“PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure containers for executing untrusted user or agent code”

Run the product headlessly / in CI for automationweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://modal.com“Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products demand.”
  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“Create a Sandbox with the specified Image, and mount a Volume”
  • [github] https://github.com/modal-labs/modal-client“The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
  • [community] https://news.ycombinator.com/item?id=49274273“Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runners, decided to build it myself on modal. it's self hosted so you can tinker with it say what resources you need explicitly and pay for what you need only'”
  • [community] https://news.ycombinator.com/item?id=49512072“Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runtime for agents and recently got grpc working with modal!'”
  • [claimed-docs] https://modal.com/docs/reference/cli“Run a command or interactive shell inside a Modal container.”
  • [probe] https://modal.com/docs/reference/cli“official CLI documented at https://modal.com/docs/reference/cli”

Plug MCP servers into this product so it can use their toolsweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Connect an agent via an official MCP serverweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Use an official CLIweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://modal.com/docs/reference/cli“Run a command or interactive shell inside a Modal container.”
  • [claimed-docs] https://modal.com/docs/reference/cli“Manage and connect to running containers.”
  • [probe] https://modal.com/docs/reference/cli“official CLI documented at https://modal.com/docs/reference/cli”
  • [claimed-docs] https://modal.com/docs/guide/sandboxes“Running commands”

Drive the product through a documented public APIweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“Create a Sandbox with the specified Image, and mount a Volume”
  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
  • [github] https://github.com/modal-labs/modal-client“The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
  • [claimed-docs] https://modal.com/docs/reference/cli“Run a command or interactive shell inside a Modal container.”
  • [claimed-docs] https://modal.com/docs/reference/cli“Manage and connect to running containers.”
  • [claimed-docs] https://modal.com/docs/sdk/py/releases.md“We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
  • [community] https://news.ycombinator.com/item?id=49512072“Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runtime for agents and recently got grpc working with modal!'”
  • [probe] https://modal.com/docs/guide/sandboxes.md“PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure containers for executing untrusted user or agent code”
  • [probe] https://modal.com/openapi.json“PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, https://modal.com/.well-known/openapi.json)”

Issue scoped/least-privilege API credentials for an agentweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://modal.com“Autonomous agents with the right tools, context, and credentials already in place — running securely in a full, isolated dev environment.”

Build against official SDKsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [github] https://github.com/modal-labs/modal-client“The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“Create a Sandbox with the specified Image, and mount a Volume”
  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”
  • [claimed-docs] https://modal.com/docs/sdk/py/releases.md“We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
  • [community] https://news.ycombinator.com/item?id=49512072“Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runtime for agents and recently got grpc working with modal!'”
  • [community] https://news.ycombinator.com/item?id=49274273“Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runners, decided to build it myself on modal. it's self hosted so you can tinker with it say what resources you need explicitly and pay for what you need only'”

Subscribe to events via webhooksweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Agent-ready = 88.0 ÷ 210 × 100 = 41.9

API quality10.0/100×0.20 of the PA blend

The programmable surface once an agent is there — machine-readable spec, interactive docs, sandbox, versioning discipline.

Explore an interactive API reference with runnable examplesweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://modal.com/openapi.json“PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, https://modal.com/.well-known/openapi.json)”
  • [claimed-docs] https://modal.com/docs/reference/cli“Run a command or interactive shell inside a Modal container.”
  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“const p = await sb.exec(["cat", "/mnt/volume/message.txt"]);”

Download a machine-readable API spec (OpenAPI or equivalent)weight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://modal.com/openapi.json“PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, https://modal.com/.well-known/openapi.json)”

Test against a sandbox environment without touching production dataweight 1

1 (weight) × 7 (quality) × 1.0 (full) = 7.0 of 10 max

  • [claimed-docs] https://modal.com“Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products demand.”
  • [claimed-docs] https://modal.com/docs/guide/sandbox-files“Filesystem Access”
  • [claimed-docs] https://modal.com/docs/guide/sandbox-snapshots“Snapshots”
  • [claimed-docs] https://modal.com/docs/guide/sandbox-networking“Networking and security”
  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
  • [claimed-docs] https://modal.com/docs/examples/agent.md“We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
  • [community] https://news.ycombinator.com/item?id=49274273“Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runners, decided to build it myself on modal. it's self hosted so you can tinker with it say what resources you need explicitly and pay for what you need only'”
  • [community] https://news.ycombinator.com/item?id=49512072“Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runtime for agents and recently got grpc working with modal!'”
  • [probe] https://modal.com/docs/guide/sandboxes.md“PROBE docs-md: HTTP 200 at https://modal.com/docs/guide/sandboxes.md # Sandboxes This page is a high-level guide to Sandboxes, secure containers for executing untrusted user or agent code”

Rely on versioned APIs with a documented deprecation policyweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://modal.com/docs/guide/sandboxes“VM Sandboxes Beta”
  • [claimed-docs] https://modal.com/docs/guide/sandboxes“Sidecars Alpha”
  • [claimed-docs] https://modal.com/docs/sdk/py/releases.md“We've added a modal.Sandbox.logs API, allowing you to fetch() logs from a specific date/time range or to tail() the most recent logs.”
  • [claimed-docs] https://modal.com/docs/sdk/py/releases.md“it's now possible to opt into a more performant Sandbox backend, which affords substantially higher creation rates and concurrency”
  • [probe] https://modal.com/openapi.json“PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, https://modal.com/.well-known/openapi.json)”

API quality = 7.0 ÷ 70 × 100 = 10.0

Openness7.2/100×0.20 of the PA blend

Can you leave, inspect, or self-host — data export, open source, portability.

Do everything through the API that I can do in the UIweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://modal.com/docs/guide/sandbox-files“Filesystem Access”
  • [claimed-docs] https://modal.com/docs/guide/sandbox-snapshots“Snapshots”
  • [claimed-docs] https://modal.com/docs/guide/sandbox-networking“Networking and security”
  • [claimed-docs] https://modal.com/docs/guide/sandboxes“Running commands”
  • [claimed-docs] https://modal.com/docs/reference/cli“Run a command or interactive shell inside a Modal container.”
  • [claimed-docs] https://modal.com/docs/reference/cli“Manage and connect to running containers.”
  • [github] https://github.com/modal-labs/modal-client“The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”
  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
  • [probe] https://modal.com/openapi.json“PROBE openapi: all candidate paths 404 (https://modal.com/openapi.json, https://modal.com/swagger.json, https://modal.com/api/openapi.json, https://modal.com/.well-known/openapi.json)”
  • [probe] https://modal.com/docs/reference/cli“official CLI documented at https://modal.com/docs/reference/cli”

Export all of my data in open formats and leaveweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Read the product's source under an open licenseweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [github] https://github.com/modal-labs/modal-client“The JS and Go SDKs allow you to use Modal Sandboxes, invoke deployed Modal Functions, and interact with some Modal platform resources.”

Self-host the core productweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Openness = 7.2 ÷ 100 × 100 = 7.2

Built-in AI18.0/100×0.15 of the PA blend

Inside-out: how agentic the product itself is for its users — built-in assistants, autonomous features.

Get AI-generated insights and suggestions from my data inside the productweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Set up automations that run autonomously in the backgroundweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://modal.com“Autonomous agents with the right tools, context, and credentials already in place — running securely in a full, isolated dev environment.”
  • [claimed-docs] https://modal.com“Spin up hundreds of thousands of concurrent rollout environments in seconds.”
  • [claimed-docs] https://modal.com/docs/examples/agent.md“We execute the agent's code in a Modal Sandbox, which allows us to run arbitrary code in a safe environment.”
  • [community] https://news.ycombinator.com/item?id=49274273“Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runners, decided to build it myself on modal. it's self hosted so you can tinker with it say what resources you need explicitly and pay for what you need only'”
  • [community] https://news.ycombinator.com/item?id=49512072“Developer built MCP-JS, a JavaScript runtime for agents, and got gRPC working to control Modal Sandboxes: 'been working on a javascript runtime for agents and recently got grpc working with modal!'”

Delegate tasks to a built-in AI assistant inside the productweight 3

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Operate the product with natural-language commandsweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://modal.com“Spin up fresh, isolated sandboxes programmatically. Custom images, any dependency — built for the latency and scale consumer AI products demand.”
  • [claimed-docs] https://modal.com/docs/reference/cli“Run a command or interactive shell inside a Modal container.”
  • [claimed-docs] https://modal.com/docs/guide/sdk-javascript-go.md“Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents.”
  • [probe] https://modal.com/docs/reference/cli“official CLI documented at https://modal.com/docs/reference/cli”

Built-in AI = 7.2 ÷ 40 × 100 = 18.0

Automation9.0/100×0.15 of the PA blend

Depth of automation primitives — rules, scheduling, bulk operations, webhooks.

Perform bulk operations across many items at onceweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://modal.com“Spin up hundreds of thousands of concurrent rollout environments in seconds.”
  • [claimed-docs] https://modal.com“Spin up hundreds of thousands of concurrent rollout environments in seconds. Fast enough to keep your GPU inference resources saturated across every episode.”
  • [claimed-docs] https://modal.com“H100s, A100s, A10Gs available on demand. Attach to any sandbox, scale to thousands of concurrent runs, pay by the second with no reserved capacity.”
  • [community] https://news.ycombinator.com/item?id=49274273“Built GitHub Actions self-hosted runners on Modal Sandboxes: 'was looking for a long time for gpu-native scalable and highly available runners, decided to build it myself on modal. it's self hosted so you can tinker with it say what resources you need explicitly and pay for what you need only'”
  • [claimed-docs] https://modal.com“Integrated logging and full visibility into every function, sandbox, and container.”

Define rules that trigger actions automatically on eventsweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Schedule recurring jobs or workflowsweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://modal.com/docs/guide/sandboxes“VM Sandboxes Beta”
  • [claimed-docs] https://modal.com/docs/guide/sandboxes“Running commands”
  • [claimed-docs] https://modal.com/docs/guide/sandbox-snapshots“Snapshots”

Version, review, and roll back my automationsweight 1

1 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 10 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Automation = 7.2 ÷ 80 × 100 = 9.0