Skip to content
ProductArena

How Devin’s scores are calculated

The full audit trail, recomputed from the verdict data at build time through the same code that produced the leaderboard: verdict × quality × story weight per cell, cells sum to dimension scores, dimensions blend into the PA Score. Every number on the product page is reproducible from this page alone; for why the formula looks like this, see the methodology.

verdict factors: full ×1.0 · partial ×0.6 · disputed ×0.3 · none ×0.0 · n/a excluded from both sides · cell points = weight × quality × factor · cell max = weight × 10

PA Score30/100

Agent-ready 49.0 × 0.30 = 14.70

API quality 5.1 × 0.20 = 1.02

Openness 6.0 × 0.20 = 1.20

Built-in AI 63.1 × 0.15 = 9.46

Automation 25.5 × 0.15 = 3.82

(14.70 + 1.02 + 1.20 + 9.46 + 3.82) ÷ (0.30 + 0.20 + 0.20 + 0.15 + 0.15) = 30.21 ÷ 1.00 = 30.2

Scores are stored to 1 decimal; the product page’s pills round to whole numbers for display. Each dimension below shows the stories, verdicts, and cited evidence behind its number.

Agent-ready49.0/100×0.30 of the PA blend

Outside-in: can YOUR agent reach and drive this product — API, MCP, CLI, headless runs, agent docs.

Point an agent at llms.txt or agent-oriented docsweight 2

2 (weight) × 9 (quality) × 1.0 (full) = 18.0 of 20 max

  • [probe] https://docs.devin.ai/llms.txt“PROBE llms.txt: HTTP 200 at https://docs.devin.ai/llms.txt # Devin Docs - [Desktop (100 pages)](https://docs.devin.ai/_llms/en/desktop.md): Documentation for Desktop. - [API (302”
  • [probe] https://docs.devin.ai/get-started/devin-intro.md“PROBE docs-md: HTTP 200 at https://docs.devin.ai/get-started/devin-intro.md > ## Documentation Index > Fetch the complete documentation index at: https://docs.devin.ai/llms.txt > Use this file to”
  • [claimed-docs] https://docs.devin.ai/onboard-devin/agents-md.md“Devin supports AGENTS.md - a simple, open standard for providing context and instructions to AI agents.”

Run the product headlessly / in CI for automationweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/sdlc-integration.md“Integrate Devin into your CI/CD pipeline to respond to findings from static analysis tools like SonarQube, Fortify, or Veracode.”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/sdlc-integration.md“With Auto-Fix enabled, Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures — creating a closed loop where PRs iterate toward merge-ready quality without you in the loop.”
  • [claimed-docs] https://docs.devin.ai/cli/sandbox.md“The --sandbox flag runs the CLI with OS-level isolation, enforcing writable paths and deny rules at the operating-system level and optionally restricting network traffic.”
  • [claimed-docs] https://docs.devin.ai/cli/index.md“a local command-line coding agent with deep Devin Cloud integration”

Plug MCP servers into this product so it can use their toolsweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”
  • [probe] https://docs.devin.ai/work-with-devin/devin-mcp“official MCP server documented at https://docs.devin.ai/work-with-devin/devin-mcp”

Connect an agent via an official MCP serverweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”
  • [probe] https://docs.devin.ai/work-with-devin/devin-mcp“official MCP server documented at https://docs.devin.ai/work-with-devin/devin-mcp”

Use an official CLIweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/cli/index.md“Devin CLI, a local command-line coding agent with deep Devin Cloud integration.”
  • [claimed-docs] https://docs.devin.ai/cli/index.md“devin -- check out this code and suggest a feasible, helpful feature”
  • [claimed-docs] https://docs.devin.ai/cli/sandbox.md“The --sandbox flag runs the CLI with OS-level isolation, enforcing writable paths and deny rules at the operating-system level and optionally restricting network traffic.”
  • [claimed-docs] https://docs.devin.ai/cli/index.md“a local command-line coding agent with deep Devin Cloud integration”
  • [probe] https://docs.devin.ai/cli/index“official CLI documented at https://docs.devin.ai/cli/index”

Drive the product through a documented public APIweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/cli/index.md“Devin CLI, a local command-line coding agent with deep Devin Cloud integration.”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”
  • [probe] https://docs.devin.ai/work-with-devin/devin-mcp“official MCP server documented at https://docs.devin.ai/work-with-devin/devin-mcp”
  • [probe] https://docs.devin.ai/cli/index“official CLI documented at https://docs.devin.ai/cli/index”
  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”

Issue scoped/least-privilege API credentials for an agentweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”

Build against official SDKsweight 2

2 (weight) × 4 (quality) × 0.6 (partial) = 4.8 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”

Subscribe to events via webhooksweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Agent-ready = 102.8 ÷ 210 × 100 = 49.0

API quality5.1/100×0.20 of the PA blend

The programmable surface once an agent is there — machine-readable spec, interactive docs, sandbox, versioning discipline.

Explore an interactive API reference with runnable examplesweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”

Download a machine-readable API spec (OpenAPI or equivalent)weight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”

Test against a sandbox environment without touching production dataweight 1

1 (weight) × 6 (quality) × 0.6 (partial) = 3.6 of 10 max

  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-handoff.md“The cloud session gets its own VM with a shell, browser, and full repo access, so it can keep going after you close your laptop.”
  • [claimed-docs] https://docs.devin.ai/onboard-devin/environment.md“Configure it once, and every session boots into that known-good state.”
  • [claimed-docs] https://docs.devin.ai/cli/sandbox.md“The --sandbox flag runs the CLI with OS-level isolation, enforcing writable paths and deny rules at the operating-system level and optionally restricting network traffic.”
  • [claimed-docs] https://docs.devin.ai/cli/sandbox.md“The `--sandbox` flag runs the CLI with OS-level isolation, enforcing writable paths and `deny` rules at the operating-system level”
  • [claimed-docs] https://docs.devin.ai/cloud/outposts/overview.md“Outposts lets you run Devin sessions inside infrastructure you control — your own VMs, containers, Kubernetes clusters, or even a Mac Mini on your desk.”
  • [claimed-docs] https://docs.devin.ai/onboard-devin/environment/blueprints.md“Devin inspects your repository, figures out which tools, runtimes, and dependencies are needed, and generates the blueprint for you.”

Rely on versioned APIs with a documented deprecation policyweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”

API quality = 3.6 ÷ 70 × 100 = 5.1

Openness6.0/100×0.20 of the PA blend

Can you leave, inspect, or self-host — data export, open source, portability.

Do everything through the API that I can do in the UIweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”
  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”

Export all of my data in open formats and leaveweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Read the product's source under an open licenseweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-handoff.md“Devin Handoff is an open-source plugin and skill that brings the same handoff workflow to any coding agent — Claude Code, Codex, Cursor, and more.”

Self-host the core productweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

  • [claimed-docs] https://docs.devin.ai/cloud/outposts/overview.md“Outposts lets you run Devin sessions inside infrastructure you control — your own VMs, containers, Kubernetes clusters, or even a Mac Mini on your desk.”

Openness = 6.0 ÷ 100 × 100 = 6.0

Built-in AI63.1/100×0.15 of the PA blend

Inside-out: how agentic the product itself is for its users — built-in assistants, autonomous features.

Get AI-generated insights and suggestions from my data inside the productweight 2

2 (weight) × 7 (quality) × 1.0 (full) = 14.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/essential-guidelines/sdlc-integration.md“Ask Devin can answer questions about code structure and dependencies, and help you scope and plan tasks before implementation.”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/sdlc-integration.md“Devin Review provides automated first-pass reviews on pull requests, checking for correctness and conformance with organizational best practices.”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/sdlc-integration.md“Use DeepWiki to navigate architecture and code with auto-generated documentation.”
  • [claimed-docs] https://docs.devin.ai/onboard-devin/index-repo.md“Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki”

Set up automations that run autonomously in the backgroundweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-handoff.md“The cloud session gets its own VM with a shell, browser, and full repo access, so it can keep going after you close your laptop.”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/sdlc-integration.md“With Auto-Fix enabled, Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures — creating a closed loop where PRs iterate toward merge-ready quality without you in the loop.”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/sdlc-integration.md“Integrate Devin into your CI/CD pipeline to respond to findings from static analysis tools like SonarQube, Fortify, or Veracode.”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Carve out independent tasks and run them simultaneously.”

Delegate tasks to a built-in AI assistant inside the productweight 3

3 (weight) × 6 (quality) × 0.6 (partial) = 10.8 of 30 max

  • [claimed-docs] https://docs.devin.ai/get-started/devin-intro“Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
  • [claimed-docs] https://docs.devin.ai/get-started/devin-intro“Devin is designed to be a conversational user interface, and allows you to follow and take over Devin's development process in the embedded IDE.”
  • [claimed-docs] https://docs.devin.ai/learn-about-devin/workflows.md“Tag Devin on a Slack or Teams thread about a bug you're discussing with coworkers”
  • [community] https://hn.algolia.com/api/v1/items/42734681“Trialed Devin, it's quite impressive when it understands code formatting and local test setup, but it always adds extraneous changes beyond the task that can break other things, and can't undo those changes if asked. Recommending cancelling it because it's not actually saving time and it's quite expensive.”
  • [community] https://hn.algolia.com/api/v1/items/42734681“One thing that surprised me is there doesn't seem to be an 'ask for help' escape hatch in Devin - it would work away for literally days on a task where any human would admit they were stuck.”
  • [community] https://hn.algolia.com/api/v1/items/42734681“I've learned to just click 'Terminate Session' immediately after spotting Devin doing something hopeless. I've managed to get real work done out of it without much effort, just checking every 10-15 minutes and typing a few lines or restarting session.”

Operate the product with natural-language commandsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/get-started/devin-intro“Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
  • [claimed-docs] https://docs.devin.ai/get-started/devin-intro“Tagging Devin on a Slack or Teams thread about a bug you're discussing with coworkers”
  • [claimed-docs] https://docs.devin.ai/get-started/devin-intro“Devin is designed to be a conversational user interface, and allows you to follow and take over Devin's development process in the embedded IDE.”
  • [claimed-docs] https://docs.devin.ai/cli/index.md“devin -- check out this code and suggest a feasible, helpful feature”
  • [claimed-docs] https://docs.devin.ai/learn-about-devin/workflows.md“Tag Devin on a Slack or Teams thread about a bug you're discussing with coworkers”
  • [community] https://hn.algolia.com/api/v1/items/42734681“Trialed Devin, it's quite impressive when it understands code formatting and local test setup, but it always adds extraneous changes beyond the task that can break other things, and can't undo those changes if asked. Recommending cancelling it because it's not actually saving time and it's quite expensive.”
  • [community] https://hn.algolia.com/api/v1/items/42734681“I've learned to just click 'Terminate Session' immediately after spotting Devin doing something hopeless. I've managed to get real work done out of it without much effort, just checking every 10-15 minutes and typing a few lines or restarting session.”

Built-in AI = 56.8 ÷ 90 × 100 = 63.1

Automation25.5/100×0.15 of the PA blend

Depth of automation primitives — rules, scheduling, bulk operations, webhooks.

Perform bulk operations across many items at onceweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Carve out independent tasks and run them simultaneously.”
  • [claimed-docs] https://docs.devin.ai/get-started/devin-intro“Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
  • [community] https://hn.algolia.com/api/v1/items/42734681“Trialed Devin, it's quite impressive when it understands code formatting and local test setup, but it always adds extraneous changes beyond the task that can break other things, and can't undo those changes if asked. Recommending cancelling it because it's not actually saving time and it's quite expensive.”

Define rules that trigger actions automatically on eventsweight 3

3 (weight) × 6 (quality) × 0.6 (partial) = 10.8 of 30 max

  • [claimed-docs] https://docs.devin.ai/essential-guidelines/sdlc-integration.md“With Auto-Fix enabled, Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures — creating a closed loop where PRs iterate toward merge-ready quality without you in the loop.”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/sdlc-integration.md“Integrate Devin into your CI/CD pipeline to respond to findings from static analysis tools like SonarQube, Fortify, or Veracode.”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixe”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”

Schedule recurring jobs or workflowsweight 2

2 (weight) × 3 (quality) × 0.6 (partial) = 3.6 of 20 max

  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp.md“gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling”

Version, review, and roll back my automationsweight 1

1 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 10 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Automation = 20.4 ÷ 80 × 100 = 25.5