Skip to content
ProductArena

How Devin’s scores are calculated

The full audit trail, recomputed from the verdict data at build time through the same code that produced the leaderboard: verdict × quality × story weight per cell, cells sum to dimension scores, dimensions blend into the PA Score. Every number on the product page is reproducible from this page alone; for why the formula looks like this, see the methodology.

verdict factors: full ×1.0 · partial ×0.6 · disputed ×0.3 · none ×0.0 · n/a excluded from both sides · cell points = weight × quality × factor · cell max = weight × 10

PA Score39/100

Agent-ready 67.0 × 0.30 = 20.10

API quality 2.6 × 0.20 = 0.52

Openness 6.0 × 0.20 = 1.20

Built-in AI 70.2 × 0.15 = 10.53

Automation 44.7 × 0.15 = 6.71

(20.10 + 0.52 + 1.20 + 10.53 + 6.71) ÷ (0.30 + 0.20 + 0.20 + 0.15 + 0.15) = 39.05 ÷ 1.00 = 39.1

Scores are stored to 1 decimal; the product page’s pills round to whole numbers for display. Each dimension below shows the stories, verdicts, and cited evidence behind its number.

Agent-ready67.0/100×0.30 of the PA blend

Outside-in: can YOUR agent reach and drive this product — API, MCP, CLI, headless runs, agent docs.

Point an agent at llms.txt or agent-oriented docsweight 2

2 (weight) × 7 (quality) × 1.0 (full) = 14.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/onboard-devin/agents-md.md“Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
  • [probe] https://docs.devin.ai/llms.txt“PROBE llms.txt: HTTP 200 at https://docs.devin.ai/llms.txt # Devin Docs - [Desktop (100 pages)](https://docs.devin.ai/_llms/en/desktop.md): Documentation for Desktop. - [API (302”

Run the product headlessly / in CI for automationweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”

Plug MCP servers into this product so it can use their toolsweight 3

3 (weight) × 9 (quality) × 1.0 (full) = 27.0 of 30 max

  • [claimed-docs] https://docs.devin.ai/cli/extensibility/mcp/overview“MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers, and any other service you can wrap in an MCP server.”
  • [claimed-docs] https://docs.devin.ai/cli/extensibility/mcp/overview“MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
  • [claimed-docs] https://docs.devin.ai/cli/extensibility/mcp/overview“Devin CLI exposes each one as a slash command”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
  • [probe] https://docs.devin.ai/cli/extensibility/mcp/overview“official MCP server documented at https://docs.devin.ai/cli/extensibility/mcp/overview”

Connect an agent via an official MCP serverweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“create sessions, manage playbooks and knowledge, set up schedules, and more”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“These tools let you explore and query documentation for any GitHub repository (public or private with authentication)”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”

Use an official CLIweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://docs.devin.ai“Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
  • [claimed-docs] https://docs.devin.ai/cli/adaptive“Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
  • [claimed-docs] https://docs.devin.ai/cli/adaptive“You can switch away from Adaptive to a specific model at any time with /model.”
  • [claimed-docs] https://docs.devin.ai/cli/extensibility/mcp/overview“MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers, and any other service you can wrap in an MCP server.”
  • [claimed-docs] https://docs.devin.ai/cli/extensibility/mcp/overview“Devin CLI exposes each one as a slash command”
  • [probe] https://docs.devin.ai/cli/adaptive“official CLI documented at https://docs.devin.ai/cli/adaptive”

Drive the product through a documented public APIweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“create sessions, manage playbooks and knowledge, set up schedules, and more”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
  • [probe] https://docs.devin.ai/llms.txt“PROBE llms.txt: HTTP 200 at https://docs.devin.ai/llms.txt # Devin Docs - [Desktop (100 pages)](https://docs.devin.ai/_llms/en/desktop.md): Documentation for Desktop. - [API (302”
  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”

Issue scoped/least-privilege API credentials for an agentweight 2

2 (weight) × 3 (quality) × 0.6 (partial) = 3.6 of 20 max

  • [claimed-docs] https://docs.devin.ai/cli/extensibility/mcp/overview“MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”

Build against official SDKsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/cli/extensibility/mcp/overview“MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers, and any other service you can wrap in an MCP server.”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“create sessions, manage playbooks and knowledge, set up schedules, and more”
  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”

Subscribe to events via webhooksweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Agent-ready = 140.6 ÷ 210 × 100 = 67.0

API quality2.6/100×0.20 of the PA blend

The programmable surface once an agent is there — machine-readable spec, interactive docs, sandbox, versioning discipline.

Explore an interactive API reference with runnable examplesweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”

Download a machine-readable API spec (OpenAPI or equivalent)weight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”

Test against a sandbox environment without touching production dataweight 1

1 (weight) × 3 (quality) × 0.6 (partial) = 1.8 of 10 max

  • [claimed-docs] https://docs.devin.ai/onboard-devin/environment.md“Configure it once, and every session boots into that known-good state.”
  • [claimed-docs] https://docs.devin.ai/onboard-devin/vpn.md“Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and internal Git hosts.”

Rely on versioned APIs with a documented deprecation policyweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”

API quality = 1.8 ÷ 70 × 100 = 2.6

Openness6.0/100×0.20 of the PA blend

Can you leave, inspect, or self-host — data export, open source, portability.

Do everything through the API that I can do in the UIweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“create sessions, manage playbooks and knowledge, set up schedules, and more”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
  • [probe] https://docs.devin.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/openapi.json, https://docs.devin.ai/.well-known/openapi.json)”

Export all of my data in open formats and leaveweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Read the product's source under an open licenseweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Self-host the core productweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

  • [claimed-docs] https://docs.devin.ai“Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
  • [claimed-docs] https://docs.devin.ai“using /handoff to send longer tasks to cloud Devin”
  • [claimed-docs] https://docs.devin.ai/onboard-devin/environment.md“Configure it once, and every session boots into that known-good state.”

Openness = 6.0 ÷ 100 × 100 = 6.0

Built-in AI70.2/100×0.15 of the PA blend

Inside-out: how agentic the product itself is for its users — built-in assistants, autonomous features.

Get AI-generated insights and suggestions from my data inside the productweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://docs.devin.ai/onboard-devin/index-repo.md“Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
  • [claimed-docs] https://docs.devin.ai“Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
  • [claimed-docs] https://docs.devin.ai/get-started/devin-intro.md“Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”

Set up automations that run autonomously in the backgroundweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“create sessions, manage playbooks and knowledge, set up schedules, and more”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

Delegate tasks to a built-in AI assistant inside the productweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

  • [claimed-docs] https://docs.devin.ai“Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
  • [claimed-docs] https://docs.devin.ai“using /handoff to send longer tasks to cloud Devin”
  • [claimed-docs] https://docs.devin.ai“Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
  • [claimed-docs] https://docs.devin.ai“Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
  • [community] https://news.ycombinator.com/item?id=39679787“From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”

Operate the product with natural-language commandsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://docs.devin.ai“Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
  • [claimed-docs] https://docs.devin.ai“Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
  • [claimed-docs] https://docs.devin.ai“Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [community] https://news.ycombinator.com/item?id=39679787“Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a try-catch. While this will get the code to run, I immediately imagined cases where it's not actually an ideal solution.”

Built-in AI = 63.2 ÷ 90 × 100 = 70.2

Automation44.7/100×0.15 of the PA blend

Depth of automation primitives — rules, scheduling, bulk operations, webhooks.

Perform bulk operations across many items at onceweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”

Define rules that trigger actions automatically on eventsweight 3

3 (weight) × 7 (quality) × 0.6 (partial) = 12.6 of 30 max

  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
  • [claimed-docs] https://docs.devin.ai/essential-guidelines/when-to-use-devin.md“Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
  • [claimed-docs] https://docs.devin.ai/api-reference/overview“The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“create sessions, manage playbooks and knowledge, set up schedules, and more”

Schedule recurring jobs or workflowsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“create sessions, manage playbooks and knowledge, set up schedules, and more”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”

Version, review, and roll back my automationsweight 1

1 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 10 max

  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“create sessions, manage playbooks and knowledge, set up schedules, and more”
  • [claimed-docs] https://docs.devin.ai/work-with-devin/devin-mcp“Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
  • [claimed-docs] https://docs.devin.ai/onboard-devin/knowledge-onboarding.md“Knowledge is the best way to share codebase-level (vs. task-level) context that can help Devin when working in your codebase.”

Automation = 35.8 ÷ 80 × 100 = 44.7