Devin vs Codegen

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Codegen
   Devin✓fullprobed7/10

   Devin explicitly supports AGENTS.md, an agent-oriented documentation file that Devin reads before starting a coding session to gain project context — directly matching the story's intent of pointing an agent at agent-oriented docs. Devin's own docs site also serves an llms.txt endpoint (verified via probe), showing familiarity with the convention, though this is Devin's own site rather than evidence Devin ingests arbitrary llms.txt files from user projects. Missing for 10: explicit documentation that Devin consumes/parses llms.txt files themselves (as opposed to AGENTS.md) and independent hands-on confirmation of this workflow.

   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.devin.ai/llms.txt # Devin Docs - [Desktop (100 pages)](https://docs.devin.ai/_llms/en/desktop.md):…”

   proof ↗⚑ flag

   Codegen✓fullprobed9/10

   A probe confirms Codegen's docs site serves a working llms.txt at https://docs.codegen.com/llms.txt (HTTP 200) listing structured doc links, directly enabling an agent to be pointed at agent-oriented docs; this is corroborated by extensive first-party documentation of agent-friendly interfaces (SDK, CLI, API). Missing for 10: independent third-party confirmation of agents actually consuming the llms.txt in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.codegen.com/llms.txt # Codegen ## Docs - [Codegen](https://docs.codegen.com/introduction/overview…”
   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round drawn
   Devin✓fullclaimed8/10

   Devin exposes a full REST API for session creation and automation (devin-docs-8, devin-docs-9), supports scheduling recurring/one-time headless runs via MCP (devin-docs-17), auto-triggers on CI failures and code review comments via Devin Review/Auto-Fix (devin-docs-20), and supports parallel/headless session orchestration (devin-docs-18). This covers CI/automation use cases well beyond interactive use. Missing for 10: no independent/hands-on CI-integration case study, and no dedicated 'CI runner' or GitHub Actions-style first-class integration doc beyond the review/auto-fix hook.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”

   proof ↗⚑ flag

   Codegen✓fullprobed8/10

   Codegen explicitly supports headless/automated usage via a Python SDK for programmatic agent invocation, a REST API with explicit CI/CD trigger support ('Trigger agents from CI/CD pipelines when builds fail'), and a CLI for scripted interactions and telemetry — all first-party documented. Missing for 10: independent/hands-on confirmation of CI usage in the wild and a working OpenAPI spec (probe found 404s), so it's vendor-doc-only evidence.

   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to Devin
   Devin✓fullprobed9/10

   Devin CLI explicitly supports connecting external MCP servers, exposing their tools as slash commands with permission controls, and docs list example integrations (Datadog, Sentry, databases, Figma, Notion, Stripe, etc.). Missing for 10: independent/hands-on community verification of MCP tool usage in practice.

   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “Devin CLI exposes each one as a slash command”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
   • [probe] “official MCP server documented at https://docs.devin.ai/cli/extensibility/mcp/overview”

   proof ↗⚑ flag

   Codegen~partialclaimed6/10

   Codegen's docs explicitly mention connecting to Slack, Linear, Figma, databases and extending capabilities 'with custom MCP tools', and separately reference provisioning MCP servers across an org — direct evidence of MCP client integration. However, the evidence is a single brief doc mention with no detailed setup guide, supported server list, or independent/hands-on confirmation of the feature working. Missing for 10: detailed MCP integration docs/tutorial, list of supported MCP servers, independent verification of the feature in practice.

   • [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
   • [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”

   proof ↗⚑ flag

4. ai-native userUse an official CLI

   weight 2 · round drawn
   Devin✓fullprobed8/10

   Devin ships an official CLI documented extensively (adaptive model routing, slash commands, MCP extensibility) confirmed by both docs and a probe hit, plus a separate Terminal/local CLI mode for quick fixes. Missing for 10: independent hands-on review of the CLI itself and more detail on installation/versioning.

   • [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
   • [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
   • [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “Devin CLI exposes each one as a slash command”
   • [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”

   proof ↗⚑ flag

   Codegen✓fullprobed8/10

   Codegen documents an official CLI (docs.codegen.com/introduction/cli) with concrete capabilities: viewing/pulling agent work, creating new agents, running Claude Code with telemetry, and managing organizations/repositories — matching an AI-native agentic workflow tool. missing for 10: independent/hands-on corroboration beyond first-party docs, and no details on installation/versioning maturity.

   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
   • [claimed-docs] “Manage organizations - Switch between organizations and configure repositories”
   • [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”

   proof ↗⚑ flag

5. ai-native userDrive the product through a documented public API

   weight 3 · round drawn
   Devin✓fullprobed8/10

   Devin publishes a documented public API for integrating into applications, automating workflows, and creating sessions programmatically (including on behalf of other users), plus an MCP server exposing session/playbook/knowledge/scheduling management to external agents. Missing for 10: a discoverable machine-readable OpenAPI/swagger spec (probe found 404s) and independent third-party corroboration of API usage beyond vendor docs.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.devin.ai/llms.txt # Devin Docs - [Desktop (100 pages)](https://docs.devin.ai/_llms/en/desktop.md):…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   Codegen✓fullprobed8/10

   Codegen documents a full API reference (agent creation, traces, CI/CD triggers), a Python SDK for programmatic agent control, and a CLI for managing agents/orgs, giving AI-native users multiple documented ways to drive the product programmatically. Missing for 10: a discoverable machine-readable OpenAPI/swagger spec (probe returned 404s) and independent third-party corroboration of the API's reliability.

   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
   • [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”

   proof ↗⚑ flag

6. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to Devin
   Devin~partialclaimed3/10

   Devin documents a permission system that restricts MCP tool access at multiple levels and an API parameter (create_as_user_id) for acting on behalf of specific users, but there is no direct documentation of issuing scoped/least-privilege API keys or credentials for the agent itself. Missing for 10: explicit API key/token scoping mechanism, documented permission tiers or roles for API credentials, and any least-privilege credential issuance workflow.

   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”

   proof ↗⚑ flag

   Codegen—none0/10

   Codegen's SDK/API uses a simple org_id + token authentication (codegen-docs-15) with no documentation of scoped, role-based, or least-privilege credential issuance, token permissions, or granular access controls. Missing for 10: any mention of scoped API keys, permission levels, or restricted-access tokens for agents.

   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

7. ai-native userBuild against official SDKs

   weight 2 · round drawn
   Devin✓fullprobed8/10

   Devin publishes a documented API reference for building integrations (create sessions, manage on behalf of users, automate workflows), plus MCP support for extensibility — clear official SDK-like surface for AI-native developers. Missing for 10: no official language-specific SDK packages (e.g. Python/JS client libraries) are cited, only REST API docs and no discoverable OpenAPI spec (probe found 404s), and no independent/hands-on corroboration of API usage.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   Codegen✓fullprobed8/10

   Codegen documents an official Python SDK for programmatic agent control (agent.run, org_id/token auth), a full API reference with traces and CI/CD triggers, and a companion CLI — all first-party docs. missing for 10: independent/hands-on corroboration beyond docs, and no public OpenAPI spec was found (probe returned 404s).

   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
   • [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”

   proof ↗⚑ flag

8. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Devin—none0/10

   No evidence in the pack mentions webhooks or event subscription mechanisms for Devin; only API session creation, MCP integrations, and CLI features are documented. As an API-driven agentic product, webhook support would be a plausible and expected capability, so its absence counts as 'none' rather than 'na'.

   ⚑ flag

   Codegen—none0/10

   No evidence in the pack mentions webhooks or event-subscription mechanisms; only SDK/API polling, CI/CD triggers, and integrations (Slack, Linear, Jira) are documented, none of which describe webhook subscriptions.

   ⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Devin
   Devin~partialclaimed6/10

   Devin's repo indexing enables 'Ask Devin' and DeepWiki features that generate AI insights and explanations from your codebase data, and Devin can proactively suggest fixes/improvements via code review auto-fix and bug triage. However, evidence is scoped only to codebase/ticket data, not broader business or analytics data, and there's no dedicated insights dashboard or proactive summarization feature documented. Missing for 10: evidence of insights beyond code (e.g., analytics/dashboards), independent corroboration of Ask Devin/DeepWiki quality, and any UI showcasing generated insights.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”

   proof ↗⚑ flag

   Codegen~partialclaimed5/10

   Codegen generates AI-driven insights specifically around code artifacts—summarizing PR diffs, running security/secrets scans, checking test coverage, and suggesting changes/comments on PRs and issues—which counts as data-driven insight generation within its coding-agent context. However, these insights are narrowly scoped to code/repo/PR data rather than broader 'your data' (e.g., databases, business metrics) despite a mention of database connectivity via MCP with no insight-generation detail there. Missing for 10: evidence of insights/suggestions generated from non-code data sources (databases, analytics), and no independent/hands-on corroboration of insight quality.

   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Devin
   Devin✓fullclaimed8/10

   Devin supports scheduling via cron expressions (recurring/one-time) and agent selection through its MCP tooling, plus API session creation and Slack/Teams triggering, enabling autonomous background automations beyond interactive chat sessions. missing for 10: independent/hands-on verification of scheduled runs actually executing unattended, and more detail on failure/retry handling for scheduled tasks.

   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

   Codegen✓fullclaimed7/10

   Codegen documents agents that trigger autonomously from CI/CD failures, Slack/Linear/Jira events, and PR activity (auto-review, auto-comment), plus SDK/CLI for programmatic background automation — this matches the story of autonomous background automations. Missing for 10: independent/hands-on verification of reliability, and no explicit scheduled/cron-trigger mechanism documented.

   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round drawn
   Devin✓fullcommunity8/10

   Devin itself functions as the built-in AI assistant to which tasks are delegated across web app, terminal, Slack/Teams, and API, with rich first-party documentation of the delegation workflow (docs-1,3,4,5,8,19). Community feedback raises reliability/skepticism concerns but does not concretely contradict that delegation itself works, so it's not disputed. Missing for 10: independent hands-on validation of delegation reliability/success rate beyond old demo critiques.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”

   proof ↗⚑ flag

   Codegen✓fullclaimed8/10

   Codegen's documentation clearly describes delegating coding tasks to a built-in AI agent that analyzes requirements, implements features, fixes bugs, writes tests, and reports progress via prompts, accessible through chat, Slack, CLI, and SDK. Missing for 10: independent/hands-on verification of the assistant's actual task-completion quality beyond vendor docs.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round drawn
   Devin✓fullcommunity8/10

   Devin is fundamentally driven by natural-language task requests across web app, Slack/Teams tagging, CLI, and API (create_as_user_id sessions), letting users describe tasks like bug fixes, features, and migrations in plain language rather than issuing structured commands. Community evidence corroborates that Devin operates via conversational/task-based prompts even amid skepticism about output quality. missing for 10: independent hands-on confirmation of NL command fidelity across all surfaces (CLI/Slack/API), and no evidence addressing ambiguous or multi-step NL instruction handling limits.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”

   proof ↗⚑ flag

   Codegen✓fullclaimed8/10

   Codegen is fundamentally natural-language driven: docs show prompt-based task execution ('Leave a review on PR #123'), Slack/Jira/Linear interactions, and SDK agent.run(prompt=...) all operate via NL commands across code, PRs, and issue trackers. This spans chat, CLI, and SDK surfaces, consistently prompt/NL-driven. Missing for 10: independent/hands-on corroboration of NL command interpretation quality and edge cases beyond first-party docs.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Devin—none0/10

   Devin documents an API reference overview and parameters (e.g. create_as_user_id) but nothing describes an interactive, in-browser 'try it' experience with runnable code examples; a probe for standard OpenAPI/Swagger interactive-doc endpoints returned 404s, suggesting no such interactive explorer exists.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   Codegen—none0/10

   Codegen has an API reference page (overview, SDK docs) but no evidence of an interactive reference with runnable/executable examples; the OpenAPI/spec probe returned 404 for all candidate paths, suggesting no interactive API explorer exists.

   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   Devin—none0/10

   Devin has a documented API (devin-docs-8, devin-docs-9) but a direct probe for OpenAPI/swagger spec files at common paths returned 404s (devin-probe-2), and no docs page offers a downloadable machine-readable spec.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   Codegen—none0/10

   Codegen has an API reference (docs-16/17/18) but explicit probes for machine-readable spec files (openapi.json, swagger.json, etc.) all returned 404, indicating no downloadable OpenAPI/machine-readable spec is available.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Codegen
   Devin~partialclaimed3/10

   Devin's docs describe each session running in its own configurable workspace/environment (devin-docs-25) and connecting via VPN to reach internal services rather than being embedded in production (devin-docs-23), which implies some environment isolation, but there is no explicit claim of a dedicated 'sandbox vs production' testing mode or safeguards against touching production data. missing for 10: explicit documentation of a production-safe/sandboxed test environment, evidence of data isolation guarantees, and independent confirmation that production systems are never touched.

   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”

   proof ↗⚑ flag

   Codegen~partialclaimed5/10

   Codegen docs state that code changes run in 'robust isolated environments' to safely execute code and test changes, which implies sandboxed testing separate from production. However, there's no explicit description of how production data is excluded or protected, no detail on the sandbox's scope/persistence, and no independent verification of this isolation claim. missing for 10: explicit production-data isolation guarantees, technical detail on sandbox architecture, independent/hands-on confirmation.

   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Devin—none0/10

   There is an API reference (devin-docs-8) but no evidence of API versioning scheme or a documented deprecation policy; the openapi.json probe returned 404s, suggesting no formal machine-readable spec either. Missing for 10: versioning scheme documentation, deprecation policy/notice, changelog for breaking changes.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   Codegen—none0/10

   There's an API reference and SDK docs, but no evidence of API versioning scheme or a documented deprecation policy; the openapi probe even returned 404s, suggesting no formal spec surfacing version info.

   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Devin
   Devin~partialclaimed6/10

   Devin's docs describe running multiple Devins in parallel for independent tasks, an API for creating sessions programmatically (including on behalf of other users), and scheduling capabilities—together supporting bulk/automated task dispatch across many items. However, there's no concrete example or independent evidence of actually orchestrating large-scale bulk operations (e.g., processing hundreds of tickets at once), and community evidence focuses on single-task demos rather than bulk workflows. Missing for 10: hands-on/independent evidence of large-scale bulk runs, explicit parallelism limits, and a documented case study of bulk item processing.

   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”

   proof ↗⚑ flag

   Codegen~partialclaimed5/10

   The Python SDK and API allow programmatic creation/monitoring of agents and triggering from CI/CD, which could be scripted in a loop to perform bulk operations, but there is no documented batch/bulk endpoint or explicit guidance for operating across many items at once. missing for 10: dedicated bulk/batch API or CLI command, documented examples of multi-item automation, and evidence of scale/performance handling for bulk workloads.

   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Devin
   Devin~partialclaimed7/10

   Devin supports event-driven automation via scheduled/cron triggers (create schedules with cron expressions, notification preferences) and via Devin Review's Auto-Fix which automatically responds to code review comments and CI failures, plus API-based workflow automation. However, these are fixed built-in trigger types (schedule, CI/review events) rather than a general user-defined rules engine for arbitrary events. Missing for 10: evidence of a flexible custom rule/trigger builder covering arbitrary event sources (e.g., webhooks, ticket status changes) beyond scheduling and code-review auto-fix, and independent confirmation these triggers work reliably in practice.

   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”

   proof ↗⚑ flag

   Codegen~partialclaimed5/10

   Codegen supports event-driven automation via CI/CD triggers (docs-18) and PR review triggers on failure (docs-20), plus SDK/API for programmatic agent invocation, but there's no evidence of a general user-facing rules engine for defining arbitrary trigger-condition-action automations across events. missing for 10: a documented rules/automation builder UI, broader event types beyond CI/CD failures and PR diffs, and independent confirmation of custom trigger logic.

   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round to Devin
   Devin✓fullclaimed8/10

   Devin's MCP documentation explicitly states scheduling support with 'cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection,' and this scheduling capability is exposed via MCP for session/playbook management. Missing for 10: independent/hands-on corroboration of recurring job execution in practice and more detail on scheduling UI/reliability beyond the docs mention.

   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”

   proof ↗⚑ flag

   Codegen—none0/10

   No evidence of any scheduling, cron-like, or recurring job/workflow capability; the docs cover SDK/API triggers, CI/CD triggers, and manual agent runs, but nothing about time-based or recurring automation.

   ⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round drawn
   Devin—none0/10

   No evidence of version history, review/diff UI, or rollback of Devin's automations (playbooks, schedules, sessions); docs mention creating/scheduling sessions and playbooks but never versioning, auditing changes, or reverting them. missing for 10: version history for playbooks/sessions, review/approval workflow for automation changes, rollback/undo mechanism.

   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “Knowledge is the best way to share codebase-level (vs. task-level) context that can help Devin when working in your codebase.”

   proof ↗⚑ flag

   Codegen—none0/10

   Evidence covers PR/code review, commits, branches, and traces/telemetry, but there is no mention of versioning automations themselves, reviewing automation history, or rolling back agent configurations/workflows. missing for 10: automation versioning system, rollback mechanism for agent workflows, audit/history UI for automations.

   ⚑ flag

End to end feature delivery

1. ai-native userHave an agent automatically generate and run tests to validate its own code changes before proposing them

   weight 2 · round drawn
   Devin~partialclaimed6/10

   Devin's docs describe autonomous CI-failure iteration and auto-fix behavior (devin-docs-20) and a terminal for executing/observing command output (devin-docs-6), implying it runs and reacts to test/build results before finalizing changes, but there is no explicit documentation of Devin proactively writing new tests and running them pre-emptively to self-validate before proposing a PR. missing for 10: explicit test-generation workflow documentation, evidence of Devin authoring test cases for new code, independent hands-on confirmation of this specific validation loop.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”

   proof ↗⚑ flag

   Codegen~partialclaimed6/10

   Docs state the agent writes tests as part of implementing features and can run code/install dependencies/test changes in isolated environments, and a PR-review flow checks test coverage before flagging issues — together implying self-validation via generated tests. However, there is no explicit end-to-end walkthrough showing the agent generating tests, running them, and iterating/fixing before proposing a PR, nor independent/hands-on confirmation. missing for 10: explicit documented workflow tying test generation+execution to pre-PR validation, independent corroboration of this loop working in practice.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”

   proof ↗⚑ flag

2. developerHave an agent autonomously diagnose and fix a reported bug

   weight 3 · round to Codegen
   Devin!disputedcontradicted5/10

   Devin's docs explicitly describe autonomous bug repro/fix workflows, Slack-triggered bug sessions, and auto-fix on CI/review failures (devin-docs-1, devin-docs-4, devin-docs-19, devin-docs-20), supporting the core story. However, independent commentary cites a low real-world resolution rate (13.8% of issues resolved) and hands-on accounts describing shallow fixes (e.g., wrapping errors in try/catch rather than真正 diagnosing root cause), directly contradicting the strength of the autonomous-fix claim (devin-comm-2, devin-comm-6, devin-comm-4). Missing for 10: independent benchmark showing high success rate, more recent hands-on verification of bug-fix quality, and resolution of the discrepancy between vendor claims and community-reported performance.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
   • [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”
   • [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”

   proof ↗⚑ flag

   Codegen~partialclaimed6/10

   Docs explicitly claim the agent can 'analyze requirements, implement features, fix bugs, write tests' and can be triggered from issue trackers/CI to autonomously act, plus run in isolated environments to test changes — directly supporting autonomous bug diagnosis and fix. However, all evidence is first-party marketing/docs with no independent or hands-on validation of end-to-end autonomous diagnosis-to-fix workflows, and no detailed walkthrough of the diagnosis step itself (e.g., root-cause analysis) beyond generic 'fix bugs' language. Missing for 10: independent/hands-on verification of successful autonomous bug fixes, detailed diagnostic reasoning process, and success-rate or case-study evidence.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

3. product-managerGo from a mockup or design to a working implementation without an engineering handoff

   weight 2 · round to Devin
   Devin~partialclaimed5/10

   Devin's docs show it can autonomously implement features from tickets (Jira/Linear) and be tagged by non-engineers on Slack/Teams, supporting the general 'PM delegates, Devin implements' workflow, and it can connect to Figma via MCP integrations. However, there is no documented mockup-to-code pipeline, no PM-specific onboarding flow, and no evidence of turning a design file into a working UI end-to-end. missing for 10: direct design/mockup ingestion workflow, PM-focused case study or walkthrough, independent evidence of successful design-to-implementation outcomes.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

   Codegen~partialclaimed4/10

   Codegen's docs show relevant building blocks — a Figma connector, Slack/Linear interfaces that let non-engineers prompt the agent, and full PR/branch creation — implying a PM could describe a design and get code without direct engineering handoff. However, there is no concrete workflow, case study, or example showing a mockup/design being turned into a working implementation end-to-end. Missing for 10: an explicit design-to-code workflow example, evidence of Figma-to-PR pipeline, and PM-specific case studies or testimonials showing the handoff-free process actually works.

   • [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”

   proof ↗⚑ flag

4. developerHave an agent implement a requested feature end-to-end, including writing tests

   weight 3 · round to Codegen
   Devin!disputedcontradicted5/10

   Devin's docs explicitly claim end-to-end feature implementation (docs-1, docs-31) and even test-writing is implied as part of full feature delivery, but independent hands-on reporting from the original demo showed only 13.8% of issues resolved and reviewers noted the work was often only ~90% correct or used hacky fixes (e.g. wrapping in try-catch) rather than proper fixes (devin-comm-2, devin-comm-4, devin-comm-6), directly contradicting the 'implements features end-to-end reliably' claim. missing for 10: verified success-rate data on test-writing specifically, more recent independent benchmarks showing improved reliability, and evidence tests are written/passed as part of the workflow rather than just code changes.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
   • [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”
   • [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”

   proof ↗⚑ flag

   Codegen✓fullclaimed8/10

   Docs explicitly describe agents analyzing requirements, implementing features, writing tests, and running/testing changes in isolated environments, plus PR/branch management for end-to-end delivery. Missing for 10: independent/hands-on validation of end-to-end feature delivery and no concrete example transcript showing a full feature+test cycle.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

Environment setup

1. developerHave an agent automatically clone the repo, install dependencies, and configure its own working environment

   weight 2 · round drawn
   Devin✓fullclaimed8/10

   Devin's docs describe indexing repos, configuring a persistent environment/workspace that boots into a known-good state every session, VPN access to internal registries/git hosts, and AGENTS.md-based setup instructions it follows before coding — together these show it autonomously clones, configures, and prepares its own working environment. missing for 10: independent/hands-on verification of the clone+dependency-install flow end-to-end, and explicit documentation of automatic dependency installation step-by-step.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”

   proof ↗⚑ flag

   Codegen✓fullclaimed8/10

   Docs describe agents that access repos via GitHub App integration, run in isolated environments, and 'safely run code, install dependencies, and test changes' automatically without complex setup, matching the story of autonomous environment setup. Missing for 10: independent/hands-on verification of environment provisioning behavior and more detail on how dependency installation is auto-configured beyond the brief doc mention.

   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”

   proof ↗⚑ flag

Interactive takeover

1. developerTake over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work

   weight 2 · round to Devin
   Devin✓fullclaimed8/10

   Docs explicitly describe multi-surface handoff: delegate via web app then take over in Devin's IDE, use /handoff to send terminal tasks to cloud Devin, jump into the Interactive Browser to help Devin navigate, and tag Devin in Slack/Teams threads to redirect work — covering editor, terminal, and browser takeover. Missing for 10: independent/hands-on confirmation of a smooth mid-task handoff experience (community evidence is generic skepticism, not a concrete handoff failure).

   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”

   proof ↗⚑ flag

   Codegen~partialclaimed5/10

   Codegen supports interacting with agents across multiple surfaces (Slack notifications/clarification requests, CLI to pull agent branches locally, Python SDK to run/monitor agents) and lets you download in-progress work to your local environment, which implies some ability to pick up and continue work. However there is no explicit documentation of pausing, redirecting, or steering a task mid-execution from editor/terminal/browser as a unified handoff flow. Missing for 10: explicit 'takeover' or redirect mid-task feature, editor/IDE integration for live task control, and evidence of seamless cross-surface handoff (Slack->CLI->browser) of the same in-progress task.

   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”

   proof ↗⚑ flag

2. developerSend follow-up instructions to an active agent session to steer its work without restarting

   weight 2 · round to Devin
   Devin~partialclaimed5/10

   Devin's docs describe delegating tasks and then 'taking over in Devin's IDE' or 'jumping in' to help with browsing tasks, implying some interactivity with an active session, but there is no explicit documentation of sending follow-up chat instructions to steer an in-progress session without restarting it. missing for 10: explicit docs on mid-session chat/steering, confirmation that follow-up messages are incorporated live, independent hands-on evidence of this workflow.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”

   proof ↗⚑ flag

   Codegen—none0/10

   No evidence describes sending follow-up instructions to an already-running agent session to redirect its work; docs only describe initiating agents via prompts, Slack notifications, or SDK task creation, not mid-session steering. Missing for 10: any documentation of a 'continue/steer/update' API, chat-like follow-up in an active run, or session-resume mechanism.

   ⚑ flag

Sandbox execution

1. developerHave an agent safely execute code and install dependencies inside an isolated sandbox

   weight 3 · round to Codegen
   Devin~partialclaimed6/10

   Docs describe Devin operating in its own workspace/VM with terminal execution, environment configuration ('known-good state'), VPN access to internal services, and multi-platform (Android/macOS/Windows) build environments, implying isolated sandboxed execution, but there is no explicit documentation of sandbox security guarantees, dependency isolation, or safety controls around code execution, and no independent/hands-on verification of sandbox isolation. missing for 10: explicit sandbox/isolation security documentation, independent verification of safe execution, details on dependency install safety and containment.

   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”
   • [claimed-docs] “Devin can build and run Android applications directly on its own machine — giving it the Android equivalent of Computer Use and browser inte…”
   • [claimed-docs] “Devin now has access to macOS virtual machines. This means Devin can now build and test iOS and macOS applications.”
   • [claimed-docs] “Devin supports Windows as a build and session platform.”

   proof ↗⚑ flag

   Codegen✓fullclaimed7/10

   First-party docs explicitly state the agent can 'Safely run code, install dependencies, and test changes in robust isolated environments,' directly matching the story. missing for 10: independent/hands-on corroboration of sandbox isolation, technical details on sandbox implementation (e.g., container tech, resource limits) beyond a single doc line.

   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”

   proof ↗⚑ flag

Approval controls

1. developerConfigure an agent to auto-approve all its actions instead of confirming each one

   weight 2 · round drawn
   Devin—none0/10

   The evidence pack describes permission systems for MCP tools (devin-docs-13) and autonomous features like Auto-Fix (devin-docs-20), but there is no documentation of a setting that lets a developer toggle between 'confirm each action' and 'auto-approve all actions' for Devin's own task execution. missing for 10: explicit auto-approve/confirmation toggle documentation, evidence of a per-action approval workflow that can be disabled, any community confirmation of this specific control.

   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

   Codegen—none0/10

   The evidence describes Codegen's autonomous capabilities (running code, committing, creating PRs, Slack/Jira integration) but never mentions any per-action confirmation mode or an auto-approve toggle/setting for oversight control. Missing for 10: any documentation of an approval/confirmation workflow, a setting to enable/disable auto-approval, or human-in-the-loop gating that this story presupposes.

   ⚑ flag

2. product-managerApprove key agent decisions from my phone while agents continue working

   weight 1 · round to Codegen
   Devin—none0/10

   No evidence describes a mobile app or phone-based approval interface for agent decisions; docs mention Slack/Teams tagging, web app, IDE, and scheduling notifications but nothing about approving decisions from a phone while sessions continue. The axis is plausible for an agentic dev tool with human-in-the-loop needs, but no supporting evidence exists.

   ⚑ flag

   Codegen~partialclaimed3/10

   Codegen supports Slack-based notifications and agent interaction (docs-2, docs-9), which could technically be used from a phone via the Slack app, but there is no explicit mobile app, push-approval workflow, or documented 'approve on the go' feature for key agent decisions. missing for 10: dedicated mobile/approval UI, evidence of approving specific agent decisions (e.g., merge/deploy gates) from a phone, and any hands-on confirmation of this workflow.

   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”

   proof ↗⚑ flag

3. engineering-leadSet tiered autonomy levels controlling what an agent can do without manual confirmation

   weight 3 · round to Devin
   Devin~partialclaimed4/10

   Devin documents a permission system for MCP/tool access that can be controlled 'at multiple levels' and features like Auto-Fix, human takeover in the IDE, and interactive browser intervention imply some control over autonomous action, but there is no documented feature specifically framed as configurable 'tiered autonomy levels' for an engineering lead to set organization-wide policies on unattended agent actions. missing for 10: explicit autonomy-tier configuration UI/API, organization-wide policy controls, and confirmation-threshold settings distinct from ad-hoc MCP tool permissions.

   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

   Codegen—none0/10

   No evidence describes configurable autonomy tiers, approval gates, or permission levels for agent actions; the docs describe what agents can do (code, PRs, Slack, Jira/Linear) but nothing about setting graduated confirmation thresholds for an engineering lead to control autonomy.

   ⚑ flag

Model control

1. ai-native userHave each task prompt automatically routed to the most suitable underlying model

   weight 2 · round to Devin
   Devin✓fullprobed8/10

   Devin's CLI has an explicit "Adaptive" model router that analyzes each prompt and automatically routes it to the best-performing model, with an override via /model if manual control is desired. This directly matches the story of automatic per-task model routing, though it's documented only for the CLI product surface and lacks independent/hands-on validation of routing quality. Missing for 10: independent corroboration of routing accuracy, and confirmation this applies across all Devin surfaces (web/IDE) not just CLI.

   • [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
   • [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
   • [claimed-docs] “Adaptive is an intelligent model router that automatically selects the best AI model for each task.”
   • [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”

   proof ↗⚑ flag

   Codegen—none0/10

   No evidence anywhere in the pack of automatic model routing per task/prompt; Codegen's docs describe agent capabilities, integrations, CLI/SDK, but nothing about selecting or routing between underlying LLMs based on task type.

   ⚑ flag

2. engineering-leadSwitch away from automatic model selection to a specific model of my choice

   weight 1 · round to Devin
   Devin✓fullprobed8/10

   Devin's docs explicitly state that Adaptive automatically routes prompts to the best model, but users can switch away from Adaptive to a specific model at any time using /model, directly matching the story. Missing for 10: independent/hands-on corroboration of the /model switch in practice and more detail on which specific models are selectable.

   • [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
   • [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
   • [claimed-docs] “Adaptive is an intelligent model router that automatically selects the best AI model for each task.”
   • [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”

   proof ↗⚑ flag

   Codegen—none0/10

   No evidence in the pack mentions model selection, choice of underlying LLM, or any setting to override automatic model routing; the docs cover integrations, CLI, SDK, and agent workflows but never reference model choice.

   ⚑ flag

Visibility monitoring

1. developerWatch what a running agent is doing in real time, including its current status

   weight 3 · round to Devin
   Devin✓fullclaimed7/10

   Devin's docs describe real-time observability features like watching the terminal execute commands with live output logs, jumping in during interactive browsing, and taking over in Devin's IDE mid-task, all indicating a live view of the agent's current activity/status. Missing for 10: no explicit dedicated 'status dashboard' documentation and no independent/hands-on corroboration of the real-time monitoring UX beyond vendor docs.

   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

   Codegen~partialclaimed6/10

   Codegen's CLI lets developers view agents and run them with 'full telemetry and monitoring,' and agents can 'report progress' via Slack, giving some real-time visibility into agent activity and status. However, there's no explicit documentation of a live dashboard/status stream showing step-by-step actions as they happen, and the closest 'trace' feature is framed as post-hoc analysis/debugging rather than live watching. Missing for 10: a documented real-time UI/dashboard showing live agent status, explicit streaming of in-progress actions, and independent/hands-on confirmation of live monitoring accuracy.

   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”

   proof ↗⚑ flag

2. developerGet notified when an agent completes a task or needs my input

   weight 2 · round to Codegen
   Devin~partialclaimed6/10

   Devin supports Slack/Teams tagging where sessions and results appear in-thread (devin-docs-4, devin-docs-19), and scheduled sessions explicitly support 'notification preferences' (devin-docs-17), implying some completion/status alerts. However there's no dedicated first-party documentation of a generic notify-on-completion or needs-input alert system (e.g., email/push) outside Slack/Teams and scheduling contexts. Missing for 10: explicit documentation of a unified notification mechanism across all session types, and independent/hands-on confirmation that notifications reliably fire when input is needed.

   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”

   proof ↗⚑ flag

   Codegen✓fullclaimed8/10

   Codegen explicitly supports sending notifications, asking for clarification, and reporting progress directly in Slack channels, plus updating statuses/comments in Jira, Linear, Clickup, and Monday.com, covering both completion and input-needed scenarios. missing for 10: independent/hands-on confirmation of real-time notification delivery, and no explicit mention of email/other channel notification options.

   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”

   proof ↗⚑ flag

Natural language task intake

1. developerDescribe a feature or bug in plain language and have it automatically turned into a scoped implementation task

   weight 3 · round drawn
   Devin✓fullcommunity7/10

   Devin's docs describe accepting natural-language asks (tickets, Slack/Teams messages, delegated tasks) and turning them into sessions/implementation tasks, which is the core intent-to-spec workflow (devin-docs-1, devin-docs-4, devin-docs-5, devin-docs-19). Community commentary raises concerns about actual task success rates and correctness (devin-comm-2, devin-comm-4), tempering confidence in scoping quality. Missing for 10: independent hands-on evidence that the auto-generated task scope/plan is accurate or reviewable before execution, and no dedicated 'plan/spec preview' feature described.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
   • [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”

   proof ↗⚑ flag

   Codegen✓fullclaimed7/10

   Codegen's docs describe taking natural-language prompts and turning them into scoped agent tasks: analyzing requirements, implementing features, fixing bugs, writing tests, and running agents from prompts via SDK/CLI/Slack/issue trackers. This directly matches the intent-to-spec story of describing a feature/bug and having it become an actionable implementation task. Missing for 10: no independent/hands-on evidence showing the quality or accuracy of the generated scoped task (e.g. how well it decomposes ambiguous requests), and no detail on how it handles clarification/scoping edge cases beyond a general 'ask for clarification' mention.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”

   proof ↗⚑ flag

2. product-managerConvert user feedback submissions into structured tasks with proposed scope

   weight 2 · round to Devin
   Devin~partialclaimed5/10

   Devin can ingest raw inputs like Linear/Jira tickets and Slack/Teams threads about bugs or feature requests and turn them into an actionable session (devin-docs-1, devin-docs-4, devin-docs-19), which covers the 'feedback-to-task' intake path, but there is no evidence of a distinct structured-task output with an explicit proposed scope/plan artifact for PM review before execution. Missing for 10: evidence of a scoping/planning deliverable (e.g., a spec doc or task breakdown with effort/scope estimate) generated prior to coding, and any PM-specific workflow or UI for reviewing/approving that scope.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

   Codegen~partialclaimed3/10

   Codegen can analyze requirements and create/link tasks in Jira, Linear, Clickup, Monday.com, which could theoretically be used to turn feedback into tasks, but there is no evidence of a feedback-intake feature or automatic scope proposal tailored for PM workflows. Missing for 10: dedicated feedback ingestion mechanism, structured scope/estimate output, PM-specific workflow examples or case studies.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”

   proof ↗⚑ flag

3. developerAttach a marked-up screenshot or mockup to a task so the agent implements the correct visual change

   weight 2 · round drawn
   Devin—none0/10

   No evidence in the pack mentions attaching screenshots, mockups, or marked-up images to a Devin task, nor any visual-diff/design-to-code workflow; documentation covers ticket delegation, terminal, Slack, MCP, and environment setup but nothing about image/mockup-based task intake.

   ⚑ flag

   Codegen—none0/10

   No evidence in the pack mentions attaching screenshots, images, or marked-up mockups to a task/prompt; Codegen's docs cover text prompts, Slack/Jira/Linear integrations, CLI/SDK, and PR review but nothing about visual/image input for implementing UI changes. Figma is mentioned only as a connectable integration, not as an image-annotation workflow.

   • [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”

   proof ↗⚑ flag

Plan approval

1. developerReview and approve an agent's implementation plan before any code changes are made

   weight 3 · round drawn
   Devin—none0/10

   The evidence pack describes Devin's task delegation, IDE handoff, terminal visibility, and MCP integrations, but nowhere documents a plan-review/approval gate before Devin begins making code changes. Missing for 10: any doc mentioning an upfront plan proposal, explicit approval step, or 'plan mode' prior to execution.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”

   proof ↗⚑ flag

   Codegen—none0/10

   No evidence in the pack of a plan-preview or approval step prior to code changes; agents appear to run prompts directly to implementation (create branches, commit code, open PRs) without a documented plan-review gate. missing for 10: any mention of a plan/preview mode, approval gate, or dry-run step before code execution.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”

   proof ↗⚑ flag

2. engineering-leadApprove a task's scope and contract before an agent is allowed to modify the repository

   weight 2 · round drawn
   Devin—none0/10

   The evidence pack shows delegation, review-after-the-fact, and IDE takeover workflows, but no documented mechanism where an engineering-lead must approve a task's scope/contract before Devin is allowed to start modifying the repository. Missing for 10: explicit pre-execution approval/gating workflow, scope/contract sign-off UI or API, evidence of blocking repo writes until lead approval.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”

   proof ↗⚑ flag

   Codegen—none0/10

   Evidence shows Codegen agents can be given prompts and run autonomously with PR review/creation, but there is no mention of any scope/contract approval gate or engineering-lead sign-off step required before an agent is allowed to modify a repository.

   ⚑ flag

Ticket driven tasking

1. developerAssign a coding task to an agent directly from an existing issue or ticket

   weight 3 · round to Devin
   Devin✓fullclaimed8/10

   Docs explicitly state Devin can be asked to tackle Linear/Jira tickets, tagged on Slack/Teams threads about bugs to start sessions, and integrated via API/MCP to create sessions from external triggers, directly supporting assigning tasks from existing issues/tickets. Missing for 10: independent hands-on verification of ticket-to-task handoff and no direct product screenshot of ticket-linking UI in evidence.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”

   proof ↗⚑ flag

   Codegen✓fullclaimed7/10

   Docs show native integrations with Jira, Linear, ClickUp and Monday.com allowing the agent to update issue statuses, add comments, link PRs and create tasks, and specifically state connecting a Linear workspace 'enables agent interactions with your issues,' which directly supports assigning work from a ticket. Missing for 10: a concrete end-to-end example of assigning a task from an issue/ticket, and independent/hands-on corroboration beyond vendor docs.

   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round drawn
   Devin~partialprobed5/10

   Devin exposes a documented API (session creation, create_as_user_id) and an MCP server that surfaces session management, playbooks, knowledge, and scheduling (devin-docs-8,9,15,29), suggesting broad but not full parity with the web/IDE UI. There's no explicit claim or evidence that UI-only features like the Interactive Browser takeover, terminal handoff, VPN setup, or environment/emulation configuration are fully API-accessible, and no OpenAPI spec was found (devin-probe-2) to confirm completeness. Missing for 10: explicit vendor statement of API/UI parity, evidence that interactive/browser/VPN/environment features are API-controllable, and independent confirmation of parity.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   Codegen~partialprobed5/10

   Codegen offers a Python SDK, CLI, and API to create/run agents, pull work, and monitor executions, showing solid programmatic control (codegen-docs-11,15,16,17,18). However, there's no evidence of a complete OpenAPI/API surface matching every UI capability (e.g., Slack/Jira/Linear integration management, PR review workflows) — the OpenAPI probe returned 404s, suggesting no public full API spec. missing for 10: OpenAPI reference confirming full UI parity, evidence that integration configs and review workflows are API-accessible, independent confirmation of API completeness.

   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   Devin—none0/10

   No evidence of any data export feature, open-format export, or account/data portability tooling in Devin's docs; evidence covers task delegation, MCP, CLI, and platform support but nothing about exporting session data, knowledge, or playbooks in open formats for migration away from the product.

   ⚑ flag

   Codegen—none0/10

   No evidence of a bulk data-export feature, open-format export of agent histories/traces, or account-portability tooling; only partial building blocks exist (pulling agent branches via CLI, retrieving traces via API) but nothing documenting a full data export/exit path.

   • [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Devin—none0/10

   Devin is a closed-source commercial SaaS product; no evidence of any open-source license or public source repository is present in the evidence pack, and the axis is applicable since AI tooling could conceivably publish open-source components. missing for 10: any open-license source release, public repo link, or licensing statement.

   ⚑ flag

   Codegen—none0/10

   No evidence of an open-license source repository or public codebase for Codegen; all evidence relates to product features, docs, CLI, and SDK usage, not source availability or licensing.

   ⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   Devin—none0/10

   Devin is exclusively delivered as a cloud SaaS product (with local CLI/terminal front-ends that hand off to cloud sessions); no evidence anywhere in the docs or community discussion describes a self-hostable core engine or on-prem deployment option. Missing for 10: any documentation of self-hosted/on-prem deployment, container/binary distribution of the core agent, or licensing for self-hosting.

   • [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”

   proof ↗⚑ flag

   Codegen—none0/10

   Codegen is presented as a cloud-hosted SaaS agent platform (GitHub App, Slack App, cloud CLI/SDK, hosted API); no evidence of any self-hosted deployment option, on-prem package, or open-source core repository. Absence of evidence for this applicable capability yields 'none'.

   ⚑ flag

Enterprise licensing

1. engineering-leadLicense an enterprise deployment with SSO and commercial support for organization-wide rollout

   weight 2 · round drawn
   Devin—none0/10

   No evidence pack items mention enterprise licensing, SSO, or commercial support offerings for Devin; only product feature docs and community sentiment are present. Missing for 10: SSO/SAML documentation, enterprise plan/pricing page, commercial support SLA details.

   ⚑ flag

   Codegen—none0/10

   No evidence in the pack mentions SSO, enterprise licensing, commercial support tiers, or org-wide rollout plans; only feature-level docs and API/CLI references are provided. Missing for 10: SSO/SAML support, enterprise plan/pricing tier documentation, commercial support SLA, org-wide admin/rollout tooling.

   ⚑ flag

Model flexibility

1. engineering-leadBring my own LLM or API key so agents run on the model of my choice

   weight 2 · round drawn
   Devin—none0/10

   Docs describe an internal 'Adaptive' model router and a /model command to pick among Devin's supported models, but there is no evidence of bringing a custom LLM or your own API key to run agents on a model of the engineering lead's choice.

   • [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
   • [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
   • [claimed-docs] “Adaptive is an intelligent model router that automatically selects the best AI model for each task.”

   proof ↗⚑ flag

   Codegen—none0/10

   No evidence pack item mentions BYO-LLM, custom API key configuration, or model selection; Codegen's docs focus on its own agents, integrations, and SDK/CLI without any option to swap the underlying model provider.

   ⚑ flag

Usage quotas

1. engineering-leadSee and manage plan-based daily task and concurrency limits for agent workflows

   weight 2 · round drawn
   Devin—none0/10

   No evidence pack item mentions plan tiers, daily task quotas, concurrency limits, or an admin console/dashboard for managing such limits; docs cover running parallel Devins and API session creation but not visibility or management of plan-based limits. Missing for 10: pricing/plan documentation, admin UI for concurrency/task caps, usage dashboards or alerts for limit management.

   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”

   proof ↗⚑ flag

   Codegen—none0/10

   No evidence pack item mentions plan-based daily task limits, concurrency limits, or any admin dashboard/controls for engineering leads to monitor or manage usage quotas; documentation covers agent capabilities, integrations, CLI, and SDK but nothing on pricing tiers or limit management.

   ⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   Devin—none0/10

   No evidence in the pack mentions data residency, regional storage options, or compliance controls for where session/data is stored; docs cover VPN connectivity, environments, and MCP but nothing about choosing a storage region. Missing for 10: any mention of data residency/region selection, compliance certifications (e.g., SOC2/GDPR data locality), or enterprise data storage controls.

   ⚑ flag

   Codegen—none0/10

   No evidence in the pack mentions data residency, region selection, or storage location options for Codegen; the docs cover integrations, CLI, SDK, and agent workflows but nothing about data storage location controls.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Devin—none0/10

   No evidence in the pack addresses data-training opt-out, privacy controls, or any policy about excluding user data from model training; all docs cover feature capabilities, MCP, CLI, and environment support instead.

   ⚑ flag

   Codegen—none0/10

   No evidence in the pack addresses data-training opt-out, privacy controls, or policies about AI model training usage; all citations concern feature functionality (agents, CLI, SDK, integrations) rather than privacy posture.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Devin—none0/10

   No evidence in the pack addresses data retention policies, deletion controls, or privacy configuration options for Devin; the docs cover features, MCP, CLI, and environments but not data governance/retention settings.

   ⚑ flag

   Codegen—none0/10

   No evidence pack items discuss data retention policies, deletion controls, or privacy/compliance settings; the docs cover functionality (agents, integrations, CLI, SDK) but nothing about controlling or deleting stored data.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Devin—none0/10

   No evidence pack item mentions telemetry opt-out, analytics settings, or usage-tracking controls for Devin; nothing addresses this privacy axis at all.

   ⚑ flag

   Codegen—none0/10

   No evidence pack item mentions a telemetry opt-out setting or privacy controls; in fact docs-12 explicitly notes the CLI runs Claude Code 'with full telemetry and monitoring,' with no mention of a way to disable it.

   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”

   proof ↗⚑ flag

Chat integration

1. developerTag an agent in a chat thread to discuss and delegate a bug or task

   weight 2 · round to Devin
   Devin✓fullclaimed8/10

   Docs explicitly describe tagging Devin in Slack/Teams threads to discuss and delegate bugs/tasks, starting sessions directly from conversations. Missing for 10: independent/hands-on corroboration of this specific chat-tagging workflow beyond vendor docs.

   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”

   proof ↗⚑ flag

   Codegen~partialclaimed6/10

   Codegen supports Slack-based chat interactions where users can message/tag the agent to report progress, ask for clarification, and delegate tasks (codegen-docs-2, codegen-docs-9), and Linear/Jira integration allows commenting and task creation to delegate bugs (codegen-docs-3, codegen-docs-10). However, evidence doesn't explicitly confirm an '@mention' tagging mechanism within an existing chat thread to delegate a specific bug — it's inferred from general Slack/Linear interaction docs rather than a documented tagging workflow. Missing for 10: explicit documentation of @-mention/tagging syntax in threads, confirmation of thread-level context retention when delegating, and independent/hands-on verification of this specific workflow.

   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”

   proof ↗⚑ flag

Knowledge context

1. developerAdd a context file describing my codebase conventions so agents generate more relevant plans and code

   weight 3 · round to Devin
   Devin✓fullclaimed9/10

   Devin explicitly supports AGENTS.md files placed in the project root that Devin reads before coding, plus a separate 'Knowledge' feature for sharing codebase-level conventions/context to improve plans and code generation. Missing for 10: independent/hands-on confirmation that this measurably improves plan/code relevance beyond vendor docs.

   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
   • [claimed-docs] “Knowledge is the best way to share codebase-level (vs. task-level) context that can help Devin when working in your codebase.”
   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”

   proof ↗⚑ flag

   Codegen—none0/10

   The evidence pack covers Codegen's integrations, CLI, SDK, and PR/issue automation, but nowhere mentions a repo-level context or rules file (e.g., conventions file) that developers can add to steer agent plans/code generation. missing for 10: any mention of a context/config file for codebase conventions, documentation on custom instructions, or repo-level rules ingestion.

   ⚑ flag

2. developerQuery generated documentation for any public or private repository

   weight 1 · round to Devin
   Devin✓fullclaimed9/10

   Devin explicitly documents 'DeepWiki'-style repo indexing and MCP tools that 'explore and query documentation for any GitHub repository (public or private with authentication)', directly matching the story, backed by repo indexing docs enabling Ask Devin/DeepWiki features. Missing for 10: independent hands-on verification of documentation query quality/accuracy beyond vendor docs.

   • [claimed-docs] “These tools let you explore and query documentation for any GitHub repository (public or private with authentication)”
   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”

   proof ↗⚑ flag

   Codegen—none0/10

   Evidence shows Codegen can 'improve documentation' as part of coding tasks, but there is no evidence of a feature to generate and then query documentation for a repository (public or private) in a Q&A style. No docs, API, or CLI reference describe a documentation-query capability.

   ⚑ flag

Project management integration

1. product-managerConnect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly

   weight 3 · round drawn
   Devin~partialclaimed6/10

   Devin explicitly supports Linear and Jira tickets natively (devin-docs-1) and its MCP extensibility (devin-docs-12, devin-docs-21) lets it connect to arbitrary tool servers including issue trackers, which could cover ClickUp/Monday.com. However, there's no first-party documentation of native ClickUp or Monday.com connectors, and no independent/hands-on evidence confirming ticket-management workflows actually work end-to-end via these integrations. missing for 10: explicit native ClickUp/Monday.com support docs, independent verification of issue-tracker workflows succeeding in practice.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”

   proof ↗⚑ flag

   Codegen~partialclaimed6/10

   Docs explicitly claim support for Jira, Linear, ClickUp, and Monday.com (update statuses, add comments, link PRs, create tasks), but detailed setup instructions are only shown for Linear (codegen-docs-10); Jira, ClickUp, and Monday.com lack dedicated connection/integration docs equivalent to Linear or Slack. Missing for 10: explicit setup guides or independent verification for Jira/ClickUp/Monday.com, and hands-on confirmation beyond first-party docs.

   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
   • [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”

   proof ↗⚑ flag

Version control integration

1. developerConnect a GitHub repository so an agent can access the code and open pull requests against it

   weight 3 · round to Codegen
   Devin~partialclaimed6/10

   Docs confirm repository indexing (public/private) and GitHub-repo access for context, plus 'Devin Review with Auto-Fix' responding to code review/CI, implying PR interaction, but there's no explicit documentation of a GitHub connection/auth flow or of Devin opening pull requests directly. missing for 10: explicit GitHub app/OAuth connection setup steps, explicit 'opens pull requests' documentation, independent/hands-on confirmation of PR creation.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “These tools let you explore and query documentation for any GitHub repository (public or private with authentication)”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”

   proof ↗⚑ flag

   Codegen✓fullclaimed8/10

   Codegen provides a GitHub App install for repo access with no complex setup, and docs describe the agent creating branches, committing code, reviewing/opening PRs, and managing repositories. missing for 10: independent/hands-on confirmation beyond vendor docs, and explicit detail on multi-repo/org-level permission scoping.

   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

2. developerGrant an agent access to my repositories with a one-click install, without complex setup

   weight 2 · round to Codegen
   Devin~partialclaimed5/10

   Devin does support connecting to repos (indexing, GitHub-based session creation, VPN access to internal git hosts) and this is described as straightforward onboarding, but there's no evidence of a literal 'one-click install' flow like a GitHub App marketplace install — setup instead involves indexing, environment configuration, AGENTS.md, and knowledge onboarding steps. Missing for 10: explicit one-click/marketplace-style install flow, independent evidence of setup simplicity, and confirmation that repo access requires no additional configuration.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”
   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”

   proof ↗⚑ flag

   Codegen✓fullclaimed8/10

   Docs explicitly state installing the GitHub App grants repo access with 'no complex setup required', directly matching the story, and this is corroborated by broader repo-integration capabilities (PR review, branch/commit management). Missing for 10: independent/hands-on confirmation of the one-click flow beyond first-party docs.

   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”

   proof ↗⚑ flag

Ci remediation

1. engineering-leadHave failed CI workflows automatically diagnosed and fixed with a proposed pull request

   weight 3 · round to Devin
   Devin✓fullclaimed7/10

   Devin explicitly documents 'Devin Review with Auto-Fix' that responds to code review comments, fixes flagged bugs, and iterates on CI failures, directly matching the story of auto-diagnosing and fixing failed CI with a proposed PR. However, this is vendor-only documentation with no independent/hands-on corroboration of the CI-fix workflow specifically. Missing for 10: independent verification of CI-fix accuracy/success rate, details on PR proposal format, and evidence of real-world reliability beyond the single docs line.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

   Codegen~partialclaimed6/10

   Docs explicitly mention triggering agents from CI/CD pipelines when builds fail and agents that fix bugs/write code and open PRs, directly supporting automated diagnosis-and-fix-PR workflows for failed CI. However, there is no dedicated end-to-end CI-failure-diagnosis feature description, no example of the proposed PR format for a failing build, and no independent/hands-on confirmation of this specific workflow. missing for 10: dedicated CI-failure-diagnosis workflow documentation, concrete example of a CI-triggered fix PR, independent verification of this specific capability.

   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

2. developerTrigger an agent from CI/CD pipelines to fix a broken build or failing test

   weight 2 · round to Codegen
   Devin~partialclaimed6/10

   Docs explicitly state Devin Review with Auto-Fix can iterate on CI failures and fix flagged bugs, and the Devin API lets you integrate Devin into applications/workflows and create sessions programmatically (including on behalf of users), which supports triggering it from automated pipelines. However, there's no explicit documented example of a CI/CD pipeline (e.g., GitHub Actions/Jenkins) directly invoking Devin on a failing build/test, and no independent/hands-on confirmation of this exact workflow. Missing for 10: explicit CI/CD trigger integration example, independent verification that pipeline-triggered fixes work reliably.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”

   proof ↗⚑ flag

   Codegen✓fullclaimed8/10

   Docs explicitly state agents can be 'Triggered from CI/CD pipelines when builds fail' and support fixing bugs/tests via API/SDK with programmatic control, which directly matches the story. Missing for 10: independent/hands-on confirmation of a real CI/CD integration example (e.g., GitHub Actions snippet) and details on how failing test context is passed to the agent.

   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”

   proof ↗⚑ flag

Diff review

1. developerConfigure an agent to automatically open a pull request when its task completes

   weight 2 · round to Codegen
   Devin~partialclaimed5/10

   Devin is known to work within GitHub workflows and create PRs as part of its task completion (implied by its coding-agent nature and features like Auto-Fix responding to review comments/CI), but the evidence pack has no explicit documentation describing a configurable setting to auto-open a pull request upon task completion. missing for 10: explicit docs describing PR-creation-on-completion configuration, settings/toggle for auto-PR behavior, and independent confirmation of this specific workflow.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

   Codegen✓fullclaimed7/10

   Codegen's agent creates branches, commits code, and manages repositories via GitHub App integration, and can be run programmatically via SDK/CLI/API to complete tasks; PR creation is a core documented capability of the agent workflow. Missing for 10: explicit first-party documentation/example showing a config flag or setting to 'auto-open PR on task completion' and independent/hands-on confirmation of this exact behavior.

   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

2. developerReview a diff of an agent's changes and approve it before it becomes a pull request

   weight 3 · round to Codegen
   Devin~partialclaimed4/10

   Docs mention delegating a task and 'taking over in Devin's IDE once it gives you a good first draft' (devin-docs-5), implying some human checkpoint before final output, and Devin's terminal/log viewing lets you watch work in progress (devin-docs-6). However there is no explicit documentation of a diff-review/approve-before-PR gate — 'Devin Review with Auto-Fix' (devin-docs-20) is about post-PR review comment handling, not a pre-PR approval step. Missing for 10: explicit diff viewer with approve/reject before PR creation, documentation of a formal approval gate, and independent/hands-on confirmation of this workflow.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

   Codegen~partialclaimed5/10

   Codegen agents create branches, commit code, and open PRs, and the agent itself can review PRs and post comments, but the evidence pack does not show a dedicated human-in-the-loop diff review/approval gate before a PR is created — the CLI lets you 'pull agent work' locally which could support manual review, but no explicit approval workflow is documented. missing for 10: explicit pre-PR diff approval UI/workflow, evidence of a human approve/reject gate, independent confirmation that PR creation is blocked pending review.

   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”

   proof ↗⚑ flag

Pr review automation

1. ai-native userHave incoming issues automatically triaged with severity suggested and routed to the right owner

   weight 2 · round drawn
   Devin—none0/10

   Devin can be pointed at Linear/Jira tickets or tagged in Slack to work a bug, but nothing in the evidence describes automatic triage logic that assesses severity or routes issues to the correct owner — these are core, specific parts of the story. Missing for 10: severity classification/scoring, ownership/routing logic, any 'triage' feature or automated inbox-sorting workflow.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”

   proof ↗⚑ flag

   Codegen—none0/10

   Evidence shows Codegen integrates with issue trackers (Jira, Linear, etc.) to update statuses, add comments, and create tasks, but there is no mention of automatic severity assessment or routing incoming issues to the correct owner. Missing for 10: automatic severity classification, owner/routing logic, any triage workflow evidence.

   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”

   proof ↗⚑ flag

2. engineering-leadHave every pull request automatically reviewed with AI-generated inline comments

   weight 3 · round to Codegen
   Devin~partialclaimed5/10

   Devin documents a 'Devin Review with Auto-Fix' feature that responds to code review comments, fixes flagged bugs, and iterates on CI failures, implying an automated review capability tied to PRs, but the evidence pack gives no detail on how review comments are generated, whether they are inline, or how coverage/setup works for 'every PR'. missing for 10: explicit documentation of inline AI-generated PR comments, configuration for gating every PR, and independent/hands-on confirmation of the review quality.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

   Codegen✓fullclaimed7/10

   Docs explicitly describe automated PR review functionality: reading the diff, summarizing changes, running security/secrets scans, checking test coverage, and posting a review comment, plus general 'Review PRs, suggest changes, comment on issues' capability tied to GitHub App integration for repo access. This matches the engineering-lead story of automatic AI-reviewed PRs with comments. Missing for 10: explicit confirmation that comments are inline (per-line) rather than a single summary comment, and independent/hands-on evidence corroborating this in production use.

   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

Readiness checks

1. engineering-leadAutomatically fix failing agent-readiness criteria in my repository

   weight 1 · round to Devin
   Devin~partialclaimed6/10

   Devin Review with Auto-Fix explicitly auto-fixes CI failures and review comments, which could cover agent-readiness gate checks if they're wired into CI/PR review, and AGENTS.md support shows Devin can consume agent-readiness config files. However there's no evidence of a dedicated 'agent-readiness criteria' scorecard or targeted remediation workflow distinct from generic CI/review auto-fix. Missing for 10: explicit agent-readiness scoring/criteria detection, evidence of automatically remediating specific agent-readiness failures (vs generic CI/review comments), and independent validation of this auto-fix loop working reliably.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”

   proof ↗⚑ flag

   Codegen~partialclaimed5/10

   Codegen agents can fix bugs, write tests, and run CI/CD-triggered fixes (codegen-docs-1, codegen-docs-18, codegen-docs-20), which could be applied to remediate failing agent-readiness checks, but there is no explicit documentation of an 'agent-readiness criteria' concept, scoring rubric, or automated fix workflow tied specifically to such criteria. missing for 10: explicit agent-readiness criteria/checklist, automated detection-to-fix pipeline for those criteria, evidence of hands-on success fixing readiness gates.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”

   proof ↗⚑ flag

2. engineering-leadRun a readiness report that evaluates how ready my repository is for autonomous agents

   weight 2 · round drawn
   Devin—none0/10

   No evidence of any 'readiness report' or scoring feature that evaluates a repository's suitability for autonomous agents; docs cover onboarding features like AGENTS.md, environment setup, indexing, and knowledge, but nothing that generates a readiness assessment/report artifact for engineering leads.

   ⚑ flag

   Codegen—none0/10

   The evidence pack shows PR review, security scanning, and CI/CD triggers, but there is no mention of a dedicated 'readiness report' feature that evaluates overall repository readiness for autonomous agents. Missing for 10: any documentation of a repo-wide readiness/scorecard report, criteria for agent-readiness, or a command/API that produces such an assessment.

   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”

   proof ↗⚑ flag

Security remediation

1. engineering-leadHave security alerts automatically validated and remediated with an opened pull request

   weight 2 · round drawn
   Devin~partialclaimed5/10

   Devin's 'Review with Auto-Fix' auto-responds to code review comments, fixes flagged bugs and iterates on CI failures, and MCP integrations extend it to security-adjacent tools like Sentry and Datadog, implying it could react to alerts and open a PR fix. However, there is no explicit documentation of a security-scanner (SAST/dependency-alert) validation-and-remediation pipeline or evidence of an actual security-alert-to-PR workflow. Missing for 10: explicit security-scanner (e.g., CodeQL/Dependabot/Snyk) integration, evidence of validating true-positive vs false-positive alerts, and a documented end-to-end 'alert triggers PR' example.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”

   proof ↗⚑ flag

   Codegen~partialclaimed5/10

   Codegen documents a security/secrets scan that posts a review comment on failure (codegen-docs-20) and general bug-fixing/PR-creation capabilities plus CI/CD-triggered agents (codegen-docs-1, codegen-docs-4, codegen-docs-18), suggesting it could act on a flagged issue and open a fix PR. However, there is no explicit evidence of ingesting external security alerts (e.g., Dependabot/CodeQL/Snyk), validating them as true positives, or automatically opening a remediation PR end-to-end — the documented scan flow stops at commenting, not remediating. missing for 10: explicit security-alert ingestion/validation workflow, evidence of an opened remediation PR tied to a specific alert, independent confirmation of this flow working in practice.

   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”

   proof ↗⚑ flag

Concurrent execution

1. engineering-leadRun many agent tasks concurrently to scale delivery throughput

   weight 3 · round to Devin
   Devin~partialclaimed7/10

   Devin docs explicitly document running multiple Devins in parallel on independent tasks and provide an API with create_as_user_id for programmatically spawning sessions on behalf of org members, directly supporting concurrent scaling of throughput. However, there's no first-party data on concurrency limits, cost/resource management at scale, or independent case studies validating throughput gains at team scale; missing for 10: quantified concurrency limits/scaling guidance, independent evidence of large-scale parallel usage, and cost/governance controls for many simultaneous sessions.

   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”

   proof ↗⚑ flag

   Codegen~partialclaimed6/10

   Codegen's SDK/API let you programmatically create and run multiple agents (agent.run, API 'full programmatic control over execution and monitoring') and CLI can view multiple running agents, implying support for parallel task execution, and CI/CD triggers suggest scaling automation into pipelines. However, there is no explicit documentation of concurrency limits, throughput guarantees, or dashboards for managing many simultaneous tasks at scale. Missing for 10: explicit concurrency/parallelism claims, scaling limits or throughput benchmarks, and independent evidence of running many agents simultaneously in practice.

   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

2. engineering-leadCreate agent sessions on behalf of other users in my organization

   weight 2 · round to Devin
   Devin✓fullclaimed8/10

   Devin's API explicitly supports creating sessions on behalf of any user in the organization via the create_as_user_id parameter, directly matching the engineering-lead use case for org-wide session delegation. Missing for 10: independent/hands-on corroboration of this specific feature and details on permission scoping/audit controls for delegated sessions.

   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”

   proof ↗⚑ flag

   Codegen~partialclaimed4/10

   Codegen's org-level SDK/CLI/API (org_id+token, 'manage organizations', programmatic agent creation) implies an engineering lead with org credentials could spin up agents at scale, but no docs explicitly describe creating or assigning a session to another named user in the org. Missing for 10: explicit user-impersonation or 'create on behalf of' API/UI feature, role-based permissions for delegating agent sessions, and any admin console for assigning sessions to teammates.

   • [claimed-docs] “Manage organizations - Switch between organizations and configure repositories”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

Deployment flexibility

1. developerUse a managed cloud offering to run agents without operating my own backend infrastructure

   weight 2 · round drawn
   Devin✓fullclaimed7/10

   Devin is delivered as a cloud-hosted agent (web app, API, session creation on behalf of users) so developers don't need to run their own backend/agent infrastructure — sessions run on Devin's managed cloud with its own configured environment, VPN access, and parallel execution. missing for 10: independent/hands-on confirmation of the managed infrastructure model, and explicit documentation of uptime/scaling/ops guarantees for the hosted service.

   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”

   proof ↗⚑ flag

   Codegen✓fullclaimed7/10

   Codegen offers cloud-run agents with isolated execution environments, API/SDK/CLI access, and no need to self-host infrastructure (codegen-docs-5, codegen-docs-11, codegen-docs-15, codegen-docs-16). This directly matches running agents without operating own backend. Missing for 10: independent/hands-on verification of the managed infrastructure's reliability at scale, and explicit pricing/SLA details for the managed cloud offering.

   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

2. engineering-leadSelf-host agent infrastructure locally, in containers, or on my own VMs

   weight 2 · round drawn
   Devin—none0/10

   Devin is delivered as a cloud-hosted SaaS agent platform; all evidence points to cloud sessions, API access, VPN connectivity into customer networks, and a CLI/terminal client for local interaction, but nothing indicates the core agent infrastructure itself can be self-hosted on-prem, in customer-managed containers, or on customer VMs. Missing for 10: any documentation of a self-hosted/on-prem deployment option, container images for the agent runtime, or VM-based installation of Devin's backend.

   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”

   proof ↗⚑ flag

   Codegen—none0/10

   Codegen is presented entirely as a cloud/SaaS agent service (GitHub App, Slack App, API, SDK, isolated cloud environments) with no evidence of a self-hostable runtime, container image, or on-prem/VM deployment option; the CLI logs local Claude Code sessions to Codegen's own cloud rather than hosting infrastructure locally.

   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”
   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

Headless automation

1. developerRun an agent headlessly inside CI/CD pipelines and shell scripts

   weight 2 · round drawn
   Devin~partialprobed6/10

   Devin's API is explicitly designed to 'integrate Devin into your applications, automate workflows, and build powerful tools' and supports creating sessions on behalf of users, which enables scripted/headless invocation, and a CLI is also documented. However, there is no explicit CI/CD pipeline example, exit-code/status handling, or GitHub Actions/shell-script integration guide in the evidence — the closest CI-related capability shown is 'Auto-Fix' reacting to CI failures rather than Devin running as a CI step itself. missing for 10: explicit CI/CD pipeline integration docs (e.g., GitHub Actions/shell script examples), documented headless/non-interactive CLI invocation with exit codes.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”

   proof ↗⚑ flag

   Codegen~partialprobed6/10

   Codegen explicitly documents triggering agents from CI/CD pipelines (e.g., on build failures) and offers a Python SDK plus CLI for programmatic/scripted invocation, which supports headless usage in pipelines and scripts. However, there's no dedicated CI/CD integration guide, no example pipeline configs (e.g., GitHub Actions YAML), and no independent/hands-on evidence confirming real-world headless CI usage. missing for 10: concrete CI/CD pipeline examples or config snippets, exit-code/status handling for scripts, independent verification of headless operation in automated pipelines.

   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”

   proof ↗⚑ flag