Devin vs OpenHands
subscription-per-seat · usage-based · credits
·open-source · free-tier · usage-based
OpenHands wins · 21–24 (27 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to OpenHandsDevin explicitly supports AGENTS.md, an agent-oriented documentation file that Devin reads before starting a coding session to gain project context — directly matching the story's intent of pointing an agent at agent-oriented docs. Devin's own docs site also serves an llms.txt endpoint (verified via probe), showing familiarity with the convention, though this is Devin's own site rather than evidence Devin ingests arbitrary llms.txt files from user projects. Missing for 10: explicit documentation that Devin consumes/parses llms.txt files themselves (as opposed to AGENTS.md) and independent hands-on confirmation of this workflow.
- [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.devin.ai/llms.txt # Devin Docs - [Desktop (100 pages)](https://docs.devin.ai/_llms/en/desktop.md):…”
A probe confirms OpenHands hosts a live llms.txt at docs.openhands.dev/llms.txt returning HTTP 200 with an explicit LLM-friendly documentation index, directly satisfying the story. Missing for 10: no independent third-party confirmation that agents actually consume/parse this file successfully in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round drawnDevin exposes a full REST API for session creation and automation (devin-docs-8, devin-docs-9), supports scheduling recurring/one-time headless runs via MCP (devin-docs-17), auto-triggers on CI failures and code review comments via Devin Review/Auto-Fix (devin-docs-20), and supports parallel/headless session orchestration (devin-docs-18). This covers CI/automation use cases well beyond interactive use. Missing for 10: no independent/hands-on CI-integration case study, and no dedicated 'CI runner' or GitHub Actions-style first-class integration doc beyond the review/auto-fix hook.
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
- [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
OpenHands supports CLI mode with auto-approve for non-interactive runs, API support for automation/scripting, webhook/schedule-triggered automations (Slack, GitHub, Linear), and can run headless in Docker/VMs/servers — all consistent with CI/automation usage. Missing for 10: an explicit first-party CI pipeline example (e.g., GitHub Actions config) or independent hands-on confirmation of headless CI runs.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [claimed-docs] “API support for automation and scripting”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round to DevinDevin CLI explicitly supports connecting external MCP servers, exposing their tools as slash commands with permission controls, and docs list example integrations (Datadog, Sentry, databases, Figma, Notion, Stripe, etc.). Missing for 10: independent/hands-on community verification of MCP tool usage in practice.
- [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
- [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
- [claimed-docs] “Devin CLI exposes each one as a slash command”
- [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
- [probe] “official MCP server documented at https://docs.devin.ai/cli/extensibility/mcp/overview”
ai-native userUse an official CLI
weight 2 · round drawnDevin ships an official CLI documented extensively (adaptive model routing, slash commands, MCP extensibility) confirmed by both docs and a probe hit, plus a separate Terminal/local CLI mode for quick fixes. Missing for 10: independent hands-on review of the CLI itself and more detail on installation/versioning.
- [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
- [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
- [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
- [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
- [claimed-docs] “Devin CLI exposes each one as a slash command”
- [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”
OpenHands documents an official CLI mode with natural language task execution, conversation resumption, and auto-approve controls, confirmed by dedicated docs pages and a probe verifying the CLI documentation page exists. Missing for 10: independent/hands-on third-party corroboration beyond vendor docs, and more detail on CLI installation/distribution mechanics.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
ai-native userDrive the product through a documented public API
weight 3 · round drawnDevin publishes a documented public API for integrating into applications, automating workflows, and creating sessions programmatically (including on behalf of other users), plus an MCP server exposing session/playbook/knowledge/scheduling management to external agents. Missing for 10: a discoverable machine-readable OpenAPI/swagger spec (probe found 404s) and independent third-party corroboration of API usage beyond vendor docs.
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
- [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.devin.ai/llms.txt # Devin Docs - [Desktop (100 pages)](https://docs.devin.ai/_llms/en/desktop.md):…”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”
OpenHands exposes a documented OpenAPI spec (openapi.json) plus an llms.txt LLM-friendly docs index, and pricing docs explicitly mention 'API support for automation and scripting,' confirming a public, documented API for programmatic/agentic control. missing for 10: independent third-party corroboration of API usage/reliability and more detailed API reference docs beyond the openapi.json probe.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
- [claimed-docs] “API support for automation and scripting”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to DevinDevin documents a permission system that restricts MCP tool access at multiple levels and an API parameter (create_as_user_id) for acting on behalf of specific users, but there is no direct documentation of issuing scoped/least-privilege API keys or credentials for the agent itself. Missing for 10: explicit API key/token scoping mechanism, documented permission tiers or roles for API credentials, and any least-privilege credential issuance workflow.
- [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
ai-native userBuild against official SDKs
weight 2 · round drawnDevin publishes a documented API reference for building integrations (create sessions, manage on behalf of users, automate workflows), plus MCP support for extensibility — clear official SDK-like surface for AI-native developers. Missing for 10: no official language-specific SDK packages (e.g. Python/JS client libraries) are cited, only REST API docs and no discoverable OpenAPI spec (probe found 404s), and no independent/hands-on corroboration of API usage.
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
- [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”
OpenHands documents an official Software Agent SDK ('a composable Python library for building agents that work with code') plus a public OpenAPI spec and llms.txt index, giving AI-native users concrete official interfaces to build against. Missing for 10: independent/third-party corroboration of SDK usage, and richer SDK-specific docs (examples, API reference depth) beyond the single description.
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
ai-native userSubscribe to events via webhooks
weight 2 · round to OpenHandsDevinnone0/10No evidence in the pack mentions webhooks or event subscription mechanisms for Devin; only API session creation, MCP integrations, and CLI features are documented. As an API-driven agentic product, webhook support would be a plausible and expected capability, so its absence counts as 'none' rather than 'na'.
GitHub README explicitly states OpenHands can 'run on a schedule or in response to webhook events' for automations, indicating webhook-triggered event subscription, but there is no dedicated documentation page detailing webhook setup, payload schema, or event types. missing for 10: dedicated webhook docs/config guide, independent/hands-on confirmation, and detail on which events can be subscribed to.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round to OpenHandsDevin's repo indexing enables 'Ask Devin' and DeepWiki features that generate AI insights and explanations from your codebase data, and Devin can proactively suggest fixes/improvements via code review auto-fix and bug triage. However, evidence is scoped only to codebase/ticket data, not broader business or analytics data, and there's no dedicated insights dashboard or proactive summarization feature documented. Missing for 10: evidence of insights beyond code (e.g., analytics/dashboards), independent corroboration of Ask Devin/DeepWiki quality, and any UI showcasing generated insights.
- [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”
OpenHands ships automation agents that inspect real data (incident logs, PR diffs, workflow failures, security alerts) and generate AI insights/suggestions such as severity assessment, review comments, root-cause analysis, and remediation PRs, directly matching the story's intent for a coding-agent product. Evidence is vendor-documented only, with no independent/hands-on corroboration of the quality of these insights. Missing for 10: independent validation of suggestion quality, a unified 'insights' UI/dashboard, and evidence of insights beyond code/security/ops contexts.
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
ai-native userSet up automations that run autonomously in the background
weight 2 · round drawnDevin supports scheduling via cron expressions (recurring/one-time) and agent selection through its MCP tooling, plus API session creation and Slack/Teams triggering, enabling autonomous background automations beyond interactive chat sessions. missing for 10: independent/hands-on verification of scheduled runs actually executing unattended, and more detail on failure/retry handling for scheduled tasks.
- [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
- [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
OpenHands supports scheduled/webhook-triggered automations integrating with Slack, GitHub, Linear, etc., and ships prebuilt autonomous workflows (bug triage, PR review, CI failure fixing, security remediation, Slack mention handling) that run without human intervention, plus API support for scripting automations. Missing for 10: independent/hands-on verification of the scheduling/webhook trigger reliability and no detailed docs excerpt on configuring schedules beyond marketing copy.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [claimed-docs] “API support for automation and scripting”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round drawnDevin itself functions as the built-in AI assistant to which tasks are delegated across web app, terminal, Slack/Teams, and API, with rich first-party documentation of the delegation workflow (docs-1,3,4,5,8,19). Community feedback raises reliability/skepticism concerns but does not concretely contradict that delegation itself works, so it's not disputed. Missing for 10: independent hands-on validation of delegation reliability/success rate beyond old demo critiques.
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
- [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
- [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
OpenHands is itself the built-in AI assistant/agent: CLI mode lets users type natural language tasks and get instant feedback, resume conversations, and auto-approve actions, and it can run locally, remote, or in the cloud with any LLM. This directly matches delegating tasks to a built-in assistant within the product. Missing for 10: independent/hands-on user reports validating the delegation experience beyond vendor docs, and more detail on task delegation depth (e.g., multi-step autonomy limits).
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [github] “Use with any LLM”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
ai-native userOperate the product with natural-language commands
weight 2 · round drawnDevin is fundamentally driven by natural-language task requests across web app, Slack/Teams tagging, CLI, and API (create_as_user_id sessions), letting users describe tasks like bug fixes, features, and migrations in plain language rather than issuing structured commands. Community evidence corroborates that Devin operates via conversational/task-based prompts even amid skepticism about output quality. missing for 10: independent hands-on confirmation of NL command fidelity across all surfaces (CLI/Slack/API), and no evidence addressing ambiguous or multi-step NL instruction handling limits.
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”
OpenHands' CLI mode explicitly lets users type natural language tasks and get instant feedback, resume conversations, and control approvals, directly matching the story, and this is corroborated by an official documented CLI probe. Missing for 10: independent/hands-on user reports validating the natural-language interaction quality beyond vendor docs.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round to OpenHandsDevinnone0/10Devin documents an API reference overview and parameters (e.g. create_as_user_id) but nothing describes an interactive, in-browser 'try it' experience with runnable code examples; a probe for standard OpenAPI/Swagger interactive-doc endpoints returned 404s, suggesting no such interactive explorer exists.
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”
Evidence shows an OpenAPI spec is served (openhands-probe-2) and API support is advertised for automation/scripting (openhands-docs-13), implying some API reference exists, but there is no evidence of an interactive documentation UI (e.g., Swagger/Redoc 'try it' console) or runnable code examples tied to that spec. missing for 10: interactive API explorer UI, runnable/try-it code examples, confirmation the openapi.json is rendered as browsable docs.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round to OpenHandsDevinnone0/10Devin has a documented API (devin-docs-8, devin-docs-9) but a direct probe for OpenAPI/swagger spec files at common paths returned 404s (devin-probe-2), and no docs page offers a downloadable machine-readable spec.
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”
A probe confirms an OpenAPI spec is served at docs.openhands.dev/openapi.json with a valid 'openapi' key, directly satisfying the machine-readable API spec requirement, and API support for automation/scripting is also documented in pricing. Missing for 10: independent third-party corroboration or detailed docs describing spec coverage/versioning.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to OpenHandsDevin's docs describe each session running in its own configurable workspace/environment (devin-docs-25) and connecting via VPN to reach internal services rather than being embedded in production (devin-docs-23), which implies some environment isolation, but there is no explicit claim of a dedicated 'sandbox vs production' testing mode or safeguards against touching production data. missing for 10: explicit documentation of a production-safe/sandboxed test environment, evidence of data isolation guarantees, and independent confirmation that production systems are never touched.
- [claimed-docs] “Configure it once, and every session boots into that known-good state.”
- [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
OpenHands can run agents in Docker/VMs (openhands-gh-5), which implies isolated execution rather than direct production access, but the evidence pack never explicitly describes a sandbox environment for testing safely against non-production data. Missing for 10: explicit documentation of sandbox/isolation guarantees, workspace-vs-production data separation, and any hands-on confirmation that production systems are protected.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnDevinnone0/10There is an API reference (devin-docs-8) but no evidence of API versioning scheme or a documented deprecation policy; the openapi.json probe returned 404s, suggesting no formal machine-readable spec either. Missing for 10: versioning scheme documentation, deprecation policy/notice, changelog for breaking changes.
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”
OpenHandsnone0/10Evidence shows OpenHands exposes an OpenAPI spec and general 'API support for automation and scripting,' and docs reference a 'V1' with legacy V0 pages excluded, but there is no documented API versioning scheme or deprecation policy anywhere in the pack.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to DevinDevin's docs describe running multiple Devins in parallel for independent tasks, an API for creating sessions programmatically (including on behalf of other users), and scheduling capabilities—together supporting bulk/automated task dispatch across many items. However, there's no concrete example or independent evidence of actually orchestrating large-scale bulk operations (e.g., processing hundreds of tickets at once), and community evidence focuses on single-task demos rather than bulk workflows. Missing for 10: hands-on/independent evidence of large-scale bulk runs, explicit parallelism limits, and a documented case study of bulk item processing.
- [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
OpenHands supports automations/workflows triggered by events (Slack, GitHub, Linear, webhooks) and API access for scripting, which enables some multi-item automation, but there's no explicit evidence of a bulk-operation feature (e.g., processing a batch list of items/tasks in one command or UI action). missing for 10: explicit bulk/batch operation feature, evidence of processing multiple items in a single invocation, UI/CLI support for batch task lists, independent confirmation of scale.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to OpenHandsDevin supports event-driven automation via scheduled/cron triggers (create schedules with cron expressions, notification preferences) and via Devin Review's Auto-Fix which automatically responds to code review comments and CI failures, plus API-based workflow automation. However, these are fixed built-in trigger types (schedule, CI/review events) rather than a general user-defined rules engine for arbitrary events. Missing for 10: evidence of a flexible custom rule/trigger builder covering arbitrary event sources (e.g., webhooks, ticket status changes) beyond scheduling and code-review auto-fix, and independent confirmation these triggers work reliably in practice.
- [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
OpenHands ships documented automation triggers—scheduled runs, webhook events, PR label events, Slack mentions, failed-workflow detection, and security-alert triage—each automatically invoking an agent action, matching the 'rules trigger actions on events' story (openhands-gh-4, openhands-docs-7 to openhands-docs-11). missing for 10: independent/hands-on verification that users can define fully custom rule logic (vs. fixed preset automations), and no evidence of a general-purpose rule-authoring UI or DSL for arbitrary event/action pairing
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
ai-native userSchedule recurring jobs or workflows
weight 2 · round drawnDevin's MCP documentation explicitly states scheduling support with 'cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection,' and this scheduling capability is exposed via MCP for session/playbook management. Missing for 10: independent/hands-on corroboration of recurring job execution in practice and more detail on scheduling UI/reliability beyond the docs mention.
- [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
- [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
- [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
OpenHands explicitly supports creating automations/workflows that run on a schedule or in response to webhook events, integrating with Slack, GitHub, Linear, etc., which directly matches recurring job scheduling. Missing for 10: independent/hands-on corroboration of the scheduling UI/config and details on job management (pause/edit/monitor recurring jobs).
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
ai-native userVersion, review, and roll back my automations
weight 1 · round drawnDevinnone0/10No evidence of version history, review/diff UI, or rollback of Devin's automations (playbooks, schedules, sessions); docs mention creating/scheduling sessions and playbooks but never versioning, auditing changes, or reverting them. missing for 10: version history for playbooks/sessions, review/approval workflow for automation changes, rollback/undo mechanism.
- [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
- [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
- [claimed-docs] “Knowledge is the best way to share codebase-level (vs. task-level) context that can help Devin when working in your codebase.”
OpenHandsnone0/10Evidence shows OpenHands can create automations/workflows (Slack, GitHub, Linear integrations, scheduled/webhook triggers) but nothing in the pack describes version history, review, or rollback mechanisms specifically for these automations themselves — no changelog, diff view, or revert feature is documented.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionAutonomous implementation
End-to-end implementation by the agent — multi-file changes, task completion
End to end feature delivery
ai-native userHave an agent automatically generate and run tests to validate its own code changes before proposing them
weight 2 · round to DevinDevin's docs describe autonomous CI-failure iteration and auto-fix behavior (devin-docs-20) and a terminal for executing/observing command output (devin-docs-6), implying it runs and reacts to test/build results before finalizing changes, but there is no explicit documentation of Devin proactively writing new tests and running them pre-emptively to self-validate before proposing a PR. missing for 10: explicit test-generation workflow documentation, evidence of Devin authoring test cases for new code, independent hands-on confirmation of this specific validation loop.
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
- [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”
OpenHandsnone0/10No evidence in the pack describes OpenHands agents autonomously generating or running tests to validate their own code changes before proposing them; the docs/GitHub excerpts cover CLI usage, multi-agent backends, automations, and pricing, but not test-generation/self-validation workflows.
developerHave an agent autonomously diagnose and fix a reported bug
weight 3 · round to OpenHandsDevindisputedcontradicted5/10Devin's docs explicitly describe autonomous bug repro/fix workflows, Slack-triggered bug sessions, and auto-fix on CI/review failures (devin-docs-1, devin-docs-4, devin-docs-19, devin-docs-20), supporting the core story. However, independent commentary cites a low real-world resolution rate (13.8% of issues resolved) and hands-on accounts describing shallow fixes (e.g., wrapping errors in try/catch rather than真正 diagnosing root cause), directly contradicting the strength of the autonomous-fix claim (devin-comm-2, devin-comm-6, devin-comm-4). Missing for 10: independent benchmark showing high success rate, more recent hands-on verification of bug-fix quality, and resolution of the discrepancy between vendor claims and community-reported performance.
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
- [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”
- [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”
OpenHands ships explicit automation for bug diagnosis and fixing: watching for new bugs/incidents and gathering logs (openhands-docs-7), detecting failed workflows/inspecting logs/identifying cause and opening a PR fix (openhands-docs-9), and general natural-language task execution via CLI (openhands-docs-1). This directly matches autonomous bug diagnosis-and-fix, though missing for 10: independent hands-on verification of fix success rate and end-to-end case studies beyond vendor docs.
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
product-managerGo from a mockup or design to a working implementation without an engineering handoff
weight 2 · round to DevinDevin's docs show it can autonomously implement features from tickets (Jira/Linear) and be tagged by non-engineers on Slack/Teams, supporting the general 'PM delegates, Devin implements' workflow, and it can connect to Figma via MCP integrations. However, there is no documented mockup-to-code pipeline, no PM-specific onboarding flow, and no evidence of turning a design file into a working UI end-to-end. missing for 10: direct design/mockup ingestion workflow, PM-focused case study or walkthrough, independent evidence of successful design-to-implementation outcomes.
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
- [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
OpenHandsnone0/10The evidence shows OpenHands executes natural-language coding tasks and automations, but nothing in the pack addresses ingesting mockups/design files or a PM-oriented, engineer-free workflow from design to implementation. missing for 10: mockup/design ingestion capability, no-code PM-facing workflow evidence, any example of design-to-code handoff elimination.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
developerHave an agent implement a requested feature end-to-end, including writing tests
weight 3 · round to OpenHandsDevindisputedcontradicted5/10Devin's docs explicitly claim end-to-end feature implementation (docs-1, docs-31) and even test-writing is implied as part of full feature delivery, but independent hands-on reporting from the original demo showed only 13.8% of issues resolved and reviewers noted the work was often only ~90% correct or used hacky fixes (e.g. wrapping in try-catch) rather than proper fixes (devin-comm-2, devin-comm-4, devin-comm-6), directly contradicting the 'implements features end-to-end reliably' claim. missing for 10: verified success-rate data on test-writing specifically, more recent independent benchmarks showing improved reliability, and evidence tests are written/passed as part of the workflow rather than just code changes.
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”
- [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
- [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”
- [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”
OpenHands is positioned as an autonomous coding agent that takes natural language tasks and works across CLI, cloud, and automation workflows (e.g., PR review, bug-fixing, incident response), implying it can implement features end-to-end, but the evidence pack lacks a concrete first-party or hands-on example of the agent writing a feature plus tests. Missing for 10: explicit documentation/demo of full feature-implementation-with-tests workflow, independent benchmark or hands-on verification of test-writing capability, and confirmation of end-to-end PR creation including tests.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [github] “Run OpenHands, Claude Code, Codex, Gemini, or any ACP-compatible agent across local, remote, and cloud backends.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
Environment setup
developerHave an agent automatically clone the repo, install dependencies, and configure its own working environment
weight 2 · round to DevinDevin's docs describe indexing repos, configuring a persistent environment/workspace that boots into a known-good state every session, VPN access to internal registries/git hosts, and AGENTS.md-based setup instructions it follows before coding — together these show it autonomously clones, configures, and prepares its own working environment. missing for 10: independent/hands-on verification of the clone+dependency-install flow end-to-end, and explicit documentation of automatic dependency installation step-by-step.
- [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
- [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
- [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
- [claimed-docs] “Configure it once, and every session boots into that known-good state.”
Evidence shows OpenHands runs agents in configurable sandboxed backends (Docker, VMs, local/remote/cloud) and supports natural-language task execution with auto-approve, which implies some autonomous environment setup, but no citation explicitly confirms the agent auto-clones repos or installs dependencies on its own. Missing for 10: explicit documentation of repo cloning, dependency installation, and end-to-end environment bootstrap steps.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Auto-approve all actions (use with caution)”
Interactive takeover
developerTake over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work
weight 2 · round to DevinDocs explicitly describe multi-surface handoff: delegate via web app then take over in Devin's IDE, use /handoff to send terminal tasks to cloud Devin, jump into the Interactive Browser to help Devin navigate, and tag Devin in Slack/Teams threads to redirect work — covering editor, terminal, and browser takeover. Missing for 10: independent/hands-on confirmation of a smooth mid-task handoff experience (community evidence is generic skepticism, not a concrete handoff failure).
- [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
- [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
- [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
- [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
OpenHands supports resuming conversations and switching between local/remote/cloud agents 'without losing focus,' implying some cross-surface continuity, but there's no explicit documentation of a developer taking over a live in-progress task from editor/terminal/browser mid-run with redirect capability. missing for 10: explicit IDE/editor integration for live takeover, concrete UI/UX description of mid-task redirect, and independent hands-on confirmation of seamless handoff across all three surfaces.
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [github] “Run OpenHands, Claude Code, Codex, Gemini, or any ACP-compatible agent across local, remote, and cloud backends.”
- [claimed-docs] “Resume previous conversations”
developerSend follow-up instructions to an active agent session to steer its work without restarting
weight 2 · round drawnDevin's docs describe delegating tasks and then 'taking over in Devin's IDE' or 'jumping in' to help with browsing tasks, implying some interactivity with an active session, but there is no explicit documentation of sending follow-up chat instructions to steer an in-progress session without restarting it. missing for 10: explicit docs on mid-session chat/steering, confirmation that follow-up messages are incorporated live, independent hands-on evidence of this workflow.
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
- [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
- [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
CLI mode docs mention typing natural language tasks with instant feedback and resuming previous conversations, which implies interactive follow-up steering, but there is no explicit evidence describing sending new instructions mid-task to an already-running/active agent session without restarting it. missing for 10: explicit documentation of mid-session steering/interrupt-and-redirect behavior while an agent is actively working, and independent/hands-on confirmation that follow-up messages are incorporated without restarting the session.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Sandbox execution
developerHave an agent safely execute code and install dependencies inside an isolated sandbox
weight 3 · round drawnDocs describe Devin operating in its own workspace/VM with terminal execution, environment configuration ('known-good state'), VPN access to internal services, and multi-platform (Android/macOS/Windows) build environments, implying isolated sandboxed execution, but there is no explicit documentation of sandbox security guarantees, dependency isolation, or safety controls around code execution, and no independent/hands-on verification of sandbox isolation. missing for 10: explicit sandbox/isolation security documentation, independent verification of safe execution, details on dependency install safety and containment.
- [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
- [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
- [claimed-docs] “Configure it once, and every session boots into that known-good state.”
- [claimed-docs] “Devin can build and run Android applications directly on its own machine — giving it the Android equivalent of Computer Use and browser inte…”
- [claimed-docs] “Devin now has access to macOS virtual machines. This means Devin can now build and test iOS and macOS applications.”
- [claimed-docs] “Devin supports Windows as a build and session platform.”
Evidence confirms OpenHands supports running agents in isolated environments (Docker/VMs) and explicitly references 'sandbox infrastructure' as part of its backend, implying code execution is sandboxed. However, there is no detailed documentation on dependency installation within the sandbox, security guarantees, or isolation mechanics beyond these brief mentions. missing for 10: explicit sandbox architecture docs, dependency-installation workflow details, isolation/security guarantees, independent verification of sandbox safety.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsHuman oversight
Keeping a human in the loop — approvals, checkpoints, interrupts
Approval controls
developerConfigure an agent to auto-approve all its actions instead of confirming each one
weight 2 · round to OpenHandsDevinnone0/10The evidence pack describes permission systems for MCP tools (devin-docs-13) and autonomous features like Auto-Fix (devin-docs-20), but there is no documentation of a setting that lets a developer toggle between 'confirm each action' and 'auto-approve all actions' for Devin's own task execution. missing for 10: explicit auto-approve/confirmation toggle documentation, evidence of a per-action approval workflow that can be disabled, any community confirmation of this specific control.
- [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
Docs explicitly list an 'Auto-approve all actions (use with caution)' setting for CLI mode, directly matching the story. Missing for 10: independent/hands-on confirmation and details on scope (per-action vs global) or how to configure it beyond CLI mode.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
product-managerApprove key agent decisions from my phone while agents continue working
weight 1 · round to OpenHandsDevinnone0/10No evidence describes a mobile app or phone-based approval interface for agent decisions; docs mention Slack/Teams tagging, web app, IDE, and scheduling notifications but nothing about approving decisions from a phone while sessions continue. The axis is plausible for an agentic dev tool with human-in-the-loop needs, but no supporting evidence exists.
OpenHands Cloud offers 'hosted cloud access from desktop and mobile' and Slack-based interaction (@mentions, replies) that could let a PM check in remotely, but there is no documented feature for approving specific in-progress agent actions/decisions via a mobile interface while the agent continues autonomously — the only approval-related control mentioned is a blanket 'auto-approve all actions' CLI flag, not selective human-in-the-loop approval. Missing for 10: explicit mobile approval/confirmation UI, human-in-the-loop decision gating documentation, evidence of push notifications or approval prompts reaching a phone.
- [claimed-docs] “Hosted cloud access from desktop and mobile”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [claimed-docs] “Auto-approve all actions (use with caution)”
engineering-leadSet tiered autonomy levels controlling what an agent can do without manual confirmation
weight 3 · round drawnDevin documents a permission system for MCP/tool access that can be controlled 'at multiple levels' and features like Auto-Fix, human takeover in the IDE, and interactive browser intervention imply some control over autonomous action, but there is no documented feature specifically framed as configurable 'tiered autonomy levels' for an engineering lead to set organization-wide policies on unattended agent actions. missing for 10: explicit autonomy-tier configuration UI/API, organization-wide policy controls, and confirmation-threshold settings distinct from ad-hoc MCP tool permissions.
- [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
Evidence shows only a binary confirmation toggle (auto-approve all actions vs manual confirmation) in CLI mode, not a tiered/granular autonomy system with multiple configurable levels for an engineering lead to set. Missing for 10: documented multi-tier permission/autonomy levels, admin controls to enforce team-wide policies, and per-action or per-risk-category confirmation settings.
- [claimed-docs] “Auto-approve all actions (use with caution)”
Model control
ai-native userHave each task prompt automatically routed to the most suitable underlying model
weight 2 · round to DevinDevin's CLI has an explicit "Adaptive" model router that analyzes each prompt and automatically routes it to the best-performing model, with an override via /model if manual control is desired. This directly matches the story of automatic per-task model routing, though it's documented only for the CLI product surface and lacks independent/hands-on validation of routing quality. Missing for 10: independent corroboration of routing accuracy, and confirmation this applies across all Devin surfaces (web/IDE) not just CLI.
- [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
- [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
- [claimed-docs] “Adaptive is an intelligent model router that automatically selects the best AI model for each task.”
- [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”
OpenHandsnone0/10Evidence shows OpenHands supports using any LLM and switching between agent backends manually, but there is no mention of automatic routing of prompts to the most suitable model based on task characteristics. Missing for 10: any model-routing/selection logic, per-task model suitability heuristics, or documentation of automatic model selection.
- [github] “Use with any LLM”
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
engineering-leadSwitch away from automatic model selection to a specific model of my choice
weight 1 · round to DevinDevin's docs explicitly state that Adaptive automatically routes prompts to the best model, but users can switch away from Adaptive to a specific model at any time using /model, directly matching the story. Missing for 10: independent/hands-on corroboration of the /model switch in practice and more detail on which specific models are selectable.
- [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
- [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
- [claimed-docs] “Adaptive is an intelligent model router that automatically selects the best AI model for each task.”
- [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”
Evidence confirms OpenHands can be used with any LLM and lets users bring their own LLM keys, implying model choice is configurable, but there is no explicit documentation of an 'automatic model selection' default or a UI/CLI toggle to override it. Missing for 10: explicit docs on default/automatic model selection behavior, step-by-step instructions for switching models, and independent confirmation of the switch working in practice.
- [github] “Use with any LLM”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Visibility monitoring
developerWatch what a running agent is doing in real time, including its current status
weight 3 · round to DevinDevin's docs describe real-time observability features like watching the terminal execute commands with live output logs, jumping in during interactive browsing, and taking over in Devin's IDE mid-task, all indicating a live view of the agent's current activity/status. Missing for 10: no explicit dedicated 'status dashboard' documentation and no independent/hands-on corroboration of the real-time monitoring UX beyond vendor docs.
- [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
- [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
OpenHands CLI mode offers instant feedback on tasks and resumable conversations, implying some real-time interaction, but there's no explicit evidence of a live status dashboard, streaming action log, or step-by-step progress view while an agent runs. missing for 10: explicit real-time status/progress UI documentation, evidence of live action streaming or step visibility, independent hands-on confirmation of watching an agent live.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
developerGet notified when an agent completes a task or needs my input
weight 2 · round drawnDevin supports Slack/Teams tagging where sessions and results appear in-thread (devin-docs-4, devin-docs-19), and scheduled sessions explicitly support 'notification preferences' (devin-docs-17), implying some completion/status alerts. However there's no dedicated first-party documentation of a generic notify-on-completion or needs-input alert system (e.g., email/push) outside Slack/Teams and scheduling contexts. Missing for 10: explicit documentation of a unified notification mechanism across all session types, and independent/hands-on confirmation that notifications reliably fire when input is needed.
- [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
- [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
- [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
OpenHands supports Slack integration where it 'replies when the agent finishes' and workflow automations tied to webhook events (Slack, GitHub, Linear), which implies notification-like behavior on task completion; CLI mode also gives instant feedback in interactive sessions. However, there's no explicit evidence of a dedicated notification system for 'needs my input' events or push/desktop alerts outside Slack context. missing for 10: explicit documentation of notifications when agent needs human input/approval, native mobile/desktop push alerts, and independent/hands-on confirmation of notification reliability.
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
Intent to spec — stories about intent to spec in this arenaIntent to spec
Stories about intent to spec in this arena
Natural language task intake
developerDescribe a feature or bug in plain language and have it automatically turned into a scoped implementation task
weight 3 · round to DevinDevin's docs describe accepting natural-language asks (tickets, Slack/Teams messages, delegated tasks) and turning them into sessions/implementation tasks, which is the core intent-to-spec workflow (devin-docs-1, devin-docs-4, devin-docs-5, devin-docs-19). Community commentary raises concerns about actual task success rates and correctness (devin-comm-2, devin-comm-4), tempering confidence in scoping quality. Missing for 10: independent hands-on evidence that the auto-generated task scope/plan is accurate or reviewable before execution, and no dedicated 'plan/spec preview' feature described.
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
- [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
- [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”
OpenHands lets users type natural-language tasks directly in the CLI and receive feedback, which is the core mechanism for turning plain-language input into agent-executed work, and automation triggers (Slack mentions, PR labels, failed workflows) show it converting informal signals into concrete PRs/tasks. However, there is no evidence of an explicit 'spec' or scoped task artifact (e.g., a generated plan, ticket, or requirements doc) being produced before implementation — missing for 10: explicit task-scoping/spec generation step, evidence of a structured plan artifact, and independent confirmation that vague bug/feature descriptions reliably become well-scoped tasks rather than direct code edits.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
product-managerConvert user feedback submissions into structured tasks with proposed scope
weight 2 · round to DevinDevin can ingest raw inputs like Linear/Jira tickets and Slack/Teams threads about bugs or feature requests and turn them into an actionable session (devin-docs-1, devin-docs-4, devin-docs-19), which covers the 'feedback-to-task' intake path, but there is no evidence of a distinct structured-task output with an explicit proposed scope/plan artifact for PM review before execution. Missing for 10: evidence of a scoping/planning deliverable (e.g., a spec doc or task breakdown with effort/scope estimate) generated prior to coding, and any PM-specific workflow or UI for reviewing/approving that scope.
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
OpenHandsnone0/10OpenHands integrates with Linear/GitHub/Slack for automations and can process natural-language tasks, so the general axis of turning input into work items is plausible, but there is no evidence of a feature that ingests user feedback and outputs a structured task with proposed scope for PM workflows.
developerAttach a marked-up screenshot or mockup to a task so the agent implements the correct visual change
weight 2 · round drawnDevinnone0/10No evidence in the pack mentions attaching screenshots, mockups, or marked-up images to a Devin task, nor any visual-diff/design-to-code workflow; documentation covers ticket delegation, terminal, Slack, MCP, and environment setup but nothing about image/mockup-based task intake.
Plan approval
developerReview and approve an agent's implementation plan before any code changes are made
weight 3 · round drawnDevinnone0/10The evidence pack describes Devin's task delegation, IDE handoff, terminal visibility, and MCP integrations, but nowhere documents a plan-review/approval gate before Devin begins making code changes. Missing for 10: any doc mentioning an upfront plan proposal, explicit approval step, or 'plan mode' prior to execution.
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
OpenHandsnone0/10The evidence pack mentions auto-approve controls and CLI usage but contains no mention of a plan-review/approval step before code changes are made; no planning-phase or approval-gate feature is documented. missing for 10: any documentation of a plan-generation step, an approval/confirmation gate prior to code edits, or user testimony confirming such a workflow exists.
engineering-leadApprove a task's scope and contract before an agent is allowed to modify the repository
weight 2 · round drawnDevinnone0/10The evidence pack shows delegation, review-after-the-fact, and IDE takeover workflows, but no documented mechanism where an engineering-lead must approve a task's scope/contract before Devin is allowed to start modifying the repository. Missing for 10: explicit pre-execution approval/gating workflow, scope/contract sign-off UI or API, evidence of blocking repo writes until lead approval.
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
- [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
OpenHandsnone0/10No evidence of a scope/contract approval workflow gating agent repository modifications; only auto-approve settings and general automation features are documented, not a pre-execution scope-approval gate for engineering leads.
- [claimed-docs] “Auto-approve all actions (use with caution)”
Ticket driven tasking
developerAssign a coding task to an agent directly from an existing issue or ticket
weight 3 · round to DevinDocs explicitly state Devin can be asked to tackle Linear/Jira tickets, tagged on Slack/Teams threads about bugs to start sessions, and integrated via API/MCP to create sessions from external triggers, directly supporting assigning tasks from existing issues/tickets. Missing for 10: independent hands-on verification of ticket-to-task handoff and no direct product screenshot of ticket-linking UI in evidence.
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
- [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
OpenHands supports automations that integrate with GitHub and Linear and can respond to webhook events, plus label-triggered PR review and bug-watching automations, implying issue/ticket-triggered agent runs are possible. However there's no explicit documentation of a workflow where a developer directly assigns a specific issue/ticket to an agent (e.g. via an 'assign to OpenHands' button or issue-comment trigger) as opposed to general automation setup. Missing for 10: explicit documentation of issue-to-agent assignment UX (e.g., GitHub issue comment/label triggering agent to pick up that specific ticket), independent/hands-on confirmation of this workflow, and ticketing system coverage beyond GitHub/Linear mentions.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to OpenHandsDevin exposes a documented API (session creation, create_as_user_id) and an MCP server that surfaces session management, playbooks, knowledge, and scheduling (devin-docs-8,9,15,29), suggesting broad but not full parity with the web/IDE UI. There's no explicit claim or evidence that UI-only features like the Interactive Browser takeover, terminal handoff, VPN setup, or environment/emulation configuration are fully API-accessible, and no OpenAPI spec was found (devin-probe-2) to confirm completeness. Missing for 10: explicit vendor statement of API/UI parity, evidence that interactive/browser/VPN/environment features are API-controllable, and independent confirmation of parity.
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
- [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”
OpenHands exposes a documented OpenAPI spec and dedicated 'API support for automation and scripting' tier, plus CLI mode with resume/auto-approve that mirrors UI conversation flows, suggesting broad API parity. However, no evidence explicitly confirms that every UI feature (e.g., cloud dashboard views, integrations config, enterprise SSO management) is fully API-accessible. Missing for 10: explicit parity documentation enumerating UI vs API feature coverage, and independent/hands-on confirmation that API can replicate all UI workflows.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnDevinnone0/10No evidence of any data export feature, open-format export, or account/data portability tooling in Devin's docs; evidence covers task delegation, MCP, CLI, and platform support but nothing about exporting session data, knowledge, or playbooks in open formats for migration away from the product.
OpenHandsnone0/10No evidence in the pack describes an explicit data export feature or open-format data portability; only conversation resume and self-hosting options are mentioned, not a way to export/leave with all user data. missing for 10: explicit export/download feature, documented open data formats, statement on data portability upon leaving the platform.
ai-native userRead the product's source under an open license
weight 2 · round to OpenHandsDevinnone0/10Devin is a closed-source commercial SaaS product; no evidence of any open-source license or public source repository is present in the evidence pack, and the axis is applicable since AI tooling could conceivably publish open-source components. missing for 10: any open-license source release, public repo link, or licensing statement.
The GitHub repository (openhands-gh-1..5) confirms the source code is publicly hosted and readable, and openhands-docs-15 explicitly references an 'OpenHands Open Source' plan/tier, implying the core project is open-licensed. However, no evidence pack item names the specific license (e.g., MIT/Apache) or points to a LICENSE file, so full open-license confirmation is unverified. Missing for 10: explicit license name/file citation, independent confirmation of license terms.
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
ai-native userSelf-host the core product
weight 3 · round to OpenHandsDevinnone0/10Devin is exclusively delivered as a cloud SaaS product (with local CLI/terminal front-ends that hand off to cloud sessions); no evidence anywhere in the docs or community discussion describes a self-hostable core engine or on-prem deployment option. Missing for 10: any documentation of self-hosted/on-prem deployment, container/binary distribution of the core agent, or licensing for self-hosting.
- [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
- [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
- [claimed-docs] “Configure it once, and every session boots into that known-good state.”
Evidence shows OpenHands can be run locally/self-hosted (Docker, VMs, or any agent server backend) as opposed to only using the managed Cloud/Enterprise offerings, and it is open-source with an SDK for building on it. Missing for 10: independent hands-on confirmation of a full self-hosted setup (e.g., a third-party report of running the entire stack without cloud dependencies) and detailed self-hosting infra requirements/documentation.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
Enterprise licensing
engineering-leadLicense an enterprise deployment with SSO and commercial support for organization-wide rollout
weight 2 · round to OpenHandsDevinnone0/10No evidence pack items mention enterprise licensing, SSO, or commercial support offerings for Devin; only product feature docs and community sentiment are present. Missing for 10: SSO/SAML documentation, enterprise plan/pricing page, commercial support SLA details.
OpenHands explicitly markets an Enterprise tier with licensed self-hosting/managed deployment and commercial support (openhands-docs-5), and the pricing page lists Enterprise SAML/SSO (openhands-docs-14). However, there is no detail on contract terms, SLA specifics, or independent confirmation of enterprise rollouts. Missing for 10: concrete SLA/support-tier documentation, case studies or third-party validation of enterprise SSO rollout, and clarity on org-wide admin/governance controls.
- [claimed-docs] “OpenHands Enterprise provides commercial capabilities and support for organizations that need licensed self-hosting or managed deployment op…”
- [claimed-docs] “Enterprise SAML / SSO”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
Model flexibility
engineering-leadBring my own LLM or API key so agents run on the model of my choice
weight 2 · round to OpenHandsDevinnone0/10Docs describe an internal 'Adaptive' model router and a /model command to pick among Devin's supported models, but there is no evidence of bringing a custom LLM or your own API key to run agents on a model of the engineering lead's choice.
- [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
- [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
- [claimed-docs] “Adaptive is an intelligent model router that automatically selects the best AI model for each task.”
GitHub docs explicitly state OpenHands can be used with any LLM, and pricing docs confirm both Open Source and Cloud plans support bringing your own LLM keys, directly matching the story. Missing for 10: independent/hands-on verification of BYO-key setup and any model-specific limitations or edge cases.
- [github] “Use with any LLM”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Usage quotas
engineering-leadSee and manage plan-based daily task and concurrency limits for agent workflows
weight 2 · round drawnDevinnone0/10No evidence pack item mentions plan tiers, daily task quotas, concurrency limits, or an admin console/dashboard for managing such limits; docs cover running parallel Devins and API session creation but not visibility or management of plan-based limits. Missing for 10: pricing/plan documentation, admin UI for concurrency/task caps, usage dashboards or alerts for limit management.
- [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
OpenHandsnone0/10Evidence shows pricing page mentions plan features like SSO, API access, and BYO LLM keys, but nothing about daily task limits, concurrency limits, or any management dashboard for such limits.
- [claimed-docs] “Hosted cloud access from desktop and mobile”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “Enterprise SAML / SSO”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnDevinnone0/10No evidence in the pack mentions data residency, regional storage options, or compliance controls for where session/data is stored; docs cover VPN connectivity, environments, and MCP but nothing about choosing a storage region. Missing for 10: any mention of data residency/region selection, compliance certifications (e.g., SOC2/GDPR data locality), or enterprise data storage controls.
ai-native userPrevent my data from being used to train AI models
weight 3 · round drawnDevinnone0/10No evidence in the pack addresses data-training opt-out, privacy controls, or any policy about excluding user data from model training; all docs cover feature capabilities, MCP, CLI, and environment support instead.
ai-native userControl data retention and deletion
weight 2 · round drawnDevinnone0/10No evidence in the pack addresses data retention policies, deletion controls, or privacy configuration options for Devin; the docs cover features, MCP, CLI, and environments but not data governance/retention settings.
OpenHandsnone0/10No evidence pack items describe data retention policies, deletion controls, or user-facing settings for managing stored conversation/data lifecycle; while OpenHands is open-source and self-hostable (implying some inherent control), no explicit retention/deletion feature or documentation is cited.
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnDevinnone0/10No evidence pack item mentions telemetry opt-out, analytics settings, or usage-tracking controls for Devin; nothing addresses this privacy axis at all.
Repo integration — stories about repo integration in this arenaRepo integration
Stories about repo integration in this arena
Chat integration
developerTag an agent in a chat thread to discuss and delegate a bug or task
weight 2 · round to DevinDocs explicitly describe tagging Devin in Slack/Teams threads to discuss and delegate bugs/tasks, starting sessions directly from conversations. Missing for 10: independent/hands-on corroboration of this specific chat-tagging workflow beyond vendor docs.
- [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
OpenHands documents a Slack integration where the agent watches for @openhands mentions, opens a conversation with the message context, and replies when done (openhands-docs-11), which matches tagging an agent in a chat thread to delegate a task. However this is limited to Slack specifically (not other chat platforms), appears tied to the Cloud/automation feature set rather than the core product, and lacks independent/hands-on corroboration. Missing for 10: support for other chat platforms (e.g., Teams, Discord), independent verification of the Slack flow working in practice, and detail on how delegated context/threading is preserved during the exchange.
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Knowledge context
developerAdd a context file describing my codebase conventions so agents generate more relevant plans and code
weight 3 · round to DevinDevin explicitly supports AGENTS.md files placed in the project root that Devin reads before coding, plus a separate 'Knowledge' feature for sharing codebase-level conventions/context to improve plans and code generation. Missing for 10: independent/hands-on confirmation that this measurably improves plan/code relevance beyond vendor docs.
- [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
- [claimed-docs] “Knowledge is the best way to share codebase-level (vs. task-level) context that can help Devin when working in your codebase.”
- [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
developerQuery generated documentation for any public or private repository
weight 1 · round to DevinDevin explicitly documents 'DeepWiki'-style repo indexing and MCP tools that 'explore and query documentation for any GitHub repository (public or private with authentication)', directly matching the story, backed by repo indexing docs enabling Ask Devin/DeepWiki features. Missing for 10: independent hands-on verification of documentation query quality/accuracy beyond vendor docs.
- [claimed-docs] “These tools let you explore and query documentation for any GitHub repository (public or private with authentication)”
- [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
- [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
- [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
Project management integration
product-managerConnect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly
weight 3 · round to DevinDevin explicitly supports Linear and Jira tickets natively (devin-docs-1) and its MCP extensibility (devin-docs-12, devin-docs-21) lets it connect to arbitrary tool servers including issue trackers, which could cover ClickUp/Monday.com. However, there's no first-party documentation of native ClickUp or Monday.com connectors, and no independent/hands-on evidence confirming ticket-management workflows actually work end-to-end via these integrations. missing for 10: explicit native ClickUp/Monday.com support docs, independent verification of issue-tracker workflows succeeding in practice.
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
- [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
- [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
OpenHands documents automation workflows that integrate with Linear (and Slack/GitHub) via webhooks/schedules, showing some issue-tracker connectivity, but Jira, ClickUp, and Monday.com are never mentioned anywhere in the evidence pack — only vague 'and more' language covers them. missing for 10: explicit Jira/ClickUp/Monday.com integrations, docs on ticket management workflows beyond Linear, evidence of two-way ticket manipulation (create/update/close) rather than just webhook triggers.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Version control integration
developerConnect a GitHub repository so an agent can access the code and open pull requests against it
weight 3 · round drawnDocs confirm repository indexing (public/private) and GitHub-repo access for context, plus 'Devin Review with Auto-Fix' responding to code review/CI, implying PR interaction, but there's no explicit documentation of a GitHub connection/auth flow or of Devin opening pull requests directly. missing for 10: explicit GitHub app/OAuth connection setup steps, explicit 'opens pull requests' documentation, independent/hands-on confirmation of PR creation.
- [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
- [claimed-docs] “These tools let you explore and query documentation for any GitHub repository (public or private with authentication)”
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
GitHub is explicitly listed as an integration target for OpenHands automations, and multiple documented workflows show the agent opening pull requests (fixing failed CI, remediating security alerts, responding to PR-review label events), which implies repo access and PR creation. However, there is no first-party documentation of the actual repo-connection/auth flow (e.g., installing a GitHub App, granting repo scopes) or hands-on confirmation that this works end-to-end. Missing for 10: explicit repo-connection setup docs, evidence of PR creation permissions/scopes, and independent verification of successful PRs opened.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
developerGrant an agent access to my repositories with a one-click install, without complex setup
weight 2 · round to DevinDevin does support connecting to repos (indexing, GitHub-based session creation, VPN access to internal git hosts) and this is described as straightforward onboarding, but there's no evidence of a literal 'one-click install' flow like a GitHub App marketplace install — setup instead involves indexing, environment configuration, AGENTS.md, and knowledge onboarding steps. Missing for 10: explicit one-click/marketplace-style install flow, independent evidence of setup simplicity, and confirmation that repo access requires no additional configuration.
- [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
- [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
- [claimed-docs] “Configure it once, and every session boots into that known-good state.”
- [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
OpenHands Cloud offers hosted access and integrations (Slack, GitHub, Linear, webhooks) suggesting some repo connection flow, but there is no concrete evidence of a one-click repo install/auth flow — no screenshots, GitHub App install steps, or onboarding walkthrough. missing for 10: documented one-click GitHub/GitLab App install flow, evidence of minimal setup steps, independent confirmation of ease of onboarding.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
- [claimed-docs] “Hosted cloud access from desktop and mobile”
Review quality gates — quality gates on changes — review flow, required checks, merge protectionReview quality gates
Quality gates on changes — review flow, required checks, merge protection
Ci remediation
engineering-leadHave failed CI workflows automatically diagnosed and fixed with a proposed pull request
weight 3 · round drawnDevin explicitly documents 'Devin Review with Auto-Fix' that responds to code review comments, fixes flagged bugs, and iterates on CI failures, directly matching the story of auto-diagnosing and fixing failed CI with a proposed PR. However, this is vendor-only documentation with no independent/hands-on corroboration of the CI-fix workflow specifically. Missing for 10: independent verification of CI-fix accuracy/success rate, details on PR proposal format, and evidence of real-world reliability beyond the single docs line.
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
OpenHands documents a dedicated automation that detects failed CI workflows, inspects logs, identifies the likely cause, and opens a pull request with a proposed fix — matching the story directly. Missing for 10: independent/hands-on verification of this automation working in practice, and detail on configuration/setup beyond the marketing description.
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
developerTrigger an agent from CI/CD pipelines to fix a broken build or failing test
weight 2 · round to OpenHandsDocs explicitly state Devin Review with Auto-Fix can iterate on CI failures and fix flagged bugs, and the Devin API lets you integrate Devin into applications/workflows and create sessions programmatically (including on behalf of users), which supports triggering it from automated pipelines. However, there's no explicit documented example of a CI/CD pipeline (e.g., GitHub Actions/Jenkins) directly invoking Devin on a failing build/test, and no independent/hands-on confirmation of this exact workflow. Missing for 10: explicit CI/CD trigger integration example, independent verification that pipeline-triggered fixes work reliably.
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
OpenHands advertises a workflow that watches for failed CI/CD workflows, inspects logs, identifies causes, and opens a PR with a fix (openhands-docs-9), plus webhook/schedule-triggered automations (openhands-gh-4) and API support for scripting (openhands-docs-13), which together support triggering an agent from CI/CD to fix broken builds. However, there is no concrete example of GitHub Actions/CI pipeline configuration, no evidence of test-failure-specific triggers, and no independent/hands-on confirmation that this works as described. Missing for 10: explicit CI pipeline integration docs/examples, test-failure-specific triggers, third-party verification.
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
Diff review
developerConfigure an agent to automatically open a pull request when its task completes
weight 2 · round to OpenHandsDevin is known to work within GitHub workflows and create PRs as part of its task completion (implied by its coding-agent nature and features like Auto-Fix responding to review comments/CI), but the evidence pack has no explicit documentation describing a configurable setting to auto-open a pull request upon task completion. missing for 10: explicit docs describing PR-creation-on-completion configuration, settings/toggle for auto-PR behavior, and independent confirmation of this specific workflow.
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
OpenHands documents workflow automations that open pull requests automatically for specific triggers (failed CI, security alerts) via [openhands-docs-9] and [openhands-docs-10], showing the underlying capability exists. However, there's no direct documentation of configuring a general coding-task agent to auto-open a PR upon arbitrary task completion — the evidence only covers specific automation templates (bug-fix, security remediation) rather than a generic 'open PR on task completion' setting. Missing for 10: explicit config option/flag for auto-PR-on-completion in standard task workflows, independent/hands-on confirmation of this behavior.
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
developerReview a diff of an agent's changes and approve it before it becomes a pull request
weight 3 · round drawnDocs mention delegating a task and 'taking over in Devin's IDE once it gives you a good first draft' (devin-docs-5), implying some human checkpoint before final output, and Devin's terminal/log viewing lets you watch work in progress (devin-docs-6). However there is no explicit documentation of a diff-review/approve-before-PR gate — 'Devin Review with Auto-Fix' (devin-docs-20) is about post-PR review comment handling, not a pre-PR approval step. Missing for 10: explicit diff viewer with approve/reject before PR creation, documentation of a formal approval gate, and independent/hands-on confirmation of this workflow.
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
- [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
OpenHands has an auto-approve/manual-approve action mode (openhands-docs-3 implies a default confirmation step exists before auto-approve is enabled) and can open PRs after agent work, suggesting some human-in-the-loop gating exists, but there's no explicit documentation of a diff review UI or an approval gate specifically before PR creation. missing for 10: explicit diff-review interface, documented approve/reject step tied to PR creation, evidence of a review-before-merge workflow.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
Pr review automation
ai-native userHave incoming issues automatically triaged with severity suggested and routed to the right owner
weight 2 · round to OpenHandsDevinnone0/10Devin can be pointed at Linear/Jira tickets or tagged in Slack to work a bug, but nothing in the evidence describes automatic triage logic that assesses severity or routes issues to the correct owner — these are core, specific parts of the story. Missing for 10: severity classification/scoring, ownership/routing logic, any 'triage' feature or automated inbox-sorting workflow.
- [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
- [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
- [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
openhands-docs-7 directly describes an automation that watches for new bugs/incidents, gathers logs and recent changes, suggests severity, and routes the issue to the right owner — matching the story closely, backed by GitHub-documented webhook/scheduled automation infrastructure (openhands-gh-4). Missing for 10: independent/hands-on verification of triage accuracy and routing correctness beyond vendor's own site copy.
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
engineering-leadHave every pull request automatically reviewed with AI-generated inline comments
weight 3 · round to OpenHandsDevin documents a 'Devin Review with Auto-Fix' feature that responds to code review comments, fixes flagged bugs, and iterates on CI failures, implying an automated review capability tied to PRs, but the evidence pack gives no detail on how review comments are generated, whether they are inline, or how coverage/setup works for 'every PR'. missing for 10: explicit documentation of inline AI-generated PR comments, configuration for gating every PR, and independent/hands-on confirmation of the review quality.
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
OpenHands documents a specific automation that watches for a configurable PR label, inspects full PR context, and posts an AI review comment, which directly matches automated PR review with AI-generated comments. However, it is label-triggered rather than automatic on every PR, and posts once per label event rather than full inline (line-by-line) comments, so it's a partial match to 'every pull request' with 'inline comments'. missing for 10: evidence of automatic triggering on all PRs without manual labeling, confirmation of true inline (line-level) code comments vs a single summary comment, and independent/hands-on verification of this workflow in practice.
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
Readiness checks
engineering-leadAutomatically fix failing agent-readiness criteria in my repository
weight 1 · round to DevinDevin Review with Auto-Fix explicitly auto-fixes CI failures and review comments, which could cover agent-readiness gate checks if they're wired into CI/PR review, and AGENTS.md support shows Devin can consume agent-readiness config files. However there's no evidence of a dedicated 'agent-readiness criteria' scorecard or targeted remediation workflow distinct from generic CI/review auto-fix. Missing for 10: explicit agent-readiness scoring/criteria detection, evidence of automatically remediating specific agent-readiness failures (vs generic CI/review comments), and independent validation of this auto-fix loop working reliably.
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
- [claimed-docs] “Configure it once, and every session boots into that known-good state.”
OpenHandsnone0/10The evidence shows OpenHands can fix failing CI workflows, remediate security alerts, and review PRs, but nothing ties this to a defined 'agent-readiness' criteria/checklist concept that an engineering lead could gate on and auto-remediate. Missing for 10: any mention of agent-readiness scoring, a checklist/criteria framework, or evidence that OpenHands detects and fixes failures against such a standard.
engineering-leadRun a readiness report that evaluates how ready my repository is for autonomous agents
weight 2 · round drawnDevinnone0/10No evidence of any 'readiness report' or scoring feature that evaluates a repository's suitability for autonomous agents; docs cover onboarding features like AGENTS.md, environment setup, indexing, and knowledge, but nothing that generates a readiness assessment/report artifact for engineering leads.
Security remediation
engineering-leadHave security alerts automatically validated and remediated with an opened pull request
weight 2 · round to OpenHandsDevin's 'Review with Auto-Fix' auto-responds to code review comments, fixes flagged bugs and iterates on CI failures, and MCP integrations extend it to security-adjacent tools like Sentry and Datadog, implying it could react to alerts and open a PR fix. However, there is no explicit documentation of a security-scanner (SAST/dependency-alert) validation-and-remediation pipeline or evidence of an actual security-alert-to-PR workflow. Missing for 10: explicit security-scanner (e.g., CodeQL/Dependabot/Snyk) integration, evidence of validating true-positive vs false-positive alerts, and a documented end-to-end 'alert triggers PR' example.
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
openhands-docs-10 explicitly describes an automation that reviews security alerts, validates the finding, updates affected code, and opens a pull request with the remediation, directly matching the story. This is a first-party vendor claim without independent/hands-on corroboration or detail on validation accuracy. Missing for 10: independent verification of remediation quality, and detail on how 'validated' findings are confirmed before code changes.
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
Scale parallelism — running many jobs at once — concurrency, fleets, queueingScale parallelism
Running many jobs at once — concurrency, fleets, queueing
Concurrent execution
engineering-leadRun many agent tasks concurrently to scale delivery throughput
weight 3 · round to DevinDevin docs explicitly document running multiple Devins in parallel on independent tasks and provide an API with create_as_user_id for programmatically spawning sessions on behalf of org members, directly supporting concurrent scaling of throughput. However, there's no first-party data on concurrency limits, cost/resource management at scale, or independent case studies validating throughput gains at team scale; missing for 10: quantified concurrency limits/scaling guidance, independent evidence of large-scale parallel usage, and cost/governance controls for many simultaneous sessions.
- [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
OpenHands supports running agents across many backends (local, Docker, VMs, cloud) and building automations triggered by schedules or webhooks (Slack, GitHub, Linear), plus API access for scripting — all of which enable running multiple tasks in parallel across infrastructure. However, there is no explicit documentation of a dashboard or orchestration layer for managing many concurrent agent runs, concurrency limits, or throughput metrics for an engineering lead. Missing for 10: explicit multi-task concurrency/orchestration UI, stated concurrency limits or scaling guarantees, and independent evidence of teams running many parallel agents successfully.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
engineering-leadCreate agent sessions on behalf of other users in my organization
weight 2 · round to DevinDevin's API explicitly supports creating sessions on behalf of any user in the organization via the create_as_user_id parameter, directly matching the engineering-lead use case for org-wide session delegation. Missing for 10: independent/hands-on corroboration of this specific feature and details on permission scoping/audit controls for delegated sessions.
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
OpenHandsnone0/10Evidence shows enterprise features like SSO/SAML and API access, which imply some multi-user org management, but there is no evidence of an engineering-lead being able to create or launch agent sessions on behalf of other users. Missing for 10: any documentation of delegated session creation, admin-initiated sessions, or role-based session provisioning for other org members.
- [claimed-docs] “Enterprise SAML / SSO”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “OpenHands Enterprise provides commercial capabilities and support for organizations that need licensed self-hosting or managed deployment op…”
Deployment flexibility
developerUse a managed cloud offering to run agents without operating my own backend infrastructure
weight 2 · round to OpenHandsDevin is delivered as a cloud-hosted agent (web app, API, session creation on behalf of users) so developers don't need to run their own backend/agent infrastructure — sessions run on Devin's managed cloud with its own configured environment, VPN access, and parallel execution. missing for 10: independent/hands-on confirmation of the managed infrastructure model, and explicit documentation of uptime/scaling/ops guarantees for the hosted service.
- [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
- [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
- [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
- [claimed-docs] “Configure it once, and every session boots into that known-good state.”
OpenHands Cloud is explicitly documented as the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure, with hosted access, API support, and bring-your-own-LLM-key options detailed on the pricing page. Missing for 10: independent/hands-on corroboration of the cloud offering's reliability and scale, and more detail on infrastructure limits or SLAs.
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
- [claimed-docs] “Hosted cloud access from desktop and mobile”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
engineering-leadSelf-host agent infrastructure locally, in containers, or on my own VMs
weight 2 · round to OpenHandsDevinnone0/10Devin is delivered as a cloud-hosted SaaS agent platform; all evidence points to cloud sessions, API access, VPN connectivity into customer networks, and a CLI/terminal client for local interaction, but nothing indicates the core agent infrastructure itself can be self-hosted on-prem, in customer-managed containers, or on customer VMs. Missing for 10: any documentation of a self-hosted/on-prem deployment option, container images for the agent runtime, or VM-based installation of Devin's backend.
- [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
GitHub docs explicitly state agents can be run locally, in Docker, on VMs, or any agent server backend, and Enterprise offers licensed self-hosting for organizations. This directly matches the engineering-lead's need for flexible self-hosted deployment. Missing for 10: independent/hands-on verification of self-hosting setup complexity and no detailed self-host deployment guide excerpt in evidence.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “OpenHands Enterprise provides commercial capabilities and support for organizations that need licensed self-hosting or managed deployment op…”
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [github] “Run OpenHands, Claude Code, Codex, Gemini, or any ACP-compatible agent across local, remote, and cloud backends.”
Headless automation
developerRun an agent headlessly inside CI/CD pipelines and shell scripts
weight 2 · round to OpenHandsDevin's API is explicitly designed to 'integrate Devin into your applications, automate workflows, and build powerful tools' and supports creating sessions on behalf of users, which enables scripted/headless invocation, and a CLI is also documented. However, there is no explicit CI/CD pipeline example, exit-code/status handling, or GitHub Actions/shell-script integration guide in the evidence — the closest CI-related capability shown is 'Auto-Fix' reacting to CI failures rather than Devin running as a CI step itself. missing for 10: explicit CI/CD pipeline integration docs (e.g., GitHub Actions/shell script examples), documented headless/non-interactive CLI invocation with exit codes.
- [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
- [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
- [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
- [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”
Docs show a CLI mode with an 'auto-approve all actions' flag for non-interactive runs, and GitHub evidence describes running agents locally/Docker/VMs and triggering automations 'on a schedule or in response to webhook events,' plus API support for automation/scripting — all consistent with headless CI/CD use. Missing for 10: an explicit CI/CD pipeline example (e.g., GitHub Actions YAML snippet) or independent hands-on confirmation of headless exit-code/scripting behavior.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “API support for automation and scripting”
Not comparable on these axes
ai-native userConnect an agent via an official MCP server
weight 3 · not comparableDevin ships an official MCP server (documented at docs.devin.ai/work-with-devin/devin-mcp) that exposes session management, playbooks, knowledge, and scheduling to any MCP-compatible agent or IDE — this is distinct from Devin's own MCP-client capability and directly satisfies the 'connect an agent via an official MCP server' story. Missing for 10: independent/hands-on confirmation that external agents successfully connect and operate through this server, and more detail on setup/auth flow beyond first-party docs.
- [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
- [claimed-docs] “These tools let you explore and query documentation for any GitHub repository (public or private with authentication)”
- [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
- [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”