OpenHands vs Codegen
open-source · free-tier · usage-based
·free-tier · subscription-per-seat · usage-based
OpenHands wins · 24–21 (27 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round drawnA probe confirms OpenHands hosts a live llms.txt at docs.openhands.dev/llms.txt returning HTTP 200 with an explicit LLM-friendly documentation index, directly satisfying the story. Missing for 10: no independent third-party confirmation that agents actually consume/parse this file successfully in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
A probe confirms Codegen's docs site serves a working llms.txt at https://docs.codegen.com/llms.txt (HTTP 200) listing structured doc links, directly enabling an agent to be pointed at agent-oriented docs; this is corroborated by extensive first-party documentation of agent-friendly interfaces (SDK, CLI, API). Missing for 10: independent third-party confirmation of agents actually consuming the llms.txt in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.codegen.com/llms.txt # Codegen ## Docs - [Codegen](https://docs.codegen.com/introduction/overview…”
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round drawnOpenHands supports CLI mode with auto-approve for non-interactive runs, API support for automation/scripting, webhook/schedule-triggered automations (Slack, GitHub, Linear), and can run headless in Docker/VMs/servers — all consistent with CI/automation usage. Missing for 10: an explicit first-party CI pipeline example (e.g., GitHub Actions config) or independent hands-on confirmation of headless CI runs.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [claimed-docs] “API support for automation and scripting”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Codegen explicitly supports headless/automated usage via a Python SDK for programmatic agent invocation, a REST API with explicit CI/CD trigger support ('Trigger agents from CI/CD pipelines when builds fail'), and a CLI for scripted interactions and telemetry — all first-party documented. Missing for 10: independent/hands-on confirmation of CI usage in the wild and a working OpenAPI spec (probe found 404s), so it's vendor-doc-only evidence.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round to CodegenOpenHandsnone0/10OpenHands is an AI coding agent, so the axis of consuming MCP servers as a client is applicable (unlike serving MCP, which would be na for an agent), but no evidence in the pack mentions MCP integration, configuration, or tool-plugging capability at all.
Codegen's docs explicitly mention connecting to Slack, Linear, Figma, databases and extending capabilities 'with custom MCP tools', and separately reference provisioning MCP servers across an org — direct evidence of MCP client integration. However, the evidence is a single brief doc mention with no detailed setup guide, supported server list, or independent/hands-on confirmation of the feature working. Missing for 10: detailed MCP integration docs/tutorial, list of supported MCP servers, independent verification of the feature in practice.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
- [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”
ai-native userUse an official CLI
weight 2 · round drawnOpenHands documents an official CLI mode with natural language task execution, conversation resumption, and auto-approve controls, confirmed by dedicated docs pages and a probe verifying the CLI documentation page exists. Missing for 10: independent/hands-on third-party corroboration beyond vendor docs, and more detail on CLI installation/distribution mechanics.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Codegen documents an official CLI (docs.codegen.com/introduction/cli) with concrete capabilities: viewing/pulling agent work, creating new agents, running Claude Code with telemetry, and managing organizations/repositories — matching an AI-native agentic workflow tool. missing for 10: independent/hands-on corroboration beyond first-party docs, and no details on installation/versioning maturity.
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “Manage organizations - Switch between organizations and configure repositories”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
ai-native userDrive the product through a documented public API
weight 3 · round drawnOpenHands exposes a documented OpenAPI spec (openapi.json) plus an llms.txt LLM-friendly docs index, and pricing docs explicitly mention 'API support for automation and scripting,' confirming a public, documented API for programmatic/agentic control. missing for 10: independent third-party corroboration of API usage/reliability and more detailed API reference docs beyond the openapi.json probe.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
- [claimed-docs] “API support for automation and scripting”
Codegen documents a full API reference (agent creation, traces, CI/CD triggers), a Python SDK for programmatic agent control, and a CLI for managing agents/orgs, giving AI-native users multiple documented ways to drive the product programmatically. Missing for 10: a discoverable machine-readable OpenAPI/swagger spec (probe returned 404s) and independent third-party corroboration of the API's reliability.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round drawnOpenHandsnone0/10Evidence shows OpenHands supports bringing your own LLM keys, SAML/SSO, and general API access, but nothing describes issuing scoped or least-privilege API credentials specifically for an agent's actions/tool access.
Codegennone0/10Codegen's SDK/API uses a simple org_id + token authentication (codegen-docs-15) with no documentation of scoped, role-based, or least-privilege credential issuance, token permissions, or granular access controls. Missing for 10: any mention of scoped API keys, permission levels, or restricted-access tokens for agents.
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userBuild against official SDKs
weight 2 · round drawnOpenHands documents an official Software Agent SDK ('a composable Python library for building agents that work with code') plus a public OpenAPI spec and llms.txt index, giving AI-native users concrete official interfaces to build against. Missing for 10: independent/third-party corroboration of SDK usage, and richer SDK-specific docs (examples, API reference depth) beyond the single description.
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
Codegen documents an official Python SDK for programmatic agent control (agent.run, org_id/token auth), a full API reference with traces and CI/CD triggers, and a companion CLI — all first-party docs. missing for 10: independent/hands-on corroboration beyond docs, and no public OpenAPI spec was found (probe returned 404s).
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
ai-native userSubscribe to events via webhooks
weight 2 · round to OpenHandsGitHub README explicitly states OpenHands can 'run on a schedule or in response to webhook events' for automations, indicating webhook-triggered event subscription, but there is no dedicated documentation page detailing webhook setup, payload schema, or event types. missing for 10: dedicated webhook docs/config guide, independent/hands-on confirmation, and detail on which events can be subscribed to.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round to OpenHandsOpenHands ships automation agents that inspect real data (incident logs, PR diffs, workflow failures, security alerts) and generate AI insights/suggestions such as severity assessment, review comments, root-cause analysis, and remediation PRs, directly matching the story's intent for a coding-agent product. Evidence is vendor-documented only, with no independent/hands-on corroboration of the quality of these insights. Missing for 10: independent validation of suggestion quality, a unified 'insights' UI/dashboard, and evidence of insights beyond code/security/ops contexts.
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
Codegen generates AI-driven insights specifically around code artifacts—summarizing PR diffs, running security/secrets scans, checking test coverage, and suggesting changes/comments on PRs and issues—which counts as data-driven insight generation within its coding-agent context. However, these insights are narrowly scoped to code/repo/PR data rather than broader 'your data' (e.g., databases, business metrics) despite a mention of database connectivity via MCP with no insight-generation detail there. Missing for 10: evidence of insights/suggestions generated from non-code data sources (databases, analytics), and no independent/hands-on corroboration of insight quality.
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
ai-native userSet up automations that run autonomously in the background
weight 2 · round to OpenHandsOpenHands supports scheduled/webhook-triggered automations integrating with Slack, GitHub, Linear, etc., and ships prebuilt autonomous workflows (bug triage, PR review, CI failure fixing, security remediation, Slack mention handling) that run without human intervention, plus API support for scripting automations. Missing for 10: independent/hands-on verification of the scheduling/webhook trigger reliability and no detailed docs excerpt on configuring schedules beyond marketing copy.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [claimed-docs] “API support for automation and scripting”
Codegen documents agents that trigger autonomously from CI/CD failures, Slack/Linear/Jira events, and PR activity (auto-review, auto-comment), plus SDK/CLI for programmatic background automation — this matches the story of autonomous background automations. Missing for 10: independent/hands-on verification of reliability, and no explicit scheduled/cron-trigger mechanism documented.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round drawnOpenHands is itself the built-in AI assistant/agent: CLI mode lets users type natural language tasks and get instant feedback, resume conversations, and auto-approve actions, and it can run locally, remote, or in the cloud with any LLM. This directly matches delegating tasks to a built-in assistant within the product. Missing for 10: independent/hands-on user reports validating the delegation experience beyond vendor docs, and more detail on task delegation depth (e.g., multi-step autonomy limits).
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [github] “Use with any LLM”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Codegen's documentation clearly describes delegating coding tasks to a built-in AI agent that analyzes requirements, implements features, fixes bugs, writes tests, and reports progress via prompts, accessible through chat, Slack, CLI, and SDK. Missing for 10: independent/hands-on verification of the assistant's actual task-completion quality beyond vendor docs.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userOperate the product with natural-language commands
weight 2 · round drawnOpenHands' CLI mode explicitly lets users type natural language tasks and get instant feedback, resume conversations, and control approvals, directly matching the story, and this is corroborated by an official documented CLI probe. Missing for 10: independent/hands-on user reports validating the natural-language interaction quality beyond vendor docs.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Codegen is fundamentally natural-language driven: docs show prompt-based task execution ('Leave a review on PR #123'), Slack/Jira/Linear interactions, and SDK agent.run(prompt=...) all operate via NL commands across code, PRs, and issue trackers. This spans chat, CLI, and SDK surfaces, consistently prompt/NL-driven. Missing for 10: independent/hands-on corroboration of NL command interpretation quality and edge cases beyond first-party docs.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round to OpenHandsEvidence shows an OpenAPI spec is served (openhands-probe-2) and API support is advertised for automation/scripting (openhands-docs-13), implying some API reference exists, but there is no evidence of an interactive documentation UI (e.g., Swagger/Redoc 'try it' console) or runnable code examples tied to that spec. missing for 10: interactive API explorer UI, runnable/try-it code examples, confirmation the openapi.json is rendered as browsable docs.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
Codegennone0/10Codegen has an API reference page (overview, SDK docs) but no evidence of an interactive reference with runnable/executable examples; the OpenAPI/spec probe returned 404 for all candidate paths, suggesting no interactive API explorer exists.
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round to OpenHandsA probe confirms an OpenAPI spec is served at docs.openhands.dev/openapi.json with a valid 'openapi' key, directly satisfying the machine-readable API spec requirement, and API support for automation/scripting is also documented in pricing. Missing for 10: independent third-party corroboration or detailed docs describing spec coverage/versioning.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
Codegennone0/10Codegen has an API reference (docs-16/17/18) but explicit probes for machine-readable spec files (openapi.json, swagger.json, etc.) all returned 404, indicating no downloadable OpenAPI/machine-readable spec is available.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to CodegenOpenHands can run agents in Docker/VMs (openhands-gh-5), which implies isolated execution rather than direct production access, but the evidence pack never explicitly describes a sandbox environment for testing safely against non-production data. Missing for 10: explicit documentation of sandbox/isolation guarantees, workspace-vs-production data separation, and any hands-on confirmation that production systems are protected.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
Codegen docs state that code changes run in 'robust isolated environments' to safely execute code and test changes, which implies sandboxed testing separate from production. However, there's no explicit description of how production data is excluded or protected, no detail on the sandbox's scope/persistence, and no independent verification of this isolation claim. missing for 10: explicit production-data isolation guarantees, technical detail on sandbox architecture, independent/hands-on confirmation.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnOpenHandsnone0/10Evidence shows OpenHands exposes an OpenAPI spec and general 'API support for automation and scripting,' and docs reference a 'V1' with legacy V0 pages excluded, but there is no documented API versioning scheme or deprecation policy anywhere in the pack.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
Codegennone0/10There's an API reference and SDK docs, but no evidence of API versioning scheme or a documented deprecation policy; the openapi probe even returned 404s, suggesting no formal spec surfacing version info.
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round drawnOpenHands supports automations/workflows triggered by events (Slack, GitHub, Linear, webhooks) and API access for scripting, which enables some multi-item automation, but there's no explicit evidence of a bulk-operation feature (e.g., processing a batch list of items/tasks in one command or UI action). missing for 10: explicit bulk/batch operation feature, evidence of processing multiple items in a single invocation, UI/CLI support for batch task lists, independent confirmation of scale.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
The Python SDK and API allow programmatic creation/monitoring of agents and triggering from CI/CD, which could be scripted in a loop to perform bulk operations, but there is no documented batch/bulk endpoint or explicit guidance for operating across many items at once. missing for 10: dedicated bulk/batch API or CLI command, documented examples of multi-item automation, and evidence of scale/performance handling for bulk workloads.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to OpenHandsOpenHands ships documented automation triggers—scheduled runs, webhook events, PR label events, Slack mentions, failed-workflow detection, and security-alert triage—each automatically invoking an agent action, matching the 'rules trigger actions on events' story (openhands-gh-4, openhands-docs-7 to openhands-docs-11). missing for 10: independent/hands-on verification that users can define fully custom rule logic (vs. fixed preset automations), and no evidence of a general-purpose rule-authoring UI or DSL for arbitrary event/action pairing
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
Codegen supports event-driven automation via CI/CD triggers (docs-18) and PR review triggers on failure (docs-20), plus SDK/API for programmatic agent invocation, but there's no evidence of a general user-facing rules engine for defining arbitrary trigger-condition-action automations across events. missing for 10: a documented rules/automation builder UI, broader event types beyond CI/CD failures and PR diffs, and independent confirmation of custom trigger logic.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userSchedule recurring jobs or workflows
weight 2 · round to OpenHandsOpenHands explicitly supports creating automations/workflows that run on a schedule or in response to webhook events, integrating with Slack, GitHub, Linear, etc., which directly matches recurring job scheduling. Missing for 10: independent/hands-on corroboration of the scheduling UI/config and details on job management (pause/edit/monitor recurring jobs).
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
ai-native userVersion, review, and roll back my automations
weight 1 · round drawnOpenHandsnone0/10Evidence shows OpenHands can create automations/workflows (Slack, GitHub, Linear integrations, scheduled/webhook triggers) but nothing in the pack describes version history, review, or rollback mechanisms specifically for these automations themselves — no changelog, diff view, or revert feature is documented.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
Codegennone0/10Evidence covers PR/code review, commits, branches, and traces/telemetry, but there is no mention of versioning automations themselves, reviewing automation history, or rolling back agent configurations/workflows. missing for 10: automation versioning system, rollback mechanism for agent workflows, audit/history UI for automations.
Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionAutonomous implementation
End-to-end implementation by the agent — multi-file changes, task completion
End to end feature delivery
ai-native userHave an agent automatically generate and run tests to validate its own code changes before proposing them
weight 2 · round to CodegenOpenHandsnone0/10No evidence in the pack describes OpenHands agents autonomously generating or running tests to validate their own code changes before proposing them; the docs/GitHub excerpts cover CLI usage, multi-agent backends, automations, and pricing, but not test-generation/self-validation workflows.
Docs state the agent writes tests as part of implementing features and can run code/install dependencies/test changes in isolated environments, and a PR-review flow checks test coverage before flagging issues — together implying self-validation via generated tests. However, there is no explicit end-to-end walkthrough showing the agent generating tests, running them, and iterating/fixing before proposing a PR, nor independent/hands-on confirmation. missing for 10: explicit documented workflow tying test generation+execution to pre-PR validation, independent corroboration of this loop working in practice.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
developerHave an agent autonomously diagnose and fix a reported bug
weight 3 · round to OpenHandsOpenHands ships explicit automation for bug diagnosis and fixing: watching for new bugs/incidents and gathering logs (openhands-docs-7), detecting failed workflows/inspecting logs/identifying cause and opening a PR fix (openhands-docs-9), and general natural-language task execution via CLI (openhands-docs-1). This directly matches autonomous bug diagnosis-and-fix, though missing for 10: independent hands-on verification of fix success rate and end-to-end case studies beyond vendor docs.
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
Docs explicitly claim the agent can 'analyze requirements, implement features, fix bugs, write tests' and can be triggered from issue trackers/CI to autonomously act, plus run in isolated environments to test changes — directly supporting autonomous bug diagnosis and fix. However, all evidence is first-party marketing/docs with no independent or hands-on validation of end-to-end autonomous diagnosis-to-fix workflows, and no detailed walkthrough of the diagnosis step itself (e.g., root-cause analysis) beyond generic 'fix bugs' language. Missing for 10: independent/hands-on verification of successful autonomous bug fixes, detailed diagnostic reasoning process, and success-rate or case-study evidence.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
product-managerGo from a mockup or design to a working implementation without an engineering handoff
weight 2 · round to CodegenOpenHandsnone0/10The evidence shows OpenHands executes natural-language coding tasks and automations, but nothing in the pack addresses ingesting mockups/design files or a PM-oriented, engineer-free workflow from design to implementation. missing for 10: mockup/design ingestion capability, no-code PM-facing workflow evidence, any example of design-to-code handoff elimination.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Codegen's docs show relevant building blocks — a Figma connector, Slack/Linear interfaces that let non-engineers prompt the agent, and full PR/branch creation — implying a PM could describe a design and get code without direct engineering handoff. However, there is no concrete workflow, case study, or example showing a mockup/design being turned into a working implementation end-to-end. Missing for 10: an explicit design-to-code workflow example, evidence of Figma-to-PR pipeline, and PM-specific case studies or testimonials showing the handoff-free process actually works.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
developerHave an agent implement a requested feature end-to-end, including writing tests
weight 3 · round to CodegenOpenHands is positioned as an autonomous coding agent that takes natural language tasks and works across CLI, cloud, and automation workflows (e.g., PR review, bug-fixing, incident response), implying it can implement features end-to-end, but the evidence pack lacks a concrete first-party or hands-on example of the agent writing a feature plus tests. Missing for 10: explicit documentation/demo of full feature-implementation-with-tests workflow, independent benchmark or hands-on verification of test-writing capability, and confirmation of end-to-end PR creation including tests.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [github] “Run OpenHands, Claude Code, Codex, Gemini, or any ACP-compatible agent across local, remote, and cloud backends.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
Docs explicitly describe agents analyzing requirements, implementing features, writing tests, and running/testing changes in isolated environments, plus PR/branch management for end-to-end delivery. Missing for 10: independent/hands-on validation of end-to-end feature delivery and no concrete example transcript showing a full feature+test cycle.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
Environment setup
developerHave an agent automatically clone the repo, install dependencies, and configure its own working environment
weight 2 · round to CodegenEvidence shows OpenHands runs agents in configurable sandboxed backends (Docker, VMs, local/remote/cloud) and supports natural-language task execution with auto-approve, which implies some autonomous environment setup, but no citation explicitly confirms the agent auto-clones repos or installs dependencies on its own. Missing for 10: explicit documentation of repo cloning, dependency installation, and end-to-end environment bootstrap steps.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Auto-approve all actions (use with caution)”
Docs describe agents that access repos via GitHub App integration, run in isolated environments, and 'safely run code, install dependencies, and test changes' automatically without complex setup, matching the story of autonomous environment setup. Missing for 10: independent/hands-on verification of environment provisioning behavior and more detail on how dependency installation is auto-configured beyond the brief doc mention.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
Interactive takeover
developerTake over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work
weight 2 · round drawnOpenHands supports resuming conversations and switching between local/remote/cloud agents 'without losing focus,' implying some cross-surface continuity, but there's no explicit documentation of a developer taking over a live in-progress task from editor/terminal/browser mid-run with redirect capability. missing for 10: explicit IDE/editor integration for live takeover, concrete UI/UX description of mid-task redirect, and independent hands-on confirmation of seamless handoff across all three surfaces.
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [github] “Run OpenHands, Claude Code, Codex, Gemini, or any ACP-compatible agent across local, remote, and cloud backends.”
- [claimed-docs] “Resume previous conversations”
Codegen supports interacting with agents across multiple surfaces (Slack notifications/clarification requests, CLI to pull agent branches locally, Python SDK to run/monitor agents) and lets you download in-progress work to your local environment, which implies some ability to pick up and continue work. However there is no explicit documentation of pausing, redirecting, or steering a task mid-execution from editor/terminal/browser as a unified handoff flow. Missing for 10: explicit 'takeover' or redirect mid-task feature, editor/IDE integration for live task control, and evidence of seamless cross-surface handoff (Slack->CLI->browser) of the same in-progress task.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
developerSend follow-up instructions to an active agent session to steer its work without restarting
weight 2 · round to OpenHandsCLI mode docs mention typing natural language tasks with instant feedback and resuming previous conversations, which implies interactive follow-up steering, but there is no explicit evidence describing sending new instructions mid-task to an already-running/active agent session without restarting it. missing for 10: explicit documentation of mid-session steering/interrupt-and-redirect behavior while an agent is actively working, and independent/hands-on confirmation that follow-up messages are incorporated without restarting the session.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Codegennone0/10No evidence describes sending follow-up instructions to an already-running agent session to redirect its work; docs only describe initiating agents via prompts, Slack notifications, or SDK task creation, not mid-session steering. Missing for 10: any documentation of a 'continue/steer/update' API, chat-like follow-up in an active run, or session-resume mechanism.
Sandbox execution
developerHave an agent safely execute code and install dependencies inside an isolated sandbox
weight 3 · round to CodegenEvidence confirms OpenHands supports running agents in isolated environments (Docker/VMs) and explicitly references 'sandbox infrastructure' as part of its backend, implying code execution is sandboxed. However, there is no detailed documentation on dependency installation within the sandbox, security guarantees, or isolation mechanics beyond these brief mentions. missing for 10: explicit sandbox architecture docs, dependency-installation workflow details, isolation/security guarantees, independent verification of sandbox safety.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
First-party docs explicitly state the agent can 'Safely run code, install dependencies, and test changes in robust isolated environments,' directly matching the story. missing for 10: independent/hands-on corroboration of sandbox isolation, technical details on sandbox implementation (e.g., container tech, resource limits) beyond a single doc line.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsHuman oversight
Keeping a human in the loop — approvals, checkpoints, interrupts
Approval controls
developerConfigure an agent to auto-approve all its actions instead of confirming each one
weight 2 · round to OpenHandsDocs explicitly list an 'Auto-approve all actions (use with caution)' setting for CLI mode, directly matching the story. Missing for 10: independent/hands-on confirmation and details on scope (per-action vs global) or how to configure it beyond CLI mode.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Codegennone0/10The evidence describes Codegen's autonomous capabilities (running code, committing, creating PRs, Slack/Jira integration) but never mentions any per-action confirmation mode or an auto-approve toggle/setting for oversight control. Missing for 10: any documentation of an approval/confirmation workflow, a setting to enable/disable auto-approval, or human-in-the-loop gating that this story presupposes.
product-managerApprove key agent decisions from my phone while agents continue working
weight 1 · round to OpenHandsOpenHands Cloud offers 'hosted cloud access from desktop and mobile' and Slack-based interaction (@mentions, replies) that could let a PM check in remotely, but there is no documented feature for approving specific in-progress agent actions/decisions via a mobile interface while the agent continues autonomously — the only approval-related control mentioned is a blanket 'auto-approve all actions' CLI flag, not selective human-in-the-loop approval. Missing for 10: explicit mobile approval/confirmation UI, human-in-the-loop decision gating documentation, evidence of push notifications or approval prompts reaching a phone.
- [claimed-docs] “Hosted cloud access from desktop and mobile”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [claimed-docs] “Auto-approve all actions (use with caution)”
Codegen supports Slack-based notifications and agent interaction (docs-2, docs-9), which could technically be used from a phone via the Slack app, but there is no explicit mobile app, push-approval workflow, or documented 'approve on the go' feature for key agent decisions. missing for 10: dedicated mobile/approval UI, evidence of approving specific agent decisions (e.g., merge/deploy gates) from a phone, and any hands-on confirmation of this workflow.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
engineering-leadSet tiered autonomy levels controlling what an agent can do without manual confirmation
weight 3 · round to OpenHandsEvidence shows only a binary confirmation toggle (auto-approve all actions vs manual confirmation) in CLI mode, not a tiered/granular autonomy system with multiple configurable levels for an engineering lead to set. Missing for 10: documented multi-tier permission/autonomy levels, admin controls to enforce team-wide policies, and per-action or per-risk-category confirmation settings.
- [claimed-docs] “Auto-approve all actions (use with caution)”
Codegennone0/10No evidence describes configurable autonomy tiers, approval gates, or permission levels for agent actions; the docs describe what agents can do (code, PRs, Slack, Jira/Linear) but nothing about setting graduated confirmation thresholds for an engineering lead to control autonomy.
Model control
ai-native userHave each task prompt automatically routed to the most suitable underlying model
weight 2 · round drawnOpenHandsnone0/10Evidence shows OpenHands supports using any LLM and switching between agent backends manually, but there is no mention of automatic routing of prompts to the most suitable model based on task characteristics. Missing for 10: any model-routing/selection logic, per-task model suitability heuristics, or documentation of automatic model selection.
- [github] “Use with any LLM”
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
engineering-leadSwitch away from automatic model selection to a specific model of my choice
weight 1 · round to OpenHandsEvidence confirms OpenHands can be used with any LLM and lets users bring their own LLM keys, implying model choice is configurable, but there is no explicit documentation of an 'automatic model selection' default or a UI/CLI toggle to override it. Missing for 10: explicit docs on default/automatic model selection behavior, step-by-step instructions for switching models, and independent confirmation of the switch working in practice.
- [github] “Use with any LLM”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Visibility monitoring
developerWatch what a running agent is doing in real time, including its current status
weight 3 · round to CodegenOpenHands CLI mode offers instant feedback on tasks and resumable conversations, implying some real-time interaction, but there's no explicit evidence of a live status dashboard, streaming action log, or step-by-step progress view while an agent runs. missing for 10: explicit real-time status/progress UI documentation, evidence of live action streaming or step visibility, independent hands-on confirmation of watching an agent live.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Codegen's CLI lets developers view agents and run them with 'full telemetry and monitoring,' and agents can 'report progress' via Slack, giving some real-time visibility into agent activity and status. However, there's no explicit documentation of a live dashboard/status stream showing step-by-step actions as they happen, and the closest 'trace' feature is framed as post-hoc analysis/debugging rather than live watching. Missing for 10: a documented real-time UI/dashboard showing live agent status, explicit streaming of in-progress actions, and independent/hands-on confirmation of live monitoring accuracy.
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
developerGet notified when an agent completes a task or needs my input
weight 2 · round to CodegenOpenHands supports Slack integration where it 'replies when the agent finishes' and workflow automations tied to webhook events (Slack, GitHub, Linear), which implies notification-like behavior on task completion; CLI mode also gives instant feedback in interactive sessions. However, there's no explicit evidence of a dedicated notification system for 'needs my input' events or push/desktop alerts outside Slack context. missing for 10: explicit documentation of notifications when agent needs human input/approval, native mobile/desktop push alerts, and independent/hands-on confirmation of notification reliability.
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
Codegen explicitly supports sending notifications, asking for clarification, and reporting progress directly in Slack channels, plus updating statuses/comments in Jira, Linear, Clickup, and Monday.com, covering both completion and input-needed scenarios. missing for 10: independent/hands-on confirmation of real-time notification delivery, and no explicit mention of email/other channel notification options.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
Intent to spec — stories about intent to spec in this arenaIntent to spec
Stories about intent to spec in this arena
Natural language task intake
developerDescribe a feature or bug in plain language and have it automatically turned into a scoped implementation task
weight 3 · round to CodegenOpenHands lets users type natural-language tasks directly in the CLI and receive feedback, which is the core mechanism for turning plain-language input into agent-executed work, and automation triggers (Slack mentions, PR labels, failed workflows) show it converting informal signals into concrete PRs/tasks. However, there is no evidence of an explicit 'spec' or scoped task artifact (e.g., a generated plan, ticket, or requirements doc) being produced before implementation — missing for 10: explicit task-scoping/spec generation step, evidence of a structured plan artifact, and independent confirmation that vague bug/feature descriptions reliably become well-scoped tasks rather than direct code edits.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
Codegen's docs describe taking natural-language prompts and turning them into scoped agent tasks: analyzing requirements, implementing features, fixing bugs, writing tests, and running agents from prompts via SDK/CLI/Slack/issue trackers. This directly matches the intent-to-spec story of describing a feature/bug and having it become an actionable implementation task. Missing for 10: no independent/hands-on evidence showing the quality or accuracy of the generated scoped task (e.g. how well it decomposes ambiguous requests), and no detail on how it handles clarification/scoping edge cases beyond a general 'ask for clarification' mention.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
product-managerConvert user feedback submissions into structured tasks with proposed scope
weight 2 · round to CodegenOpenHandsnone0/10OpenHands integrates with Linear/GitHub/Slack for automations and can process natural-language tasks, so the general axis of turning input into work items is plausible, but there is no evidence of a feature that ingests user feedback and outputs a structured task with proposed scope for PM workflows.
Codegen can analyze requirements and create/link tasks in Jira, Linear, Clickup, Monday.com, which could theoretically be used to turn feedback into tasks, but there is no evidence of a feedback-intake feature or automatic scope proposal tailored for PM workflows. Missing for 10: dedicated feedback ingestion mechanism, structured scope/estimate output, PM-specific workflow examples or case studies.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
developerAttach a marked-up screenshot or mockup to a task so the agent implements the correct visual change
weight 2 · round drawnOpenHandsnone0/10No evidence anywhere in the pack that OpenHands supports attaching images, screenshots, or mockups to a task, or that the agent can interpret visual markup to drive implementation; documentation focuses on text-based CLI tasks, integrations, and automations.
Codegennone0/10No evidence in the pack mentions attaching screenshots, images, or marked-up mockups to a task/prompt; Codegen's docs cover text prompts, Slack/Jira/Linear integrations, CLI/SDK, and PR review but nothing about visual/image input for implementing UI changes. Figma is mentioned only as a connectable integration, not as an image-annotation workflow.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
Plan approval
developerReview and approve an agent's implementation plan before any code changes are made
weight 3 · round drawnOpenHandsnone0/10The evidence pack mentions auto-approve controls and CLI usage but contains no mention of a plan-review/approval step before code changes are made; no planning-phase or approval-gate feature is documented. missing for 10: any documentation of a plan-generation step, an approval/confirmation gate prior to code edits, or user testimony confirming such a workflow exists.
Codegennone0/10No evidence in the pack of a plan-preview or approval step prior to code changes; agents appear to run prompts directly to implementation (create branches, commit code, open PRs) without a documented plan-review gate. missing for 10: any mention of a plan/preview mode, approval gate, or dry-run step before code execution.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
engineering-leadApprove a task's scope and contract before an agent is allowed to modify the repository
weight 2 · round drawnOpenHandsnone0/10No evidence of a scope/contract approval workflow gating agent repository modifications; only auto-approve settings and general automation features are documented, not a pre-execution scope-approval gate for engineering leads.
- [claimed-docs] “Auto-approve all actions (use with caution)”
Ticket driven tasking
developerAssign a coding task to an agent directly from an existing issue or ticket
weight 3 · round to CodegenOpenHands supports automations that integrate with GitHub and Linear and can respond to webhook events, plus label-triggered PR review and bug-watching automations, implying issue/ticket-triggered agent runs are possible. However there's no explicit documentation of a workflow where a developer directly assigns a specific issue/ticket to an agent (e.g. via an 'assign to OpenHands' button or issue-comment trigger) as opposed to general automation setup. Missing for 10: explicit documentation of issue-to-agent assignment UX (e.g., GitHub issue comment/label triggering agent to pick up that specific ticket), independent/hands-on confirmation of this workflow, and ticketing system coverage beyond GitHub/Linear mentions.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
Docs show native integrations with Jira, Linear, ClickUp and Monday.com allowing the agent to update issue statuses, add comments, link PRs and create tasks, and specifically state connecting a Linear workspace 'enables agent interactions with your issues,' which directly supports assigning work from a ticket. Missing for 10: a concrete end-to-end example of assigning a task from an issue/ticket, and independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to OpenHandsOpenHands exposes a documented OpenAPI spec and dedicated 'API support for automation and scripting' tier, plus CLI mode with resume/auto-approve that mirrors UI conversation flows, suggesting broad API parity. However, no evidence explicitly confirms that every UI feature (e.g., cloud dashboard views, integrations config, enterprise SSO management) is fully API-accessible. Missing for 10: explicit parity documentation enumerating UI vs API feature coverage, and independent/hands-on confirmation that API can replicate all UI workflows.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Codegen offers a Python SDK, CLI, and API to create/run agents, pull work, and monitor executions, showing solid programmatic control (codegen-docs-11,15,16,17,18). However, there's no evidence of a complete OpenAPI/API surface matching every UI capability (e.g., Slack/Jira/Linear integration management, PR review workflows) — the OpenAPI probe returned 404s, suggesting no public full API spec. missing for 10: OpenAPI reference confirming full UI parity, evidence that integration configs and review workflows are API-accessible, independent confirmation of API completeness.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnOpenHandsnone0/10No evidence in the pack describes an explicit data export feature or open-format data portability; only conversation resume and self-hosting options are mentioned, not a way to export/leave with all user data. missing for 10: explicit export/download feature, documented open data formats, statement on data portability upon leaving the platform.
Codegennone0/10No evidence of a bulk data-export feature, open-format export of agent histories/traces, or account-portability tooling; only partial building blocks exist (pulling agent branches via CLI, retrieving traces via API) but nothing documenting a full data export/exit path.
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
ai-native userRead the product's source under an open license
weight 2 · round to OpenHandsThe GitHub repository (openhands-gh-1..5) confirms the source code is publicly hosted and readable, and openhands-docs-15 explicitly references an 'OpenHands Open Source' plan/tier, implying the core project is open-licensed. However, no evidence pack item names the specific license (e.g., MIT/Apache) or points to a LICENSE file, so full open-license confirmation is unverified. Missing for 10: explicit license name/file citation, independent confirmation of license terms.
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
ai-native userSelf-host the core product
weight 3 · round to OpenHandsEvidence shows OpenHands can be run locally/self-hosted (Docker, VMs, or any agent server backend) as opposed to only using the managed Cloud/Enterprise offerings, and it is open-source with an SDK for building on it. Missing for 10: independent hands-on confirmation of a full self-hosted setup (e.g., a third-party report of running the entire stack without cloud dependencies) and detailed self-hosting infra requirements/documentation.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
Enterprise licensing
engineering-leadLicense an enterprise deployment with SSO and commercial support for organization-wide rollout
weight 2 · round to OpenHandsOpenHands explicitly markets an Enterprise tier with licensed self-hosting/managed deployment and commercial support (openhands-docs-5), and the pricing page lists Enterprise SAML/SSO (openhands-docs-14). However, there is no detail on contract terms, SLA specifics, or independent confirmation of enterprise rollouts. Missing for 10: concrete SLA/support-tier documentation, case studies or third-party validation of enterprise SSO rollout, and clarity on org-wide admin/governance controls.
- [claimed-docs] “OpenHands Enterprise provides commercial capabilities and support for organizations that need licensed self-hosting or managed deployment op…”
- [claimed-docs] “Enterprise SAML / SSO”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
Codegennone0/10No evidence in the pack mentions SSO, enterprise licensing, commercial support tiers, or org-wide rollout plans; only feature-level docs and API/CLI references are provided. Missing for 10: SSO/SAML support, enterprise plan/pricing tier documentation, commercial support SLA, org-wide admin/rollout tooling.
Model flexibility
engineering-leadBring my own LLM or API key so agents run on the model of my choice
weight 2 · round to OpenHandsGitHub docs explicitly state OpenHands can be used with any LLM, and pricing docs confirm both Open Source and Cloud plans support bringing your own LLM keys, directly matching the story. Missing for 10: independent/hands-on verification of BYO-key setup and any model-specific limitations or edge cases.
- [github] “Use with any LLM”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Usage quotas
engineering-leadSee and manage plan-based daily task and concurrency limits for agent workflows
weight 2 · round drawnOpenHandsnone0/10Evidence shows pricing page mentions plan features like SSO, API access, and BYO LLM keys, but nothing about daily task limits, concurrency limits, or any management dashboard for such limits.
- [claimed-docs] “Hosted cloud access from desktop and mobile”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “Enterprise SAML / SSO”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Codegennone0/10No evidence pack item mentions plan-based daily task limits, concurrency limits, or any admin dashboard/controls for engineering leads to monitor or manage usage quotas; documentation covers agent capabilities, integrations, CLI, and SDK but nothing on pricing tiers or limit management.
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnOpenHandsnone0/10No evidence of data residency/region selection controls anywhere in the pack; only self-hosting options (local/Docker/VM) are mentioned, which is a workaround, not a documented region-choice feature for the managed/cloud offering.
ai-native userPrevent my data from being used to train AI models
weight 3 · round drawnOpenHandsnone0/10No evidence in the pack addresses data usage for AI model training, opt-out policies, or any privacy commitments regarding training data; the pack only covers CLI usage, deployment options, and integrations.
ai-native userControl data retention and deletion
weight 2 · round drawnOpenHandsnone0/10No evidence pack items describe data retention policies, deletion controls, or user-facing settings for managing stored conversation/data lifecycle; while OpenHands is open-source and self-hostable (implying some inherent control), no explicit retention/deletion feature or documentation is cited.
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnOpenHandsnone0/10No evidence in the pack mentions telemetry, usage tracking, analytics, or opt-out settings; the pack covers CLI features, agent orchestration, and pricing tiers only.
Codegennone0/10No evidence pack item mentions a telemetry opt-out setting or privacy controls; in fact docs-12 explicitly notes the CLI runs Claude Code 'with full telemetry and monitoring,' with no mention of a way to disable it.
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
Repo integration — stories about repo integration in this arenaRepo integration
Stories about repo integration in this arena
Chat integration
developerTag an agent in a chat thread to discuss and delegate a bug or task
weight 2 · round drawnOpenHands documents a Slack integration where the agent watches for @openhands mentions, opens a conversation with the message context, and replies when done (openhands-docs-11), which matches tagging an agent in a chat thread to delegate a task. However this is limited to Slack specifically (not other chat platforms), appears tied to the Cloud/automation feature set rather than the core product, and lacks independent/hands-on corroboration. Missing for 10: support for other chat platforms (e.g., Teams, Discord), independent verification of the Slack flow working in practice, and detail on how delegated context/threading is preserved during the exchange.
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Codegen supports Slack-based chat interactions where users can message/tag the agent to report progress, ask for clarification, and delegate tasks (codegen-docs-2, codegen-docs-9), and Linear/Jira integration allows commenting and task creation to delegate bugs (codegen-docs-3, codegen-docs-10). However, evidence doesn't explicitly confirm an '@mention' tagging mechanism within an existing chat thread to delegate a specific bug — it's inferred from general Slack/Linear interaction docs rather than a documented tagging workflow. Missing for 10: explicit documentation of @-mention/tagging syntax in threads, confirmation of thread-level context retention when delegating, and independent/hands-on verification of this specific workflow.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
Knowledge context
developerAdd a context file describing my codebase conventions so agents generate more relevant plans and code
weight 3 · round drawnOpenHandsnone0/10No evidence in this pack mentions a repo-level context/convention file (e.g., microagents, custom instructions, or similar) that developers can add to guide agent behavior; the pack only covers CLI usage, deployment, integrations, and pricing.
Codegennone0/10The evidence pack covers Codegen's integrations, CLI, SDK, and PR/issue automation, but nowhere mentions a repo-level context or rules file (e.g., conventions file) that developers can add to steer agent plans/code generation. missing for 10: any mention of a context/config file for codebase conventions, documentation on custom instructions, or repo-level rules ingestion.
developerQuery generated documentation for any public or private repository
weight 1 · round drawnOpenHandsnone0/10No evidence of a documentation-generation or repo-doc-querying feature; OpenHands is a coding agent focused on tasks, automations, and integrations, not a repo documentation search/query tool.
Codegennone0/10Evidence shows Codegen can 'improve documentation' as part of coding tasks, but there is no evidence of a feature to generate and then query documentation for a repository (public or private) in a Q&A style. No docs, API, or CLI reference describe a documentation-query capability.
Project management integration
product-managerConnect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly
weight 3 · round to CodegenOpenHands documents automation workflows that integrate with Linear (and Slack/GitHub) via webhooks/schedules, showing some issue-tracker connectivity, but Jira, ClickUp, and Monday.com are never mentioned anywhere in the evidence pack — only vague 'and more' language covers them. missing for 10: explicit Jira/ClickUp/Monday.com integrations, docs on ticket management workflows beyond Linear, evidence of two-way ticket manipulation (create/update/close) rather than just webhook triggers.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Docs explicitly claim support for Jira, Linear, ClickUp, and Monday.com (update statuses, add comments, link PRs, create tasks), but detailed setup instructions are only shown for Linear (codegen-docs-10); Jira, ClickUp, and Monday.com lack dedicated connection/integration docs equivalent to Linear or Slack. Missing for 10: explicit setup guides or independent verification for Jira/ClickUp/Monday.com, and hands-on confirmation beyond first-party docs.
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
Version control integration
developerConnect a GitHub repository so an agent can access the code and open pull requests against it
weight 3 · round to CodegenGitHub is explicitly listed as an integration target for OpenHands automations, and multiple documented workflows show the agent opening pull requests (fixing failed CI, remediating security alerts, responding to PR-review label events), which implies repo access and PR creation. However, there is no first-party documentation of the actual repo-connection/auth flow (e.g., installing a GitHub App, granting repo scopes) or hands-on confirmation that this works end-to-end. Missing for 10: explicit repo-connection setup docs, evidence of PR creation permissions/scopes, and independent verification of successful PRs opened.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
Codegen provides a GitHub App install for repo access with no complex setup, and docs describe the agent creating branches, committing code, reviewing/opening PRs, and managing repositories. missing for 10: independent/hands-on confirmation beyond vendor docs, and explicit detail on multi-repo/org-level permission scoping.
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
developerGrant an agent access to my repositories with a one-click install, without complex setup
weight 2 · round to CodegenOpenHands Cloud offers hosted access and integrations (Slack, GitHub, Linear, webhooks) suggesting some repo connection flow, but there is no concrete evidence of a one-click repo install/auth flow — no screenshots, GitHub App install steps, or onboarding walkthrough. missing for 10: documented one-click GitHub/GitLab App install flow, evidence of minimal setup steps, independent confirmation of ease of onboarding.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
- [claimed-docs] “Hosted cloud access from desktop and mobile”
Docs explicitly state installing the GitHub App grants repo access with 'no complex setup required', directly matching the story, and this is corroborated by broader repo-integration capabilities (PR review, branch/commit management). Missing for 10: independent/hands-on confirmation of the one-click flow beyond first-party docs.
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
Review quality gates — quality gates on changes — review flow, required checks, merge protectionReview quality gates
Quality gates on changes — review flow, required checks, merge protection
Ci remediation
engineering-leadHave failed CI workflows automatically diagnosed and fixed with a proposed pull request
weight 3 · round to OpenHandsOpenHands documents a dedicated automation that detects failed CI workflows, inspects logs, identifies the likely cause, and opens a pull request with a proposed fix — matching the story directly. Missing for 10: independent/hands-on verification of this automation working in practice, and detail on configuration/setup beyond the marketing description.
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Docs explicitly mention triggering agents from CI/CD pipelines when builds fail and agents that fix bugs/write code and open PRs, directly supporting automated diagnosis-and-fix-PR workflows for failed CI. However, there is no dedicated end-to-end CI-failure-diagnosis feature description, no example of the proposed PR format for a failing build, and no independent/hands-on confirmation of this specific workflow. missing for 10: dedicated CI-failure-diagnosis workflow documentation, concrete example of a CI-triggered fix PR, independent verification of this specific capability.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
developerTrigger an agent from CI/CD pipelines to fix a broken build or failing test
weight 2 · round to CodegenOpenHands advertises a workflow that watches for failed CI/CD workflows, inspects logs, identifies causes, and opens a PR with a fix (openhands-docs-9), plus webhook/schedule-triggered automations (openhands-gh-4) and API support for scripting (openhands-docs-13), which together support triggering an agent from CI/CD to fix broken builds. However, there is no concrete example of GitHub Actions/CI pipeline configuration, no evidence of test-failure-specific triggers, and no independent/hands-on confirmation that this works as described. Missing for 10: explicit CI pipeline integration docs/examples, test-failure-specific triggers, third-party verification.
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
Docs explicitly state agents can be 'Triggered from CI/CD pipelines when builds fail' and support fixing bugs/tests via API/SDK with programmatic control, which directly matches the story. Missing for 10: independent/hands-on confirmation of a real CI/CD integration example (e.g., GitHub Actions snippet) and details on how failing test context is passed to the agent.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
Diff review
developerConfigure an agent to automatically open a pull request when its task completes
weight 2 · round to CodegenOpenHands documents workflow automations that open pull requests automatically for specific triggers (failed CI, security alerts) via [openhands-docs-9] and [openhands-docs-10], showing the underlying capability exists. However, there's no direct documentation of configuring a general coding-task agent to auto-open a PR upon arbitrary task completion — the evidence only covers specific automation templates (bug-fix, security remediation) rather than a generic 'open PR on task completion' setting. Missing for 10: explicit config option/flag for auto-PR-on-completion in standard task workflows, independent/hands-on confirmation of this behavior.
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Codegen's agent creates branches, commits code, and manages repositories via GitHub App integration, and can be run programmatically via SDK/CLI/API to complete tasks; PR creation is a core documented capability of the agent workflow. Missing for 10: explicit first-party documentation/example showing a config flag or setting to 'auto-open PR on task completion' and independent/hands-on confirmation of this exact behavior.
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
developerReview a diff of an agent's changes and approve it before it becomes a pull request
weight 3 · round to CodegenOpenHands has an auto-approve/manual-approve action mode (openhands-docs-3 implies a default confirmation step exists before auto-approve is enabled) and can open PRs after agent work, suggesting some human-in-the-loop gating exists, but there's no explicit documentation of a diff review UI or an approval gate specifically before PR creation. missing for 10: explicit diff-review interface, documented approve/reject step tied to PR creation, evidence of a review-before-merge workflow.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
Codegen agents create branches, commit code, and open PRs, and the agent itself can review PRs and post comments, but the evidence pack does not show a dedicated human-in-the-loop diff review/approval gate before a PR is created — the CLI lets you 'pull agent work' locally which could support manual review, but no explicit approval workflow is documented. missing for 10: explicit pre-PR diff approval UI/workflow, evidence of a human approve/reject gate, independent confirmation that PR creation is blocked pending review.
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
Pr review automation
ai-native userHave incoming issues automatically triaged with severity suggested and routed to the right owner
weight 2 · round to OpenHandsopenhands-docs-7 directly describes an automation that watches for new bugs/incidents, gathers logs and recent changes, suggests severity, and routes the issue to the right owner — matching the story closely, backed by GitHub-documented webhook/scheduled automation infrastructure (openhands-gh-4). Missing for 10: independent/hands-on verification of triage accuracy and routing correctness beyond vendor's own site copy.
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Codegennone0/10Evidence shows Codegen integrates with issue trackers (Jira, Linear, etc.) to update statuses, add comments, and create tasks, but there is no mention of automatic severity assessment or routing incoming issues to the correct owner. Missing for 10: automatic severity classification, owner/routing logic, any triage workflow evidence.
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
engineering-leadHave every pull request automatically reviewed with AI-generated inline comments
weight 3 · round to CodegenOpenHands documents a specific automation that watches for a configurable PR label, inspects full PR context, and posts an AI review comment, which directly matches automated PR review with AI-generated comments. However, it is label-triggered rather than automatic on every PR, and posts once per label event rather than full inline (line-by-line) comments, so it's a partial match to 'every pull request' with 'inline comments'. missing for 10: evidence of automatic triggering on all PRs without manual labeling, confirmation of true inline (line-level) code comments vs a single summary comment, and independent/hands-on verification of this workflow in practice.
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
Docs explicitly describe automated PR review functionality: reading the diff, summarizing changes, running security/secrets scans, checking test coverage, and posting a review comment, plus general 'Review PRs, suggest changes, comment on issues' capability tied to GitHub App integration for repo access. This matches the engineering-lead story of automatic AI-reviewed PRs with comments. Missing for 10: explicit confirmation that comments are inline (per-line) rather than a single summary comment, and independent/hands-on evidence corroborating this in production use.
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
Readiness checks
engineering-leadAutomatically fix failing agent-readiness criteria in my repository
weight 1 · round to CodegenOpenHandsnone0/10The evidence shows OpenHands can fix failing CI workflows, remediate security alerts, and review PRs, but nothing ties this to a defined 'agent-readiness' criteria/checklist concept that an engineering lead could gate on and auto-remediate. Missing for 10: any mention of agent-readiness scoring, a checklist/criteria framework, or evidence that OpenHands detects and fixes failures against such a standard.
Codegen agents can fix bugs, write tests, and run CI/CD-triggered fixes (codegen-docs-1, codegen-docs-18, codegen-docs-20), which could be applied to remediate failing agent-readiness checks, but there is no explicit documentation of an 'agent-readiness criteria' concept, scoring rubric, or automated fix workflow tied specifically to such criteria. missing for 10: explicit agent-readiness criteria/checklist, automated detection-to-fix pipeline for those criteria, evidence of hands-on success fixing readiness gates.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
engineering-leadRun a readiness report that evaluates how ready my repository is for autonomous agents
weight 2 · round drawnOpenHandsnone0/10No evidence of a repository readiness assessment/report feature for autonomous agents; OpenHands' evidence covers task execution, automations, and integrations but nothing about scoring or auditing repo readiness.
Codegennone0/10The evidence pack shows PR review, security scanning, and CI/CD triggers, but there is no mention of a dedicated 'readiness report' feature that evaluates overall repository readiness for autonomous agents. Missing for 10: any documentation of a repo-wide readiness/scorecard report, criteria for agent-readiness, or a command/API that produces such an assessment.
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
Security remediation
engineering-leadHave security alerts automatically validated and remediated with an opened pull request
weight 2 · round to OpenHandsopenhands-docs-10 explicitly describes an automation that reviews security alerts, validates the finding, updates affected code, and opens a pull request with the remediation, directly matching the story. This is a first-party vendor claim without independent/hands-on corroboration or detail on validation accuracy. Missing for 10: independent verification of remediation quality, and detail on how 'validated' findings are confirmed before code changes.
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
Codegen documents a security/secrets scan that posts a review comment on failure (codegen-docs-20) and general bug-fixing/PR-creation capabilities plus CI/CD-triggered agents (codegen-docs-1, codegen-docs-4, codegen-docs-18), suggesting it could act on a flagged issue and open a fix PR. However, there is no explicit evidence of ingesting external security alerts (e.g., Dependabot/CodeQL/Snyk), validating them as true positives, or automatically opening a remediation PR end-to-end — the documented scan flow stops at commenting, not remediating. missing for 10: explicit security-alert ingestion/validation workflow, evidence of an opened remediation PR tied to a specific alert, independent confirmation of this flow working in practice.
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
Scale parallelism — running many jobs at once — concurrency, fleets, queueingScale parallelism
Running many jobs at once — concurrency, fleets, queueing
Concurrent execution
engineering-leadRun many agent tasks concurrently to scale delivery throughput
weight 3 · round drawnOpenHands supports running agents across many backends (local, Docker, VMs, cloud) and building automations triggered by schedules or webhooks (Slack, GitHub, Linear), plus API access for scripting — all of which enable running multiple tasks in parallel across infrastructure. However, there is no explicit documentation of a dashboard or orchestration layer for managing many concurrent agent runs, concurrency limits, or throughput metrics for an engineering lead. Missing for 10: explicit multi-task concurrency/orchestration UI, stated concurrency limits or scaling guarantees, and independent evidence of teams running many parallel agents successfully.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
Codegen's SDK/API let you programmatically create and run multiple agents (agent.run, API 'full programmatic control over execution and monitoring') and CLI can view multiple running agents, implying support for parallel task execution, and CI/CD triggers suggest scaling automation into pipelines. However, there is no explicit documentation of concurrency limits, throughput guarantees, or dashboards for managing many simultaneous tasks at scale. Missing for 10: explicit concurrency/parallelism claims, scaling limits or throughput benchmarks, and independent evidence of running many agents simultaneously in practice.
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
engineering-leadCreate agent sessions on behalf of other users in my organization
weight 2 · round to CodegenOpenHandsnone0/10Evidence shows enterprise features like SSO/SAML and API access, which imply some multi-user org management, but there is no evidence of an engineering-lead being able to create or launch agent sessions on behalf of other users. Missing for 10: any documentation of delegated session creation, admin-initiated sessions, or role-based session provisioning for other org members.
- [claimed-docs] “Enterprise SAML / SSO”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “OpenHands Enterprise provides commercial capabilities and support for organizations that need licensed self-hosting or managed deployment op…”
Codegen's org-level SDK/CLI/API (org_id+token, 'manage organizations', programmatic agent creation) implies an engineering lead with org credentials could spin up agents at scale, but no docs explicitly describe creating or assigning a session to another named user in the org. Missing for 10: explicit user-impersonation or 'create on behalf of' API/UI feature, role-based permissions for delegating agent sessions, and any admin console for assigning sessions to teammates.
- [claimed-docs] “Manage organizations - Switch between organizations and configure repositories”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
Deployment flexibility
developerUse a managed cloud offering to run agents without operating my own backend infrastructure
weight 2 · round to OpenHandsOpenHands Cloud is explicitly documented as the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure, with hosted access, API support, and bring-your-own-LLM-key options detailed on the pricing page. Missing for 10: independent/hands-on corroboration of the cloud offering's reliability and scale, and more detail on infrastructure limits or SLAs.
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
- [claimed-docs] “Hosted cloud access from desktop and mobile”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Codegen offers cloud-run agents with isolated execution environments, API/SDK/CLI access, and no need to self-host infrastructure (codegen-docs-5, codegen-docs-11, codegen-docs-15, codegen-docs-16). This directly matches running agents without operating own backend. Missing for 10: independent/hands-on verification of the managed infrastructure's reliability at scale, and explicit pricing/SLA details for the managed cloud offering.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
engineering-leadSelf-host agent infrastructure locally, in containers, or on my own VMs
weight 2 · round to OpenHandsGitHub docs explicitly state agents can be run locally, in Docker, on VMs, or any agent server backend, and Enterprise offers licensed self-hosting for organizations. This directly matches the engineering-lead's need for flexible self-hosted deployment. Missing for 10: independent/hands-on verification of self-hosting setup complexity and no detailed self-host deployment guide excerpt in evidence.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “OpenHands Enterprise provides commercial capabilities and support for organizations that need licensed self-hosting or managed deployment op…”
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [github] “Run OpenHands, Claude Code, Codex, Gemini, or any ACP-compatible agent across local, remote, and cloud backends.”
Codegennone0/10Codegen is presented entirely as a cloud/SaaS agent service (GitHub App, Slack App, API, SDK, isolated cloud environments) with no evidence of a self-hostable runtime, container image, or on-prem/VM deployment option; the CLI logs local Claude Code sessions to Codegen's own cloud rather than hosting infrastructure locally.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
Headless automation
developerRun an agent headlessly inside CI/CD pipelines and shell scripts
weight 2 · round to OpenHandsDocs show a CLI mode with an 'auto-approve all actions' flag for non-interactive runs, and GitHub evidence describes running agents locally/Docker/VMs and triggering automations 'on a schedule or in response to webhook events,' plus API support for automation/scripting — all consistent with headless CI/CD use. Missing for 10: an explicit CI/CD pipeline example (e.g., GitHub Actions YAML snippet) or independent hands-on confirmation of headless exit-code/scripting behavior.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “API support for automation and scripting”
Codegen explicitly documents triggering agents from CI/CD pipelines (e.g., on build failures) and offers a Python SDK plus CLI for programmatic/scripted invocation, which supports headless usage in pipelines and scripts. However, there's no dedicated CI/CD integration guide, no example pipeline configs (e.g., GitHub Actions YAML), and no independent/hands-on evidence confirming real-world headless CI usage. missing for 10: concrete CI/CD pipeline examples or config snippets, exit-code/status handling for scripts, independent verification of headless operation in automated pipelines.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
Not comparable on these axes
ai-native userConnect an agent via an official MCP server
weight 3 · not comparableOpenHandsn/aOpenHands is itself an AI coding agent, so serving as an MCP server for other agents is a different product role (agent-as-client, not server) — this axis is out of scope per the agent-role exception. No evidence shows OpenHands running as an MCP server endpoint.
Codegenn/aCodegen is itself an AI coding agent, so serving as an MCP server for other agents to connect to is a different product role (client vs server) per the category rule. Evidence only shows Codegen acting as an MCP client (connecting to Slack/Linear/Figma/custom MCP tools) or provisioning MCP servers for its own org's Claude Code instances, not exposing itself as an official MCP server endpoint for external agents to connect to.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
- [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”