Foreloop vs OpenHands
waitlist · early-access
·open-source · free-tier · usage-based
OpenHands wins · 16–35 (14 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to OpenHandsForeloopnone0/10No evidence pack item mentions llms.txt or an agent-oriented docs endpoint that agents can be pointed at; docs reference CLI, MCP endpoints, and skills but not an llms.txt-style discoverable doc file.
A probe confirms OpenHands hosts a live llms.txt at docs.openhands.dev/llms.txt returning HTTP 200 with an explicit LLM-friendly documentation index, directly satisfying the story. Missing for 10: no independent third-party confirmation that agents actually consume/parse this file successfully in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to OpenHandsForeloop ships a global CLI with a stable, additive `--json` output on every command, and docs explicitly state it can be run 'from anything that can run one, including coding agents and cron,' which is direct evidence of headless/CI-style automation support. However, there's no CI-specific documentation (e.g. GitHub Actions example, env-based auth for non-interactive contexts) and the approval-gate design ('nothing runs until a person approves a contract') implies a human-in-the-loop checkpoint that could block fully unattended pipelines. missing for 10: CI/pipeline-specific setup docs or examples, confirmation that approval gates can be bypassed or pre-authorized for headless runs, independent/hands-on verification of CI usage.
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [probe] “official CLI documented at https://foreloop.com/install”
OpenHands supports CLI mode with auto-approve for non-interactive runs, API support for automation/scripting, webhook/schedule-triggered automations (Slack, GitHub, Linear), and can run headless in Docker/VMs/servers — all consistent with CI/automation usage. Missing for 10: an explicit first-party CI pipeline example (e.g., GitHub Actions config) or independent hands-on confirmation of headless CI runs.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [claimed-docs] “API support for automation and scripting”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round drawnForeloopnone0/10All MCP-related evidence describes Foreloop exposing its own MCP server endpoint (/api/app/public/v1/mcp) so that external MCP clients (agents) can call into it — this is the reverse of the story, which asks whether the user can plug external MCP servers into Foreloop so it can use their tools. No evidence shows Foreloop acting as an MCP client that consumes other servers' tools.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
ai-native userUse an official CLI
weight 2 · round drawnForeloop ships an official globally-installable CLI (npm i -g foreloop) that manages projects, loops, intentions, stories, and tasks, supports scripting via `--json` with a stable additive schema, and is explicitly designed to be run 'from anything that can run one, including coding agents and cron,' making it clearly AI-native/agentic-workflow friendly. missing for 10: independent/hands-on corroboration beyond vendor docs, and more detail on full command surface.
- [claimed-docs] “npm i -g foreloop # installs globally, so the CLI works in any directory”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [probe] “official CLI documented at https://foreloop.com/install”
OpenHands documents an official CLI mode with natural language task execution, conversation resumption, and auto-approve controls, confirmed by dedicated docs pages and a probe verifying the CLI documentation page exists. Missing for 10: independent/hands-on third-party corroboration beyond vendor docs, and more detail on CLI installation/distribution mechanics.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
ai-native userDrive the product through a documented public API
weight 3 · round drawnForeloop documents a public MCP endpoint (/api/app/public/v1/mcp over streamable HTTP with just a URL and API key) plus a CLI with stable, additive `--json` output usable from any scriptable context including cron and coding agents, giving AI-native users two documented programmatic surfaces. missing for 10: no independent/third-party corroboration of the API's stability or real-world agent integration beyond first-party docs, and no formal OpenAPI/REST reference is shown alongside the MCP/CLI docs.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [probe] “official CLI documented at https://foreloop.com/install”
OpenHands exposes a documented OpenAPI spec (openapi.json) plus an llms.txt LLM-friendly docs index, and pricing docs explicitly mention 'API support for automation and scripting,' confirming a public, documented API for programmatic/agentic control. missing for 10: independent third-party corroboration of API usage/reliability and more detailed API reference docs beyond the openapi.json probe.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
- [claimed-docs] “API support for automation and scripting”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to ForeloopThe docs mention a 'write-only' API key type (flpk_YOUR_WRITE_ONLY_KEY) which implies some credential scoping, and the MCP endpoint for agents requires 'an API key', but there is no explicit documentation of issuing least-privilege or role-scoped credentials specifically for agent access to repos/tasks. missing for 10: explicit agent-scoped API key creation/management flow, documented permission levels (read/write/task-only), and evidence of per-agent credential rotation or revocation.
- [claimed-docs] “await initForeloopFeedback({ apiKey: "flpk_YOUR_WRITE_ONLY_KEY", reporterName: signedInUser?.email, });”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
ai-native userBuild against official SDKs
weight 2 · round to OpenHandsForeloop exposes an official CLI, a JS feedback SDK (initForeloopFeedback), and an MCP endpoint for agent clients, giving AI-native builders official integration surfaces beyond just docs. However, this is all first-party vendor documentation with no independent corroboration or SDK versioning/language-coverage details (e.g. only JS snippet shown, no Python/other language SDKs). Missing for 10: independent/hands-on verification of SDK usage, multi-language SDK coverage, and evidence of SDK stability/versioning beyond the JS snippet.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “await initForeloopFeedback({ apiKey: "flpk_YOUR_WRITE_ONLY_KEY", reporterName: signedInUser?.email, });”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [probe] “official CLI documented at https://foreloop.com/install”
OpenHands documents an official Software Agent SDK ('a composable Python library for building agents that work with code') plus a public OpenAPI spec and llms.txt index, giving AI-native users concrete official interfaces to build against. Missing for 10: independent/third-party corroboration of SDK usage, and richer SDK-specific docs (examples, API reference depth) beyond the single description.
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
ai-native userSubscribe to events via webhooks
weight 2 · round to OpenHandsForeloopnone0/10The evidence pack documents MCP connectivity, CLI, and feedback widget integrations, but nowhere mentions webhooks or event subscriptions that an AI-native user could subscribe to; no push/callback mechanism is described.
GitHub README explicitly states OpenHands can 'run on a schedule or in response to webhook events' for automations, indicating webhook-triggered event subscription, but there is no dedicated documentation page detailing webhook setup, payload schema, or event types. missing for 10: dedicated webhook docs/config guide, independent/hands-on confirmation, and detail on which events can be subscribed to.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round to OpenHandsForeloop's agent reads submitted feedback/reports and proposes intentions and tasks from that data (foreloop-docs-12, foreloop-docs-4), which is a form of AI-generated suggestion from collected data, but this is limited to feedback-to-task triage rather than broader analytics or insight generation across product data. Missing for 10: evidence of AI summarizing trends/patterns across historical data, dashboards or reports surfacing insights beyond individual feedback items, and independent corroboration of insight quality.
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
OpenHands ships automation agents that inspect real data (incident logs, PR diffs, workflow failures, security alerts) and generate AI insights/suggestions such as severity assessment, review comments, root-cause analysis, and remediation PRs, directly matching the story's intent for a coding-agent product. Evidence is vendor-documented only, with no independent/hands-on corroboration of the quality of these insights. Missing for 10: independent validation of suggestion quality, a unified 'insights' UI/dashboard, and evidence of insights beyond code/security/ops contexts.
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
ai-native userSet up automations that run autonomously in the background
weight 2 · round to OpenHandsForeloop supports background-style automation via its CLI (usable from cron or other schedulers) and agents that autonomously claim and work tasks in a repo, but it explicitly gates all agent actions behind a mandatory human approval step ('Nothing runs until a person approves a contract'), meaning true unattended autonomy is limited. Missing for 10: concrete documentation of scheduled/recurring automation setups, evidence of fully unattended runs without approval gating, and details on monitoring/alerting for background tasks.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Approve the big calls from your phone.”
OpenHands supports scheduled/webhook-triggered automations integrating with Slack, GitHub, Linear, etc., and ships prebuilt autonomous workflows (bug triage, PR review, CI failure fixing, security remediation, Slack mention handling) that run without human intervention, plus API support for scripting automations. Missing for 10: independent/hands-on verification of the scheduling/webhook trigger reliability and no detailed docs excerpt on configuring schedules beyond marketing copy.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [claimed-docs] “API support for automation and scripting”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round to OpenHandsForeloopnone0/10Foreloop's documented model relies on external agents (Claude Code or Codex) running on the user's own machine and connecting via MCP to claim tasks — there is no built-in AI assistant shipped inside Foreloop itself that a user delegates tasks to directly.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
OpenHands is itself the built-in AI assistant/agent: CLI mode lets users type natural language tasks and get instant feedback, resume conversations, and auto-approve actions, and it can run locally, remote, or in the cloud with any LLM. This directly matches delegating tasks to a built-in assistant within the product. Missing for 10: independent/hands-on user reports validating the delegation experience beyond vendor docs, and more detail on task delegation depth (e.g., multi-step autonomy limits).
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [github] “Use with any LLM”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
ai-native userOperate the product with natural-language commands
weight 2 · round to OpenHandsForeloop lets users describe problems in plain language (e.g., "Users can't tell which invoices are overdue") to kick off an intention that an agent turns into tasks, showing some natural-language entry point, but its actual operating surface is a structured CLI (with --json flags) and an MCP interface for agents, not a general natural-language command interface for the product as a whole. Missing for 10: evidence of NL commands controlling arbitrary product actions (not just initial feedback/intention creation), and any hands-on/independent confirmation of this NL flow working.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
OpenHands' CLI mode explicitly lets users type natural language tasks and get instant feedback, resume conversations, and control approvals, directly matching the story, and this is corroborated by an official documented CLI probe. Missing for 10: independent/hands-on user reports validating the natural-language interaction quality beyond vendor docs.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round to OpenHandsForeloopnone0/10The evidence pack shows CLI docs, MCP endpoint info, and install snippets, but nowhere describes an interactive API reference (e.g., Swagger/OpenAPI explorer, try-it-now console) with runnable examples.
Evidence shows an OpenAPI spec is served (openhands-probe-2) and API support is advertised for automation/scripting (openhands-docs-13), implying some API reference exists, but there is no evidence of an interactive documentation UI (e.g., Swagger/Redoc 'try it' console) or runnable code examples tied to that spec. missing for 10: interactive API explorer UI, runnable/try-it code examples, confirmation the openapi.json is rendered as browsable docs.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round to OpenHandsForeloopnone0/10Evidence shows Foreloop exposes an MCP endpoint and CLI with --json output, but there is no mention of a downloadable OpenAPI spec or any machine-readable API specification document for its REST/API surface.
A probe confirms an OpenAPI spec is served at docs.openhands.dev/openapi.json with a valid 'openapi' key, directly satisfying the machine-readable API spec requirement, and API support for automation/scripting is also documented in pricing. Missing for 10: independent third-party corroboration or detailed docs describing spec coverage/versioning.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round to ForeloopThe docs mention a stability guarantee for CLI `--json` output ('has a stable shape. Additive changes only') and the MCP/API endpoint path includes a 'v1' segment, implying some versioning discipline, but there is no explicit documented deprecation policy, versioning scheme, or migration guidance for the API itself. missing for 10: explicit API version numbering scheme, documented deprecation timeline/process, changelog or migration guides, evidence beyond CLI JSON stability.
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
OpenHandsnone0/10Evidence shows OpenHands exposes an OpenAPI spec and general 'API support for automation and scripting,' and docs reference a 'V1' with legacy V0 pages excluded, but there is no documented API versioning scheme or deprecation policy anywhere in the pack.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.openhands.dev/llms.txt # OpenHands Docs > LLM-friendly index of OpenHands documentation (V1). Lega…”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to OpenHandsForeloopnone0/10Foreloop's docs describe per-task workflows (claim one task, agent does the work, opens one PR) and a CLI/MCP interface, but nothing in the evidence describes performing an operation across many items/tasks simultaneously (e.g., batch approve, bulk edit, multi-task claim). The axis is plausible for a CLI/automation tool but no evidence of bulk operation support exists.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
OpenHands supports automations/workflows triggered by events (Slack, GitHub, Linear, webhooks) and API access for scripting, which enables some multi-item automation, but there's no explicit evidence of a bulk-operation feature (e.g., processing a batch list of items/tasks in one command or UI action). missing for 10: explicit bulk/batch operation feature, evidence of processing multiple items in a single invocation, UI/CLI support for batch task lists, independent confirmation of scale.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to OpenHandsForeloopnone0/10Foreloop's docs describe human-approved task workflows (agents claim tasks, but 'nothing runs until a person approves a contract'), not a rules engine for automatic event-triggered actions; the closest mention is CLI usability from cron, which is not evidence of a native rules/trigger system. missing for 10: any documented rule-definition syntax, event-trigger configuration, or automation-without-approval capability.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
OpenHands ships documented automation triggers—scheduled runs, webhook events, PR label events, Slack mentions, failed-workflow detection, and security-alert triage—each automatically invoking an agent action, matching the 'rules trigger actions on events' story (openhands-gh-4, openhands-docs-7 to openhands-docs-11). missing for 10: independent/hands-on verification that users can define fully custom rule logic (vs. fixed preset automations), and no evidence of a general-purpose rule-authoring UI or DSL for arbitrary event/action pairing
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
ai-native userSchedule recurring jobs or workflows
weight 2 · round to OpenHandsThe only relevant evidence is a single line noting the CLI can be run 'from anything that can run one, including coding agents and cron,' implying users could wire up their own cron-based scheduling — but Foreloop itself documents no native recurring-job/workflow scheduler, retry policy, or schedule management UI. Missing for 10: a first-party scheduling feature, documented recurring-workflow configuration, and evidence of built-in cron/interval management within the product.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
OpenHands explicitly supports creating automations/workflows that run on a schedule or in response to webhook events, integrating with Slack, GitHub, Linear, etc., which directly matches recurring job scheduling. Missing for 10: independent/hands-on corroboration of the scheduling UI/config and details on job management (pause/edit/monitor recurring jobs).
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
ai-native userVersion, review, and roll back my automations
weight 1 · round to ForeloopForeloop's workflow requires human approval before any agent-driven change runs (foreloop-docs-7) and changes are delivered as GitHub pull requests (foreloop-docs-3, foreloop-docs-15), which inherently gives code-level versioning and review via git. However, there is no documentation of versioning, reviewing, or rolling back the automations/tasks/loops themselves (as opposed to the resulting code diffs) — no rollback command, task history, or automation-versioning feature is evidenced. Missing for 10: explicit automation/task versioning, a documented rollback mechanism for tasks or loops, and audit/history views beyond the live agent status list.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
OpenHandsnone0/10Evidence shows OpenHands can create automations/workflows (Slack, GitHub, Linear integrations, scheduled/webhook triggers) but nothing in the pack describes version history, review, or rollback mechanisms specifically for these automations themselves — no changelog, diff view, or revert feature is documented.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionAutonomous implementation
End-to-end implementation by the agent — multi-file changes, task completion
End to end feature delivery
ai-native userHave an agent automatically generate and run tests to validate its own code changes before proposing them
weight 2 · round drawnForeloopnone0/10No evidence describes Foreloop's agents generating or running tests to validate code changes before proposing them; the docs describe task claiming, PR opening, and human approval gates but never mention automated test generation/execution as a validation step.
OpenHandsnone0/10No evidence in the pack describes OpenHands agents autonomously generating or running tests to validate their own code changes before proposing them; the docs/GitHub excerpts cover CLI usage, multi-agent backends, automations, and pricing, but not test-generation/self-validation workflows.
developerHave an agent autonomously diagnose and fix a reported bug
weight 3 · round to OpenHandsForeloop's documented workflow directly supports this: a bug report captured via the feedback widget (with page, metadata, screenshot) becomes an intention/task that an agent (Claude Code or Codex) claims, works on in a repo checkout, and resolves by opening a PR, without the reporter needing to know which files change. Human approval gates task initiation but the diagnosis-and-fix work itself is agent-driven end to end. Missing for 10: independent/hands-on evidence that agents actually diagnose root causes correctly, and more detail on how the agent investigates ambiguous bugs beyond the general task pipeline description.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
OpenHands ships explicit automation for bug diagnosis and fixing: watching for new bugs/incidents and gathering logs (openhands-docs-7), detecting failed workflows/inspecting logs/identifying cause and opening a PR fix (openhands-docs-9), and general natural-language task execution via CLI (openhands-docs-1). This directly matches autonomous bug diagnosis-and-fix, though missing for 10: independent hands-on verification of fix success rate and end-to-end case studies beyond vendor docs.
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
product-managerGo from a mockup or design to a working implementation without an engineering handoff
weight 2 · round to ForeloopForeloop lets a non-engineer describe a need in plain language (e.g. "Users can't tell which invoices are overdue") or flag a page via a feedback widget with screenshot, and an agent turns that into tasks and a PR without the PM needing to specify files — this covers a 'no engineering handoff' description-to-implementation flow. However, the evidence never shows ingesting an actual mockup/design file (e.g. Figma) as the starting artifact, and a human still must approve the contract before code changes run, so it's not a fully autonomous design-to-code path. Missing for 10: direct mockup/design-file import as an input, evidence of end-to-end completion without any review gate, and a demonstrated design-to-code case study.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
OpenHandsnone0/10The evidence shows OpenHands executes natural-language coding tasks and automations, but nothing in the pack addresses ingesting mockups/design files or a PM-oriented, engineer-free workflow from design to implementation. missing for 10: mockup/design ingestion capability, no-code PM-facing workflow evidence, any example of design-to-code handoff elimination.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
developerHave an agent implement a requested feature end-to-end, including writing tests
weight 3 · round to ForeloopForeloop orchestrates tasks and lets an agent (Claude Code/Codex) claim a task, do the work, and open a PR in the user's repo, which covers end-to-end feature implementation via an external coding agent — but Foreloop itself doesn't write code or tests; it delegates that entirely to the connected agent, and there's no explicit mention of test-writing behavior or verification. missing for 10: explicit evidence that generated PRs include tests, independent/hands-on confirmation of successful end-to-end feature delivery, and detail on how test coverage is verified before approval.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
OpenHands is positioned as an autonomous coding agent that takes natural language tasks and works across CLI, cloud, and automation workflows (e.g., PR review, bug-fixing, incident response), implying it can implement features end-to-end, but the evidence pack lacks a concrete first-party or hands-on example of the agent writing a feature plus tests. Missing for 10: explicit documentation/demo of full feature-implementation-with-tests workflow, independent benchmark or hands-on verification of test-writing capability, and confirmation of end-to-end PR creation including tests.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [github] “Run OpenHands, Claude Code, Codex, Gemini, or any ACP-compatible agent across local, remote, and cloud backends.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
Environment setup
developerHave an agent automatically clone the repo, install dependencies, and configure its own working environment
weight 2 · round drawnForeloop's docs describe the developer running an agent 'in a checkout of your repository on your own machine' and using `foreloop init` to install a task-executor skill into that existing checkout — implying the human, not the agent, does the initial clone/setup, with the agent only claiming and working tasks afterward. There's no documentation of the agent autonomously cloning the repo, installing dependencies, or provisioning its own environment from scratch. Missing for 10: evidence of agent-initiated repo cloning, dependency installation steps, and autonomous environment configuration (e.g., a sandboxed or cloud checkout flow) rather than reliance on a pre-existing local checkout.
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
Evidence shows OpenHands runs agents in configurable sandboxed backends (Docker, VMs, local/remote/cloud) and supports natural-language task execution with auto-approve, which implies some autonomous environment setup, but no citation explicitly confirms the agent auto-clones repos or installs dependencies on its own. Missing for 10: explicit documentation of repo cloning, dependency installation, and end-to-end environment bootstrap steps.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Auto-approve all actions (use with caution)”
Interactive takeover
developerTake over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work
weight 2 · round to OpenHandsForeloop lets agents run locally in a checkout and claim/report tasks, and humans can approve contracts or view agent status via a dashboard row per worker, but there's no evidence of a live handoff mechanism to pause, take over, or redirect an in-progress task from editor/terminal/browser mid-execution. missing for 10: explicit takeover/interrupt UX, editor-integrated control of a running agent, terminal command to seize an active task, and any hands-on confirmation of mid-task redirection.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “Approve the big calls from your phone.”
OpenHands supports resuming conversations and switching between local/remote/cloud agents 'without losing focus,' implying some cross-surface continuity, but there's no explicit documentation of a developer taking over a live in-progress task from editor/terminal/browser mid-run with redirect capability. missing for 10: explicit IDE/editor integration for live takeover, concrete UI/UX description of mid-task redirect, and independent hands-on confirmation of seamless handoff across all three surfaces.
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [github] “Run OpenHands, Claude Code, Codex, Gemini, or any ACP-compatible agent across local, remote, and cloud backends.”
- [claimed-docs] “Resume previous conversations”
developerSend follow-up instructions to an active agent session to steer its work without restarting
weight 2 · round to OpenHandsForeloopnone0/10The evidence describes agents claiming tasks, opening PRs, and reporting status, plus approval gates and skill-writing to correct recurring mistakes — but nothing describes sending live follow-up/steering instructions to an already-running agent session mid-task without restarting it.
CLI mode docs mention typing natural language tasks with instant feedback and resuming previous conversations, which implies interactive follow-up steering, but there is no explicit evidence describing sending new instructions mid-task to an already-running/active agent session without restarting it. missing for 10: explicit documentation of mid-session steering/interrupt-and-redirect behavior while an agent is actively working, and independent/hands-on confirmation that follow-up messages are incorporated without restarting the session.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
Sandbox execution
developerHave an agent safely execute code and install dependencies inside an isolated sandbox
weight 3 · round to OpenHandsForeloopnone0/10Foreloop's docs explicitly describe agents running 'in a checkout of your repository on your own machine' (foreloop-docs-3), with no mention of sandboxing, containerization, or isolated execution environments for code/dependency installation. Missing for 10: any documentation of sandbox/container isolation, dependency install safety, or resource/network restrictions during agent execution.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
Evidence confirms OpenHands supports running agents in isolated environments (Docker/VMs) and explicitly references 'sandbox infrastructure' as part of its backend, implying code execution is sandboxed. However, there is no detailed documentation on dependency installation within the sandbox, security guarantees, or isolation mechanics beyond these brief mentions. missing for 10: explicit sandbox architecture docs, dependency-installation workflow details, isolation/security guarantees, independent verification of sandbox safety.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsHuman oversight
Keeping a human in the loop — approvals, checkpoints, interrupts
Approval controls
developerConfigure an agent to auto-approve all its actions instead of confirming each one
weight 2 · round to OpenHandsForeloopnone0/10Foreloop's docs explicitly state that human approval of a contract is a deliberate, mandatory gate before any agent action runs ('Nothing runs until a person approves a contract. Approval is the only gate'), with no mention of a setting to bypass or auto-approve this gate. This indicates the product does not offer an auto-approve configuration, so the story's capability is unsupported.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Approve the big calls from your phone.”
Docs explicitly list an 'Auto-approve all actions (use with caution)' setting for CLI mode, directly matching the story. Missing for 10: independent/hands-on confirmation and details on scope (per-action vs global) or how to configure it beyond CLI mode.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
product-managerApprove key agent decisions from my phone while agents continue working
weight 1 · round to ForeloopForeloop explicitly markets phone-based approval of 'big calls' and documents that no agent work runs until a human approves a contract, which matches the core ask. However, evidence is limited to marketing taglines with no detail on the mobile UI/notification flow, how approvals interact with agents continuing to work in parallel, or independent corroboration. Missing for 10: concrete mobile-app/notification workflow details, evidence of multi-agent concurrent operation while awaiting approval, and third-party confirmation.
- [claimed-docs] “Approve the big calls from your phone.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
OpenHands Cloud offers 'hosted cloud access from desktop and mobile' and Slack-based interaction (@mentions, replies) that could let a PM check in remotely, but there is no documented feature for approving specific in-progress agent actions/decisions via a mobile interface while the agent continues autonomously — the only approval-related control mentioned is a blanket 'auto-approve all actions' CLI flag, not selective human-in-the-loop approval. Missing for 10: explicit mobile approval/confirmation UI, human-in-the-loop decision gating documentation, evidence of push notifications or approval prompts reaching a phone.
- [claimed-docs] “Hosted cloud access from desktop and mobile”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [claimed-docs] “Auto-approve all actions (use with caution)”
engineering-leadSet tiered autonomy levels controlling what an agent can do without manual confirmation
weight 3 · round to OpenHandsForeloopnone0/10Foreloop documents a single approval gate before any agent work starts ('Nothing runs until a person approves a contract') and lets you approve from your phone, but there is no evidence of configurable tiered autonomy levels (e.g., auto-approving certain task types while requiring confirmation for others).
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Approve the big calls from your phone.”
Evidence shows only a binary confirmation toggle (auto-approve all actions vs manual confirmation) in CLI mode, not a tiered/granular autonomy system with multiple configurable levels for an engineering lead to set. Missing for 10: documented multi-tier permission/autonomy levels, admin controls to enforce team-wide policies, and per-action or per-risk-category confirmation settings.
- [claimed-docs] “Auto-approve all actions (use with caution)”
Visibility monitoring
developerWatch what a running agent is doing in real time, including its current status
weight 3 · round to ForeloopThe docs describe a Development/Agents view listing each worker's agent identity, checkout name, current task, and last message — a real-time-ish status view (foreloop-docs-8). However, there's no evidence of live streaming updates, logs, or granular in-progress activity beyond a status line, and no independent/hands-on confirmation of real-time refresh behavior. missing for 10: evidence of live/streaming updates or detailed activity logs, independent confirmation of real-time refresh, granularity beyond a single status line per agent.
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
OpenHands CLI mode offers instant feedback on tasks and resumable conversations, implying some real-time interaction, but there's no explicit evidence of a live status dashboard, streaming action log, or step-by-step progress view while an agent runs. missing for 10: explicit real-time status/progress UI documentation, evidence of live action streaming or step visibility, independent hands-on confirmation of watching an agent live.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
developerGet notified when an agent completes a task or needs my input
weight 2 · round to OpenHandsForeloop provides a dashboard showing agent status ('last thing it said') and mentions approving 'big calls from your phone', implying some notification/approval flow, but there is no explicit documentation of push/email/Slack notifications when a task completes or needs input. missing for 10: explicit notification channel (email/Slack/push) documentation, confirmation of real-time alerts rather than manual dashboard polling, evidence of notification content for 'needs input' events.
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “Approve the big calls from your phone.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
OpenHands supports Slack integration where it 'replies when the agent finishes' and workflow automations tied to webhook events (Slack, GitHub, Linear), which implies notification-like behavior on task completion; CLI mode also gives instant feedback in interactive sessions. However, there's no explicit evidence of a dedicated notification system for 'needs my input' events or push/desktop alerts outside Slack context. missing for 10: explicit documentation of notifications when agent needs human input/approval, native mobile/desktop push alerts, and independent/hands-on confirmation of notification reliability.
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
Intent to spec — stories about intent to spec in this arenaIntent to spec
Stories about intent to spec in this arena
Natural language task intake
developerDescribe a feature or bug in plain language and have it automatically turned into a scoped implementation task
weight 3 · round to ForeloopDocs describe plain-language input ('Users can't tell which invoices are overdue') turning into scoped, approvable tasks via 'intentions' that an agent fills in and proposes as tasks, gated by human approval before an agent acts on the repo. This directly matches intent-to-spec workflow with first-party documentation across multiple pages. Missing for 10: independent/hands-on verification of the resulting task's scoping quality and no example of the full generated task artifact.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
OpenHands lets users type natural-language tasks directly in the CLI and receive feedback, which is the core mechanism for turning plain-language input into agent-executed work, and automation triggers (Slack mentions, PR labels, failed workflows) show it converting informal signals into concrete PRs/tasks. However, there is no evidence of an explicit 'spec' or scoped task artifact (e.g., a generated plan, ticket, or requirements doc) being produced before implementation — missing for 10: explicit task-scoping/spec generation step, evidence of a structured plan artifact, and independent confirmation that vague bug/feature descriptions reliably become well-scoped tasks rather than direct code edits.
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
product-managerConvert user feedback submissions into structured tasks with proposed scope
weight 2 · round to ForeloopDocs show a feedback-collection widget that captures user submissions, and an agent-driven 'intention' workflow that reads the submission, fills in an intention, and proposes tasks (foreloop-docs-12), with human approval gating execution (foreloop-docs-7). This directly matches converting feedback into structured tasks with proposed scope, though evidence is entirely vendor docs with no independent/hands-on corroboration of output quality or scope structure. Missing for 10: independent/hands-on verification of the feedback-to-task pipeline, and concrete example of the 'proposed scope' structure/fields.
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
OpenHandsnone0/10OpenHands integrates with Linear/GitHub/Slack for automations and can process natural-language tasks, so the general axis of turning input into work items is plausible, but there is no evidence of a feature that ingests user feedback and outputs a structured task with proposed scope for PM workflows.
developerAttach a marked-up screenshot or mockup to a task so the agent implements the correct visual change
weight 2 · round to ForeloopThe feedback widget lets a customer point at an element and attach an optional screenshot with metadata so 'the model fixes the right thing' (foreloop-docs-21), which is adjacent to the story but is customer-bug-report flow, not a developer attaching an annotated mockup directly to a task. There's no documentation of markup/annotation tools, mockup uploads, or a developer-facing task-attachment workflow for visual specs. Missing for 10: developer-facing screenshot/mockup attachment to tasks, annotation/markup capability, and evidence the agent parses visual markup to implement precise UI changes.
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
Plan approval
developerReview and approve an agent's implementation plan before any code changes are made
weight 3 · round to ForeloopDocs explicitly describe a 'contract' approval gate before any agent touches the repository ('Nothing runs until a person approves a contract... an agent is about to change your repository'), and mention approving 'big calls' from a phone, indicating a review/approve workflow prior to code changes. Missing for 10: independent/hands-on confirmation of the approval UI showing the actual implementation plan content and evidence of what happens if a plan is rejected/revised.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Approve the big calls from your phone.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
OpenHandsnone0/10The evidence pack mentions auto-approve controls and CLI usage but contains no mention of a plan-review/approval step before code changes are made; no planning-phase or approval-gate feature is documented. missing for 10: any documentation of a plan-generation step, an approval/confirmation gate prior to code edits, or user testimony confirming such a workflow exists.
engineering-leadApprove a task's scope and contract before an agent is allowed to modify the repository
weight 2 · round to ForeloopForeloop's docs explicitly state approval of a contract is a mandatory gate before any agent touches the repository ('Nothing runs until a person approves a contract... an agent is about to change your repository'), and agents only claim 'approved tasks' from the repo. This directly matches the engineering-lead approval-gate story, reinforced by mobile approval flow. Missing for 10: independent/hands-on verification that the gate is enforced in practice (not just documented) and detail on what 'scope and contract' specifically contains.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Approve the big calls from your phone.”
OpenHandsnone0/10No evidence of a scope/contract approval workflow gating agent repository modifications; only auto-approve settings and general automation features are documented, not a pre-execution scope-approval gate for engineering leads.
- [claimed-docs] “Auto-approve all actions (use with caution)”
Ticket driven tasking
developerAssign a coding task to an agent directly from an existing issue or ticket
weight 3 · round to OpenHandsForeloopnone0/10Foreloop's workflow starts from user feedback reports or free-text intentions that get turned into tasks an agent claims (foreloop-docs-4, foreloop-docs-12), but there is no evidence of importing or linking an existing GitHub issue/Jira ticket as the task source for an agent to pick up.
OpenHands supports automations that integrate with GitHub and Linear and can respond to webhook events, plus label-triggered PR review and bug-watching automations, implying issue/ticket-triggered agent runs are possible. However there's no explicit documentation of a workflow where a developer directly assigns a specific issue/ticket to an agent (e.g. via an 'assign to OpenHands' button or issue-comment trigger) as opposed to general automation setup. Missing for 10: explicit documentation of issue-to-agent assignment UX (e.g., GitHub issue comment/label triggering agent to pick up that specific ticket), independent/hands-on confirmation of this workflow, and ticketing system coverage beyond GitHub/Linear mentions.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round drawnForeloop ships a CLI covering projects, loops, intentions, stories and tasks with a stable `--json` output on every command, plus an MCP endpoint for agent clients, suggesting broad API/CLI parity with the UI. However, there's no explicit statement or independent confirmation that literally every UI action (e.g., approvals, feedback widget config, agent monitoring dashboard views) is exposed via API/CLI. missing for 10: explicit UI-to-API parity documentation or audit, independent/hands-on verification, coverage of approval workflow and dashboard views via API.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [probe] “official CLI documented at https://foreloop.com/install”
OpenHands exposes a documented OpenAPI spec and dedicated 'API support for automation and scripting' tier, plus CLI mode with resume/auto-approve that mirrors UI conversation flows, suggesting broad API parity. However, no evidence explicitly confirms that every UI feature (e.g., cloud dashboard views, integrations config, enterprise SSO management) is fully API-accessible. Missing for 10: explicit parity documentation enumerating UI vs API feature coverage, and independent/hands-on confirmation that API can replicate all UI workflows.
- [probe] “PROBE openapi: HTTP 200 at https://docs.openhands.dev/openapi.json — contains "openapi" key”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “Type natural language tasks and receive instant feedback”
- [claimed-docs] “Resume previous conversations”
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnForeloopnone0/10No evidence of a bulk data export feature or open-format export of projects/tasks/intentions for migration away from Foreloop; the closest hints (--json CLI output, MCP API access) are for programmatic access, not a documented full-data export/leave capability.
OpenHandsnone0/10No evidence in the pack describes an explicit data export feature or open-format data portability; only conversation resume and self-hosting options are mentioned, not a way to export/leave with all user data. missing for 10: explicit export/download feature, documented open data formats, statement on data portability upon leaving the platform.
ai-native userRead the product's source under an open license
weight 2 · round to OpenHandsForeloopnone0/10No evidence of an open-source license or public source code repository; Foreloop is presented as a closed SaaS product with a CLI/widget/MCP server, but nothing indicates the source is available under an open license.
The GitHub repository (openhands-gh-1..5) confirms the source code is publicly hosted and readable, and openhands-docs-15 explicitly references an 'OpenHands Open Source' plan/tier, implying the core project is open-licensed. However, no evidence pack item names the specific license (e.g., MIT/Apache) or points to a LICENSE file, so full open-license confirmation is unverified. Missing for 10: explicit license name/file citation, independent confirmation of license terms.
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
ai-native userSelf-host the core product
weight 3 · round to OpenHandsForeloopnone0/10All evidence describes Foreloop as a hosted SaaS (with a CLI/MCP endpoint at foreloop.com/api) that agents connect to remotely; there is no mention of a self-hostable core server, on-prem deployment, or open-source release. missing for 10: any documentation of a self-hosted deployment option, container/server image, or license permitting self-hosting.
Evidence shows OpenHands can be run locally/self-hosted (Docker, VMs, or any agent server backend) as opposed to only using the managed Cloud/Enterprise offerings, and it is open-source with an SDK for building on it. Missing for 10: independent hands-on confirmation of a full self-hosted setup (e.g., a third-party report of running the entire stack without cloud dependencies) and detailed self-hosting infra requirements/documentation.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
Enterprise licensing
engineering-leadLicense an enterprise deployment with SSO and commercial support for organization-wide rollout
weight 2 · round to OpenHandsForeloopnone0/10No evidence in the pack mentions enterprise licensing, SSO, or commercial support tiers; all citations focus on CLI usage, MCP integration, and agent workflows. This is a plausible axis for a SaaS product like Foreloop (organizations often need SSO/enterprise support), so absence of evidence yields 'none' rather than 'na'.
OpenHands explicitly markets an Enterprise tier with licensed self-hosting/managed deployment and commercial support (openhands-docs-5), and the pricing page lists Enterprise SAML/SSO (openhands-docs-14). However, there is no detail on contract terms, SLA specifics, or independent confirmation of enterprise rollouts. Missing for 10: concrete SLA/support-tier documentation, case studies or third-party validation of enterprise SSO rollout, and clarity on org-wide admin/governance controls.
- [claimed-docs] “OpenHands Enterprise provides commercial capabilities and support for organizations that need licensed self-hosting or managed deployment op…”
- [claimed-docs] “Enterprise SAML / SSO”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
Model flexibility
engineering-leadBring my own LLM or API key so agents run on the model of my choice
weight 2 · round to OpenHandsForeloopnone0/10The docs describe running third-party agents (Claude Code or Codex) locally against Foreloop's task API, but there's no evidence Foreloop lets a user configure or supply their own LLM/API key for Foreloop's own model usage (e.g., intention generation, feedback triage). Missing for 10: any documentation of model/provider selection, API key configuration UI, or support for alternate LLM backends.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
GitHub docs explicitly state OpenHands can be used with any LLM, and pricing docs confirm both Open Source and Cloud plans support bringing your own LLM keys, directly matching the story. Missing for 10: independent/hands-on verification of BYO-key setup and any model-specific limitations or edge cases.
- [github] “Use with any LLM”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Usage quotas
engineering-leadSee and manage plan-based daily task and concurrency limits for agent workflows
weight 2 · round drawnForeloopnone0/10No evidence in the pack mentions plan-based daily task limits, concurrency limits, or any pricing/plan management dashboard for engineering leads; the docs focus on agent workflow mechanics (task claiming, approvals, MCP) with no reference to usage caps or plan tiers.
OpenHandsnone0/10Evidence shows pricing page mentions plan features like SSO, API access, and BYO LLM keys, but nothing about daily task limits, concurrency limits, or any management dashboard for such limits.
- [claimed-docs] “Hosted cloud access from desktop and mobile”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “Enterprise SAML / SSO”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnForeloopnone0/10No evidence anywhere in the pack mentions data residency, region selection, or storage location options; Foreloop's docs cover CLI, MCP, agents, and feedback widgets but never address where data is stored or any regional controls.
ai-native userControl data retention and deletion
weight 2 · round drawnForeloopnone0/10No evidence pack items mention data retention policies, deletion controls, data export, or privacy settings for user/feedback data collected by Foreloop; all evidence focuses on agent workflows, CLI usage, and MCP integration.
OpenHandsnone0/10No evidence pack items describe data retention policies, deletion controls, or user-facing settings for managing stored conversation/data lifecycle; while OpenHands is open-source and self-hostable (implying some inherent control), no explicit retention/deletion feature or documentation is cited.
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnForeloopnone0/10No evidence in the pack mentions telemetry, usage tracking, analytics collection, or an opt-out mechanism for Foreloop; the docs cover CLI usage, agent workflows, and feedback widgets but never address data collection or privacy controls.
Repo integration — stories about repo integration in this arenaRepo integration
Stories about repo integration in this arena
Chat integration
developerTag an agent in a chat thread to discuss and delegate a bug or task
weight 2 · round to OpenHandsForeloopnone0/10Foreloop's workflow is built around agents polling for approved tasks/contracts and reporting status via a dashboard (docs-3, docs-7, docs-8), not around tagging or chatting with agents in a thread. No evidence describes a chat-thread interface or an @-mention mechanism to delegate work conversationally.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
OpenHands documents a Slack integration where the agent watches for @openhands mentions, opens a conversation with the message context, and replies when done (openhands-docs-11), which matches tagging an agent in a chat thread to delegate a task. However this is limited to Slack specifically (not other chat platforms), appears tied to the Cloud/automation feature set rather than the core product, and lacks independent/hands-on corroboration. Missing for 10: support for other chat platforms (e.g., Teams, Discord), independent verification of the Slack flow working in practice, and detail on how delegated context/threading is preserved during the exchange.
- [claimed-docs] “Watch Slack channels for @openhands mentions, open a conversation with the message context, and reply when the agent finishes.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Knowledge context
developerAdd a context file describing my codebase conventions so agents generate more relevant plans and code
weight 3 · round to ForeloopForeloop's docs describe a 'skill' mechanism ('Write a skill when you notice yourself giving an agent the same correction twice') and an installed task-executor skill in the checkout, which loosely maps to encoding project-specific conventions for agents, but there's no explicit documentation of a dedicated context file (e.g., conventions.md) for describing codebase style/conventions. Missing for 10: explicit context-file feature, guidance on structuring codebase conventions, evidence that agents consume such a file specifically for improving plan/code relevance.
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “Write a skill when you notice yourself giving an agent the same correction twice.”
Project management integration
product-managerConnect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly
weight 3 · round to OpenHandsForeloopnone0/10Foreloop's evidence covers GitHub integration, its own task/loop tracking, MCP server, and feedback widgets, but there is no mention of connecting to Jira, Linear, ClickUp, or Monday.com at all — Foreloop appears to use its own internal issue/task system rather than integrating external issue trackers.
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
OpenHands documents automation workflows that integrate with Linear (and Slack/GitHub) via webhooks/schedules, showing some issue-tracker connectivity, but Jira, ClickUp, and Monday.com are never mentioned anywhere in the evidence pack — only vague 'and more' language covers them. missing for 10: explicit Jira/ClickUp/Monday.com integrations, docs on ticket management workflows beyond Linear, evidence of two-way ticket manipulation (create/update/close) rather than just webhook triggers.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Version control integration
developerConnect a GitHub repository so an agent can access the code and open pull requests against it
weight 3 · round to ForeloopDocs explicitly describe connecting GitHub so Foreloop can read code and agents can open PRs against it, with a full workflow (agent claims task in a checkout, does work, opens a PR from the user's GitHub account, gated by human approval). Missing for 10: independent/hands-on verification of the GitHub connection flow and no screenshots or third-party confirmation that PRs are actually opened successfully.
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
GitHub is explicitly listed as an integration target for OpenHands automations, and multiple documented workflows show the agent opening pull requests (fixing failed CI, remediating security alerts, responding to PR-review label events), which implies repo access and PR creation. However, there is no first-party documentation of the actual repo-connection/auth flow (e.g., installing a GitHub App, granting repo scopes) or hands-on confirmation that this works end-to-end. Missing for 10: explicit repo-connection setup docs, evidence of PR creation permissions/scopes, and independent verification of successful PRs opened.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
developerGrant an agent access to my repositories with a one-click install, without complex setup
weight 2 · round to ForeloopDocs confirm connecting GitHub gives Foreloop read/PR access to repos (foreloop-docs-15) and that agents run against a checkout, but achieving this requires installing a global CLI, running foreloop init, configuring MCP with an API key, and running an agent locally — multiple manual steps rather than a true one-click flow. missing for 10: explicit one-click OAuth install flow, evidence of a simple GitHub App install button, independent confirmation of setup simplicity.
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “npm i -g foreloop # installs globally, so the CLI works in any directory”
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
OpenHands Cloud offers hosted access and integrations (Slack, GitHub, Linear, webhooks) suggesting some repo connection flow, but there is no concrete evidence of a one-click repo install/auth flow — no screenshots, GitHub App install steps, or onboarding walkthrough. missing for 10: documented one-click GitHub/GitLab App install flow, evidence of minimal setup steps, independent confirmation of ease of onboarding.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
- [claimed-docs] “Hosted cloud access from desktop and mobile”
Review quality gates — quality gates on changes — review flow, required checks, merge protectionReview quality gates
Quality gates on changes — review flow, required checks, merge protection
Ci remediation
engineering-leadHave failed CI workflows automatically diagnosed and fixed with a proposed pull request
weight 3 · round to OpenHandsForeloopnone0/10Foreloop's documented workflow is task/intention-based (feedback → task → agent → PR), with no evidence of CI pipeline integration, failed-build detection, or automated diagnosis-and-fix triggered by CI failures. The axis is plausible for an agent-driven PR tool, but nothing in the evidence pack shows Foreloop watching or reacting to CI status.
OpenHands documents a dedicated automation that detects failed CI workflows, inspects logs, identifies the likely cause, and opens a pull request with a proposed fix — matching the story directly. Missing for 10: independent/hands-on verification of this automation working in practice, and detail on configuration/setup beyond the marketing description.
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
developerTrigger an agent from CI/CD pipelines to fix a broken build or failing test
weight 2 · round to OpenHandsForeloopnone0/10Foreloop's docs describe agents claiming human-approved tasks and opening PRs from a local checkout, with a CLI usable from "anything that can run one, including coding agents and cron" (foreloop-docs-16), but there is no evidence of CI/CD pipeline integration or of the agent being triggered specifically to fix broken builds or failing tests — its workflow centers on approved feature/task work, not automated CI failure remediation.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
OpenHands advertises a workflow that watches for failed CI/CD workflows, inspects logs, identifies causes, and opens a PR with a fix (openhands-docs-9), plus webhook/schedule-triggered automations (openhands-gh-4) and API support for scripting (openhands-docs-13), which together support triggering an agent from CI/CD to fix broken builds. However, there is no concrete example of GitHub Actions/CI pipeline configuration, no evidence of test-failure-specific triggers, and no independent/hands-on confirmation that this works as described. Missing for 10: explicit CI pipeline integration docs/examples, test-failure-specific triggers, third-party verification.
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “The Software Agent SDK is a composable Python library for building agents that work with code.”
Diff review
developerConfigure an agent to automatically open a pull request when its task completes
weight 2 · round to ForeloopDocs clearly describe the agent workflow: it claims a task, does the work, and opens a pull request from the user's GitHub account, with GitHub connection enabling PR creation, gated by human approval of the contract before work starts. Missing for 10: independent/hands-on confirmation that PRs are reliably opened on task completion, and no detail on configuring PR-specific settings (e.g., target branch, draft PRs).
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
OpenHands documents workflow automations that open pull requests automatically for specific triggers (failed CI, security alerts) via [openhands-docs-9] and [openhands-docs-10], showing the underlying capability exists. However, there's no direct documentation of configuring a general coding-task agent to auto-open a PR upon arbitrary task completion — the evidence only covers specific automation templates (bug-fix, security remediation) rather than a generic 'open PR on task completion' setting. Missing for 10: explicit config option/flag for auto-PR-on-completion in standard task workflows, independent/hands-on confirmation of this behavior.
- [claimed-docs] “Detect failed workflows, inspect logs, identify the likely cause, and open a pull request with a proposed fix.”
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
developerReview a diff of an agent's changes and approve it before it becomes a pull request
weight 3 · round to OpenHandsForeloopnone0/10Foreloop's 'approval' gate (foreloop-docs-7) is a pre-work contract approval, not a diff review after the agent finishes changes — the agent then 'opens a pull request from your GitHub account' directly (foreloop-docs-3), with no documented step where a person reviews the diff before the PR is created. No evidence describes a diff-preview/approve-before-PR workflow.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
OpenHands has an auto-approve/manual-approve action mode (openhands-docs-3 implies a default confirmation step exists before auto-approve is enabled) and can open PRs after agent work, suggesting some human-in-the-loop gating exists, but there's no explicit documentation of a diff review UI or an approval gate specifically before PR creation. missing for 10: explicit diff-review interface, documented approve/reject step tied to PR creation, evidence of a review-before-merge workflow.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
Pr review automation
ai-native userHave incoming issues automatically triaged with severity suggested and routed to the right owner
weight 2 · round to OpenHandsForeloopnone0/10Foreloop's docs describe turning user feedback into intentions/tasks that agents pick up and claim (foreloop-docs-12, foreloop-docs-4), but there is no evidence of automatic severity classification or routing to a specific human owner — tasks are self-claimed by agents rather than triaged and assigned. missing for 10: severity scoring, owner/routing logic, any triage workflow evidence.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
openhands-docs-7 directly describes an automation that watches for new bugs/incidents, gathers logs and recent changes, suggests severity, and routes the issue to the right owner — matching the story closely, backed by GitHub-documented webhook/scheduled automation infrastructure (openhands-gh-4). Missing for 10: independent/hands-on verification of triage accuracy and routing correctness beyond vendor's own site copy.
- [claimed-docs] “Watch for new bugs or incidents, gather logs and recent changes, suggest severity, and route the issue to the right owner.”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
Readiness checks
engineering-leadAutomatically fix failing agent-readiness criteria in my repository
weight 1 · round drawnForeloopnone0/10Foreloop's docs describe a task/feedback pipeline where agents claim approved tasks and open PRs, but nothing in the evidence pack mentions detecting or automatically remediating 'agent-readiness criteria' (e.g., AGENTS.md, tool-permission manifests, readiness scoring) in a repository. Missing for 10: any readiness-criteria scanning/grading feature, evidence of automatic detection of failing criteria, and evidence of automated fixes tied to such criteria.
OpenHandsnone0/10The evidence shows OpenHands can fix failing CI workflows, remediate security alerts, and review PRs, but nothing ties this to a defined 'agent-readiness' criteria/checklist concept that an engineering lead could gate on and auto-remediate. Missing for 10: any mention of agent-readiness scoring, a checklist/criteria framework, or evidence that OpenHands detects and fixes failures against such a standard.
engineering-leadRun a readiness report that evaluates how ready my repository is for autonomous agents
weight 2 · round drawnForeloopnone0/10No evidence in the pack describes a 'readiness report' or any repository-readiness scoring/assessment feature for autonomous agents; Foreloop's docs cover task claiming, MCP connectivity, feedback widgets, and approval gates, but nothing about evaluating repo readiness.
Security remediation
engineering-leadHave security alerts automatically validated and remediated with an opened pull request
weight 2 · round to OpenHandsForeloopnone0/10Foreloop's evidence describes a general feedback/task-to-PR agent workflow (bug reports, feature requests) with human approval gates, but nothing in the pack mentions ingesting security alerts (e.g., from SAST/SCA/dependency scanners), validating them, or triggering remediation PRs specifically for vulnerabilities. The axis is plausible for a task-automation platform, but no evidence shows this specific capability.
openhands-docs-10 explicitly describes an automation that reviews security alerts, validates the finding, updates affected code, and opens a pull request with the remediation, directly matching the story. This is a first-party vendor claim without independent/hands-on corroboration or detail on validation accuracy. Missing for 10: independent verification of remediation quality, and detail on how 'validated' findings are confirmed before code changes.
- [claimed-docs] “Review security alerts, validate the finding, update affected code, and open a pull request with the remediation.”
Scale parallelism — running many jobs at once — concurrency, fleets, queueingScale parallelism
Running many jobs at once — concurrency, fleets, queueing
Concurrent execution
engineering-leadRun many agent tasks concurrently to scale delivery throughput
weight 3 · round to OpenHandsThe docs show that multiple agent workers can each claim tasks and run in parallel checkouts, with a dashboard listing one row per worker (foreloop-docs-8), implying some capacity for concurrent agent execution across a team. However, there is no explicit product feature for orchestrating, scaling, or managing many concurrent tasks at once, no throughput/scale claims, and no evidence of built-in concurrency limits or fleet management tooling. missing for 10: explicit support/documentation for running many agents simultaneously at scale, concurrency controls, throughput metrics, or a scheduler/queue for parallel task dispatch.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
OpenHands supports running agents across many backends (local, Docker, VMs, cloud) and building automations triggered by schedules or webhooks (Slack, GitHub, Linear), plus API access for scripting — all of which enable running multiple tasks in parallel across infrastructure. However, there is no explicit documentation of a dashboard or orchestration layer for managing many concurrent agent runs, concurrency limits, or throughput metrics for an engineering lead. Missing for 10: explicit multi-task concurrency/orchestration UI, stated concurrency limits or scaling guarantees, and independent evidence of teams running many parallel agents successfully.
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
engineering-leadCreate agent sessions on behalf of other users in my organization
weight 2 · round drawnForeloopnone0/10The evidence describes agents run locally by an individual user in their own repo checkout, claiming tasks and opening PRs from their own GitHub account, but nothing describes an engineering-lead provisioning or launching agent sessions on behalf of other org members. Missing for 10: any admin/org-level session creation API or UI, delegated-session documentation, or multi-user session management evidence.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
OpenHandsnone0/10Evidence shows enterprise features like SSO/SAML and API access, which imply some multi-user org management, but there is no evidence of an engineering-lead being able to create or launch agent sessions on behalf of other users. Missing for 10: any documentation of delegated session creation, admin-initiated sessions, or role-based session provisioning for other org members.
- [claimed-docs] “Enterprise SAML / SSO”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “OpenHands Enterprise provides commercial capabilities and support for organizations that need licensed self-hosting or managed deployment op…”
Deployment flexibility
developerUse a managed cloud offering to run agents without operating my own backend infrastructure
weight 2 · round to OpenHandsForeloopnone0/10Foreloop's own docs state the opposite of this story: agents (Claude Code or Codex) run 'in a checkout of your repository on your own machine,' with Foreloop only coordinating tasks and approvals — there is no evidence of a managed cloud runtime that executes agents on Foreloop's infrastructure.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
OpenHands Cloud is explicitly documented as the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure, with hosted access, API support, and bring-your-own-LLM-key options detailed on the pricing page. Missing for 10: independent/hands-on corroboration of the cloud offering's reliability and scale, and more detail on infrastructure limits or SLAs.
- [claimed-docs] “OpenHands Cloud is the managed commercial service for running OpenHands without operating your own backend and sandbox infrastructure.”
- [claimed-docs] “Hosted cloud access from desktop and mobile”
- [claimed-docs] “API support for automation and scripting”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
Headless automation
developerRun an agent headlessly inside CI/CD pipelines and shell scripts
weight 2 · round to OpenHandsDocs mention the CLI is invocable from 'anything that can run one, including coding agents and cron' and support scriptable --json output, suggesting headless invocation is possible (foreloop-docs-16, foreloop-docs-9). However, other docs state that 'nothing runs until a person approves a contract' (foreloop-docs-7), implying a human-in-the-loop gate that complicates fully unattended CI/CD execution, and there is no explicit CI/CD pipeline example or corroboration. Missing for 10: explicit CI/CD/pipeline example, evidence of non-interactive approval flow, independent confirmation of headless operation.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
Docs show a CLI mode with an 'auto-approve all actions' flag for non-interactive runs, and GitHub evidence describes running agents locally/Docker/VMs and triggering automations 'on a schedule or in response to webhook events,' plus API support for automation/scripting — all consistent with headless CI/CD use. Missing for 10: an explicit CI/CD pipeline example (e.g., GitHub Actions YAML snippet) or independent hands-on confirmation of headless exit-code/scripting behavior.
- [claimed-docs] “Auto-approve all actions (use with caution)”
- [probe] “official CLI documented at https://docs.openhands.dev/usage/how-to/cli-mode”
- [github] “Create automations and workflows that integrate with Slack, GitHub, Linear, and more. Run on a schedule or in response to webhook events”
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “API support for automation and scripting”
Not comparable on these axes
ai-native userConnect an agent via an official MCP server
weight 3 · not comparableForeloop documents a first-party MCP server endpoint (/api/app/public/v1/mcp over streamable HTTP) that any MCP client can connect to with just a URL and API key, explicitly enabling agents to claim tasks and work in repos. This is first-party documentation only, with no independent/hands-on corroboration of the MCP connection working in practice. Missing for 10: independent verification that an external MCP client successfully connects and operates against this endpoint.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
ai-native userTest against a sandbox environment without touching production data
weight 1 · not comparableForeloopn/aForeloop is a project/task-management and agent-orchestration tool for coding work (tasks, PRs, feedback widgets); it has no concept of a sandbox testing environment vs production data. This is a category error for its product type — the axis of 'sandbox vs production data testing' doesn't apply.
OpenHands can run agents in Docker/VMs (openhands-gh-5), which implies isolated execution rather than direct production access, but the evidence pack never explicitly describes a sandbox environment for testing safely against non-production data. Missing for 10: explicit documentation of sandbox/isolation guarantees, workspace-vs-production data separation, and any hands-on confirmation that production systems are protected.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
ai-native userHave each task prompt automatically routed to the most suitable underlying model
weight 2 · not comparableForeloopn/aForeloop is a task-management/agent-orchestration layer that hands work to whatever coding agent (Claude Code, Codex) the developer runs locally — it does not itself route prompts across multiple underlying LLMs. Automatic model routing per task is a wrong axis for this product's category (it's a task/PM layer, not a multi-model gateway).
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
OpenHandsnone0/10Evidence shows OpenHands supports using any LLM and switching between agent backends manually, but there is no mention of automatic routing of prompts to the most suitable model based on task characteristics. Missing for 10: any model-routing/selection logic, per-task model suitability heuristics, or documentation of automatic model selection.
- [github] “Use with any LLM”
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
engineering-leadSwitch away from automatic model selection to a specific model of my choice
weight 1 · not comparableForeloopn/aForeloop orchestrates tasks/agents (Claude Code, Codex) run by the user rather than offering its own model-selection setting; there is no evidence of an internal model picker, so this axis is a category error for this product's role as an agent-orchestration/task platform.
Evidence confirms OpenHands can be used with any LLM and lets users bring their own LLM keys, implying model choice is configurable, but there is no explicit documentation of an 'automatic model selection' default or a UI/CLI toggle to override it. Missing for 10: explicit docs on default/automatic model selection behavior, step-by-step instructions for switching models, and independent confirmation of the switch working in practice.
- [github] “Use with any LLM”
- [claimed-docs] “OpenHands Open Source and OpenHands Cloud plans offer support to bring your own LLM keys.”
ai-native userPrevent my data from being used to train AI models
weight 3 · not comparableForeloopn/aForeloop is a task/agent orchestration and feedback-collection tool for software repos, not a data-processing or AI-model-training service; the evidence pack contains no data-usage or model-training policy at all, and this axis is a category error for this product type.
developerQuery generated documentation for any public or private repository
weight 1 · not comparableForeloopn/aForeloop is a task/agent orchestration and feedback-collection tool that connects to GitHub repos to let agents claim tasks and open PRs; it does not generate or expose queryable documentation for repositories. This story targets a documentation-generation/query product category, which is a different axis from Foreloop's core offering.
engineering-leadHave every pull request automatically reviewed with AI-generated inline comments
weight 3 · not comparableForeloopn/aForeloop is a task-management/agent-orchestration tool that lets agents claim tasks and open PRs from feedback, not a PR-review/code-review product; there's no evidence it reviews pull requests or adds inline review comments — this is a different product category (task orchestration vs. review-quality-gates).
OpenHands documents a specific automation that watches for a configurable PR label, inspects full PR context, and posts an AI review comment, which directly matches automated PR review with AI-generated comments. However, it is label-triggered rather than automatic on every PR, and posts once per label event rather than full inline (line-by-line) comments, so it's a partial match to 'every pull request' with 'inline comments'. missing for 10: evidence of automatic triggering on all PRs without manual labeling, confirmation of true inline (line-level) code comments vs a single summary comment, and independent/hands-on verification of this workflow in practice.
- [claimed-docs] “Watch for a configurable label on pull requests, inspect full PR context, and post an AI review comment once per label event.”
engineering-leadSelf-host agent infrastructure locally, in containers, or on my own VMs
weight 2 · not comparableForeloopn/aForeloop is a hosted SaaS orchestration layer (agents run locally in your repo checkout and talk to Foreloop's cloud API/MCP endpoint) — there is no evidence of a self-hostable server/backend component; the product's core value is the hosted service coordinating tasks, so 'self-hosting agent infrastructure' is a category mismatch rather than a missing feature.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
GitHub docs explicitly state agents can be run locally, in Docker, on VMs, or any agent server backend, and Enterprise offers licensed self-hosting for organizations. This directly matches the engineering-lead's need for flexible self-hosted deployment. Missing for 10: independent/hands-on verification of self-hosting setup complexity and no detailed self-host deployment guide excerpt in evidence.
- [github] “Run agents locally, in Docker, on VMs, or anywhere you can run an agent server backend”
- [claimed-docs] “OpenHands Enterprise provides commercial capabilities and support for organizations that need licensed self-hosting or managed deployment op…”
- [github] “Switch between local, remote, and cloud agents without losing focus”
- [github] “Run OpenHands, Claude Code, Codex, Gemini, or any ACP-compatible agent across local, remote, and cloud backends.”