Foreloop vs Codegen
Codegen wins · 15–29 (21 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to CodegenForeloopnone0/10No evidence pack item mentions llms.txt or an agent-oriented docs endpoint that agents can be pointed at; docs reference CLI, MCP endpoints, and skills but not an llms.txt-style discoverable doc file.
A probe confirms Codegen's docs site serves a working llms.txt at https://docs.codegen.com/llms.txt (HTTP 200) listing structured doc links, directly enabling an agent to be pointed at agent-oriented docs; this is corroborated by extensive first-party documentation of agent-friendly interfaces (SDK, CLI, API). Missing for 10: independent third-party confirmation of agents actually consuming the llms.txt in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.codegen.com/llms.txt # Codegen ## Docs - [Codegen](https://docs.codegen.com/introduction/overview…”
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to CodegenForeloop ships a global CLI with a stable, additive `--json` output on every command, and docs explicitly state it can be run 'from anything that can run one, including coding agents and cron,' which is direct evidence of headless/CI-style automation support. However, there's no CI-specific documentation (e.g. GitHub Actions example, env-based auth for non-interactive contexts) and the approval-gate design ('nothing runs until a person approves a contract') implies a human-in-the-loop checkpoint that could block fully unattended pipelines. missing for 10: CI/pipeline-specific setup docs or examples, confirmation that approval gates can be bypassed or pre-authorized for headless runs, independent/hands-on verification of CI usage.
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [probe] “official CLI documented at https://foreloop.com/install”
Codegen explicitly supports headless/automated usage via a Python SDK for programmatic agent invocation, a REST API with explicit CI/CD trigger support ('Trigger agents from CI/CD pipelines when builds fail'), and a CLI for scripted interactions and telemetry — all first-party documented. Missing for 10: independent/hands-on confirmation of CI usage in the wild and a working OpenAPI spec (probe found 404s), so it's vendor-doc-only evidence.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round to CodegenForeloopnone0/10All MCP-related evidence describes Foreloop exposing its own MCP server endpoint (/api/app/public/v1/mcp) so that external MCP clients (agents) can call into it — this is the reverse of the story, which asks whether the user can plug external MCP servers into Foreloop so it can use their tools. No evidence shows Foreloop acting as an MCP client that consumes other servers' tools.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
Codegen's docs explicitly mention connecting to Slack, Linear, Figma, databases and extending capabilities 'with custom MCP tools', and separately reference provisioning MCP servers across an org — direct evidence of MCP client integration. However, the evidence is a single brief doc mention with no detailed setup guide, supported server list, or independent/hands-on confirmation of the feature working. Missing for 10: detailed MCP integration docs/tutorial, list of supported MCP servers, independent verification of the feature in practice.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
- [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”
ai-native userUse an official CLI
weight 2 · round drawnForeloop ships an official globally-installable CLI (npm i -g foreloop) that manages projects, loops, intentions, stories, and tasks, supports scripting via `--json` with a stable additive schema, and is explicitly designed to be run 'from anything that can run one, including coding agents and cron,' making it clearly AI-native/agentic-workflow friendly. missing for 10: independent/hands-on corroboration beyond vendor docs, and more detail on full command surface.
- [claimed-docs] “npm i -g foreloop # installs globally, so the CLI works in any directory”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [probe] “official CLI documented at https://foreloop.com/install”
Codegen documents an official CLI (docs.codegen.com/introduction/cli) with concrete capabilities: viewing/pulling agent work, creating new agents, running Claude Code with telemetry, and managing organizations/repositories — matching an AI-native agentic workflow tool. missing for 10: independent/hands-on corroboration beyond first-party docs, and no details on installation/versioning maturity.
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “Manage organizations - Switch between organizations and configure repositories”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
ai-native userDrive the product through a documented public API
weight 3 · round drawnForeloop documents a public MCP endpoint (/api/app/public/v1/mcp over streamable HTTP with just a URL and API key) plus a CLI with stable, additive `--json` output usable from any scriptable context including cron and coding agents, giving AI-native users two documented programmatic surfaces. missing for 10: no independent/third-party corroboration of the API's stability or real-world agent integration beyond first-party docs, and no formal OpenAPI/REST reference is shown alongside the MCP/CLI docs.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [probe] “official CLI documented at https://foreloop.com/install”
Codegen documents a full API reference (agent creation, traces, CI/CD triggers), a Python SDK for programmatic agent control, and a CLI for managing agents/orgs, giving AI-native users multiple documented ways to drive the product programmatically. Missing for 10: a discoverable machine-readable OpenAPI/swagger spec (probe returned 404s) and independent third-party corroboration of the API's reliability.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to ForeloopThe docs mention a 'write-only' API key type (flpk_YOUR_WRITE_ONLY_KEY) which implies some credential scoping, and the MCP endpoint for agents requires 'an API key', but there is no explicit documentation of issuing least-privilege or role-scoped credentials specifically for agent access to repos/tasks. missing for 10: explicit agent-scoped API key creation/management flow, documented permission levels (read/write/task-only), and evidence of per-agent credential rotation or revocation.
- [claimed-docs] “await initForeloopFeedback({ apiKey: "flpk_YOUR_WRITE_ONLY_KEY", reporterName: signedInUser?.email, });”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
Codegennone0/10Codegen's SDK/API uses a simple org_id + token authentication (codegen-docs-15) with no documentation of scoped, role-based, or least-privilege credential issuance, token permissions, or granular access controls. Missing for 10: any mention of scoped API keys, permission levels, or restricted-access tokens for agents.
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userBuild against official SDKs
weight 2 · round to CodegenForeloop exposes an official CLI, a JS feedback SDK (initForeloopFeedback), and an MCP endpoint for agent clients, giving AI-native builders official integration surfaces beyond just docs. However, this is all first-party vendor documentation with no independent corroboration or SDK versioning/language-coverage details (e.g. only JS snippet shown, no Python/other language SDKs). Missing for 10: independent/hands-on verification of SDK usage, multi-language SDK coverage, and evidence of SDK stability/versioning beyond the JS snippet.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “await initForeloopFeedback({ apiKey: "flpk_YOUR_WRITE_ONLY_KEY", reporterName: signedInUser?.email, });”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [probe] “official CLI documented at https://foreloop.com/install”
Codegen documents an official Python SDK for programmatic agent control (agent.run, org_id/token auth), a full API reference with traces and CI/CD triggers, and a companion CLI — all first-party docs. missing for 10: independent/hands-on corroboration beyond docs, and no public OpenAPI spec was found (probe returned 404s).
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
ai-native userSubscribe to events via webhooks
weight 2 · round drawnForeloopnone0/10The evidence pack documents MCP connectivity, CLI, and feedback widget integrations, but nowhere mentions webhooks or event subscriptions that an AI-native user could subscribe to; no push/callback mechanism is described.
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round drawnForeloop's agent reads submitted feedback/reports and proposes intentions and tasks from that data (foreloop-docs-12, foreloop-docs-4), which is a form of AI-generated suggestion from collected data, but this is limited to feedback-to-task triage rather than broader analytics or insight generation across product data. Missing for 10: evidence of AI summarizing trends/patterns across historical data, dashboards or reports surfacing insights beyond individual feedback items, and independent corroboration of insight quality.
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
Codegen generates AI-driven insights specifically around code artifacts—summarizing PR diffs, running security/secrets scans, checking test coverage, and suggesting changes/comments on PRs and issues—which counts as data-driven insight generation within its coding-agent context. However, these insights are narrowly scoped to code/repo/PR data rather than broader 'your data' (e.g., databases, business metrics) despite a mention of database connectivity via MCP with no insight-generation detail there. Missing for 10: evidence of insights/suggestions generated from non-code data sources (databases, analytics), and no independent/hands-on corroboration of insight quality.
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
ai-native userSet up automations that run autonomously in the background
weight 2 · round to CodegenForeloop supports background-style automation via its CLI (usable from cron or other schedulers) and agents that autonomously claim and work tasks in a repo, but it explicitly gates all agent actions behind a mandatory human approval step ('Nothing runs until a person approves a contract'), meaning true unattended autonomy is limited. Missing for 10: concrete documentation of scheduled/recurring automation setups, evidence of fully unattended runs without approval gating, and details on monitoring/alerting for background tasks.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Approve the big calls from your phone.”
Codegen documents agents that trigger autonomously from CI/CD failures, Slack/Linear/Jira events, and PR activity (auto-review, auto-comment), plus SDK/CLI for programmatic background automation — this matches the story of autonomous background automations. Missing for 10: independent/hands-on verification of reliability, and no explicit scheduled/cron-trigger mechanism documented.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round to CodegenForeloopnone0/10Foreloop's documented model relies on external agents (Claude Code or Codex) running on the user's own machine and connecting via MCP to claim tasks — there is no built-in AI assistant shipped inside Foreloop itself that a user delegates tasks to directly.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
Codegen's documentation clearly describes delegating coding tasks to a built-in AI agent that analyzes requirements, implements features, fixes bugs, writes tests, and reports progress via prompts, accessible through chat, Slack, CLI, and SDK. Missing for 10: independent/hands-on verification of the assistant's actual task-completion quality beyond vendor docs.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userOperate the product with natural-language commands
weight 2 · round to CodegenForeloop lets users describe problems in plain language (e.g., "Users can't tell which invoices are overdue") to kick off an intention that an agent turns into tasks, showing some natural-language entry point, but its actual operating surface is a structured CLI (with --json flags) and an MCP interface for agents, not a general natural-language command interface for the product as a whole. Missing for 10: evidence of NL commands controlling arbitrary product actions (not just initial feedback/intention creation), and any hands-on/independent confirmation of this NL flow working.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
Codegen is fundamentally natural-language driven: docs show prompt-based task execution ('Leave a review on PR #123'), Slack/Jira/Linear interactions, and SDK agent.run(prompt=...) all operate via NL commands across code, PRs, and issue trackers. This spans chat, CLI, and SDK surfaces, consistently prompt/NL-driven. Missing for 10: independent/hands-on corroboration of NL command interpretation quality and edge cases beyond first-party docs.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnForeloopnone0/10The evidence pack shows CLI docs, MCP endpoint info, and install snippets, but nowhere describes an interactive API reference (e.g., Swagger/OpenAPI explorer, try-it-now console) with runnable examples.
Codegennone0/10Codegen has an API reference page (overview, SDK docs) but no evidence of an interactive reference with runnable/executable examples; the OpenAPI/spec probe returned 404 for all candidate paths, suggesting no interactive API explorer exists.
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round drawnForeloopnone0/10Evidence shows Foreloop exposes an MCP endpoint and CLI with --json output, but there is no mention of a downloadable OpenAPI spec or any machine-readable API specification document for its REST/API surface.
Codegennone0/10Codegen has an API reference (docs-16/17/18) but explicit probes for machine-readable spec files (openapi.json, swagger.json, etc.) all returned 404, indicating no downloadable OpenAPI/machine-readable spec is available.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round to ForeloopThe docs mention a stability guarantee for CLI `--json` output ('has a stable shape. Additive changes only') and the MCP/API endpoint path includes a 'v1' segment, implying some versioning discipline, but there is no explicit documented deprecation policy, versioning scheme, or migration guidance for the API itself. missing for 10: explicit API version numbering scheme, documented deprecation timeline/process, changelog or migration guides, evidence beyond CLI JSON stability.
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
Codegennone0/10There's an API reference and SDK docs, but no evidence of API versioning scheme or a documented deprecation policy; the openapi probe even returned 404s, suggesting no formal spec surfacing version info.
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to CodegenForeloopnone0/10Foreloop's docs describe per-task workflows (claim one task, agent does the work, opens one PR) and a CLI/MCP interface, but nothing in the evidence describes performing an operation across many items/tasks simultaneously (e.g., batch approve, bulk edit, multi-task claim). The axis is plausible for a CLI/automation tool but no evidence of bulk operation support exists.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
The Python SDK and API allow programmatic creation/monitoring of agents and triggering from CI/CD, which could be scripted in a loop to perform bulk operations, but there is no documented batch/bulk endpoint or explicit guidance for operating across many items at once. missing for 10: dedicated bulk/batch API or CLI command, documented examples of multi-item automation, and evidence of scale/performance handling for bulk workloads.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to CodegenForeloopnone0/10Foreloop's docs describe human-approved task workflows (agents claim tasks, but 'nothing runs until a person approves a contract'), not a rules engine for automatic event-triggered actions; the closest mention is CLI usability from cron, which is not evidence of a native rules/trigger system. missing for 10: any documented rule-definition syntax, event-trigger configuration, or automation-without-approval capability.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
Codegen supports event-driven automation via CI/CD triggers (docs-18) and PR review triggers on failure (docs-20), plus SDK/API for programmatic agent invocation, but there's no evidence of a general user-facing rules engine for defining arbitrary trigger-condition-action automations across events. missing for 10: a documented rules/automation builder UI, broader event types beyond CI/CD failures and PR diffs, and independent confirmation of custom trigger logic.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userSchedule recurring jobs or workflows
weight 2 · round to ForeloopThe only relevant evidence is a single line noting the CLI can be run 'from anything that can run one, including coding agents and cron,' implying users could wire up their own cron-based scheduling — but Foreloop itself documents no native recurring-job/workflow scheduler, retry policy, or schedule management UI. Missing for 10: a first-party scheduling feature, documented recurring-workflow configuration, and evidence of built-in cron/interval management within the product.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
ai-native userVersion, review, and roll back my automations
weight 1 · round to ForeloopForeloop's workflow requires human approval before any agent-driven change runs (foreloop-docs-7) and changes are delivered as GitHub pull requests (foreloop-docs-3, foreloop-docs-15), which inherently gives code-level versioning and review via git. However, there is no documentation of versioning, reviewing, or rolling back the automations/tasks/loops themselves (as opposed to the resulting code diffs) — no rollback command, task history, or automation-versioning feature is evidenced. Missing for 10: explicit automation/task versioning, a documented rollback mechanism for tasks or loops, and audit/history views beyond the live agent status list.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
Codegennone0/10Evidence covers PR/code review, commits, branches, and traces/telemetry, but there is no mention of versioning automations themselves, reviewing automation history, or rolling back agent configurations/workflows. missing for 10: automation versioning system, rollback mechanism for agent workflows, audit/history UI for automations.
Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionAutonomous implementation
End-to-end implementation by the agent — multi-file changes, task completion
End to end feature delivery
ai-native userHave an agent automatically generate and run tests to validate its own code changes before proposing them
weight 2 · round to CodegenForeloopnone0/10No evidence describes Foreloop's agents generating or running tests to validate code changes before proposing them; the docs describe task claiming, PR opening, and human approval gates but never mention automated test generation/execution as a validation step.
Docs state the agent writes tests as part of implementing features and can run code/install dependencies/test changes in isolated environments, and a PR-review flow checks test coverage before flagging issues — together implying self-validation via generated tests. However, there is no explicit end-to-end walkthrough showing the agent generating tests, running them, and iterating/fixing before proposing a PR, nor independent/hands-on confirmation. missing for 10: explicit documented workflow tying test generation+execution to pre-PR validation, independent corroboration of this loop working in practice.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
developerHave an agent autonomously diagnose and fix a reported bug
weight 3 · round to ForeloopForeloop's documented workflow directly supports this: a bug report captured via the feedback widget (with page, metadata, screenshot) becomes an intention/task that an agent (Claude Code or Codex) claims, works on in a repo checkout, and resolves by opening a PR, without the reporter needing to know which files change. Human approval gates task initiation but the diagnosis-and-fix work itself is agent-driven end to end. Missing for 10: independent/hands-on evidence that agents actually diagnose root causes correctly, and more detail on how the agent investigates ambiguous bugs beyond the general task pipeline description.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
Docs explicitly claim the agent can 'analyze requirements, implement features, fix bugs, write tests' and can be triggered from issue trackers/CI to autonomously act, plus run in isolated environments to test changes — directly supporting autonomous bug diagnosis and fix. However, all evidence is first-party marketing/docs with no independent or hands-on validation of end-to-end autonomous diagnosis-to-fix workflows, and no detailed walkthrough of the diagnosis step itself (e.g., root-cause analysis) beyond generic 'fix bugs' language. Missing for 10: independent/hands-on verification of successful autonomous bug fixes, detailed diagnostic reasoning process, and success-rate or case-study evidence.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
product-managerGo from a mockup or design to a working implementation without an engineering handoff
weight 2 · round to ForeloopForeloop lets a non-engineer describe a need in plain language (e.g. "Users can't tell which invoices are overdue") or flag a page via a feedback widget with screenshot, and an agent turns that into tasks and a PR without the PM needing to specify files — this covers a 'no engineering handoff' description-to-implementation flow. However, the evidence never shows ingesting an actual mockup/design file (e.g. Figma) as the starting artifact, and a human still must approve the contract before code changes run, so it's not a fully autonomous design-to-code path. Missing for 10: direct mockup/design-file import as an input, evidence of end-to-end completion without any review gate, and a demonstrated design-to-code case study.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
Codegen's docs show relevant building blocks — a Figma connector, Slack/Linear interfaces that let non-engineers prompt the agent, and full PR/branch creation — implying a PM could describe a design and get code without direct engineering handoff. However, there is no concrete workflow, case study, or example showing a mockup/design being turned into a working implementation end-to-end. Missing for 10: an explicit design-to-code workflow example, evidence of Figma-to-PR pipeline, and PM-specific case studies or testimonials showing the handoff-free process actually works.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
developerHave an agent implement a requested feature end-to-end, including writing tests
weight 3 · round to CodegenForeloop orchestrates tasks and lets an agent (Claude Code/Codex) claim a task, do the work, and open a PR in the user's repo, which covers end-to-end feature implementation via an external coding agent — but Foreloop itself doesn't write code or tests; it delegates that entirely to the connected agent, and there's no explicit mention of test-writing behavior or verification. missing for 10: explicit evidence that generated PRs include tests, independent/hands-on confirmation of successful end-to-end feature delivery, and detail on how test coverage is verified before approval.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
Docs explicitly describe agents analyzing requirements, implementing features, writing tests, and running/testing changes in isolated environments, plus PR/branch management for end-to-end delivery. Missing for 10: independent/hands-on validation of end-to-end feature delivery and no concrete example transcript showing a full feature+test cycle.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
Environment setup
developerHave an agent automatically clone the repo, install dependencies, and configure its own working environment
weight 2 · round to CodegenForeloop's docs describe the developer running an agent 'in a checkout of your repository on your own machine' and using `foreloop init` to install a task-executor skill into that existing checkout — implying the human, not the agent, does the initial clone/setup, with the agent only claiming and working tasks afterward. There's no documentation of the agent autonomously cloning the repo, installing dependencies, or provisioning its own environment from scratch. Missing for 10: evidence of agent-initiated repo cloning, dependency installation steps, and autonomous environment configuration (e.g., a sandboxed or cloud checkout flow) rather than reliance on a pre-existing local checkout.
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
Docs describe agents that access repos via GitHub App integration, run in isolated environments, and 'safely run code, install dependencies, and test changes' automatically without complex setup, matching the story of autonomous environment setup. Missing for 10: independent/hands-on verification of environment provisioning behavior and more detail on how dependency installation is auto-configured beyond the brief doc mention.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
Interactive takeover
developerTake over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work
weight 2 · round to CodegenForeloop lets agents run locally in a checkout and claim/report tasks, and humans can approve contracts or view agent status via a dashboard row per worker, but there's no evidence of a live handoff mechanism to pause, take over, or redirect an in-progress task from editor/terminal/browser mid-execution. missing for 10: explicit takeover/interrupt UX, editor-integrated control of a running agent, terminal command to seize an active task, and any hands-on confirmation of mid-task redirection.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “Approve the big calls from your phone.”
Codegen supports interacting with agents across multiple surfaces (Slack notifications/clarification requests, CLI to pull agent branches locally, Python SDK to run/monitor agents) and lets you download in-progress work to your local environment, which implies some ability to pick up and continue work. However there is no explicit documentation of pausing, redirecting, or steering a task mid-execution from editor/terminal/browser as a unified handoff flow. Missing for 10: explicit 'takeover' or redirect mid-task feature, editor/IDE integration for live task control, and evidence of seamless cross-surface handoff (Slack->CLI->browser) of the same in-progress task.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
developerSend follow-up instructions to an active agent session to steer its work without restarting
weight 2 · round drawnForeloopnone0/10The evidence describes agents claiming tasks, opening PRs, and reporting status, plus approval gates and skill-writing to correct recurring mistakes — but nothing describes sending live follow-up/steering instructions to an already-running agent session mid-task without restarting it.
Codegennone0/10No evidence describes sending follow-up instructions to an already-running agent session to redirect its work; docs only describe initiating agents via prompts, Slack notifications, or SDK task creation, not mid-session steering. Missing for 10: any documentation of a 'continue/steer/update' API, chat-like follow-up in an active run, or session-resume mechanism.
Sandbox execution
developerHave an agent safely execute code and install dependencies inside an isolated sandbox
weight 3 · round to CodegenForeloopnone0/10Foreloop's docs explicitly describe agents running 'in a checkout of your repository on your own machine' (foreloop-docs-3), with no mention of sandboxing, containerization, or isolated execution environments for code/dependency installation. Missing for 10: any documentation of sandbox/container isolation, dependency install safety, or resource/network restrictions during agent execution.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
First-party docs explicitly state the agent can 'Safely run code, install dependencies, and test changes in robust isolated environments,' directly matching the story. missing for 10: independent/hands-on corroboration of sandbox isolation, technical details on sandbox implementation (e.g., container tech, resource limits) beyond a single doc line.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsHuman oversight
Keeping a human in the loop — approvals, checkpoints, interrupts
Approval controls
developerConfigure an agent to auto-approve all its actions instead of confirming each one
weight 2 · round drawnForeloopnone0/10Foreloop's docs explicitly state that human approval of a contract is a deliberate, mandatory gate before any agent action runs ('Nothing runs until a person approves a contract. Approval is the only gate'), with no mention of a setting to bypass or auto-approve this gate. This indicates the product does not offer an auto-approve configuration, so the story's capability is unsupported.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Approve the big calls from your phone.”
Codegennone0/10The evidence describes Codegen's autonomous capabilities (running code, committing, creating PRs, Slack/Jira integration) but never mentions any per-action confirmation mode or an auto-approve toggle/setting for oversight control. Missing for 10: any documentation of an approval/confirmation workflow, a setting to enable/disable auto-approval, or human-in-the-loop gating that this story presupposes.
product-managerApprove key agent decisions from my phone while agents continue working
weight 1 · round to ForeloopForeloop explicitly markets phone-based approval of 'big calls' and documents that no agent work runs until a human approves a contract, which matches the core ask. However, evidence is limited to marketing taglines with no detail on the mobile UI/notification flow, how approvals interact with agents continuing to work in parallel, or independent corroboration. Missing for 10: concrete mobile-app/notification workflow details, evidence of multi-agent concurrent operation while awaiting approval, and third-party confirmation.
- [claimed-docs] “Approve the big calls from your phone.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
Codegen supports Slack-based notifications and agent interaction (docs-2, docs-9), which could technically be used from a phone via the Slack app, but there is no explicit mobile app, push-approval workflow, or documented 'approve on the go' feature for key agent decisions. missing for 10: dedicated mobile/approval UI, evidence of approving specific agent decisions (e.g., merge/deploy gates) from a phone, and any hands-on confirmation of this workflow.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
engineering-leadSet tiered autonomy levels controlling what an agent can do without manual confirmation
weight 3 · round drawnForeloopnone0/10Foreloop documents a single approval gate before any agent work starts ('Nothing runs until a person approves a contract') and lets you approve from your phone, but there is no evidence of configurable tiered autonomy levels (e.g., auto-approving certain task types while requiring confirmation for others).
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Approve the big calls from your phone.”
Codegennone0/10No evidence describes configurable autonomy tiers, approval gates, or permission levels for agent actions; the docs describe what agents can do (code, PRs, Slack, Jira/Linear) but nothing about setting graduated confirmation thresholds for an engineering lead to control autonomy.
Visibility monitoring
developerWatch what a running agent is doing in real time, including its current status
weight 3 · round drawnThe docs describe a Development/Agents view listing each worker's agent identity, checkout name, current task, and last message — a real-time-ish status view (foreloop-docs-8). However, there's no evidence of live streaming updates, logs, or granular in-progress activity beyond a status line, and no independent/hands-on confirmation of real-time refresh behavior. missing for 10: evidence of live/streaming updates or detailed activity logs, independent confirmation of real-time refresh, granularity beyond a single status line per agent.
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
Codegen's CLI lets developers view agents and run them with 'full telemetry and monitoring,' and agents can 'report progress' via Slack, giving some real-time visibility into agent activity and status. However, there's no explicit documentation of a live dashboard/status stream showing step-by-step actions as they happen, and the closest 'trace' feature is framed as post-hoc analysis/debugging rather than live watching. Missing for 10: a documented real-time UI/dashboard showing live agent status, explicit streaming of in-progress actions, and independent/hands-on confirmation of live monitoring accuracy.
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
developerGet notified when an agent completes a task or needs my input
weight 2 · round to CodegenForeloop provides a dashboard showing agent status ('last thing it said') and mentions approving 'big calls from your phone', implying some notification/approval flow, but there is no explicit documentation of push/email/Slack notifications when a task completes or needs input. missing for 10: explicit notification channel (email/Slack/push) documentation, confirmation of real-time alerts rather than manual dashboard polling, evidence of notification content for 'needs input' events.
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “Approve the big calls from your phone.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
Codegen explicitly supports sending notifications, asking for clarification, and reporting progress directly in Slack channels, plus updating statuses/comments in Jira, Linear, Clickup, and Monday.com, covering both completion and input-needed scenarios. missing for 10: independent/hands-on confirmation of real-time notification delivery, and no explicit mention of email/other channel notification options.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
Intent to spec — stories about intent to spec in this arenaIntent to spec
Stories about intent to spec in this arena
Natural language task intake
developerDescribe a feature or bug in plain language and have it automatically turned into a scoped implementation task
weight 3 · round to ForeloopDocs describe plain-language input ('Users can't tell which invoices are overdue') turning into scoped, approvable tasks via 'intentions' that an agent fills in and proposes as tasks, gated by human approval before an agent acts on the repo. This directly matches intent-to-spec workflow with first-party documentation across multiple pages. Missing for 10: independent/hands-on verification of the resulting task's scoping quality and no example of the full generated task artifact.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
Codegen's docs describe taking natural-language prompts and turning them into scoped agent tasks: analyzing requirements, implementing features, fixing bugs, writing tests, and running agents from prompts via SDK/CLI/Slack/issue trackers. This directly matches the intent-to-spec story of describing a feature/bug and having it become an actionable implementation task. Missing for 10: no independent/hands-on evidence showing the quality or accuracy of the generated scoped task (e.g. how well it decomposes ambiguous requests), and no detail on how it handles clarification/scoping edge cases beyond a general 'ask for clarification' mention.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
product-managerConvert user feedback submissions into structured tasks with proposed scope
weight 2 · round to ForeloopDocs show a feedback-collection widget that captures user submissions, and an agent-driven 'intention' workflow that reads the submission, fills in an intention, and proposes tasks (foreloop-docs-12), with human approval gating execution (foreloop-docs-7). This directly matches converting feedback into structured tasks with proposed scope, though evidence is entirely vendor docs with no independent/hands-on corroboration of output quality or scope structure. Missing for 10: independent/hands-on verification of the feedback-to-task pipeline, and concrete example of the 'proposed scope' structure/fields.
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
Codegen can analyze requirements and create/link tasks in Jira, Linear, Clickup, Monday.com, which could theoretically be used to turn feedback into tasks, but there is no evidence of a feedback-intake feature or automatic scope proposal tailored for PM workflows. Missing for 10: dedicated feedback ingestion mechanism, structured scope/estimate output, PM-specific workflow examples or case studies.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
developerAttach a marked-up screenshot or mockup to a task so the agent implements the correct visual change
weight 2 · round to ForeloopThe feedback widget lets a customer point at an element and attach an optional screenshot with metadata so 'the model fixes the right thing' (foreloop-docs-21), which is adjacent to the story but is customer-bug-report flow, not a developer attaching an annotated mockup directly to a task. There's no documentation of markup/annotation tools, mockup uploads, or a developer-facing task-attachment workflow for visual specs. Missing for 10: developer-facing screenshot/mockup attachment to tasks, annotation/markup capability, and evidence the agent parses visual markup to implement precise UI changes.
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
Codegennone0/10No evidence in the pack mentions attaching screenshots, images, or marked-up mockups to a task/prompt; Codegen's docs cover text prompts, Slack/Jira/Linear integrations, CLI/SDK, and PR review but nothing about visual/image input for implementing UI changes. Figma is mentioned only as a connectable integration, not as an image-annotation workflow.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
Plan approval
developerReview and approve an agent's implementation plan before any code changes are made
weight 3 · round to ForeloopDocs explicitly describe a 'contract' approval gate before any agent touches the repository ('Nothing runs until a person approves a contract... an agent is about to change your repository'), and mention approving 'big calls' from a phone, indicating a review/approve workflow prior to code changes. Missing for 10: independent/hands-on confirmation of the approval UI showing the actual implementation plan content and evidence of what happens if a plan is rejected/revised.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Approve the big calls from your phone.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
Codegennone0/10No evidence in the pack of a plan-preview or approval step prior to code changes; agents appear to run prompts directly to implementation (create branches, commit code, open PRs) without a documented plan-review gate. missing for 10: any mention of a plan/preview mode, approval gate, or dry-run step before code execution.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
engineering-leadApprove a task's scope and contract before an agent is allowed to modify the repository
weight 2 · round to ForeloopForeloop's docs explicitly state approval of a contract is a mandatory gate before any agent touches the repository ('Nothing runs until a person approves a contract... an agent is about to change your repository'), and agents only claim 'approved tasks' from the repo. This directly matches the engineering-lead approval-gate story, reinforced by mobile approval flow. Missing for 10: independent/hands-on verification that the gate is enforced in practice (not just documented) and detail on what 'scope and contract' specifically contains.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Approve the big calls from your phone.”
Ticket driven tasking
developerAssign a coding task to an agent directly from an existing issue or ticket
weight 3 · round to CodegenForeloopnone0/10Foreloop's workflow starts from user feedback reports or free-text intentions that get turned into tasks an agent claims (foreloop-docs-4, foreloop-docs-12), but there is no evidence of importing or linking an existing GitHub issue/Jira ticket as the task source for an agent to pick up.
Docs show native integrations with Jira, Linear, ClickUp and Monday.com allowing the agent to update issue statuses, add comments, link PRs and create tasks, and specifically state connecting a Linear workspace 'enables agent interactions with your issues,' which directly supports assigning work from a ticket. Missing for 10: a concrete end-to-end example of assigning a task from an issue/ticket, and independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to ForeloopForeloop ships a CLI covering projects, loops, intentions, stories and tasks with a stable `--json` output on every command, plus an MCP endpoint for agent clients, suggesting broad API/CLI parity with the UI. However, there's no explicit statement or independent confirmation that literally every UI action (e.g., approvals, feedback widget config, agent monitoring dashboard views) is exposed via API/CLI. missing for 10: explicit UI-to-API parity documentation or audit, independent/hands-on verification, coverage of approval workflow and dashboard views via API.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [probe] “official CLI documented at https://foreloop.com/install”
Codegen offers a Python SDK, CLI, and API to create/run agents, pull work, and monitor executions, showing solid programmatic control (codegen-docs-11,15,16,17,18). However, there's no evidence of a complete OpenAPI/API surface matching every UI capability (e.g., Slack/Jira/Linear integration management, PR review workflows) — the OpenAPI probe returned 404s, suggesting no public full API spec. missing for 10: OpenAPI reference confirming full UI parity, evidence that integration configs and review workflows are API-accessible, independent confirmation of API completeness.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnForeloopnone0/10No evidence of a bulk data export feature or open-format export of projects/tasks/intentions for migration away from Foreloop; the closest hints (--json CLI output, MCP API access) are for programmatic access, not a documented full-data export/leave capability.
Codegennone0/10No evidence of a bulk data-export feature, open-format export of agent histories/traces, or account-portability tooling; only partial building blocks exist (pulling agent branches via CLI, retrieving traces via API) but nothing documenting a full data export/exit path.
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
ai-native userRead the product's source under an open license
weight 2 · round drawnForeloopnone0/10No evidence of an open-source license or public source code repository; Foreloop is presented as a closed SaaS product with a CLI/widget/MCP server, but nothing indicates the source is available under an open license.
ai-native userSelf-host the core product
weight 3 · round drawnForeloopnone0/10All evidence describes Foreloop as a hosted SaaS (with a CLI/MCP endpoint at foreloop.com/api) that agents connect to remotely; there is no mention of a self-hostable core server, on-prem deployment, or open-source release. missing for 10: any documentation of a self-hosted deployment option, container/server image, or license permitting self-hosting.
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
Enterprise licensing
engineering-leadLicense an enterprise deployment with SSO and commercial support for organization-wide rollout
weight 2 · round drawnForeloopnone0/10No evidence in the pack mentions enterprise licensing, SSO, or commercial support tiers; all citations focus on CLI usage, MCP integration, and agent workflows. This is a plausible axis for a SaaS product like Foreloop (organizations often need SSO/enterprise support), so absence of evidence yields 'none' rather than 'na'.
Codegennone0/10No evidence in the pack mentions SSO, enterprise licensing, commercial support tiers, or org-wide rollout plans; only feature-level docs and API/CLI references are provided. Missing for 10: SSO/SAML support, enterprise plan/pricing tier documentation, commercial support SLA, org-wide admin/rollout tooling.
Model flexibility
engineering-leadBring my own LLM or API key so agents run on the model of my choice
weight 2 · round drawnForeloopnone0/10The docs describe running third-party agents (Claude Code or Codex) locally against Foreloop's task API, but there's no evidence Foreloop lets a user configure or supply their own LLM/API key for Foreloop's own model usage (e.g., intention generation, feedback triage). Missing for 10: any documentation of model/provider selection, API key configuration UI, or support for alternate LLM backends.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
Usage quotas
engineering-leadSee and manage plan-based daily task and concurrency limits for agent workflows
weight 2 · round drawnForeloopnone0/10No evidence in the pack mentions plan-based daily task limits, concurrency limits, or any pricing/plan management dashboard for engineering leads; the docs focus on agent workflow mechanics (task claiming, approvals, MCP) with no reference to usage caps or plan tiers.
Codegennone0/10No evidence pack item mentions plan-based daily task limits, concurrency limits, or any admin dashboard/controls for engineering leads to monitor or manage usage quotas; documentation covers agent capabilities, integrations, CLI, and SDK but nothing on pricing tiers or limit management.
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnForeloopnone0/10No evidence anywhere in the pack mentions data residency, region selection, or storage location options; Foreloop's docs cover CLI, MCP, agents, and feedback widgets but never address where data is stored or any regional controls.
ai-native userControl data retention and deletion
weight 2 · round drawnForeloopnone0/10No evidence pack items mention data retention policies, deletion controls, data export, or privacy settings for user/feedback data collected by Foreloop; all evidence focuses on agent workflows, CLI usage, and MCP integration.
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnForeloopnone0/10No evidence in the pack mentions telemetry, usage tracking, analytics collection, or an opt-out mechanism for Foreloop; the docs cover CLI usage, agent workflows, and feedback widgets but never address data collection or privacy controls.
Codegennone0/10No evidence pack item mentions a telemetry opt-out setting or privacy controls; in fact docs-12 explicitly notes the CLI runs Claude Code 'with full telemetry and monitoring,' with no mention of a way to disable it.
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
Repo integration — stories about repo integration in this arenaRepo integration
Stories about repo integration in this arena
Chat integration
developerTag an agent in a chat thread to discuss and delegate a bug or task
weight 2 · round to CodegenForeloopnone0/10Foreloop's workflow is built around agents polling for approved tasks/contracts and reporting status via a dashboard (docs-3, docs-7, docs-8), not around tagging or chatting with agents in a thread. No evidence describes a chat-thread interface or an @-mention mechanism to delegate work conversationally.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
Codegen supports Slack-based chat interactions where users can message/tag the agent to report progress, ask for clarification, and delegate tasks (codegen-docs-2, codegen-docs-9), and Linear/Jira integration allows commenting and task creation to delegate bugs (codegen-docs-3, codegen-docs-10). However, evidence doesn't explicitly confirm an '@mention' tagging mechanism within an existing chat thread to delegate a specific bug — it's inferred from general Slack/Linear interaction docs rather than a documented tagging workflow. Missing for 10: explicit documentation of @-mention/tagging syntax in threads, confirmation of thread-level context retention when delegating, and independent/hands-on verification of this specific workflow.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
Knowledge context
developerAdd a context file describing my codebase conventions so agents generate more relevant plans and code
weight 3 · round to ForeloopForeloop's docs describe a 'skill' mechanism ('Write a skill when you notice yourself giving an agent the same correction twice') and an installed task-executor skill in the checkout, which loosely maps to encoding project-specific conventions for agents, but there's no explicit documentation of a dedicated context file (e.g., conventions.md) for describing codebase style/conventions. Missing for 10: explicit context-file feature, guidance on structuring codebase conventions, evidence that agents consume such a file specifically for improving plan/code relevance.
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “Write a skill when you notice yourself giving an agent the same correction twice.”
Codegennone0/10The evidence pack covers Codegen's integrations, CLI, SDK, and PR/issue automation, but nowhere mentions a repo-level context or rules file (e.g., conventions file) that developers can add to steer agent plans/code generation. missing for 10: any mention of a context/config file for codebase conventions, documentation on custom instructions, or repo-level rules ingestion.
Project management integration
product-managerConnect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly
weight 3 · round to CodegenForeloopnone0/10Foreloop's evidence covers GitHub integration, its own task/loop tracking, MCP server, and feedback widgets, but there is no mention of connecting to Jira, Linear, ClickUp, or Monday.com at all — Foreloop appears to use its own internal issue/task system rather than integrating external issue trackers.
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
Docs explicitly claim support for Jira, Linear, ClickUp, and Monday.com (update statuses, add comments, link PRs, create tasks), but detailed setup instructions are only shown for Linear (codegen-docs-10); Jira, ClickUp, and Monday.com lack dedicated connection/integration docs equivalent to Linear or Slack. Missing for 10: explicit setup guides or independent verification for Jira/ClickUp/Monday.com, and hands-on confirmation beyond first-party docs.
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
Version control integration
developerConnect a GitHub repository so an agent can access the code and open pull requests against it
weight 3 · round to CodegenDocs explicitly describe connecting GitHub so Foreloop can read code and agents can open PRs against it, with a full workflow (agent claims task in a checkout, does work, opens a PR from the user's GitHub account, gated by human approval). Missing for 10: independent/hands-on verification of the GitHub connection flow and no screenshots or third-party confirmation that PRs are actually opened successfully.
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
Codegen provides a GitHub App install for repo access with no complex setup, and docs describe the agent creating branches, committing code, reviewing/opening PRs, and managing repositories. missing for 10: independent/hands-on confirmation beyond vendor docs, and explicit detail on multi-repo/org-level permission scoping.
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
developerGrant an agent access to my repositories with a one-click install, without complex setup
weight 2 · round to CodegenDocs confirm connecting GitHub gives Foreloop read/PR access to repos (foreloop-docs-15) and that agents run against a checkout, but achieving this requires installing a global CLI, running foreloop init, configuring MCP with an API key, and running an agent locally — multiple manual steps rather than a true one-click flow. missing for 10: explicit one-click OAuth install flow, evidence of a simple GitHub App install button, independent confirmation of setup simplicity.
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “npm i -g foreloop # installs globally, so the CLI works in any directory”
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
Docs explicitly state installing the GitHub App grants repo access with 'no complex setup required', directly matching the story, and this is corroborated by broader repo-integration capabilities (PR review, branch/commit management). Missing for 10: independent/hands-on confirmation of the one-click flow beyond first-party docs.
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
Review quality gates — quality gates on changes — review flow, required checks, merge protectionReview quality gates
Quality gates on changes — review flow, required checks, merge protection
Ci remediation
engineering-leadHave failed CI workflows automatically diagnosed and fixed with a proposed pull request
weight 3 · round to CodegenForeloopnone0/10Foreloop's documented workflow is task/intention-based (feedback → task → agent → PR), with no evidence of CI pipeline integration, failed-build detection, or automated diagnosis-and-fix triggered by CI failures. The axis is plausible for an agent-driven PR tool, but nothing in the evidence pack shows Foreloop watching or reacting to CI status.
Docs explicitly mention triggering agents from CI/CD pipelines when builds fail and agents that fix bugs/write code and open PRs, directly supporting automated diagnosis-and-fix-PR workflows for failed CI. However, there is no dedicated end-to-end CI-failure-diagnosis feature description, no example of the proposed PR format for a failing build, and no independent/hands-on confirmation of this specific workflow. missing for 10: dedicated CI-failure-diagnosis workflow documentation, concrete example of a CI-triggered fix PR, independent verification of this specific capability.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
developerTrigger an agent from CI/CD pipelines to fix a broken build or failing test
weight 2 · round to CodegenForeloopnone0/10Foreloop's docs describe agents claiming human-approved tasks and opening PRs from a local checkout, with a CLI usable from "anything that can run one, including coding agents and cron" (foreloop-docs-16), but there is no evidence of CI/CD pipeline integration or of the agent being triggered specifically to fix broken builds or failing tests — its workflow centers on approved feature/task work, not automated CI failure remediation.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
Docs explicitly state agents can be 'Triggered from CI/CD pipelines when builds fail' and support fixing bugs/tests via API/SDK with programmatic control, which directly matches the story. Missing for 10: independent/hands-on confirmation of a real CI/CD integration example (e.g., GitHub Actions snippet) and details on how failing test context is passed to the agent.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
Diff review
developerConfigure an agent to automatically open a pull request when its task completes
weight 2 · round to ForeloopDocs clearly describe the agent workflow: it claims a task, does the work, and opens a pull request from the user's GitHub account, with GitHub connection enabling PR creation, gated by human approval of the contract before work starts. Missing for 10: independent/hands-on confirmation that PRs are reliably opened on task completion, and no detail on configuring PR-specific settings (e.g., target branch, draft PRs).
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
Codegen's agent creates branches, commits code, and manages repositories via GitHub App integration, and can be run programmatically via SDK/CLI/API to complete tasks; PR creation is a core documented capability of the agent workflow. Missing for 10: explicit first-party documentation/example showing a config flag or setting to 'auto-open PR on task completion' and independent/hands-on confirmation of this exact behavior.
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
developerReview a diff of an agent's changes and approve it before it becomes a pull request
weight 3 · round to CodegenForeloopnone0/10Foreloop's 'approval' gate (foreloop-docs-7) is a pre-work contract approval, not a diff review after the agent finishes changes — the agent then 'opens a pull request from your GitHub account' directly (foreloop-docs-3), with no documented step where a person reviews the diff before the PR is created. No evidence describes a diff-preview/approve-before-PR workflow.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
Codegen agents create branches, commit code, and open PRs, and the agent itself can review PRs and post comments, but the evidence pack does not show a dedicated human-in-the-loop diff review/approval gate before a PR is created — the CLI lets you 'pull agent work' locally which could support manual review, but no explicit approval workflow is documented. missing for 10: explicit pre-PR diff approval UI/workflow, evidence of a human approve/reject gate, independent confirmation that PR creation is blocked pending review.
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
Pr review automation
ai-native userHave incoming issues automatically triaged with severity suggested and routed to the right owner
weight 2 · round drawnForeloopnone0/10Foreloop's docs describe turning user feedback into intentions/tasks that agents pick up and claim (foreloop-docs-12, foreloop-docs-4), but there is no evidence of automatic severity classification or routing to a specific human owner — tasks are self-claimed by agents rather than triaged and assigned. missing for 10: severity scoring, owner/routing logic, any triage workflow evidence.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
Codegennone0/10Evidence shows Codegen integrates with issue trackers (Jira, Linear, etc.) to update statuses, add comments, and create tasks, but there is no mention of automatic severity assessment or routing incoming issues to the correct owner. Missing for 10: automatic severity classification, owner/routing logic, any triage workflow evidence.
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
Readiness checks
engineering-leadAutomatically fix failing agent-readiness criteria in my repository
weight 1 · round to CodegenForeloopnone0/10Foreloop's docs describe a task/feedback pipeline where agents claim approved tasks and open PRs, but nothing in the evidence pack mentions detecting or automatically remediating 'agent-readiness criteria' (e.g., AGENTS.md, tool-permission manifests, readiness scoring) in a repository. Missing for 10: any readiness-criteria scanning/grading feature, evidence of automatic detection of failing criteria, and evidence of automated fixes tied to such criteria.
Codegen agents can fix bugs, write tests, and run CI/CD-triggered fixes (codegen-docs-1, codegen-docs-18, codegen-docs-20), which could be applied to remediate failing agent-readiness checks, but there is no explicit documentation of an 'agent-readiness criteria' concept, scoring rubric, or automated fix workflow tied specifically to such criteria. missing for 10: explicit agent-readiness criteria/checklist, automated detection-to-fix pipeline for those criteria, evidence of hands-on success fixing readiness gates.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
engineering-leadRun a readiness report that evaluates how ready my repository is for autonomous agents
weight 2 · round drawnForeloopnone0/10No evidence in the pack describes a 'readiness report' or any repository-readiness scoring/assessment feature for autonomous agents; Foreloop's docs cover task claiming, MCP connectivity, feedback widgets, and approval gates, but nothing about evaluating repo readiness.
Codegennone0/10The evidence pack shows PR review, security scanning, and CI/CD triggers, but there is no mention of a dedicated 'readiness report' feature that evaluates overall repository readiness for autonomous agents. Missing for 10: any documentation of a repo-wide readiness/scorecard report, criteria for agent-readiness, or a command/API that produces such an assessment.
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
Security remediation
engineering-leadHave security alerts automatically validated and remediated with an opened pull request
weight 2 · round to CodegenForeloopnone0/10Foreloop's evidence describes a general feedback/task-to-PR agent workflow (bug reports, feature requests) with human approval gates, but nothing in the pack mentions ingesting security alerts (e.g., from SAST/SCA/dependency scanners), validating them, or triggering remediation PRs specifically for vulnerabilities. The axis is plausible for a task-automation platform, but no evidence shows this specific capability.
Codegen documents a security/secrets scan that posts a review comment on failure (codegen-docs-20) and general bug-fixing/PR-creation capabilities plus CI/CD-triggered agents (codegen-docs-1, codegen-docs-4, codegen-docs-18), suggesting it could act on a flagged issue and open a fix PR. However, there is no explicit evidence of ingesting external security alerts (e.g., Dependabot/CodeQL/Snyk), validating them as true positives, or automatically opening a remediation PR end-to-end — the documented scan flow stops at commenting, not remediating. missing for 10: explicit security-alert ingestion/validation workflow, evidence of an opened remediation PR tied to a specific alert, independent confirmation of this flow working in practice.
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
Scale parallelism — running many jobs at once — concurrency, fleets, queueingScale parallelism
Running many jobs at once — concurrency, fleets, queueing
Concurrent execution
engineering-leadRun many agent tasks concurrently to scale delivery throughput
weight 3 · round to CodegenThe docs show that multiple agent workers can each claim tasks and run in parallel checkouts, with a dashboard listing one row per worker (foreloop-docs-8), implying some capacity for concurrent agent execution across a team. However, there is no explicit product feature for orchestrating, scaling, or managing many concurrent tasks at once, no throughput/scale claims, and no evidence of built-in concurrency limits or fleet management tooling. missing for 10: explicit support/documentation for running many agents simultaneously at scale, concurrency controls, throughput metrics, or a scheduler/queue for parallel task dispatch.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
Codegen's SDK/API let you programmatically create and run multiple agents (agent.run, API 'full programmatic control over execution and monitoring') and CLI can view multiple running agents, implying support for parallel task execution, and CI/CD triggers suggest scaling automation into pipelines. However, there is no explicit documentation of concurrency limits, throughput guarantees, or dashboards for managing many simultaneous tasks at scale. Missing for 10: explicit concurrency/parallelism claims, scaling limits or throughput benchmarks, and independent evidence of running many agents simultaneously in practice.
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
engineering-leadCreate agent sessions on behalf of other users in my organization
weight 2 · round to CodegenForeloopnone0/10The evidence describes agents run locally by an individual user in their own repo checkout, claiming tasks and opening PRs from their own GitHub account, but nothing describes an engineering-lead provisioning or launching agent sessions on behalf of other org members. Missing for 10: any admin/org-level session creation API or UI, delegated-session documentation, or multi-user session management evidence.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
Codegen's org-level SDK/CLI/API (org_id+token, 'manage organizations', programmatic agent creation) implies an engineering lead with org credentials could spin up agents at scale, but no docs explicitly describe creating or assigning a session to another named user in the org. Missing for 10: explicit user-impersonation or 'create on behalf of' API/UI feature, role-based permissions for delegating agent sessions, and any admin console for assigning sessions to teammates.
- [claimed-docs] “Manage organizations - Switch between organizations and configure repositories”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
Deployment flexibility
developerUse a managed cloud offering to run agents without operating my own backend infrastructure
weight 2 · round to CodegenForeloopnone0/10Foreloop's own docs state the opposite of this story: agents (Claude Code or Codex) run 'in a checkout of your repository on your own machine,' with Foreloop only coordinating tasks and approvals — there is no evidence of a managed cloud runtime that executes agents on Foreloop's infrastructure.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
Codegen offers cloud-run agents with isolated execution environments, API/SDK/CLI access, and no need to self-host infrastructure (codegen-docs-5, codegen-docs-11, codegen-docs-15, codegen-docs-16). This directly matches running agents without operating own backend. Missing for 10: independent/hands-on verification of the managed infrastructure's reliability at scale, and explicit pricing/SLA details for the managed cloud offering.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
Headless automation
developerRun an agent headlessly inside CI/CD pipelines and shell scripts
weight 2 · round to CodegenDocs mention the CLI is invocable from 'anything that can run one, including coding agents and cron' and support scriptable --json output, suggesting headless invocation is possible (foreloop-docs-16, foreloop-docs-9). However, other docs state that 'nothing runs until a person approves a contract' (foreloop-docs-7), implying a human-in-the-loop gate that complicates fully unattended CI/CD execution, and there is no explicit CI/CD pipeline example or corroboration. Missing for 10: explicit CI/CD/pipeline example, evidence of non-interactive approval flow, independent confirmation of headless operation.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
Codegen explicitly documents triggering agents from CI/CD pipelines (e.g., on build failures) and offers a Python SDK plus CLI for programmatic/scripted invocation, which supports headless usage in pipelines and scripts. However, there's no dedicated CI/CD integration guide, no example pipeline configs (e.g., GitHub Actions YAML), and no independent/hands-on evidence confirming real-world headless CI usage. missing for 10: concrete CI/CD pipeline examples or config snippets, exit-code/status handling for scripts, independent verification of headless operation in automated pipelines.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
Not comparable on these axes
ai-native userConnect an agent via an official MCP server
weight 3 · not comparableForeloop documents a first-party MCP server endpoint (/api/app/public/v1/mcp over streamable HTTP) that any MCP client can connect to with just a URL and API key, explicitly enabling agents to claim tasks and work in repos. This is first-party documentation only, with no independent/hands-on corroboration of the MCP connection working in practice. Missing for 10: independent verification that an external MCP client successfully connects and operates against this endpoint.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
Codegenn/aCodegen is itself an AI coding agent, so serving as an MCP server for other agents to connect to is a different product role (client vs server) per the category rule. Evidence only shows Codegen acting as an MCP client (connecting to Slack/Linear/Figma/custom MCP tools) or provisioning MCP servers for its own org's Claude Code instances, not exposing itself as an official MCP server endpoint for external agents to connect to.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
- [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”
ai-native userTest against a sandbox environment without touching production data
weight 1 · not comparableForeloopn/aForeloop is a project/task-management and agent-orchestration tool for coding work (tasks, PRs, feedback widgets); it has no concept of a sandbox testing environment vs production data. This is a category error for its product type — the axis of 'sandbox vs production data testing' doesn't apply.
Codegen docs state that code changes run in 'robust isolated environments' to safely execute code and test changes, which implies sandboxed testing separate from production. However, there's no explicit description of how production data is excluded or protected, no detail on the sandbox's scope/persistence, and no independent verification of this isolation claim. missing for 10: explicit production-data isolation guarantees, technical detail on sandbox architecture, independent/hands-on confirmation.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
ai-native userHave each task prompt automatically routed to the most suitable underlying model
weight 2 · not comparableForeloopn/aForeloop is a task-management/agent-orchestration layer that hands work to whatever coding agent (Claude Code, Codex) the developer runs locally — it does not itself route prompts across multiple underlying LLMs. Automatic model routing per task is a wrong axis for this product's category (it's a task/PM layer, not a multi-model gateway).
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
engineering-leadSwitch away from automatic model selection to a specific model of my choice
weight 1 · not comparableForeloopn/aForeloop orchestrates tasks/agents (Claude Code, Codex) run by the user rather than offering its own model-selection setting; there is no evidence of an internal model picker, so this axis is a category error for this product's role as an agent-orchestration/task platform.
ai-native userPrevent my data from being used to train AI models
weight 3 · not comparableForeloopn/aForeloop is a task/agent orchestration and feedback-collection tool for software repos, not a data-processing or AI-model-training service; the evidence pack contains no data-usage or model-training policy at all, and this axis is a category error for this product type.
developerQuery generated documentation for any public or private repository
weight 1 · not comparableForeloopn/aForeloop is a task/agent orchestration and feedback-collection tool that connects to GitHub repos to let agents claim tasks and open PRs; it does not generate or expose queryable documentation for repositories. This story targets a documentation-generation/query product category, which is a different axis from Foreloop's core offering.
Codegennone0/10Evidence shows Codegen can 'improve documentation' as part of coding tasks, but there is no evidence of a feature to generate and then query documentation for a repository (public or private) in a Q&A style. No docs, API, or CLI reference describe a documentation-query capability.
engineering-leadHave every pull request automatically reviewed with AI-generated inline comments
weight 3 · not comparableForeloopn/aForeloop is a task-management/agent-orchestration tool that lets agents claim tasks and open PRs from feedback, not a PR-review/code-review product; there's no evidence it reviews pull requests or adds inline review comments — this is a different product category (task orchestration vs. review-quality-gates).
Docs explicitly describe automated PR review functionality: reading the diff, summarizing changes, running security/secrets scans, checking test coverage, and posting a review comment, plus general 'Review PRs, suggest changes, comment on issues' capability tied to GitHub App integration for repo access. This matches the engineering-lead story of automatic AI-reviewed PRs with comments. Missing for 10: explicit confirmation that comments are inline (per-line) rather than a single summary comment, and independent/hands-on evidence corroborating this in production use.
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
engineering-leadSelf-host agent infrastructure locally, in containers, or on my own VMs
weight 2 · not comparableForeloopn/aForeloop is a hosted SaaS orchestration layer (agents run locally in your repo checkout and talk to Foreloop's cloud API/MCP endpoint) — there is no evidence of a self-hostable server/backend component; the product's core value is the hosted service coordinating tasks, so 'self-hosting agent infrastructure' is a category mismatch rather than a missing feature.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
Codegennone0/10Codegen is presented entirely as a cloud/SaaS agent service (GitHub App, Slack App, API, SDK, isolated cloud environments) with no evidence of a self-hostable runtime, container image, or on-prem/VM deployment option; the CLI logs local Claude Code sessions to Codegen's own cloud rather than hosting infrastructure locally.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”