Foreloop vs Devin

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Devin
   Foreloop—none0/10

   No evidence pack item mentions llms.txt or an agent-oriented docs endpoint that agents can be pointed at; docs reference CLI, MCP endpoints, and skills but not an llms.txt-style discoverable doc file.

   ⚑ flag

   Devin✓fullprobed7/10

   Devin explicitly supports AGENTS.md, an agent-oriented documentation file that Devin reads before starting a coding session to gain project context — directly matching the story's intent of pointing an agent at agent-oriented docs. Devin's own docs site also serves an llms.txt endpoint (verified via probe), showing familiarity with the convention, though this is Devin's own site rather than evidence Devin ingests arbitrary llms.txt files from user projects. Missing for 10: explicit documentation that Devin consumes/parses llms.txt files themselves (as opposed to AGENTS.md) and independent hands-on confirmation of this workflow.

   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.devin.ai/llms.txt # Devin Docs - [Desktop (100 pages)](https://docs.devin.ai/_llms/en/desktop.md):…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Devin
   Foreloop~partialprobed6/10

   Foreloop ships a global CLI with a stable, additive `--json` output on every command, and docs explicitly state it can be run 'from anything that can run one, including coding agents and cron,' which is direct evidence of headless/CI-style automation support. However, there's no CI-specific documentation (e.g. GitHub Actions example, env-based auth for non-interactive contexts) and the approval-gate design ('nothing runs until a person approves a contract') implies a human-in-the-loop checkpoint that could block fully unattended pipelines. missing for 10: CI/pipeline-specific setup docs or examples, confirmation that approval gates can be bypassed or pre-authorized for headless runs, independent/hands-on verification of CI usage.

   • [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
   • [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [probe] “official CLI documented at https://foreloop.com/install”

   proof ↗⚑ flag

   Devin✓fullclaimed8/10

   Devin exposes a full REST API for session creation and automation (devin-docs-8, devin-docs-9), supports scheduling recurring/one-time headless runs via MCP (devin-docs-17), auto-triggers on CI failures and code review comments via Devin Review/Auto-Fix (devin-docs-20), and supports parallel/headless session orchestration (devin-docs-18). This covers CI/automation use cases well beyond interactive use. Missing for 10: no independent/hands-on CI-integration case study, and no dedicated 'CI runner' or GitHub Actions-style first-class integration doc beyond the review/auto-fix hook.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to Devin
   Foreloop—none0/10

   All MCP-related evidence describes Foreloop exposing its own MCP server endpoint (/api/app/public/v1/mcp) so that external MCP clients (agents) can call into it — this is the reverse of the story, which asks whether the user can plug external MCP servers into Foreloop so it can use their tools. No evidence shows Foreloop acting as an MCP client that consumes other servers' tools.

   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”

   proof ↗⚑ flag

   Devin✓fullprobed9/10

   Devin CLI explicitly supports connecting external MCP servers, exposing their tools as slash commands with permission controls, and docs list example integrations (Datadog, Sentry, databases, Figma, Notion, Stripe, etc.). Missing for 10: independent/hands-on community verification of MCP tool usage in practice.

   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “Devin CLI exposes each one as a slash command”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
   • [probe] “official MCP server documented at https://docs.devin.ai/cli/extensibility/mcp/overview”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round drawn
   Foreloop✓fullclaimed8/10

   Foreloop documents a first-party MCP server endpoint (/api/app/public/v1/mcp over streamable HTTP) that any MCP client can connect to with just a URL and API key, explicitly enabling agents to claim tasks and work in repos. This is first-party documentation only, with no independent/hands-on corroboration of the MCP connection working in practice. Missing for 10: independent verification that an external MCP client successfully connects and operates against this endpoint.

   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”

   proof ↗⚑ flag

   Devin✓fullclaimed8/10

   Devin ships an official MCP server (documented at docs.devin.ai/work-with-devin/devin-mcp) that exposes session management, playbooks, knowledge, and scheduling to any MCP-compatible agent or IDE — this is distinct from Devin's own MCP-client capability and directly satisfies the 'connect an agent via an official MCP server' story. Missing for 10: independent/hands-on confirmation that external agents successfully connect and operate through this server, and more detail on setup/auth flow beyond first-party docs.

   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “These tools let you explore and query documentation for any GitHub repository (public or private with authentication)”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round drawn
   Foreloop✓fullprobed8/10

   Foreloop ships an official globally-installable CLI (npm i -g foreloop) that manages projects, loops, intentions, stories, and tasks, supports scripting via `--json` with a stable additive schema, and is explicitly designed to be run 'from anything that can run one, including coding agents and cron,' making it clearly AI-native/agentic-workflow friendly. missing for 10: independent/hands-on corroboration beyond vendor docs, and more detail on full command surface.

   • [claimed-docs] “npm i -g foreloop # installs globally, so the CLI works in any directory”
   • [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
   • [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
   • [probe] “official CLI documented at https://foreloop.com/install”

   proof ↗⚑ flag

   Devin✓fullprobed8/10

   Devin ships an official CLI documented extensively (adaptive model routing, slash commands, MCP extensibility) confirmed by both docs and a probe hit, plus a separate Terminal/local CLI mode for quick fixes. Missing for 10: independent hands-on review of the CLI itself and more detail on installation/versioning.

   • [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
   • [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
   • [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “Devin CLI exposes each one as a slash command”
   • [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round drawn
   Foreloop✓fullprobed8/10

   Foreloop documents a public MCP endpoint (/api/app/public/v1/mcp over streamable HTTP with just a URL and API key) plus a CLI with stable, additive `--json` output usable from any scriptable context including cron and coding agents, giving AI-native users two documented programmatic surfaces. missing for 10: no independent/third-party corroboration of the API's stability or real-world agent integration beyond first-party docs, and no formal OpenAPI/REST reference is shown alongside the MCP/CLI docs.

   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
   • [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [probe] “official CLI documented at https://foreloop.com/install”

   proof ↗⚑ flag

   Devin✓fullprobed8/10

   Devin publishes a documented public API for integrating into applications, automating workflows, and creating sessions programmatically (including on behalf of other users), plus an MCP server exposing session/playbook/knowledge/scheduling management to external agents. Missing for 10: a discoverable machine-readable OpenAPI/swagger spec (probe found 404s) and independent third-party corroboration of API usage beyond vendor docs.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.devin.ai/llms.txt # Devin Docs - [Desktop (100 pages)](https://docs.devin.ai/_llms/en/desktop.md):…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   Foreloop~partialclaimed3/10

   The docs mention a 'write-only' API key type (flpk_YOUR_WRITE_ONLY_KEY) which implies some credential scoping, and the MCP endpoint for agents requires 'an API key', but there is no explicit documentation of issuing least-privilege or role-scoped credentials specifically for agent access to repos/tasks. missing for 10: explicit agent-scoped API key creation/management flow, documented permission levels (read/write/task-only), and evidence of per-agent credential rotation or revocation.

   • [claimed-docs] “await initForeloopFeedback({ apiKey: "flpk_YOUR_WRITE_ONLY_KEY", reporterName: signedInUser?.email, });”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”

   proof ↗⚑ flag

   Devin~partialclaimed3/10

   Devin documents a permission system that restricts MCP tool access at multiple levels and an API parameter (create_as_user_id) for acting on behalf of specific users, but there is no direct documentation of issuing scoped/least-privilege API keys or credentials for the agent itself. Missing for 10: explicit API key/token scoping mechanism, documented permission tiers or roles for API credentials, and any least-privilege credential issuance workflow.

   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Devin
   Foreloop~partialprobed6/10

   Foreloop exposes an official CLI, a JS feedback SDK (initForeloopFeedback), and an MCP endpoint for agent clients, giving AI-native builders official integration surfaces beyond just docs. However, this is all first-party vendor documentation with no independent corroboration or SDK versioning/language-coverage details (e.g. only JS snippet shown, no Python/other language SDKs). Missing for 10: independent/hands-on verification of SDK usage, multi-language SDK coverage, and evidence of SDK stability/versioning beyond the JS snippet.

   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
   • [claimed-docs] “await initForeloopFeedback({ apiKey: "flpk_YOUR_WRITE_ONLY_KEY", reporterName: signedInUser?.email, });”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
   • [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
   • [probe] “official CLI documented at https://foreloop.com/install”

   proof ↗⚑ flag

   Devin✓fullprobed8/10

   Devin publishes a documented API reference for building integrations (create sessions, manage on behalf of users, automate workflows), plus MCP support for extensibility — clear official SDK-like surface for AI-native developers. Missing for 10: no official language-specific SDK packages (e.g. Python/JS client libraries) are cited, only REST API docs and no discoverable OpenAPI spec (probe found 404s), and no independent/hands-on corroboration of API usage.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Foreloop—none0/10

   The evidence pack documents MCP connectivity, CLI, and feedback widget integrations, but nowhere mentions webhooks or event subscriptions that an AI-native user could subscribe to; no push/callback mechanism is described.

   ⚑ flag

   Devin—none0/10

   No evidence in the pack mentions webhooks or event subscription mechanisms for Devin; only API session creation, MCP integrations, and CLI features are documented. As an API-driven agentic product, webhook support would be a plausible and expected capability, so its absence counts as 'none' rather than 'na'.

   ⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Devin
   Foreloop~partialclaimed5/10

   Foreloop's agent reads submitted feedback/reports and proposes intentions and tasks from that data (foreloop-docs-12, foreloop-docs-4), which is a form of AI-generated suggestion from collected data, but this is limited to feedback-to-task triage rather than broader analytics or insight generation across product data. Missing for 10: evidence of AI summarizing trends/patterns across historical data, dashboards or reports surfacing insights beyond individual feedback items, and independent corroboration of insight quality.

   • [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
   • [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
   • [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”

   proof ↗⚑ flag

   Devin~partialclaimed6/10

   Devin's repo indexing enables 'Ask Devin' and DeepWiki features that generate AI insights and explanations from your codebase data, and Devin can proactively suggest fixes/improvements via code review auto-fix and bug triage. However, evidence is scoped only to codebase/ticket data, not broader business or analytics data, and there's no dedicated insights dashboard or proactive summarization feature documented. Missing for 10: evidence of insights beyond code (e.g., analytics/dashboards), independent corroboration of Ask Devin/DeepWiki quality, and any UI showcasing generated insights.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Devin
   Foreloop~partialclaimed5/10

   Foreloop supports background-style automation via its CLI (usable from cron or other schedulers) and agents that autonomously claim and work tasks in a repo, but it explicitly gates all agent actions behind a mandatory human approval step ('Nothing runs until a person approves a contract'), meaning true unattended autonomy is limited. Missing for 10: concrete documentation of scheduled/recurring automation setups, evidence of fully unattended runs without approval gating, and details on monitoring/alerting for background tasks.

   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Approve the big calls from your phone.”

   proof ↗⚑ flag

   Devin✓fullclaimed8/10

   Devin supports scheduling via cron expressions (recurring/one-time) and agent selection through its MCP tooling, plus API session creation and Slack/Teams triggering, enabling autonomous background automations beyond interactive chat sessions. missing for 10: independent/hands-on verification of scheduled runs actually executing unattended, and more detail on failure/retry handling for scheduled tasks.

   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round to Devin
   Foreloop—none0/10

   Foreloop's documented model relies on external agents (Claude Code or Codex) running on the user's own machine and connecting via MCP to claim tasks — there is no built-in AI assistant shipped inside Foreloop itself that a user delegates tasks to directly.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”

   proof ↗⚑ flag

   Devin✓fullcommunity8/10

   Devin itself functions as the built-in AI assistant to which tasks are delegated across web app, terminal, Slack/Teams, and API, with rich first-party documentation of the delegation workflow (docs-1,3,4,5,8,19). Community feedback raises reliability/skepticism concerns but does not concretely contradict that delegation itself works, so it's not disputed. Missing for 10: independent hands-on validation of delegation reliability/success rate beyond old demo critiques.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”

   proof ↗⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round to Devin
   Foreloop~partialclaimed5/10

   Foreloop lets users describe problems in plain language (e.g., "Users can't tell which invoices are overdue") to kick off an intention that an agent turns into tasks, showing some natural-language entry point, but its actual operating surface is a structured CLI (with --json flags) and an MCP interface for agents, not a general natural-language command interface for the product as a whole. Missing for 10: evidence of NL commands controlling arbitrary product actions (not just initial feedback/intention creation), and any hands-on/independent confirmation of this NL flow working.

   • [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
   • [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”

   proof ↗⚑ flag

   Devin✓fullcommunity8/10

   Devin is fundamentally driven by natural-language task requests across web app, Slack/Teams tagging, CLI, and API (create_as_user_id sessions), letting users describe tasks like bug fixes, features, and migrations in plain language rather than issuing structured commands. Community evidence corroborates that Devin operates via conversational/task-based prompts even amid skepticism about output quality. missing for 10: independent hands-on confirmation of NL command fidelity across all surfaces (CLI/Slack/API), and no evidence addressing ambiguous or multi-step NL instruction handling limits.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Foreloop—none0/10

   The evidence pack shows CLI docs, MCP endpoint info, and install snippets, but nowhere describes an interactive API reference (e.g., Swagger/OpenAPI explorer, try-it-now console) with runnable examples.

   ⚑ flag

   Devin—none0/10

   Devin documents an API reference overview and parameters (e.g. create_as_user_id) but nothing describes an interactive, in-browser 'try it' experience with runnable code examples; a probe for standard OpenAPI/Swagger interactive-doc endpoints returned 404s, suggesting no such interactive explorer exists.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   Foreloop—none0/10

   Evidence shows Foreloop exposes an MCP endpoint and CLI with --json output, but there is no mention of a downloadable OpenAPI spec or any machine-readable API specification document for its REST/API surface.

   ⚑ flag

   Devin—none0/10

   Devin has a documented API (devin-docs-8, devin-docs-9) but a direct probe for OpenAPI/swagger spec files at common paths returned 404s (devin-probe-2), and no docs page offers a downloadable machine-readable spec.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

3. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round to Foreloop
   Foreloop~partialclaimed3/10

   The docs mention a stability guarantee for CLI `--json` output ('has a stable shape. Additive changes only') and the MCP/API endpoint path includes a 'v1' segment, implying some versioning discipline, but there is no explicit documented deprecation policy, versioning scheme, or migration guidance for the API itself. missing for 10: explicit API version numbering scheme, documented deprecation timeline/process, changelog or migration guides, evidence beyond CLI JSON stability.

   • [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
   • [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”

   proof ↗⚑ flag

   Devin—none0/10

   There is an API reference (devin-docs-8) but no evidence of API versioning scheme or a documented deprecation policy; the openapi.json probe returned 404s, suggesting no formal machine-readable spec either. Missing for 10: versioning scheme documentation, deprecation policy/notice, changelog for breaking changes.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Devin
   Foreloop—none0/10

   Foreloop's docs describe per-task workflows (claim one task, agent does the work, opens one PR) and a CLI/MCP interface, but nothing in the evidence describes performing an operation across many items/tasks simultaneously (e.g., batch approve, bulk edit, multi-task claim). The axis is plausible for a CLI/automation tool but no evidence of bulk operation support exists.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
   • [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”

   proof ↗⚑ flag

   Devin~partialclaimed6/10

   Devin's docs describe running multiple Devins in parallel for independent tasks, an API for creating sessions programmatically (including on behalf of other users), and scheduling capabilities—together supporting bulk/automated task dispatch across many items. However, there's no concrete example or independent evidence of actually orchestrating large-scale bulk operations (e.g., processing hundreds of tickets at once), and community evidence focuses on single-task demos rather than bulk workflows. Missing for 10: hands-on/independent evidence of large-scale bulk runs, explicit parallelism limits, and a documented case study of bulk item processing.

   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Devin
   Foreloop—none0/10

   Foreloop's docs describe human-approved task workflows (agents claim tasks, but 'nothing runs until a person approves a contract'), not a rules engine for automatic event-triggered actions; the closest mention is CLI usability from cron, which is not evidence of a native rules/trigger system. missing for 10: any documented rule-definition syntax, event-trigger configuration, or automation-without-approval capability.

   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”

   proof ↗⚑ flag

   Devin~partialclaimed7/10

   Devin supports event-driven automation via scheduled/cron triggers (create schedules with cron expressions, notification preferences) and via Devin Review's Auto-Fix which automatically responds to code review comments and CI failures, plus API-based workflow automation. However, these are fixed built-in trigger types (schedule, CI/review events) rather than a general user-defined rules engine for arbitrary events. Missing for 10: evidence of a flexible custom rule/trigger builder covering arbitrary event sources (e.g., webhooks, ticket status changes) beyond scheduling and code-review auto-fix, and independent confirmation these triggers work reliably in practice.

   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round to Devin
   Foreloop~partialclaimed3/10

   The only relevant evidence is a single line noting the CLI can be run 'from anything that can run one, including coding agents and cron,' implying users could wire up their own cron-based scheduling — but Foreloop itself documents no native recurring-job/workflow scheduler, retry policy, or schedule management UI. Missing for 10: a first-party scheduling feature, documented recurring-workflow configuration, and evidence of built-in cron/interval management within the product.

   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”

   proof ↗⚑ flag

   Devin✓fullclaimed8/10

   Devin's MCP documentation explicitly states scheduling support with 'cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection,' and this scheduling capability is exposed via MCP for session/playbook management. Missing for 10: independent/hands-on corroboration of recurring job execution in practice and more detail on scheduling UI/reliability beyond the docs mention.

   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”

   proof ↗⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round to Foreloop
   Foreloop~partialclaimed4/10

   Foreloop's workflow requires human approval before any agent-driven change runs (foreloop-docs-7) and changes are delivered as GitHub pull requests (foreloop-docs-3, foreloop-docs-15), which inherently gives code-level versioning and review via git. However, there is no documentation of versioning, reviewing, or rolling back the automations/tasks/loops themselves (as opposed to the resulting code diffs) — no rollback command, task history, or automation-versioning feature is evidenced. Missing for 10: explicit automation/task versioning, a documented rollback mechanism for tasks or loops, and audit/history views beyond the live agent status list.

   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
   • [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”

   proof ↗⚑ flag

   Devin—none0/10

   No evidence of version history, review/diff UI, or rollback of Devin's automations (playbooks, schedules, sessions); docs mention creating/scheduling sessions and playbooks but never versioning, auditing changes, or reverting them. missing for 10: version history for playbooks/sessions, review/approval workflow for automation changes, rollback/undo mechanism.

   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “Knowledge is the best way to share codebase-level (vs. task-level) context that can help Devin when working in your codebase.”

   proof ↗⚑ flag

End to end feature delivery

1. ai-native userHave an agent automatically generate and run tests to validate its own code changes before proposing them

   weight 2 · round to Devin
   Foreloop—none0/10

   No evidence describes Foreloop's agents generating or running tests to validate code changes before proposing them; the docs describe task claiming, PR opening, and human approval gates but never mention automated test generation/execution as a validation step.

   ⚑ flag

   Devin~partialclaimed6/10

   Devin's docs describe autonomous CI-failure iteration and auto-fix behavior (devin-docs-20) and a terminal for executing/observing command output (devin-docs-6), implying it runs and reacts to test/build results before finalizing changes, but there is no explicit documentation of Devin proactively writing new tests and running them pre-emptively to self-validate before proposing a PR. missing for 10: explicit test-generation workflow documentation, evidence of Devin authoring test cases for new code, independent hands-on confirmation of this specific validation loop.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”

   proof ↗⚑ flag

2. developerHave an agent autonomously diagnose and fix a reported bug

   weight 3 · round to Foreloop
   Foreloop✓fullclaimed7/10

   Foreloop's documented workflow directly supports this: a bug report captured via the feedback widget (with page, metadata, screenshot) becomes an intention/task that an agent (Claude Code or Codex) claims, works on in a repo checkout, and resolves by opening a PR, without the reporter needing to know which files change. Human approval gates task initiation but the diagnosis-and-fix work itself is agent-driven end to end. Missing for 10: independent/hands-on evidence that agents actually diagnose root causes correctly, and more detail on how the agent investigates ambiguous bugs beyond the general task pipeline description.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
   • [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
   • [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
   • [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”

   proof ↗⚑ flag

   Devin!disputedcontradicted5/10

   Devin's docs explicitly describe autonomous bug repro/fix workflows, Slack-triggered bug sessions, and auto-fix on CI/review failures (devin-docs-1, devin-docs-4, devin-docs-19, devin-docs-20), supporting the core story. However, independent commentary cites a low real-world resolution rate (13.8% of issues resolved) and hands-on accounts describing shallow fixes (e.g., wrapping errors in try/catch rather than真正 diagnosing root cause), directly contradicting the strength of the autonomous-fix claim (devin-comm-2, devin-comm-6, devin-comm-4). Missing for 10: independent benchmark showing high success rate, more recent hands-on verification of bug-fix quality, and resolution of the discrepancy between vendor claims and community-reported performance.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
   • [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”
   • [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”

   proof ↗⚑ flag

3. product-managerGo from a mockup or design to a working implementation without an engineering handoff

   weight 2 · round to Foreloop
   Foreloop~partialclaimed6/10

   Foreloop lets a non-engineer describe a need in plain language (e.g. "Users can't tell which invoices are overdue") or flag a page via a feedback widget with screenshot, and an agent turns that into tasks and a PR without the PM needing to specify files — this covers a 'no engineering handoff' description-to-implementation flow. However, the evidence never shows ingesting an actual mockup/design file (e.g. Figma) as the starting artifact, and a human still must approve the contract before code changes run, so it's not a fully autonomous design-to-code path. Missing for 10: direct mockup/design-file import as an input, evidence of end-to-end completion without any review gate, and a demonstrated design-to-code case study.

   • [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
   • [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
   • [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”

   proof ↗⚑ flag

   Devin~partialclaimed5/10

   Devin's docs show it can autonomously implement features from tickets (Jira/Linear) and be tagged by non-engineers on Slack/Teams, supporting the general 'PM delegates, Devin implements' workflow, and it can connect to Figma via MCP integrations. However, there is no documented mockup-to-code pipeline, no PM-specific onboarding flow, and no evidence of turning a design file into a working UI end-to-end. missing for 10: direct design/mockup ingestion workflow, PM-focused case study or walkthrough, independent evidence of successful design-to-implementation outcomes.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

4. developerHave an agent implement a requested feature end-to-end, including writing tests

   weight 3 · round to Foreloop
   Foreloop~partialclaimed6/10

   Foreloop orchestrates tasks and lets an agent (Claude Code/Codex) claim a task, do the work, and open a PR in the user's repo, which covers end-to-end feature implementation via an external coding agent — but Foreloop itself doesn't write code or tests; it delegates that entirely to the connected agent, and there's no explicit mention of test-writing behavior or verification. missing for 10: explicit evidence that generated PRs include tests, independent/hands-on confirmation of successful end-to-end feature delivery, and detail on how test coverage is verified before approval.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”

   proof ↗⚑ flag

   Devin!disputedcontradicted5/10

   Devin's docs explicitly claim end-to-end feature implementation (docs-1, docs-31) and even test-writing is implied as part of full feature delivery, but independent hands-on reporting from the original demo showed only 13.8% of issues resolved and reviewers noted the work was often only ~90% correct or used hacky fixes (e.g. wrapping in try-catch) rather than proper fixes (devin-comm-2, devin-comm-4, devin-comm-6), directly contradicting the 'implements features end-to-end reliably' claim. missing for 10: verified success-rate data on test-writing specifically, more recent independent benchmarks showing improved reliability, and evidence tests are written/passed as part of the workflow rather than just code changes.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
   • [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”
   • [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”

   proof ↗⚑ flag

Environment setup

1. developerHave an agent automatically clone the repo, install dependencies, and configure its own working environment

   weight 2 · round to Devin
   Foreloop~partialclaimed4/10

   Foreloop's docs describe the developer running an agent 'in a checkout of your repository on your own machine' and using `foreloop init` to install a task-executor skill into that existing checkout — implying the human, not the agent, does the initial clone/setup, with the agent only claiming and working tasks afterward. There's no documentation of the agent autonomously cloning the repo, installing dependencies, or provisioning its own environment from scratch. Missing for 10: evidence of agent-initiated repo cloning, dependency installation steps, and autonomous environment configuration (e.g., a sandboxed or cloud checkout flow) rather than reliance on a pre-existing local checkout.

   • [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”

   proof ↗⚑ flag

   Devin✓fullclaimed8/10

   Devin's docs describe indexing repos, configuring a persistent environment/workspace that boots into a known-good state every session, VPN access to internal registries/git hosts, and AGENTS.md-based setup instructions it follows before coding — together these show it autonomously clones, configures, and prepares its own working environment. missing for 10: independent/hands-on verification of the clone+dependency-install flow end-to-end, and explicit documentation of automatic dependency installation step-by-step.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”

   proof ↗⚑ flag

Interactive takeover

1. developerTake over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work

   weight 2 · round to Devin
   Foreloop~partialclaimed4/10

   Foreloop lets agents run locally in a checkout and claim/report tasks, and humans can approve contracts or view agent status via a dashboard row per worker, but there's no evidence of a live handoff mechanism to pause, take over, or redirect an in-progress task from editor/terminal/browser mid-execution. missing for 10: explicit takeover/interrupt UX, editor-integrated control of a running agent, terminal command to seize an active task, and any hands-on confirmation of mid-task redirection.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
   • [claimed-docs] “Approve the big calls from your phone.”

   proof ↗⚑ flag

   Devin✓fullclaimed8/10

   Docs explicitly describe multi-surface handoff: delegate via web app then take over in Devin's IDE, use /handoff to send terminal tasks to cloud Devin, jump into the Interactive Browser to help Devin navigate, and tag Devin in Slack/Teams threads to redirect work — covering editor, terminal, and browser takeover. Missing for 10: independent/hands-on confirmation of a smooth mid-task handoff experience (community evidence is generic skepticism, not a concrete handoff failure).

   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”

   proof ↗⚑ flag

2. developerSend follow-up instructions to an active agent session to steer its work without restarting

   weight 2 · round to Devin
   Foreloop—none0/10

   The evidence describes agents claiming tasks, opening PRs, and reporting status, plus approval gates and skill-writing to correct recurring mistakes — but nothing describes sending live follow-up/steering instructions to an already-running agent session mid-task without restarting it.

   ⚑ flag

   Devin~partialclaimed5/10

   Devin's docs describe delegating tasks and then 'taking over in Devin's IDE' or 'jumping in' to help with browsing tasks, implying some interactivity with an active session, but there is no explicit documentation of sending follow-up chat instructions to steer an in-progress session without restarting it. missing for 10: explicit docs on mid-session chat/steering, confirmation that follow-up messages are incorporated live, independent hands-on evidence of this workflow.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”

   proof ↗⚑ flag

Sandbox execution

1. developerHave an agent safely execute code and install dependencies inside an isolated sandbox

   weight 3 · round to Devin
   Foreloop—none0/10

   Foreloop's docs explicitly describe agents running 'in a checkout of your repository on your own machine' (foreloop-docs-3), with no mention of sandboxing, containerization, or isolated execution environments for code/dependency installation. Missing for 10: any documentation of sandbox/container isolation, dependency install safety, or resource/network restrictions during agent execution.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”

   proof ↗⚑ flag

   Devin~partialclaimed6/10

   Docs describe Devin operating in its own workspace/VM with terminal execution, environment configuration ('known-good state'), VPN access to internal services, and multi-platform (Android/macOS/Windows) build environments, implying isolated sandboxed execution, but there is no explicit documentation of sandbox security guarantees, dependency isolation, or safety controls around code execution, and no independent/hands-on verification of sandbox isolation. missing for 10: explicit sandbox/isolation security documentation, independent verification of safe execution, details on dependency install safety and containment.

   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”
   • [claimed-docs] “Devin can build and run Android applications directly on its own machine — giving it the Android equivalent of Computer Use and browser inte…”
   • [claimed-docs] “Devin now has access to macOS virtual machines. This means Devin can now build and test iOS and macOS applications.”
   • [claimed-docs] “Devin supports Windows as a build and session platform.”

   proof ↗⚑ flag

Approval controls

1. developerConfigure an agent to auto-approve all its actions instead of confirming each one

   weight 2 · round drawn
   Foreloop—none0/10

   Foreloop's docs explicitly state that human approval of a contract is a deliberate, mandatory gate before any agent action runs ('Nothing runs until a person approves a contract. Approval is the only gate'), with no mention of a setting to bypass or auto-approve this gate. This indicates the product does not offer an auto-approve configuration, so the story's capability is unsupported.

   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “Approve the big calls from your phone.”

   proof ↗⚑ flag

   Devin—none0/10

   The evidence pack describes permission systems for MCP tools (devin-docs-13) and autonomous features like Auto-Fix (devin-docs-20), but there is no documentation of a setting that lets a developer toggle between 'confirm each action' and 'auto-approve all actions' for Devin's own task execution. missing for 10: explicit auto-approve/confirmation toggle documentation, evidence of a per-action approval workflow that can be disabled, any community confirmation of this specific control.

   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

2. product-managerApprove key agent decisions from my phone while agents continue working

   weight 1 · round to Foreloop
   Foreloop~partialclaimed5/10

   Foreloop explicitly markets phone-based approval of 'big calls' and documents that no agent work runs until a human approves a contract, which matches the core ask. However, evidence is limited to marketing taglines with no detail on the mobile UI/notification flow, how approvals interact with agents continuing to work in parallel, or independent corroboration. Missing for 10: concrete mobile-app/notification workflow details, evidence of multi-agent concurrent operation while awaiting approval, and third-party confirmation.

   • [claimed-docs] “Approve the big calls from your phone.”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”

   proof ↗⚑ flag

   Devin—none0/10

   No evidence describes a mobile app or phone-based approval interface for agent decisions; docs mention Slack/Teams tagging, web app, IDE, and scheduling notifications but nothing about approving decisions from a phone while sessions continue. The axis is plausible for an agentic dev tool with human-in-the-loop needs, but no supporting evidence exists.

   ⚑ flag

3. engineering-leadSet tiered autonomy levels controlling what an agent can do without manual confirmation

   weight 3 · round to Devin
   Foreloop—none0/10

   Foreloop documents a single approval gate before any agent work starts ('Nothing runs until a person approves a contract') and lets you approve from your phone, but there is no evidence of configurable tiered autonomy levels (e.g., auto-approving certain task types while requiring confirmation for others).

   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “Approve the big calls from your phone.”

   proof ↗⚑ flag

   Devin~partialclaimed4/10

   Devin documents a permission system for MCP/tool access that can be controlled 'at multiple levels' and features like Auto-Fix, human takeover in the IDE, and interactive browser intervention imply some control over autonomous action, but there is no documented feature specifically framed as configurable 'tiered autonomy levels' for an engineering lead to set organization-wide policies on unattended agent actions. missing for 10: explicit autonomy-tier configuration UI/API, organization-wide policy controls, and confirmation-threshold settings distinct from ad-hoc MCP tool permissions.

   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

Visibility monitoring

1. developerWatch what a running agent is doing in real time, including its current status

   weight 3 · round to Devin
   Foreloop~partialclaimed6/10

   The docs describe a Development/Agents view listing each worker's agent identity, checkout name, current task, and last message — a real-time-ish status view (foreloop-docs-8). However, there's no evidence of live streaming updates, logs, or granular in-progress activity beyond a status line, and no independent/hands-on confirmation of real-time refresh behavior. missing for 10: evidence of live/streaming updates or detailed activity logs, independent confirmation of real-time refresh, granularity beyond a single status line per agent.

   • [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”

   proof ↗⚑ flag

   Devin✓fullclaimed7/10

   Devin's docs describe real-time observability features like watching the terminal execute commands with live output logs, jumping in during interactive browsing, and taking over in Devin's IDE mid-task, all indicating a live view of the agent's current activity/status. Missing for 10: no explicit dedicated 'status dashboard' documentation and no independent/hands-on corroboration of the real-time monitoring UX beyond vendor docs.

   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

2. developerGet notified when an agent completes a task or needs my input

   weight 2 · round to Devin
   Foreloop~partialclaimed3/10

   Foreloop provides a dashboard showing agent status ('last thing it said') and mentions approving 'big calls from your phone', implying some notification/approval flow, but there is no explicit documentation of push/email/Slack notifications when a task completes or needs input. missing for 10: explicit notification channel (email/Slack/push) documentation, confirmation of real-time alerts rather than manual dashboard polling, evidence of notification content for 'needs input' events.

   • [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
   • [claimed-docs] “Approve the big calls from your phone.”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”

   proof ↗⚑ flag

   Devin~partialclaimed6/10

   Devin supports Slack/Teams tagging where sessions and results appear in-thread (devin-docs-4, devin-docs-19), and scheduled sessions explicitly support 'notification preferences' (devin-docs-17), implying some completion/status alerts. However there's no dedicated first-party documentation of a generic notify-on-completion or needs-input alert system (e.g., email/push) outside Slack/Teams and scheduling contexts. Missing for 10: explicit documentation of a unified notification mechanism across all session types, and independent/hands-on confirmation that notifications reliably fire when input is needed.

   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”

   proof ↗⚑ flag

Natural language task intake

1. developerDescribe a feature or bug in plain language and have it automatically turned into a scoped implementation task

   weight 3 · round to Foreloop
   Foreloop✓fullclaimed8/10

   Docs describe plain-language input ('Users can't tell which invoices are overdue') turning into scoped, approvable tasks via 'intentions' that an agent fills in and proposes as tasks, gated by human approval before an agent acts on the repo. This directly matches intent-to-spec workflow with first-party documentation across multiple pages. Missing for 10: independent/hands-on verification of the resulting task's scoping quality and no example of the full generated task artifact.

   • [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
   • [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”

   proof ↗⚑ flag

   Devin✓fullcommunity7/10

   Devin's docs describe accepting natural-language asks (tickets, Slack/Teams messages, delegated tasks) and turning them into sessions/implementation tasks, which is the core intent-to-spec workflow (devin-docs-1, devin-docs-4, devin-docs-5, devin-docs-19). Community commentary raises concerns about actual task success rates and correctness (devin-comm-2, devin-comm-4), tempering confidence in scoping quality. Missing for 10: independent hands-on evidence that the auto-generated task scope/plan is accurate or reviewable before execution, and no dedicated 'plan/spec preview' feature described.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
   • [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”

   proof ↗⚑ flag

2. product-managerConvert user feedback submissions into structured tasks with proposed scope

   weight 2 · round to Foreloop
   Foreloop✓fullclaimed7/10

   Docs show a feedback-collection widget that captures user submissions, and an agent-driven 'intention' workflow that reads the submission, fills in an intention, and proposes tasks (foreloop-docs-12), with human approval gating execution (foreloop-docs-7). This directly matches converting feedback into structured tasks with proposed scope, though evidence is entirely vendor docs with no independent/hands-on corroboration of output quality or scope structure. Missing for 10: independent/hands-on verification of the feedback-to-task pipeline, and concrete example of the 'proposed scope' structure/fields.

   • [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
   • [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”

   proof ↗⚑ flag

   Devin~partialclaimed5/10

   Devin can ingest raw inputs like Linear/Jira tickets and Slack/Teams threads about bugs or feature requests and turn them into an actionable session (devin-docs-1, devin-docs-4, devin-docs-19), which covers the 'feedback-to-task' intake path, but there is no evidence of a distinct structured-task output with an explicit proposed scope/plan artifact for PM review before execution. Missing for 10: evidence of a scoping/planning deliverable (e.g., a spec doc or task breakdown with effort/scope estimate) generated prior to coding, and any PM-specific workflow or UI for reviewing/approving that scope.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

3. developerAttach a marked-up screenshot or mockup to a task so the agent implements the correct visual change

   weight 2 · round to Foreloop
   Foreloop~partialclaimed4/10

   The feedback widget lets a customer point at an element and attach an optional screenshot with metadata so 'the model fixes the right thing' (foreloop-docs-21), which is adjacent to the story but is customer-bug-report flow, not a developer attaching an annotated mockup directly to a task. There's no documentation of markup/annotation tools, mockup uploads, or a developer-facing task-attachment workflow for visual specs. Missing for 10: developer-facing screenshot/mockup attachment to tasks, annotation/markup capability, and evidence the agent parses visual markup to implement precise UI changes.

   • [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”

   proof ↗⚑ flag

   Devin—none0/10

   No evidence in the pack mentions attaching screenshots, mockups, or marked-up images to a Devin task, nor any visual-diff/design-to-code workflow; documentation covers ticket delegation, terminal, Slack, MCP, and environment setup but nothing about image/mockup-based task intake.

   ⚑ flag

Plan approval

1. developerReview and approve an agent's implementation plan before any code changes are made

   weight 3 · round to Foreloop
   Foreloop✓fullclaimed8/10

   Docs explicitly describe a 'contract' approval gate before any agent touches the repository ('Nothing runs until a person approves a contract... an agent is about to change your repository'), and mention approving 'big calls' from a phone, indicating a review/approve workflow prior to code changes. Missing for 10: independent/hands-on confirmation of the approval UI showing the actual implementation plan content and evidence of what happens if a plan is rejected/revised.

   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “Approve the big calls from your phone.”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”

   proof ↗⚑ flag

   Devin—none0/10

   The evidence pack describes Devin's task delegation, IDE handoff, terminal visibility, and MCP integrations, but nowhere documents a plan-review/approval gate before Devin begins making code changes. Missing for 10: any doc mentioning an upfront plan proposal, explicit approval step, or 'plan mode' prior to execution.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”

   proof ↗⚑ flag

2. engineering-leadApprove a task's scope and contract before an agent is allowed to modify the repository

   weight 2 · round to Foreloop
   Foreloop✓fullclaimed8/10

   Foreloop's docs explicitly state approval of a contract is a mandatory gate before any agent touches the repository ('Nothing runs until a person approves a contract... an agent is about to change your repository'), and agents only claim 'approved tasks' from the repo. This directly matches the engineering-lead approval-gate story, reinforced by mobile approval flow. Missing for 10: independent/hands-on verification that the gate is enforced in practice (not just documented) and detail on what 'scope and contract' specifically contains.

   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Approve the big calls from your phone.”

   proof ↗⚑ flag

   Devin—none0/10

   The evidence pack shows delegation, review-after-the-fact, and IDE takeover workflows, but no documented mechanism where an engineering-lead must approve a task's scope/contract before Devin is allowed to start modifying the repository. Missing for 10: explicit pre-execution approval/gating workflow, scope/contract sign-off UI or API, evidence of blocking repo writes until lead approval.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”

   proof ↗⚑ flag

Ticket driven tasking

1. developerAssign a coding task to an agent directly from an existing issue or ticket

   weight 3 · round to Devin
   Foreloop—none0/10

   Foreloop's workflow starts from user feedback reports or free-text intentions that get turned into tasks an agent claims (foreloop-docs-4, foreloop-docs-12), but there is no evidence of importing or linking an existing GitHub issue/Jira ticket as the task source for an agent to pick up.

   ⚑ flag

   Devin✓fullclaimed8/10

   Docs explicitly state Devin can be asked to tackle Linear/Jira tickets, tagged on Slack/Teams threads about bugs to start sessions, and integrated via API/MCP to create sessions from external triggers, directly supporting assigning tasks from existing issues/tickets. Missing for 10: independent hands-on verification of ticket-to-task handoff and no direct product screenshot of ticket-linking UI in evidence.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Foreloop
   Foreloop~partialprobed6/10

   Foreloop ships a CLI covering projects, loops, intentions, stories and tasks with a stable `--json` output on every command, plus an MCP endpoint for agent clients, suggesting broad API/CLI parity with the UI. However, there's no explicit statement or independent confirmation that literally every UI action (e.g., approvals, feedback widget config, agent monitoring dashboard views) is exposed via API/CLI. missing for 10: explicit UI-to-API parity documentation or audit, independent/hands-on verification, coverage of approval workflow and dashboard views via API.

   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
   • [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
   • [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
   • [probe] “official CLI documented at https://foreloop.com/install”

   proof ↗⚑ flag

   Devin~partialprobed5/10

   Devin exposes a documented API (session creation, create_as_user_id) and an MCP server that surfaces session management, playbooks, knowledge, and scheduling (devin-docs-8,9,15,29), suggesting broad but not full parity with the web/IDE UI. There's no explicit claim or evidence that UI-only features like the Interactive Browser takeover, terminal handoff, VPN setup, or environment/emulation configuration are fully API-accessible, and no OpenAPI spec was found (devin-probe-2) to confirm completeness. Missing for 10: explicit vendor statement of API/UI parity, evidence that interactive/browser/VPN/environment features are API-controllable, and independent confirmation of parity.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   Foreloop—none0/10

   No evidence of a bulk data export feature or open-format export of projects/tasks/intentions for migration away from Foreloop; the closest hints (--json CLI output, MCP API access) are for programmatic access, not a documented full-data export/leave capability.

   ⚑ flag

   Devin—none0/10

   No evidence of any data export feature, open-format export, or account/data portability tooling in Devin's docs; evidence covers task delegation, MCP, CLI, and platform support but nothing about exporting session data, knowledge, or playbooks in open formats for migration away from the product.

   ⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Foreloop—none0/10

   No evidence of an open-source license or public source code repository; Foreloop is presented as a closed SaaS product with a CLI/widget/MCP server, but nothing indicates the source is available under an open license.

   ⚑ flag

   Devin—none0/10

   Devin is a closed-source commercial SaaS product; no evidence of any open-source license or public source repository is present in the evidence pack, and the axis is applicable since AI tooling could conceivably publish open-source components. missing for 10: any open-license source release, public repo link, or licensing statement.

   ⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   Foreloop—none0/10

   All evidence describes Foreloop as a hosted SaaS (with a CLI/MCP endpoint at foreloop.com/api) that agents connect to remotely; there is no mention of a self-hostable core server, on-prem deployment, or open-source release. missing for 10: any documentation of a self-hosted deployment option, container/server image, or license permitting self-hosting.

   ⚑ flag

   Devin—none0/10

   Devin is exclusively delivered as a cloud SaaS product (with local CLI/terminal front-ends that hand off to cloud sessions); no evidence anywhere in the docs or community discussion describes a self-hostable core engine or on-prem deployment option. Missing for 10: any documentation of self-hosted/on-prem deployment, container/binary distribution of the core agent, or licensing for self-hosting.

   • [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”

   proof ↗⚑ flag

Enterprise licensing

1. engineering-leadLicense an enterprise deployment with SSO and commercial support for organization-wide rollout

   weight 2 · round drawn
   Foreloop—none0/10

   No evidence in the pack mentions enterprise licensing, SSO, or commercial support tiers; all citations focus on CLI usage, MCP integration, and agent workflows. This is a plausible axis for a SaaS product like Foreloop (organizations often need SSO/enterprise support), so absence of evidence yields 'none' rather than 'na'.

   ⚑ flag

   Devin—none0/10

   No evidence pack items mention enterprise licensing, SSO, or commercial support offerings for Devin; only product feature docs and community sentiment are present. Missing for 10: SSO/SAML documentation, enterprise plan/pricing page, commercial support SLA details.

   ⚑ flag

Model flexibility

1. engineering-leadBring my own LLM or API key so agents run on the model of my choice

   weight 2 · round drawn
   Foreloop—none0/10

   The docs describe running third-party agents (Claude Code or Codex) locally against Foreloop's task API, but there's no evidence Foreloop lets a user configure or supply their own LLM/API key for Foreloop's own model usage (e.g., intention generation, feedback triage). Missing for 10: any documentation of model/provider selection, API key configuration UI, or support for alternate LLM backends.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”

   proof ↗⚑ flag

   Devin—none0/10

   Docs describe an internal 'Adaptive' model router and a /model command to pick among Devin's supported models, but there is no evidence of bringing a custom LLM or your own API key to run agents on a model of the engineering lead's choice.

   • [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
   • [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
   • [claimed-docs] “Adaptive is an intelligent model router that automatically selects the best AI model for each task.”

   proof ↗⚑ flag

Usage quotas

1. engineering-leadSee and manage plan-based daily task and concurrency limits for agent workflows

   weight 2 · round drawn
   Foreloop—none0/10

   No evidence in the pack mentions plan-based daily task limits, concurrency limits, or any pricing/plan management dashboard for engineering leads; the docs focus on agent workflow mechanics (task claiming, approvals, MCP) with no reference to usage caps or plan tiers.

   ⚑ flag

   Devin—none0/10

   No evidence pack item mentions plan tiers, daily task quotas, concurrency limits, or an admin console/dashboard for managing such limits; docs cover running parallel Devins and API session creation but not visibility or management of plan-based limits. Missing for 10: pricing/plan documentation, admin UI for concurrency/task caps, usage dashboards or alerts for limit management.

   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   Foreloop—none0/10

   No evidence anywhere in the pack mentions data residency, region selection, or storage location options; Foreloop's docs cover CLI, MCP, agents, and feedback widgets but never address where data is stored or any regional controls.

   ⚑ flag

   Devin—none0/10

   No evidence in the pack mentions data residency, regional storage options, or compliance controls for where session/data is stored; docs cover VPN connectivity, environments, and MCP but nothing about choosing a storage region. Missing for 10: any mention of data residency/region selection, compliance certifications (e.g., SOC2/GDPR data locality), or enterprise data storage controls.

   ⚑ flag

2. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Foreloop—none0/10

   No evidence pack items mention data retention policies, deletion controls, data export, or privacy settings for user/feedback data collected by Foreloop; all evidence focuses on agent workflows, CLI usage, and MCP integration.

   ⚑ flag

   Devin—none0/10

   No evidence in the pack addresses data retention policies, deletion controls, or privacy configuration options for Devin; the docs cover features, MCP, CLI, and environments but not data governance/retention settings.

   ⚑ flag

3. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Foreloop—none0/10

   No evidence in the pack mentions telemetry, usage tracking, analytics collection, or an opt-out mechanism for Foreloop; the docs cover CLI usage, agent workflows, and feedback widgets but never address data collection or privacy controls.

   ⚑ flag

   Devin—none0/10

   No evidence pack item mentions telemetry opt-out, analytics settings, or usage-tracking controls for Devin; nothing addresses this privacy axis at all.

   ⚑ flag

Chat integration

1. developerTag an agent in a chat thread to discuss and delegate a bug or task

   weight 2 · round to Devin
   Foreloop—none0/10

   Foreloop's workflow is built around agents polling for approved tasks/contracts and reporting status via a dashboard (docs-3, docs-7, docs-8), not around tagging or chatting with agents in a thread. No evidence describes a chat-thread interface or an @-mention mechanism to delegate work conversationally.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”

   proof ↗⚑ flag

   Devin✓fullclaimed8/10

   Docs explicitly describe tagging Devin in Slack/Teams threads to discuss and delegate bugs/tasks, starting sessions directly from conversations. Missing for 10: independent/hands-on corroboration of this specific chat-tagging workflow beyond vendor docs.

   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”

   proof ↗⚑ flag

Knowledge context

1. developerAdd a context file describing my codebase conventions so agents generate more relevant plans and code

   weight 3 · round to Devin
   Foreloop~partialclaimed4/10

   Foreloop's docs describe a 'skill' mechanism ('Write a skill when you notice yourself giving an agent the same correction twice') and an installed task-executor skill in the checkout, which loosely maps to encoding project-specific conventions for agents, but there's no explicit documentation of a dedicated context file (e.g., conventions.md) for describing codebase style/conventions. Missing for 10: explicit context-file feature, guidance on structuring codebase conventions, evidence that agents consume such a file specifically for improving plan/code relevance.

   • [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
   • [claimed-docs] “Write a skill when you notice yourself giving an agent the same correction twice.”

   proof ↗⚑ flag

   Devin✓fullclaimed9/10

   Devin explicitly supports AGENTS.md files placed in the project root that Devin reads before coding, plus a separate 'Knowledge' feature for sharing codebase-level conventions/context to improve plans and code generation. Missing for 10: independent/hands-on confirmation that this measurably improves plan/code relevance beyond vendor docs.

   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
   • [claimed-docs] “Knowledge is the best way to share codebase-level (vs. task-level) context that can help Devin when working in your codebase.”
   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”

   proof ↗⚑ flag

Project management integration

1. product-managerConnect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly

   weight 3 · round to Devin
   Foreloop—none0/10

   Foreloop's evidence covers GitHub integration, its own task/loop tracking, MCP server, and feedback widgets, but there is no mention of connecting to Jira, Linear, ClickUp, or Monday.com at all — Foreloop appears to use its own internal issue/task system rather than integrating external issue trackers.

   • [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”

   proof ↗⚑ flag

   Devin~partialclaimed6/10

   Devin explicitly supports Linear and Jira tickets natively (devin-docs-1) and its MCP extensibility (devin-docs-12, devin-docs-21) lets it connect to arbitrary tool servers including issue trackers, which could cover ClickUp/Monday.com. However, there's no first-party documentation of native ClickUp or Monday.com connectors, and no independent/hands-on evidence confirming ticket-management workflows actually work end-to-end via these integrations. missing for 10: explicit native ClickUp/Monday.com support docs, independent verification of issue-tracker workflows succeeding in practice.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”

   proof ↗⚑ flag

Version control integration

1. developerConnect a GitHub repository so an agent can access the code and open pull requests against it

   weight 3 · round to Foreloop
   Foreloop✓fullclaimed7/10

   Docs explicitly describe connecting GitHub so Foreloop can read code and agents can open PRs against it, with a full workflow (agent claims task in a checkout, does work, opens a PR from the user's GitHub account, gated by human approval). Missing for 10: independent/hands-on verification of the GitHub connection flow and no screenshots or third-party confirmation that PRs are actually opened successfully.

   • [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”

   proof ↗⚑ flag

   Devin~partialclaimed6/10

   Docs confirm repository indexing (public/private) and GitHub-repo access for context, plus 'Devin Review with Auto-Fix' responding to code review/CI, implying PR interaction, but there's no explicit documentation of a GitHub connection/auth flow or of Devin opening pull requests directly. missing for 10: explicit GitHub app/OAuth connection setup steps, explicit 'opens pull requests' documentation, independent/hands-on confirmation of PR creation.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “These tools let you explore and query documentation for any GitHub repository (public or private with authentication)”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”

   proof ↗⚑ flag

2. developerGrant an agent access to my repositories with a one-click install, without complex setup

   weight 2 · round drawn
   Foreloop~partialclaimed5/10

   Docs confirm connecting GitHub gives Foreloop read/PR access to repos (foreloop-docs-15) and that agents run against a checkout, but achieving this requires installing a global CLI, running foreloop init, configuring MCP with an API key, and running an agent locally — multiple manual steps rather than a true one-click flow. missing for 10: explicit one-click OAuth install flow, evidence of a simple GitHub App install button, independent confirmation of setup simplicity.

   • [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
   • [claimed-docs] “npm i -g foreloop # installs globally, so the CLI works in any directory”
   • [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
   • [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”

   proof ↗⚑ flag

   Devin~partialclaimed5/10

   Devin does support connecting to repos (indexing, GitHub-based session creation, VPN access to internal git hosts) and this is described as straightforward onboarding, but there's no evidence of a literal 'one-click install' flow like a GitHub App marketplace install — setup instead involves indexing, environment configuration, AGENTS.md, and knowledge onboarding steps. Missing for 10: explicit one-click/marketplace-style install flow, independent evidence of setup simplicity, and confirmation that repo access requires no additional configuration.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”
   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”

   proof ↗⚑ flag

Ci remediation

1. engineering-leadHave failed CI workflows automatically diagnosed and fixed with a proposed pull request

   weight 3 · round to Devin
   Foreloop—none0/10

   Foreloop's documented workflow is task/intention-based (feedback → task → agent → PR), with no evidence of CI pipeline integration, failed-build detection, or automated diagnosis-and-fix triggered by CI failures. The axis is plausible for an agent-driven PR tool, but nothing in the evidence pack shows Foreloop watching or reacting to CI status.

   ⚑ flag

   Devin✓fullclaimed7/10

   Devin explicitly documents 'Devin Review with Auto-Fix' that responds to code review comments, fixes flagged bugs, and iterates on CI failures, directly matching the story of auto-diagnosing and fixing failed CI with a proposed PR. However, this is vendor-only documentation with no independent/hands-on corroboration of the CI-fix workflow specifically. Missing for 10: independent verification of CI-fix accuracy/success rate, details on PR proposal format, and evidence of real-world reliability beyond the single docs line.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

2. developerTrigger an agent from CI/CD pipelines to fix a broken build or failing test

   weight 2 · round to Devin
   Foreloop—none0/10

   Foreloop's docs describe agents claiming human-approved tasks and opening PRs from a local checkout, with a CLI usable from "anything that can run one, including coding agents and cron" (foreloop-docs-16), but there is no evidence of CI/CD pipeline integration or of the agent being triggered specifically to fix broken builds or failing tests — its workflow centers on approved feature/task work, not automated CI failure remediation.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”

   proof ↗⚑ flag

   Devin~partialclaimed6/10

   Docs explicitly state Devin Review with Auto-Fix can iterate on CI failures and fix flagged bugs, and the Devin API lets you integrate Devin into applications/workflows and create sessions programmatically (including on behalf of users), which supports triggering it from automated pipelines. However, there's no explicit documented example of a CI/CD pipeline (e.g., GitHub Actions/Jenkins) directly invoking Devin on a failing build/test, and no independent/hands-on confirmation of this exact workflow. Missing for 10: explicit CI/CD trigger integration example, independent verification that pipeline-triggered fixes work reliably.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”

   proof ↗⚑ flag

Diff review

1. developerConfigure an agent to automatically open a pull request when its task completes

   weight 2 · round to Foreloop
   Foreloop✓fullclaimed8/10

   Docs clearly describe the agent workflow: it claims a task, does the work, and opens a pull request from the user's GitHub account, with GitHub connection enabling PR creation, gated by human approval of the contract before work starts. Missing for 10: independent/hands-on confirmation that PRs are reliably opened on task completion, and no detail on configuring PR-specific settings (e.g., target branch, draft PRs).

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”

   proof ↗⚑ flag

   Devin~partialclaimed5/10

   Devin is known to work within GitHub workflows and create PRs as part of its task completion (implied by its coding-agent nature and features like Auto-Fix responding to review comments/CI), but the evidence pack has no explicit documentation describing a configurable setting to auto-open a pull request upon task completion. missing for 10: explicit docs describing PR-creation-on-completion configuration, settings/toggle for auto-PR behavior, and independent confirmation of this specific workflow.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

2. developerReview a diff of an agent's changes and approve it before it becomes a pull request

   weight 3 · round to Devin
   Foreloop—none0/10

   Foreloop's 'approval' gate (foreloop-docs-7) is a pre-work contract approval, not a diff review after the agent finishes changes — the agent then 'opens a pull request from your GitHub account' directly (foreloop-docs-3), with no documented step where a person reviews the diff before the PR is created. No evidence describes a diff-preview/approve-before-PR workflow.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”

   proof ↗⚑ flag

   Devin~partialclaimed4/10

   Docs mention delegating a task and 'taking over in Devin's IDE once it gives you a good first draft' (devin-docs-5), implying some human checkpoint before final output, and Devin's terminal/log viewing lets you watch work in progress (devin-docs-6). However there is no explicit documentation of a diff-review/approve-before-PR gate — 'Devin Review with Auto-Fix' (devin-docs-20) is about post-PR review comment handling, not a pre-PR approval step. Missing for 10: explicit diff viewer with approve/reject before PR creation, documentation of a formal approval gate, and independent/hands-on confirmation of this workflow.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

Pr review automation

1. ai-native userHave incoming issues automatically triaged with severity suggested and routed to the right owner

   weight 2 · round drawn
   Foreloop—none0/10

   Foreloop's docs describe turning user feedback into intentions/tasks that agents pick up and claim (foreloop-docs-12, foreloop-docs-4), but there is no evidence of automatic severity classification or routing to a specific human owner — tasks are self-claimed by agents rather than triaged and assigned. missing for 10: severity scoring, owner/routing logic, any triage workflow evidence.

   • [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
   • [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”

   proof ↗⚑ flag

   Devin—none0/10

   Devin can be pointed at Linear/Jira tickets or tagged in Slack to work a bug, but nothing in the evidence describes automatic triage logic that assesses severity or routes issues to the correct owner — these are core, specific parts of the story. Missing for 10: severity classification/scoring, ownership/routing logic, any 'triage' feature or automated inbox-sorting workflow.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”

   proof ↗⚑ flag

Readiness checks

1. engineering-leadAutomatically fix failing agent-readiness criteria in my repository

   weight 1 · round to Devin
   Foreloop—none0/10

   Foreloop's docs describe a task/feedback pipeline where agents claim approved tasks and open PRs, but nothing in the evidence pack mentions detecting or automatically remediating 'agent-readiness criteria' (e.g., AGENTS.md, tool-permission manifests, readiness scoring) in a repository. Missing for 10: any readiness-criteria scanning/grading feature, evidence of automatic detection of failing criteria, and evidence of automated fixes tied to such criteria.

   ⚑ flag

   Devin~partialclaimed6/10

   Devin Review with Auto-Fix explicitly auto-fixes CI failures and review comments, which could cover agent-readiness gate checks if they're wired into CI/PR review, and AGENTS.md support shows Devin can consume agent-readiness config files. However there's no evidence of a dedicated 'agent-readiness criteria' scorecard or targeted remediation workflow distinct from generic CI/review auto-fix. Missing for 10: explicit agent-readiness scoring/criteria detection, evidence of automatically remediating specific agent-readiness failures (vs generic CI/review comments), and independent validation of this auto-fix loop working reliably.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”

   proof ↗⚑ flag

2. engineering-leadRun a readiness report that evaluates how ready my repository is for autonomous agents

   weight 2 · round drawn
   Foreloop—none0/10

   No evidence in the pack describes a 'readiness report' or any repository-readiness scoring/assessment feature for autonomous agents; Foreloop's docs cover task claiming, MCP connectivity, feedback widgets, and approval gates, but nothing about evaluating repo readiness.

   ⚑ flag

   Devin—none0/10

   No evidence of any 'readiness report' or scoring feature that evaluates a repository's suitability for autonomous agents; docs cover onboarding features like AGENTS.md, environment setup, indexing, and knowledge, but nothing that generates a readiness assessment/report artifact for engineering leads.

   ⚑ flag

Security remediation

1. engineering-leadHave security alerts automatically validated and remediated with an opened pull request

   weight 2 · round to Devin
   Foreloop—none0/10

   Foreloop's evidence describes a general feedback/task-to-PR agent workflow (bug reports, feature requests) with human approval gates, but nothing in the pack mentions ingesting security alerts (e.g., from SAST/SCA/dependency scanners), validating them, or triggering remediation PRs specifically for vulnerabilities. The axis is plausible for a task-automation platform, but no evidence shows this specific capability.

   ⚑ flag

   Devin~partialclaimed5/10

   Devin's 'Review with Auto-Fix' auto-responds to code review comments, fixes flagged bugs and iterates on CI failures, and MCP integrations extend it to security-adjacent tools like Sentry and Datadog, implying it could react to alerts and open a PR fix. However, there is no explicit documentation of a security-scanner (SAST/dependency-alert) validation-and-remediation pipeline or evidence of an actual security-alert-to-PR workflow. Missing for 10: explicit security-scanner (e.g., CodeQL/Dependabot/Snyk) integration, evidence of validating true-positive vs false-positive alerts, and a documented end-to-end 'alert triggers PR' example.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”

   proof ↗⚑ flag

Concurrent execution

1. engineering-leadRun many agent tasks concurrently to scale delivery throughput

   weight 3 · round to Devin
   Foreloop~partialclaimed4/10

   The docs show that multiple agent workers can each claim tasks and run in parallel checkouts, with a dashboard listing one row per worker (foreloop-docs-8), implying some capacity for concurrent agent execution across a team. However, there is no explicit product feature for orchestrating, scaling, or managing many concurrent tasks at once, no throughput/scale claims, and no evidence of built-in concurrency limits or fleet management tooling. missing for 10: explicit support/documentation for running many agents simultaneously at scale, concurrency controls, throughput metrics, or a scheduler/queue for parallel task dispatch.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”

   proof ↗⚑ flag

   Devin~partialclaimed7/10

   Devin docs explicitly document running multiple Devins in parallel on independent tasks and provide an API with create_as_user_id for programmatically spawning sessions on behalf of org members, directly supporting concurrent scaling of throughput. However, there's no first-party data on concurrency limits, cost/resource management at scale, or independent case studies validating throughput gains at team scale; missing for 10: quantified concurrency limits/scaling guidance, independent evidence of large-scale parallel usage, and cost/governance controls for many simultaneous sessions.

   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”

   proof ↗⚑ flag

2. engineering-leadCreate agent sessions on behalf of other users in my organization

   weight 2 · round to Devin
   Foreloop—none0/10

   The evidence describes agents run locally by an individual user in their own repo checkout, claiming tasks and opening PRs from their own GitHub account, but nothing describes an engineering-lead provisioning or launching agent sessions on behalf of other org members. Missing for 10: any admin/org-level session creation API or UI, delegated-session documentation, or multi-user session management evidence.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
   • [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”

   proof ↗⚑ flag

   Devin✓fullclaimed8/10

   Devin's API explicitly supports creating sessions on behalf of any user in the organization via the create_as_user_id parameter, directly matching the engineering-lead use case for org-wide session delegation. Missing for 10: independent/hands-on corroboration of this specific feature and details on permission scoping/audit controls for delegated sessions.

   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”

   proof ↗⚑ flag

Deployment flexibility

1. developerUse a managed cloud offering to run agents without operating my own backend infrastructure

   weight 2 · round to Devin
   Foreloop—none0/10

   Foreloop's own docs state the opposite of this story: agents (Claude Code or Codex) run 'in a checkout of your repository on your own machine,' with Foreloop only coordinating tasks and approvals — there is no evidence of a managed cloud runtime that executes agents on Foreloop's infrastructure.

   • [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
   • [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”

   proof ↗⚑ flag

   Devin✓fullclaimed7/10

   Devin is delivered as a cloud-hosted agent (web app, API, session creation on behalf of users) so developers don't need to run their own backend/agent infrastructure — sessions run on Devin's managed cloud with its own configured environment, VPN access, and parallel execution. missing for 10: independent/hands-on confirmation of the managed infrastructure model, and explicit documentation of uptime/scaling/ops guarantees for the hosted service.

   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”

   proof ↗⚑ flag

Headless automation

1. developerRun an agent headlessly inside CI/CD pipelines and shell scripts

   weight 2 · round to Devin
   Foreloop~partialclaimed5/10

   Docs mention the CLI is invocable from 'anything that can run one, including coding agents and cron' and support scriptable --json output, suggesting headless invocation is possible (foreloop-docs-16, foreloop-docs-9). However, other docs state that 'nothing runs until a person approves a contract' (foreloop-docs-7), implying a human-in-the-loop gate that complicates fully unattended CI/CD execution, and there is no explicit CI/CD pipeline example or corroboration. Missing for 10: explicit CI/CD/pipeline example, evidence of non-interactive approval flow, independent confirmation of headless operation.

   • [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
   • [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
   • [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”

   proof ↗⚑ flag

   Devin~partialprobed6/10

   Devin's API is explicitly designed to 'integrate Devin into your applications, automate workflows, and build powerful tools' and supports creating sessions on behalf of users, which enables scripted/headless invocation, and a CLI is also documented. However, there is no explicit CI/CD pipeline example, exit-code/status handling, or GitHub Actions/shell-script integration guide in the evidence — the closest CI-related capability shown is 'Auto-Fix' reacting to CI failures rather than Devin running as a CI step itself. missing for 10: explicit CI/CD pipeline integration docs (e.g., GitHub Actions/shell script examples), documented headless/non-interactive CLI invocation with exit codes.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”

   proof ↗⚑ flag