Devin vs HumanLayer

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Devin
   Devin✓fullprobed7/10

   Devin explicitly supports AGENTS.md, an agent-oriented documentation file that Devin reads before starting a coding session to gain project context — directly matching the story's intent of pointing an agent at agent-oriented docs. Devin's own docs site also serves an llms.txt endpoint (verified via probe), showing familiarity with the convention, though this is Devin's own site rather than evidence Devin ingests arbitrary llms.txt files from user projects. Missing for 10: explicit documentation that Devin consumes/parses llms.txt files themselves (as opposed to AGENTS.md) and independent hands-on confirmation of this workflow.

   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.devin.ai/llms.txt # Devin Docs - [Desktop (100 pages)](https://docs.devin.ai/_llms/en/desktop.md):…”

   proof ↗⚑ flag

   HumanLayer—none0/10

   A direct probe for llms.txt returned 404, and no evidence pack item mentions agent-oriented documentation formats or an alternative machine-readable docs endpoint.

   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round drawn
   Devin✓fullclaimed8/10

   Devin exposes a full REST API for session creation and automation (devin-docs-8, devin-docs-9), supports scheduling recurring/one-time headless runs via MCP (devin-docs-17), auto-triggers on CI failures and code review comments via Devin Review/Auto-Fix (devin-docs-20), and supports parallel/headless session orchestration (devin-docs-18). This covers CI/automation use cases well beyond interactive use. Missing for 10: no independent/hands-on CI-integration case study, and no dedicated 'CI runner' or GitHub Actions-style first-class integration doc beyond the review/auto-fix hook.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”

   proof ↗⚑ flag

   HumanLayer✓fullprobed8/10

   Docs explicitly describe running HumanLayer headlessly via `humanlayer automation run` from CI jobs, cron machines, or scripts, plus launch tokens for non-interactive/non-PTY environments, and remote daemon hosts on cloud VMs or servers, directly matching the CI/automation story. Missing for 10: independent/hands-on confirmation of CI usage and more detail on auth/config specifics for automated pipelines.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to Devin
   Devin✓fullprobed9/10

   Devin CLI explicitly supports connecting external MCP servers, exposing their tools as slash commands with permission controls, and docs list example integrations (Datadog, Sentry, databases, Figma, Notion, Stripe, etc.). Missing for 10: independent/hands-on community verification of MCP tool usage in practice.

   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “Devin CLI exposes each one as a slash command”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
   • [probe] “official MCP server documented at https://docs.devin.ai/cli/extensibility/mcp/overview”

   proof ↗⚑ flag

   HumanLayer—none0/10

   No evidence anywhere in the pack mentions MCP servers or the ability to plug external tool servers into HumanLayer; integrations mentioned (Jira, Slack, GitHub, Linear) are task-source connectors, not MCP tool servers. Missing for 10: any mention of MCP protocol support, MCP server configuration, or tool-plugin mechanism.

   ⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round to Devin
   Devin✓fullclaimed8/10

   Devin ships an official MCP server (documented at docs.devin.ai/work-with-devin/devin-mcp) that exposes session management, playbooks, knowledge, and scheduling to any MCP-compatible agent or IDE — this is distinct from Devin's own MCP-client capability and directly satisfies the 'connect an agent via an official MCP server' story. Missing for 10: independent/hands-on confirmation that external agents successfully connect and operate through this server, and more detail on setup/auth flow beyond first-party docs.

   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “These tools let you explore and query documentation for any GitHub repository (public or private with authentication)”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer is a platform/control-plane for running and overseeing coding-agent sessions (Claude Code, Codex) rather than itself being an agent, so an official MCP server is a fair, applicable axis. The evidence pack documents many integrations (Jira, Slack, GitHub, Linear), a CLI, and remote daemons, but no MCP server offering is mentioned anywhere, and API/OpenAPI probes returned 404s. missing for 10: any first-party MCP server documentation, endpoint, or 'mcp serve' style capability.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to Devin
   Devin✓fullprobed8/10

   Devin ships an official CLI documented extensively (adaptive model routing, slash commands, MCP extensibility) confirmed by both docs and a probe hit, plus a separate Terminal/local CLI mode for quick fixes. Missing for 10: independent hands-on review of the CLI itself and more detail on installation/versioning.

   • [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
   • [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
   • [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “Devin CLI exposes each one as a slash command”
   • [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”

   proof ↗⚑ flag

   HumanLayer✓fullprobed7/10

   HumanLayer documents an official CLI (e.g. `humanlayer automation run`, launch tokens, remote-daemon control) used for CI, cron, and scripted agentic workflows, confirmed by a dedicated CLI doc page. missing for 10: no independent/hands-on verification of the CLI, no full command reference, and no evidence of broader CLI feature parity with the app.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to Devin
   Devin✓fullprobed8/10

   Devin publishes a documented public API for integrating into applications, automating workflows, and creating sessions programmatically (including on behalf of other users), plus an MCP server exposing session/playbook/knowledge/scheduling management to external agents. Missing for 10: a discoverable machine-readable OpenAPI/swagger spec (probe found 404s) and independent third-party corroboration of API usage beyond vendor docs.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.devin.ai/llms.txt # Devin Docs - [Desktop (100 pages)](https://docs.devin.ai/_llms/en/desktop.md):…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   HumanLayer~partialprobed4/10

   HumanLayer documents a CLI (`humanlayer automation run`, launch tokens, remote daemons) that lets automation environments drive sessions programmatically, which is a form of documented programmatic control, but there is no evidence of a documented public REST/OpenAPI API — probes explicitly found openapi.json/swagger.json/llms.txt all 404. missing for 10: a documented HTTP/OpenAPI public API spec, SDK/client library docs, and independent confirmation of API usage.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   Devin~partialclaimed3/10

   Devin documents a permission system that restricts MCP tool access at multiple levels and an API parameter (create_as_user_id) for acting on behalf of specific users, but there is no direct documentation of issuing scoped/least-privilege API keys or credentials for the agent itself. Missing for 10: explicit API key/token scoping mechanism, documented permission tiers or roles for API credentials, and any least-privilege credential issuance workflow.

   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed3/10

   The only relevant evidence is a mention of a 'launch token' scoped to a single non-interactive command, which implies some least-privilege token issuance, but there is no documentation of a broader credential/permission model, scopes, or API key management for agents. missing for 10: explicit least-privilege credential scoping model, permission granularity, revocation/rotation mechanisms, and any independent corroboration.

   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Devin
   Devin✓fullprobed8/10

   Devin publishes a documented API reference for building integrations (create sessions, manage on behalf of users, automate workflows), plus MCP support for extensibility — clear official SDK-like surface for AI-native developers. Missing for 10: no official language-specific SDK packages (e.g. Python/JS client libraries) are cited, only REST API docs and no discoverable OpenAPI spec (probe found 404s), and no independent/hands-on corroboration of API usage.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   HumanLayer—none0/10

   The evidence pack covers HumanLayer's CLI, workspace config, and third-party integrations (Jira, Slack, GitHub, Linear) but contains no mention of an official SDK (Python, TypeScript, etc.) for building against HumanLayer programmatically. Probes for openapi.json and llms.txt both returned 404, further suggesting no discoverable API/SDK surface.

   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Devin—none0/10

   No evidence in the pack mentions webhooks or event subscription mechanisms for Devin; only API session creation, MCP integrations, and CLI features are documented. As an API-driven agentic product, webhook support would be a plausible and expected capability, so its absence counts as 'none' rather than 'na'.

   ⚑ flag

   HumanLayer—none0/10

   No evidence pack item mentions webhooks or event subscription mechanisms; integrations described (Slack, Jira, Linear, GitHub) are inbound task-creation connectors, not outbound webhook events, and API/OpenAPI probes returned 404s.

   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Devin
   Devin~partialclaimed6/10

   Devin's repo indexing enables 'Ask Devin' and DeepWiki features that generate AI insights and explanations from your codebase data, and Devin can proactively suggest fixes/improvements via code review auto-fix and bug triage. However, evidence is scoped only to codebase/ticket data, not broader business or analytics data, and there's no dedicated insights dashboard or proactive summarization feature documented. Missing for 10: evidence of insights beyond code (e.g., analytics/dashboards), independent corroboration of Ask Devin/DeepWiki quality, and any UI showcasing generated insights.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”

   proof ↗⚑ flag

   HumanLayer~partialclaimed4/10

   HumanLayer's agents do generate task artifacts, draft PRs, and comments derived from a user's codebase/tickets, and 'Advanced Metrics' surfaces usage/cost/productivity data, which loosely resembles data-derived insight. However there is no evidence of dedicated AI-generated analytical insights or proactive suggestions distinct from executing assigned coding tasks. Missing for 10: explicit insight/analytics generation from user data, proactive suggestion features beyond task execution, and any first-party or community evidence of an 'insights' capability.

   • [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”
   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Devin
   Devin✓fullclaimed8/10

   Devin supports scheduling via cron expressions (recurring/one-time) and agent selection through its MCP tooling, plus API session creation and Slack/Teams triggering, enabling autonomous background automations beyond interactive chat sessions. missing for 10: independent/hands-on verification of scheduled runs actually executing unattended, and more detail on failure/retry handling for scheduled tasks.

   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed7/10

   Docs explicitly describe `humanlayer automation run` for running Cloud-visible coding sessions from CI jobs, cron machines, or scripts, plus remote daemons and launch tokens for non-interactive/background execution contexts, directly supporting autonomous background automations. Missing for 10: independent/hands-on verification of long-running background automations, native scheduling UI, and clarity on how human-approval gates interact with continuous autonomous runs.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”

   proof ↗⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round to Devin
   Devin✓fullcommunity8/10

   Devin itself functions as the built-in AI assistant to which tasks are delegated across web app, terminal, Slack/Teams, and API, with rich first-party documentation of the delegation workflow (docs-1,3,4,5,8,19). Community feedback raises reliability/skepticism concerns but does not concretely contradict that delegation itself works, so it's not disputed. Missing for 10: independent hands-on validation of delegation reliability/success rate beyond old demo critiques.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed7/10

   HumanLayer's core product model is task delegation to AI coding agents (Claude Code, Codex, Bedrock-backed Claude) via 'sessions', with workflow modes (Oneshot, RPI, PRD-Oriented, Freeform) and automation runs from CI/cron, directly matching 'delegate tasks to a built-in AI assistant'. missing for 10: independent/hands-on verification of the delegation experience beyond vendor docs, and clarity on how autonomous vs supervised the assistant is in practice.

   • [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”

   proof ↗⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round to Devin
   Devin✓fullcommunity8/10

   Devin is fundamentally driven by natural-language task requests across web app, Slack/Teams tagging, CLI, and API (create_as_user_id sessions), letting users describe tasks like bug fixes, features, and migrations in plain language rather than issuing structured commands. Community evidence corroborates that Devin operates via conversational/task-based prompts even amid skepticism about output quality. missing for 10: independent hands-on confirmation of NL command fidelity across all surfaces (CLI/Slack/API), and no evidence addressing ambiguous or multi-step NL instruction handling limits.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed7/10

   HumanLayer's core interaction model is giving natural-language instructions to agent sessions (Claude Code, Codex) to create tasks, configure workspaces, and choose workflow phases, as shown by the example NL workspace-config prompt and workflow-selection docs. Missing for 10: independent/hands-on corroboration of NL command robustness and no evidence of a broader NL command surface beyond task/workflow setup.

   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Devin—none0/10

   Devin documents an API reference overview and parameters (e.g. create_as_user_id) but nothing describes an interactive, in-browser 'try it' experience with runnable code examples; a probe for standard OpenAPI/Swagger interactive-doc endpoints returned 404s, suggesting no such interactive explorer exists.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   HumanLayer—none0/10

   No evidence of an interactive API reference or runnable examples; probes explicitly show no OpenAPI/swagger spec and no llms.txt found, and docs are guide/tutorial style rather than an API reference sandbox.

   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   Devin—none0/10

   Devin has a documented API (devin-docs-8, devin-docs-9) but a direct probe for OpenAPI/swagger spec files at common paths returned 404s (devin-probe-2), and no docs page offers a downloadable machine-readable spec.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   HumanLayer—none0/10

   A direct probe for OpenAPI/swagger specs at all standard locations returned 404s, and no evidence pack item shows a downloadable machine-readable API spec being offered.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Devin
   Devin~partialclaimed3/10

   Devin's docs describe each session running in its own configurable workspace/environment (devin-docs-25) and connecting via VPN to reach internal services rather than being embedded in production (devin-docs-23), which implies some environment isolation, but there is no explicit claim of a dedicated 'sandbox vs production' testing mode or safeguards against touching production data. missing for 10: explicit documentation of a production-safe/sandboxed test environment, evidence of data isolation guarantees, and independent confirmation that production systems are never touched.

   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's docs describe remote daemons, workspaces, and automation sessions, but there is no mention of a sandbox environment, staging/test data isolation, or any mechanism to separate test runs from production data. missing for 10: explicit sandbox/staging environment documentation, data isolation guarantees, evidence of test-vs-production separation.

   ⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Devin—none0/10

   There is an API reference (devin-docs-8) but no evidence of API versioning scheme or a documented deprecation policy; the openapi.json probe returned 404s, suggesting no formal machine-readable spec either. Missing for 10: versioning scheme documentation, deprecation policy/notice, changelog for breaking changes.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   HumanLayer—none0/10

   No evidence of API versioning scheme or a documented deprecation policy; probes for openapi.json/llms.txt returned 404s and no API reference or changelog covering versioning/deprecation is present.

   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Devin
   Devin~partialclaimed6/10

   Devin's docs describe running multiple Devins in parallel for independent tasks, an API for creating sessions programmatically (including on behalf of other users), and scheduling capabilities—together supporting bulk/automated task dispatch across many items. However, there's no concrete example or independent evidence of actually orchestrating large-scale bulk operations (e.g., processing hundreds of tickets at once), and community evidence focuses on single-task demos rather than bulk workflows. Missing for 10: hands-on/independent evidence of large-scale bulk runs, explicit parallelism limits, and a documented case study of bulk item processing.

   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's documentation consistently frames work as single tasks/sessions ('one small task', 'one task on a remote machine', 'one Cloud-visible coding session') with per-task review and approval workflows; there is no mention of batch/bulk operations spanning many items at once. missing for 10: any documented bulk-action API/CLI flag, batch approval mechanism, or multi-item automation workflow.

   • [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Devin
   Devin~partialclaimed7/10

   Devin supports event-driven automation via scheduled/cron triggers (create schedules with cron expressions, notification preferences) and via Devin Review's Auto-Fix which automatically responds to code review comments and CI failures, plus API-based workflow automation. However, these are fixed built-in trigger types (schedule, CI/review events) rather than a general user-defined rules engine for arbitrary events. Missing for 10: evidence of a flexible custom rule/trigger builder covering arbitrary event sources (e.g., webhooks, ticket status changes) beyond scheduling and code-review auto-fix, and independent confirmation these triggers work reliably in practice.

   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer integrations (GitHub, Jira, Linear) create tasks automatically from external events like new issues/tickets, and `humanlayer automation run` lets sessions be triggered from CI jobs, cron, or scripts — both are forms of event-driven automation. However, there's no evidence of a general-purpose rules/conditions engine letting users define arbitrary trigger-condition-action logic; the automation is limited to fixed integration hooks and script-based invocation. Missing for 10: a documented rule-definition interface (conditions, filters, custom triggers) beyond fixed platform integrations, and independent confirmation these event-triggers work reliably in practice.

   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round to Devin
   Devin✓fullclaimed8/10

   Devin's MCP documentation explicitly states scheduling support with 'cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection,' and this scheduling capability is exposed via MCP for session/playbook management. Missing for 10: independent/hands-on corroboration of recurring job execution in practice and more detail on scheduling UI/reliability beyond the docs mention.

   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed4/10

   HumanLayer's `automation run` CLI lets you trigger a Cloud-visible coding session from a cron machine or CI job, implying recurring runs are possible via external schedulers, but there is no documented native scheduling/recurrence feature (no cron syntax, interval config, or job queue) inside HumanLayer itself. missing for 10: built-in recurring scheduler, interval/cron configuration options, evidence of persistent recurring workflow management.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round to HumanLayer
   Devin—none0/10

   No evidence of version history, review/diff UI, or rollback of Devin's automations (playbooks, schedules, sessions); docs mention creating/scheduling sessions and playbooks but never versioning, auditing changes, or reverting them. missing for 10: version history for playbooks/sessions, review/approval workflow for automation changes, rollback/undo mechanism.

   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “Knowledge is the best way to share codebase-level (vs. task-level) context that can help Devin when working in your codebase.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   HumanLayer's task model provides review (comments, PR draft creation) and history that persists across sessions, giving some review/audit capability, but there is no explicit documentation of versioning workflow definitions or rolling back an automation to a prior version. missing for 10: explicit automation versioning/history diffing, a documented rollback mechanism, and independent confirmation these review features extend to full automation lifecycle management.

   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”

   proof ↗⚑ flag

End to end feature delivery

1. ai-native userHave an agent automatically generate and run tests to validate its own code changes before proposing them

   weight 2 · round to Devin
   Devin~partialclaimed6/10

   Devin's docs describe autonomous CI-failure iteration and auto-fix behavior (devin-docs-20) and a terminal for executing/observing command output (devin-docs-6), implying it runs and reacts to test/build results before finalizing changes, but there is no explicit documentation of Devin proactively writing new tests and running them pre-emptively to self-validate before proposing a PR. missing for 10: explicit test-generation workflow documentation, evidence of Devin authoring test cases for new code, independent hands-on confirmation of this specific validation loop.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”

   proof ↗⚑ flag

   HumanLayer—none0/10

   No evidence in the pack describes HumanLayer agents automatically writing or executing tests to self-validate code changes; docs cover task/session management, integrations, and workflow phases but not test generation/execution as a validation step.

   ⚑ flag

2. developerHave an agent autonomously diagnose and fix a reported bug

   weight 3 · round to HumanLayer
   Devin!disputedcontradicted5/10

   Devin's docs explicitly describe autonomous bug repro/fix workflows, Slack-triggered bug sessions, and auto-fix on CI/review failures (devin-docs-1, devin-docs-4, devin-docs-19, devin-docs-20), supporting the core story. However, independent commentary cites a low real-world resolution rate (13.8% of issues resolved) and hands-on accounts describing shallow fixes (e.g., wrapping errors in try/catch rather than真正 diagnosing root cause), directly contradicting the strength of the autonomous-fix claim (devin-comm-2, devin-comm-6, devin-comm-4). Missing for 10: independent benchmark showing high success rate, more recent hands-on verification of bug-fix quality, and resolution of the discrepancy between vendor claims and community-reported performance.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
   • [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”
   • [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”

   proof ↗⚑ flag

   HumanLayer~partialcommunity6/10

   HumanLayer supports creating tasks directly from GitHub/Jira/Linear issues and then running an agent session (Claude Code/Codex) against the linked repo, with an 'Oneshot' workflow phase designed for small, clear changes — a plausible bug-fix pipeline. However, the product's core design is human-in-the-loop with approval gates rather than fully autonomous action, and there's no end-to-end documented example of an agent independently diagnosing a bug from a ticket and shipping a fix without human review. Missing for 10: a concrete autonomous bug-diagnosis-and-fix walkthrough, and clarity on how much human approval is required mid-flow.

   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”

   proof ↗⚑ flag

3. product-managerGo from a mockup or design to a working implementation without an engineering handoff

   weight 2 · round to Devin
   Devin~partialclaimed5/10

   Devin's docs show it can autonomously implement features from tickets (Jira/Linear) and be tagged by non-engineers on Slack/Teams, supporting the general 'PM delegates, Devin implements' workflow, and it can connect to Figma via MCP integrations. However, there is no documented mockup-to-code pipeline, no PM-specific onboarding flow, and no evidence of turning a design file into a working UI end-to-end. missing for 10: direct design/mockup ingestion workflow, PM-focused case study or walkthrough, independent evidence of successful design-to-implementation outcomes.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's evidence is entirely about developer-facing workflows: running coding sessions, connecting Jira/Linear/GitHub/Slack, managing remote daemons, and CLI automation for engineers overseeing coding agents. Nothing in the pack shows a mockup/design import capability, a no-code interface, or any path for a non-engineer product manager to turn a design into a working implementation without engineering involvement — in fact the workflow language (RPI, PRD-oriented, Oneshot) and workspace configs assume an engineering operator. Missing for evidence of delivering this story: mockup/design ingestion, PM-oriented no-code UI, and any case study of a non-engineer shipping code end-to-end.

   • [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”

   proof ↗⚑ flag

4. developerHave an agent implement a requested feature end-to-end, including writing tests

   weight 3 · round to HumanLayer
   Devin!disputedcontradicted5/10

   Devin's docs explicitly claim end-to-end feature implementation (docs-1, docs-31) and even test-writing is implied as part of full feature delivery, but independent hands-on reporting from the original demo showed only 13.8% of issues resolved and reviewers noted the work was often only ~90% correct or used hacky fixes (e.g. wrapping in try-catch) rather than proper fixes (devin-comm-2, devin-comm-4, devin-comm-6), directly contradicting the 'implements features end-to-end reliably' claim. missing for 10: verified success-rate data on test-writing specifically, more recent independent benchmarks showing improved reliability, and evidence tests are written/passed as part of the workflow rather than just code changes.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Language migrations (e.g. JavaScript to TypeScript) Framework upgrades (e.g. Angular 16 -> 18)”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
   • [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”
   • [community] “Clearly an extremely impressive demo... I notice in the first demo Devin hits a KeyError and decides to resolve it by wrapping the code in a…”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   HumanLayer clearly supports end-to-end autonomous coding sessions (Oneshot, RPI, PRD-Oriented workflows) that implement tasks using agents like Claude Code and Codex, including structured phases and sub-agents for research/plan/implement, which implies substantial feature work can be delegated (humanlayer-docs-6, humanlayer-docs-9, humanlayer-docs-10, humanlayer-docs-14). However, no evidence explicitly confirms the agent writes or runs tests as part of the workflow, and no hands-on account demonstrates a full feature-plus-tests delivery. Missing for 10: explicit documentation or example showing test generation/execution as part of the implementation flow, and independent verification of end-to-end feature completion including tests.

   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

Environment setup

1. developerHave an agent automatically clone the repo, install dependencies, and configure its own working environment

   weight 2 · round to Devin
   Devin✓fullclaimed8/10

   Devin's docs describe indexing repos, configuring a persistent environment/workspace that boots into a known-good state every session, VPN access to internal registries/git hosts, and AGENTS.md-based setup instructions it follows before coding — together these show it autonomously clones, configures, and prepares its own working environment. missing for 10: independent/hands-on verification of the clone+dependency-install flow end-to-end, and explicit documentation of automatic dependency installation step-by-step.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   Docs describe workspace configuration (workspace.json, multi-repo setups) and remote hosts that must have access to code/tools/credentials, and one example prompt asks the agent about 'setup commands or local files to copy,' implying some environment configuration ability. However, there is no explicit description of the agent autonomously cloning a repo or installing dependencies end-to-end without human setup of the host/workspace first. Missing for 10: explicit documentation of automatic repo cloning, dependency installation steps, and end-to-end environment bootstrap without prior manual host/workspace configuration.

   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

Interactive takeover

1. developerTake over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work

   weight 2 · round to Devin
   Devin✓fullclaimed8/10

   Docs explicitly describe multi-surface handoff: delegate via web app then take over in Devin's IDE, use /handoff to send terminal tasks to cloud Devin, jump into the Interactive Browser to help Devin navigate, and tag Devin in Slack/Teams threads to redirect work — covering editor, terminal, and browser takeover. Missing for 10: independent/hands-on confirmation of a smooth mid-task handoff experience (community evidence is generic skepticism, not a concrete handoff failure).

   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed7/10

   Docs describe tasks with session/history continuity 'across agents and workstations', remote control from app.humanlayer.com on any machine or phone, and CLI-driven remote daemons for terminal/server contexts, all pointing to genuine hand-off of in-progress work between editor (Claude Code), terminal (remote daemon/CLI), and browser (web app). Live multiplayer draft recovery further shows shared/continuable session state. missing for 10: independent/hands-on confirmation of a live takeover mid-task, and explicit description of an in-editor (IDE plugin) takeover UI rather than just CLI/app/web.

   • [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”

   proof ↗⚑ flag

2. developerSend follow-up instructions to an active agent session to steer its work without restarting

   weight 2 · round to Devin
   Devin~partialclaimed5/10

   Devin's docs describe delegating tasks and then 'taking over in Devin's IDE' or 'jumping in' to help with browsing tasks, implying some interactivity with an active session, but there is no explicit documentation of sending follow-up chat instructions to steer an in-progress session without restarting it. missing for 10: explicit docs on mid-session chat/steering, confirmation that follow-up messages are incorporated live, independent hands-on evidence of this workflow.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed4/10

   HumanLayer's task/session model implies ongoing interaction with agents across a task's lifetime (docs-5) and release notes mention live prompt collaboration during sessions (docs-18), suggesting some capacity to interact with an active session, but there is no explicit documentation of sending mid-session follow-up instructions to steer a running agent without restarting it. missing for 10: explicit docs on injecting new instructions into a live/running session, confirmation the agent incorporates such input without restart, and independent/hands-on verification of this steering behavior.

   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”

   proof ↗⚑ flag

Sandbox execution

1. developerHave an agent safely execute code and install dependencies inside an isolated sandbox

   weight 3 · round to Devin
   Devin~partialclaimed6/10

   Docs describe Devin operating in its own workspace/VM with terminal execution, environment configuration ('known-good state'), VPN access to internal services, and multi-platform (Android/macOS/Windows) build environments, implying isolated sandboxed execution, but there is no explicit documentation of sandbox security guarantees, dependency isolation, or safety controls around code execution, and no independent/hands-on verification of sandbox isolation. missing for 10: explicit sandbox/isolation security documentation, independent verification of safe execution, details on dependency install safety and containment.

   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”
   • [claimed-docs] “Devin can build and run Android applications directly on its own machine — giving it the Android equivalent of Computer Use and browser inte…”
   • [claimed-docs] “Devin now has access to macOS virtual machines. This means Devin can now build and test iOS and macOS applications.”
   • [claimed-docs] “Devin supports Windows as a build and session platform.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer docs describe running sessions on remote hosts (cloud VM, workstation, private-network machine) and automation environments, but there is no mention of an isolated/sandboxed execution environment for running code or installing dependencies safely — the host selection is about access/credentials, not isolation guarantees. Missing for full/partial: any explicit sandbox, container, or isolation mechanism; no evidence of dependency-install safety controls.

   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

Approval controls

1. developerConfigure an agent to auto-approve all its actions instead of confirming each one

   weight 2 · round to HumanLayer
   Devin—none0/10

   The evidence pack describes permission systems for MCP tools (devin-docs-13) and autonomous features like Auto-Fix (devin-docs-20), but there is no documentation of a setting that lets a developer toggle between 'confirm each action' and 'auto-approve all actions' for Devin's own task execution. missing for 10: explicit auto-approve/confirmation toggle documentation, evidence of a per-action approval workflow that can be disabled, any community confirmation of this specific control.

   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

   HumanLayer~partialclaimed4/10

   HumanLayer's core premise is human-in-the-loop approval, and docs mention workflow phases like 'Oneshot' for low-risk changes and automation sessions (humanlayer automation run) that run non-interactively without confirmation, implying some auto-approve capability exists, but no explicit documentation of a configurable 'auto-approve all actions' toggle or setting is shown. missing for 10: explicit config/flag to disable per-action confirmation entirely, documentation confirming automation sessions skip all human review rather than just running unattended, and independent confirmation this works as intended.

   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

2. product-managerApprove key agent decisions from my phone while agents continue working

   weight 1 · round to HumanLayer
   Devin—none0/10

   No evidence describes a mobile app or phone-based approval interface for agent decisions; docs mention Slack/Teams tagging, web app, IDE, and scheduling notifications but nothing about approving decisions from a phone while sessions continue. The axis is plausible for an agentic dev tool with human-in-the-loop needs, but no supporting evidence exists.

   ⚑ flag

   HumanLayer✓fullcommunity8/10

   Docs explicitly describe controlling and continuing a running agent task from a phone via app.humanlayer.com, with tasks providing a shared review/comment history across devices, directly matching the phone-approval-while-agent-continues story; community sentiment corroborates the human-approval-of-agent-actions use case. Missing for 10: a hands-on/independent account specifically confirming the mobile approval UI in practice, and explicit documentation of an 'approve/deny decision' action (vs. general task control) on mobile.

   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”

   proof ↗⚑ flag

3. engineering-leadSet tiered autonomy levels controlling what an agent can do without manual confirmation

   weight 3 · round to HumanLayer
   Devin~partialclaimed4/10

   Devin documents a permission system for MCP/tool access that can be controlled 'at multiple levels' and features like Auto-Fix, human takeover in the IDE, and interactive browser intervention imply some control over autonomous action, but there is no documented feature specifically framed as configurable 'tiered autonomy levels' for an engineering lead to set organization-wide policies on unattended agent actions. missing for 10: explicit autonomy-tier configuration UI/API, organization-wide policy controls, and confirmation-threshold settings distinct from ad-hoc MCP tool permissions.

   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   HumanLayer's workflow-phases doc shows tiered approaches (Oneshot for low-risk changes vs. more review for unclear/risky changes) and skills-workflows lets users select Oneshot/RPI/PRD-Oriented/Freeform modes, which map to different levels of autonomy vs. oversight. However, there's no explicit documentation of a formal 'autonomy level' setting per agent/task with configurable confirmation thresholds, and no independent evidence confirming this tiered control works as an oversight mechanism in practice. missing for 10: explicit named autonomy-tier configuration (e.g., low/medium/high) tied to confirmation gating, evidence of engineering-lead-level policy controls across a team, and independent/hands-on validation that these workflow phases actually reduce unnecessary confirmations without sacrificing safety.

   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”

   proof ↗⚑ flag

Model control

1. ai-native userHave each task prompt automatically routed to the most suitable underlying model

   weight 2 · round to Devin
   Devin✓fullprobed8/10

   Devin's CLI has an explicit "Adaptive" model router that analyzes each prompt and automatically routes it to the best-performing model, with an override via /model if manual control is desired. This directly matches the story of automatic per-task model routing, though it's documented only for the CLI product surface and lacks independent/hands-on validation of routing quality. Missing for 10: independent corroboration of routing accuracy, and confirmation this applies across all Devin surfaces (web/IDE) not just CLI.

   • [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
   • [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
   • [claimed-docs] “Adaptive is an intelligent model router that automatically selects the best AI model for each task.”
   • [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”

   proof ↗⚑ flag

   HumanLayer—none0/10

   Evidence shows HumanLayer lets users manually select or configure which model/backend to use (Claude via Bedrock, Codex, RPI sub-agents) but there is no evidence of automatic routing of a task prompt to the 'most suitable' model based on task characteristics.

   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
   • [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
   • [claimed-docs] “HumanLayer sessions can run Claude through Amazon Bedrock instead of the Anthropic API.”

   proof ↗⚑ flag

2. engineering-leadSwitch away from automatic model selection to a specific model of my choice

   weight 1 · round to Devin
   Devin✓fullprobed8/10

   Devin's docs explicitly state that Adaptive automatically routes prompts to the best model, but users can switch away from Adaptive to a specific model at any time using /model, directly matching the story. Missing for 10: independent/hands-on corroboration of the /model switch in practice and more detail on which specific models are selectable.

   • [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
   • [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
   • [claimed-docs] “Adaptive is an intelligent model router that automatically selects the best AI model for each task.”
   • [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   Docs show explicit model/backend selection — choosing Codex ("install, authenticate, select, and check Codex") or running Claude via Amazon Bedrock instead of the Anthropic API, plus subagent model registration — indicating an engineering lead can pick a specific model rather than a default. However, there is no explicit documentation of an 'automatic' default-selection mode being toggled off, so the framing of 'switching away from automatic' isn't directly evidenced. Missing for 10: explicit mention of an automatic/default model-selection setting and a documented UI/CLI flag to override it, independent confirmation of model-switching behavior.

   • [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
   • [claimed-docs] “HumanLayer sessions can run Claude through Amazon Bedrock instead of the Anthropic API.”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”

   proof ↗⚑ flag

Visibility monitoring

1. developerWatch what a running agent is doing in real time, including its current status

   weight 3 · round to Devin
   Devin✓fullclaimed7/10

   Devin's docs describe real-time observability features like watching the terminal execute commands with live output logs, jumping in during interactive browsing, and taking over in Devin's IDE mid-task, all indicating a live view of the agent's current activity/status. Missing for 10: no explicit dedicated 'status dashboard' documentation and no independent/hands-on corroboration of the real-time monitoring UX beyond vendor docs.

   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer's docs describe remote-daemon control from app.humanlayer.com (including from a phone), live multiplayer session viewing with presence/read-only viewing, and task/session history — all suggesting real-time visibility into agent activity. However, there's no explicit documentation of a dedicated live status/log stream or dashboard showing granular agent state (e.g., current tool call, progress bar) beyond session/task views. missing for 10: explicit real-time status/log streaming documentation, independent hands-on confirmation of live monitoring UX.

   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

2. developerGet notified when an agent completes a task or needs my input

   weight 2 · round drawn
   Devin~partialclaimed6/10

   Devin supports Slack/Teams tagging where sessions and results appear in-thread (devin-docs-4, devin-docs-19), and scheduled sessions explicitly support 'notification preferences' (devin-docs-17), implying some completion/status alerts. However there's no dedicated first-party documentation of a generic notify-on-completion or needs-input alert system (e.g., email/push) outside Slack/Teams and scheduling contexts. Missing for 10: explicit documentation of a unified notification mechanism across all session types, and independent/hands-on confirmation that notifications reliably fire when input is needed.

   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Supports cron expressions for recurring schedules, one-time scheduling, notification preferences, and agent selection”
   • [claimed-docs] “You can jump in to help Devin navigate through browsing tasks via the Interactive Browser.”

   proof ↗⚑ flag

   HumanLayer~partialcommunity6/10

   HumanLayer supports Slack/GitHub/Jira/Linear integrations that push task artifact updates and human-in-the-loop approvals, implying notification when tasks progress or need input, and its core design centers on human oversight of agent work. However, there is no explicit documentation of a dedicated 'task complete' or 'needs input' notification/alert mechanism (e.g., push notification, email, or webhook triggered specifically on completion/approval-required events) beyond generic artifact updates in Slack. missing for 10: explicit completion/needs-input notification docs, evidence of notification latency/reliability, independent/hands-on confirmation of notification delivery.

   • [claimed-docs] “Connect Slack to send HumanLayer task artifact updates into the channels where your team already works.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”

   proof ↗⚑ flag

Natural language task intake

1. developerDescribe a feature or bug in plain language and have it automatically turned into a scoped implementation task

   weight 3 · round to Devin
   Devin✓fullcommunity7/10

   Devin's docs describe accepting natural-language asks (tickets, Slack/Teams messages, delegated tasks) and turning them into sessions/implementation tasks, which is the core intent-to-spec workflow (devin-docs-1, devin-docs-4, devin-docs-5, devin-docs-19). Community commentary raises concerns about actual task success rates and correctness (devin-comm-2, devin-comm-4), tempering confidence in scoping quality. Missing for 10: independent hands-on evidence that the auto-generated task scope/plan is accurate or reviewable before execution, and no dedicated 'plan/spec preview' feature described.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [community] “From the graph at the end: 13.8% of issues resolved. Devin may need some additional help for awhile.”
   • [community] “Although the demos are impressive, they seem short and limited in scope... How often will the solution only be 90% correct, which is 100% no…”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer's task model (docs-5) and workflow-phase selection (docs-6, docs-10) show that a task is created and can be routed through 'Oneshot' for small clear changes or heavier RPI/PRD-oriented flows for ambiguous work, and RPI sub-agents (docs-9) imply a research→plan→implement pipeline that turns a description into a scoped plan. However, there is no explicit walkthrough showing a raw plain-language bug/feature description being automatically parsed into a scoped implementation task end-to-end, and integrations (Jira/Linear/GitHub) mostly create tasks from existing tickets rather than free-form language input. Missing for 10: a concrete example or tutorial of plain-language-to-scoped-task conversion, and independent/hands-on confirmation that this pipeline works as described.

   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”

   proof ↗⚑ flag

2. product-managerConvert user feedback submissions into structured tasks with proposed scope

   weight 2 · round to Devin
   Devin~partialclaimed5/10

   Devin can ingest raw inputs like Linear/Jira tickets and Slack/Teams threads about bugs or feature requests and turn them into an actionable session (devin-docs-1, devin-docs-4, devin-docs-19), which covers the 'feedback-to-task' intake path, but there is no evidence of a distinct structured-task output with an explicit proposed scope/plan artifact for PM review before execution. Missing for 10: evidence of a scoping/planning deliverable (e.g., a spec doc or task breakdown with effort/scope estimate) generated prior to coding, and any PM-specific workflow or UI for reviewing/approving that scope.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed4/10

   HumanLayer supports creating tasks from external issue trackers (Jira, Linear, GitHub) which could serve as a proxy for user feedback submissions, and tasks include shared files/scope info, but there's no evidence of a dedicated feature for ingesting raw user feedback (e.g., support tickets, survey responses) and auto-structuring it into a task with a proposed scope specifically tailored for PM workflows. missing for 10: dedicated feedback-ingestion mechanism, explicit 'proposed scope' generation from unstructured feedback text, PM-specific workflow templates, and any hands-on/community validation of this specific use case.

   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

3. developerAttach a marked-up screenshot or mockup to a task so the agent implements the correct visual change

   weight 2 · round to HumanLayer
   Devin—none0/10

   No evidence in the pack mentions attaching screenshots, mockups, or marked-up images to a Devin task, nor any visual-diff/design-to-code workflow; documentation covers ticket delegation, terminal, Slack, MCP, and environment setup but nothing about image/mockup-based task intake.

   ⚑ flag

   HumanLayer~partialclaimed4/10

   Docs confirm images can be pasted into the new task composer as attachments (humanlayer-docs-22), which supports attaching a screenshot to a task, but there is no evidence of markup/annotation tooling or of the agent parsing visual annotations to implement a corresponding UI change. Missing for 10: annotation/markup capability for screenshots, evidence the agent interprets visual markup into a specific implementation, and any hands-on example of this workflow succeeding.

   • [claimed-docs] “Image paste in new tasks — Paste images straight into the new task composer as attachments.”

   proof ↗⚑ flag

Plan approval

1. developerReview and approve an agent's implementation plan before any code changes are made

   weight 3 · round to HumanLayer
   Devin—none0/10

   The evidence pack describes Devin's task delegation, IDE handoff, terminal visibility, and MCP integrations, but nowhere documents a plan-review/approval gate before Devin begins making code changes. Missing for 10: any doc mentioning an upfront plan proposal, explicit approval step, or 'plan mode' prior to execution.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer's workflow-phases and RPI sub-agent docs describe planning phases (e.g., 'a change with unclear behavior or code shape needs more review before implementation') and PRD-Oriented/RPI workflows imply a plan stage before code changes, with tasks providing 'one place for comments and review.' However, no evidence explicitly shows a dedicated plan-approval gate/UI step where a developer reviews and approves a plan artifact before implementation begins. missing for 10: explicit documentation of a plan-approval step/UI, first-party example of blocking implementation until plan is approved, independent/hands-on confirmation of this specific gate.

   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

2. engineering-leadApprove a task's scope and contract before an agent is allowed to modify the repository

   weight 2 · round to HumanLayer
   Devin—none0/10

   The evidence pack shows delegation, review-after-the-fact, and IDE takeover workflows, but no documented mechanism where an engineering-lead must approve a task's scope/contract before Devin is allowed to start modifying the repository. Missing for 10: explicit pre-execution approval/gating workflow, scope/contract sign-off UI or API, evidence of blocking repo writes until lead approval.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”

   proof ↗⚑ flag

   HumanLayer~partialcommunity6/10

   HumanLayer's workflow-phases doc explicitly supports scoping review before implementation (e.g., 'a change with unclear behavior or code shape needs more review before implementation'), and tasks/sessions provide a structured place for comments and review prior to agent execution, plus human-in-the-loop approval is core to the product's value prop per community discussion. However, there's no explicit documented feature for an engineering-lead specifically approving a 'scope and contract' artifact as a gating step before repo modification — it's inferred from general workflow-phase and review mechanics rather than a dedicated scope-approval gate. missing for 10: explicit documentation of a formal scope/contract approval step tied to lead sign-off, evidence of blocking repo writes until such approval, and independent/hands-on confirmation this gate works as intended.

   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”

   proof ↗⚑ flag

Ticket driven tasking

1. developerAssign a coding task to an agent directly from an existing issue or ticket

   weight 3 · round drawn
   Devin✓fullclaimed8/10

   Docs explicitly state Devin can be asked to tackle Linear/Jira tickets, tagged on Slack/Teams threads about bugs to start sessions, and integrated via API/MCP to create sessions from external triggers, directly supporting assigning tasks from existing issues/tickets. Missing for 10: independent hands-on verification of ticket-to-task handoff and no direct product screenshot of ticket-linking UI in evidence.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed8/10

   Docs explicitly describe connecting GitHub, Jira, and Linear so HumanLayer creates tasks directly from issues/tickets and links artifacts back to the source, directly matching the story of assigning agent work from an existing ticket. Missing for 10: independent/hands-on confirmation that this ticket-to-task flow works reliably in practice, and more detail on the actual assignment UX.

   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Devin
   Devin~partialprobed5/10

   Devin exposes a documented API (session creation, create_as_user_id) and an MCP server that surfaces session management, playbooks, knowledge, and scheduling (devin-docs-8,9,15,29), suggesting broad but not full parity with the web/IDE UI. There's no explicit claim or evidence that UI-only features like the Interactive Browser takeover, terminal handoff, VPN setup, or environment/emulation configuration are fully API-accessible, and no OpenAPI spec was found (devin-probe-2) to confirm completeness. Missing for 10: explicit vendor statement of API/UI parity, evidence that interactive/browser/VPN/environment features are API-controllable, and independent confirmation of parity.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”
   • [claimed-docs] “it gives any MCP-compatible AI agent or IDE full access to session management, playbooks, knowledge, and scheduling.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.devin.ai/openapi.json, https://docs.devin.ai/swagger.json, https://docs.devin.ai/api/op…”

   proof ↗⚑ flag

   HumanLayer~partialprobed3/10

   HumanLayer offers a CLI (`humanlayer automation run`) and remote daemon controls that let you launch and manage sessions outside the UI, but there is no documented public API/OpenAPI spec (both openapi.json and llms.txt probes 404), and UI-only features like live multiplayer drafts, keyboard navigation, and image paste have no CLI/API equivalent documented. missing for 10: a documented REST/GraphQL API or OpenAPI spec, confirmation that all UI actions (draft PRs, multiplayer editing, metrics views) are exposed programmatically, and independent verification of API-UI parity.

   • [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
   • [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”
   • [claimed-docs] “Keyboard navigation for changed files — Move through the PR changes tree with J/K, N/P, G shortcuts, and Enter.”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   Devin—none0/10

   No evidence of any data export feature, open-format export, or account/data portability tooling in Devin's docs; evidence covers task delegation, MCP, CLI, and platform support but nothing about exporting session data, knowledge, or playbooks in open formats for migration away from the product.

   ⚑ flag

   HumanLayer—none0/10

   No evidence of data export functionality or open-format data portability; docs focus on tasks, workflows, and integrations, with no mention of exporting user data or account deletion/leaving.

   ⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Devin—none0/10

   Devin is a closed-source commercial SaaS product; no evidence of any open-source license or public source repository is present in the evidence pack, and the axis is applicable since AI tooling could conceivably publish open-source components. missing for 10: any open-license source release, public repo link, or licensing statement.

   ⚑ flag

   HumanLayer—none0/10

   No evidence pack item mentions an open-source license or public source repository for HumanLayer; all evidence is docs, community discussion, or probes about APIs/CLI. Absence of evidence for this applicable axis yields none.

   ⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   Devin—none0/10

   Devin is exclusively delivered as a cloud SaaS product (with local CLI/terminal front-ends that hand off to cloud sessions); no evidence anywhere in the docs or community discussion describes a self-hostable core engine or on-prem deployment option. Missing for 10: any documentation of self-hosted/on-prem deployment, container/binary distribution of the core agent, or licensing for self-hosting.

   • [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”
   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   Evidence shows HumanLayer's daemon/agent execution can run on a user-controlled host (cloud VM, workstation, private network), but the core control plane is explicitly tied to the hosted app.humanlayer.com service — no docs describe self-hosting that core product. missing for 10: no self-hosted control-plane/server option, no on-prem deployment guide, no Docker/Helm chart or license for running the full stack independently.

   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

Enterprise licensing

1. engineering-leadLicense an enterprise deployment with SSO and commercial support for organization-wide rollout

   weight 2 · round drawn
   Devin—none0/10

   No evidence pack items mention enterprise licensing, SSO, or commercial support offerings for Devin; only product feature docs and community sentiment are present. Missing for 10: SSO/SAML documentation, enterprise plan/pricing page, commercial support SLA details.

   ⚑ flag

   HumanLayer—none0/10

   No evidence pack item mentions SSO, enterprise licensing, org-wide rollout, or commercial support tiers; docs cover tutorials, integrations, and workflow features only, and community discussion focuses on pricing/skepticism unrelated to enterprise licensing terms.

   ⚑ flag

Model flexibility

1. engineering-leadBring my own LLM or API key so agents run on the model of my choice

   weight 2 · round to HumanLayer
   Devin—none0/10

   Docs describe an internal 'Adaptive' model router and a /model command to pick among Devin's supported models, but there is no evidence of bringing a custom LLM or your own API key to run agents on a model of the engineering lead's choice.

   • [claimed-docs] “Adaptive analyzes your prompt and routes it to the model that will deliver the best result.”
   • [claimed-docs] “You can switch away from Adaptive to a specific model at any time with /model.”
   • [claimed-docs] “Adaptive is an intelligent model router that automatically selects the best AI model for each task.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   Docs show HumanLayer sessions can use different backends/models — Claude Code, OpenAI Codex, and Claude via Amazon Bedrock instead of the Anthropic API — indicating some flexibility in model/provider choice, which implies bringing your own credentials for these paths. However, there is no explicit doc describing a generic 'bring your own API key' mechanism for arbitrary LLMs or a pricing-tier note tying this to cost savings for engineering leads. Missing for 10: explicit BYO-API-key configuration docs, support for arbitrary/third-party model providers beyond Claude/Codex/Bedrock, and any pricing-related messaging about cost control via own keys.

   • [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
   • [claimed-docs] “HumanLayer sessions can run Claude through Amazon Bedrock instead of the Anthropic API.”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”

   proof ↗⚑ flag

Usage quotas

1. engineering-leadSee and manage plan-based daily task and concurrency limits for agent workflows

   weight 2 · round drawn
   Devin—none0/10

   No evidence pack item mentions plan tiers, daily task quotas, concurrency limits, or an admin console/dashboard for managing such limits; docs cover running parallel Devins and API session creation but not visibility or management of plan-based limits. Missing for 10: pricing/plan documentation, admin UI for concurrency/task caps, usage dashboards or alerts for limit management.

   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”

   proof ↗⚑ flag

   HumanLayer—none0/10

   The evidence pack has no mention of plan-based daily task/concurrency limits or any admin controls for managing such limits; only a vague reference to 'Advanced Metrics for all paid plans' which covers usage/cost/productivity viewing, not concurrency or daily task limits management.

   • [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   Devin—none0/10

   No evidence in the pack mentions data residency, regional storage options, or compliance controls for where session/data is stored; docs cover VPN connectivity, environments, and MCP but nothing about choosing a storage region. Missing for 10: any mention of data residency/region selection, compliance certifications (e.g., SOC2/GDPR data locality), or enterprise data storage controls.

   ⚑ flag

   HumanLayer—none0/10

   No evidence in the pack addresses data region/residency selection or storage location controls; docs cover hosts, integrations, and workflows but never data residency options.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Devin—none0/10

   No evidence in the pack addresses data-training opt-out, privacy controls, or any policy about excluding user data from model training; all docs cover feature capabilities, MCP, CLI, and environment support instead.

   ⚑ flag

   HumanLayer—none0/10

   No evidence in the pack addresses data usage for AI model training, opt-out policies, or privacy commitments regarding training data; the docs focus on task workflows, integrations, and remote sessions.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Devin—none0/10

   No evidence in the pack addresses data retention policies, deletion controls, or privacy configuration options for Devin; the docs cover features, MCP, CLI, and environments but not data governance/retention settings.

   ⚑ flag

   HumanLayer—none0/10

   No evidence pack items mention data retention policies, deletion controls, or privacy settings for HumanLayer; the docs focus on task/session workflows, integrations, and remote daemons rather than data governance.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Devin—none0/10

   No evidence pack item mentions telemetry opt-out, analytics settings, or usage-tracking controls for Devin; nothing addresses this privacy axis at all.

   ⚑ flag

   HumanLayer—none0/10

   No evidence in the pack addresses telemetry, usage tracking, or opt-out controls for HumanLayer; docs cover integrations, workflows, and CLI usage but nothing about privacy/telemetry settings. Missing for 10: any mention of telemetry collection, opt-out mechanism, privacy policy, or data-handling documentation.

   ⚑ flag

Chat integration

1. developerTag an agent in a chat thread to discuss and delegate a bug or task

   weight 2 · round to Devin
   Devin✓fullclaimed8/10

   Docs explicitly describe tagging Devin in Slack/Teams threads to discuss and delegate bugs/tasks, starting sessions directly from conversations. Missing for 10: independent/hands-on corroboration of this specific chat-tagging workflow beyond vendor docs.

   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's Slack integration only pushes task-artifact updates into channels (docs-4) and other integrations (GitHub, Jira, Linear) create tasks from tickets/issues, not from tagging an agent inside a chat thread. There is no evidence of a chat-native @mention or in-thread delegation workflow for discussing/assigning tasks to an agent.

   • [claimed-docs] “Connect Slack to send HumanLayer task artifact updates into the channels where your team already works.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”

   proof ↗⚑ flag

Knowledge context

1. developerAdd a context file describing my codebase conventions so agents generate more relevant plans and code

   weight 3 · round to Devin
   Devin✓fullclaimed9/10

   Devin explicitly supports AGENTS.md files placed in the project root that Devin reads before coding, plus a separate 'Knowledge' feature for sharing codebase-level conventions/context to improve plans and code generation. Missing for 10: independent/hands-on confirmation that this measurably improves plan/code relevance beyond vendor docs.

   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”
   • [claimed-docs] “Knowledge is the best way to share codebase-level (vs. task-level) context that can help Devin when working in your codebase.”
   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   The docs describe workspace-level config files (workspace.json/workspace.local.json) for team/machine settings and multi-repo setup, but there is no evidence of a dedicated context file for describing codebase conventions to improve agent-generated plans/code.

   • [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”

   proof ↗⚑ flag

Project management integration

1. product-managerConnect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly

   weight 3 · round drawn
   Devin~partialclaimed6/10

   Devin explicitly supports Linear and Jira tickets natively (devin-docs-1) and its MCP extensibility (devin-docs-12, devin-docs-21) lets it connect to arbitrary tool servers including issue trackers, which could cover ClickUp/Monday.com. However, there's no first-party documentation of native ClickUp or Monday.com connectors, and no independent/hands-on evidence confirming ticket-management workflows actually work end-to-end via these integrations. missing for 10: explicit native ClickUp/Monday.com support docs, independent verification of issue-tracker workflows succeeding in practice.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “MCP (Model Context Protocol) lets you connect external tool servers to Devin CLI, giving the agent access to APIs, databases, issue trackers…”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”
   • [claimed-docs] “MCP tools are subject to the same permission system as built-in tools. You can control access at multiple levels”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer documents native Jira Cloud and Linear integrations that create tasks from tickets and sync/link status back to source issues, directly matching the story for those trackers. However, there is no evidence of ClickUp or Monday.com integrations, so the story is only partially delivered. Missing for 10: ClickUp integration docs, Monday.com integration docs, independent/hands-on verification of ticket sync working in practice.

   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”

   proof ↗⚑ flag

Version control integration

1. developerConnect a GitHub repository so an agent can access the code and open pull requests against it

   weight 3 · round to HumanLayer
   Devin~partialclaimed6/10

   Docs confirm repository indexing (public/private) and GitHub-repo access for context, plus 'Devin Review with Auto-Fix' responding to code review/CI, implying PR interaction, but there's no explicit documentation of a GitHub connection/auth flow or of Devin opening pull requests directly. missing for 10: explicit GitHub app/OAuth connection setup steps, explicit 'opens pull requests' documentation, independent/hands-on confirmation of PR creation.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “These tools let you explore and query documentation for any GitHub repository (public or private with authentication)”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed8/10

   Docs describe connecting GitHub to create tasks from issues and link artifacts back (humanlayer-docs-16), plus agent sessions can access code via configured workspaces/repos (humanlayer-docs-8, humanlayer-docs-11) and open draft PRs directly from the session UI (humanlayer-docs-19). missing for 10: independent/hands-on corroboration of the GitHub connection flow and PR-opening working end-to-end beyond first-party docs.

   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”

   proof ↗⚑ flag

2. developerGrant an agent access to my repositories with a one-click install, without complex setup

   weight 2 · round to Devin
   Devin~partialclaimed5/10

   Devin does support connecting to repos (indexing, GitHub-based session creation, VPN access to internal git hosts) and this is described as straightforward onboarding, but there's no evidence of a literal 'one-click install' flow like a GitHub App marketplace install — setup instead involves indexing, environment configuration, AGENTS.md, and knowledge onboarding steps. Missing for 10: explicit one-click/marketplace-style install flow, independent evidence of setup simplicity, and confirmation that repo access requires no additional configuration.

   • [claimed-docs] “Indexing your repositories allows Devin to understand your codebase and enables powerful features like Ask Devin and DeepWiki.”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”
   • [claimed-docs] “Just put an AGENTS.md file in your project root (or anywhere else). Devin will look for the file before it starts coding.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   Docs describe GitHub/Jira/Linear integrations for creating tasks from issues, but connecting a repo requires selecting a host, configuring workspace.json/workspace.local.json, and setting up remote daemons or multi-repo workspaces — none of this is framed as a one-click, no-setup install. No evidence pack item claims or demonstrates a one-click repo access flow.

   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”

   proof ↗⚑ flag

Ci remediation

1. engineering-leadHave failed CI workflows automatically diagnosed and fixed with a proposed pull request

   weight 3 · round to Devin
   Devin✓fullclaimed7/10

   Devin explicitly documents 'Devin Review with Auto-Fix' that responds to code review comments, fixes flagged bugs, and iterates on CI failures, directly matching the story of auto-diagnosing and fixing failed CI with a proposed PR. However, this is vendor-only documentation with no independent/hands-on corroboration of the CI-fix workflow specifically. Missing for 10: independent verification of CI-fix accuracy/success rate, details on PR proposal format, and evidence of real-world reliability beyond the single docs line.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   HumanLayer supports running automation sessions from CI (`humanlayer automation run`) and can open draft PRs from a session's diff/GitHub tab, and GitHub integration links tasks to issues—so the building blocks for a CI-triggered fix-and-PR flow exist. However there is no direct evidence of automatic diagnosis of failed CI logs/errors or a documented end-to-end 'CI failure → agent diagnosis → PR' pipeline. Missing for 10: explicit CI-failure-detection/diagnosis workflow docs, example of a failing pipeline auto-triggering a session, and confirmation the resulting PR addresses the CI failure specifically.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”

   proof ↗⚑ flag

2. developerTrigger an agent from CI/CD pipelines to fix a broken build or failing test

   weight 2 · round drawn
   Devin~partialclaimed6/10

   Docs explicitly state Devin Review with Auto-Fix can iterate on CI failures and fix flagged bugs, and the Devin API lets you integrate Devin into applications/workflows and create sessions programmatically (including on behalf of users), which supports triggering it from automated pipelines. However, there's no explicit documented example of a CI/CD pipeline (e.g., GitHub Actions/Jenkins) directly invoking Devin on a failing build/test, and no independent/hands-on confirmation of this exact workflow. Missing for 10: explicit CI/CD trigger integration example, independent verification that pipeline-triggered fixes work reliably.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer docs explicitly describe `humanlayer automation run` for running a Cloud-visible coding session 'from any automation environment — a CI job, a cron machine, or a script on a server,' plus launch tokens for non-interactive command execution, directly supporting CI/CD-triggered agent runs. However, there is no explicit example or integration guide showing a CI failure (broken build/failing test) triggering the agent to diagnose and fix it, nor independent/hands-on confirmation of this workflow. missing for 10: a concrete CI/CD pipeline example tied to build/test failures, evidence of automatic failure detection triggering the agent, and independent verification of this automation flow working in practice.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”

   proof ↗⚑ flag

Diff review

1. developerConfigure an agent to automatically open a pull request when its task completes

   weight 2 · round to Devin
   Devin~partialclaimed5/10

   Devin is known to work within GitHub workflows and create PRs as part of its task completion (implied by its coding-agent nature and features like Auto-Fix responding to review comments/CI), but the evidence pack has no explicit documentation describing a configurable setting to auto-open a pull request upon task completion. missing for 10: explicit docs describing PR-creation-on-completion configuration, settings/toggle for auto-PR behavior, and independent confirmation of this specific workflow.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed4/10

   Docs confirm HumanLayer can open a draft pull request from a session (humanlayer-docs-19) and integrates with GitHub for issue-linked tasks (humanlayer-docs-16), but the evidence describes PR creation as a manual 'ask the session agent' action from the UI, not an automatic trigger fired upon task completion. Missing for 10: explicit configuration option/workflow setting for automatic PR creation on task completion, and any evidence of it happening without a manual prompt.

   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”

   proof ↗⚑ flag

2. developerReview a diff of an agent's changes and approve it before it becomes a pull request

   weight 3 · round to HumanLayer
   Devin~partialclaimed4/10

   Docs mention delegating a task and 'taking over in Devin's IDE once it gives you a good first draft' (devin-docs-5), implying some human checkpoint before final output, and Devin's terminal/log viewing lets you watch work in progress (devin-docs-6). However there is no explicit documentation of a diff-review/approve-before-PR gate — 'Devin Review with Auto-Fix' (devin-docs-20) is about post-PR review comment handling, not a pre-PR approval step. Missing for 10: explicit diff viewer with approve/reject before PR creation, documentation of a formal approval gate, and independent/hands-on confirmation of this workflow.

   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “Devin’s terminal, where you can watch commands being executed and view output logs. You can also copy the shell output for debugging purpose…”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

   HumanLayer~partialcommunity7/10

   Release notes explicitly reference a diff view and 'draft PR creation' workflow (docs-19, docs-21), and tasks include 'One place for comments and review' (docs-5), indicating a review-before-PR mechanism. However, there's no detailed documentation of an explicit approve/reject gate tied specifically to diff review prior to PR creation, and community discussion raises concerns about automation bias in approvals (comm-3) without disputing the core capability. Missing for 10: explicit step-by-step approval workflow docs, independent hands-on verification of the diff-review-then-approve flow, and clarity on how rejection/edits are handled before PR creation.

   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Keyboard navigation for changed files — Move through the PR changes tree with J/K, N/P, G shortcuts, and Enter.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”
   • [community] “User raised concern about automation bias: once an agent proves reliable, humans may rubber-stamp approvals without careful review, letting …”

   proof ↗⚑ flag

Pr review automation

1. ai-native userHave incoming issues automatically triaged with severity suggested and routed to the right owner

   weight 2 · round drawn
   Devin—none0/10

   Devin can be pointed at Linear/Jira tickets or tagged in Slack to work a bug, but nothing in the evidence describes automatic triage logic that assesses severity or routes issues to the correct owner — these are core, specific parts of the story. Missing for 10: severity classification/scoring, ownership/routing logic, any 'triage' feature or automated inbox-sorting workflow.

   • [claimed-docs] “Ask Devin to tackle Linear/Jira tickets, implement entirely new features, repro and fix bugs, build internal tools, and more!”
   • [claimed-docs] “Tagging Devin on a Slack or Teams thread about a bug you’re discussing with coworkers”
   • [claimed-docs] “Tag Devin on Slack or Teams: Start sessions directly from conversations about bugs, feature requests, or questions.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer connects issue trackers (Jira, GitHub, Linear) to create tasks from tickets, but there is no evidence of automatic triage, severity classification, or routing to an owner — integrations only create/link tasks, not assess or assign severity/ownership.

   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”

   proof ↗⚑ flag

2. engineering-leadHave every pull request automatically reviewed with AI-generated inline comments

   weight 3 · round to Devin
   Devin~partialclaimed5/10

   Devin documents a 'Devin Review with Auto-Fix' feature that responds to code review comments, fixes flagged bugs, and iterates on CI failures, implying an automated review capability tied to PRs, but the evidence pack gives no detail on how review comments are generated, whether they are inline, or how coverage/setup works for 'every PR'. missing for 10: explicit documentation of inline AI-generated PR comments, configuration for gating every PR, and independent/hands-on confirmation of the review quality.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's evidence covers task/session management, human-in-the-loop approvals, draft PR creation, and a PR diff-viewing UI, but there is no mention of an automated review process that generates inline review comments on every pull request. Missing for 10: no evidence of automatic PR review triggers, no mention of AI-generated inline comments, no review-quality-gate CI integration for PRs.

   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Keyboard navigation for changed files — Move through the PR changes tree with J/K, N/P, G shortcuts, and Enter.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”

   proof ↗⚑ flag

Readiness checks

1. engineering-leadRun a readiness report that evaluates how ready my repository is for autonomous agents

   weight 2 · round drawn
   Devin—none0/10

   No evidence of any 'readiness report' or scoring feature that evaluates a repository's suitability for autonomous agents; docs cover onboarding features like AGENTS.md, environment setup, indexing, and knowledge, but nothing that generates a readiness assessment/report artifact for engineering leads.

   ⚑ flag

   HumanLayer—none0/10

   No evidence of any readiness-report or repository-readiness-assessment feature; HumanLayer's docs focus on task/session management, integrations, and human-in-the-loop workflows, not a diagnostic scan of a repo's agent-readiness.

   ⚑ flag

Security remediation

1. engineering-leadHave security alerts automatically validated and remediated with an opened pull request

   weight 2 · round to Devin
   Devin~partialclaimed5/10

   Devin's 'Review with Auto-Fix' auto-responds to code review comments, fixes flagged bugs and iterates on CI failures, and MCP integrations extend it to security-adjacent tools like Sentry and Datadog, implying it could react to alerts and open a PR fix. However, there is no explicit documentation of a security-scanner (SAST/dependency-alert) validation-and-remediation pipeline or evidence of an actual security-alert-to-PR workflow. Missing for 10: explicit security-scanner (e.g., CodeQL/Dependabot/Snyk) integration, evidence of validating true-positive vs false-positive alerts, and a documented end-to-end 'alert triggers PR' example.

   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [claimed-docs] “Extend Devin's reach with MCP integrations: Connect Devin to Datadog, Sentry, databases, Figma, Notion, Stripe, and hundreds of other tool”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's docs show generic task creation from GitHub/Jira/Linear issues and draft PR creation from agent sessions, but there is no evidence of any security-alert-specific validation or automated vulnerability remediation workflow (no CVE, dependency-alert, or security-scanner integration mentioned).

   ⚑ flag

Concurrent execution

1. engineering-leadRun many agent tasks concurrently to scale delivery throughput

   weight 3 · round to Devin
   Devin~partialclaimed7/10

   Devin docs explicitly document running multiple Devins in parallel on independent tasks and provide an API with create_as_user_id for programmatically spawning sessions on behalf of org members, directly supporting concurrent scaling of throughput. However, there's no first-party data on concurrency limits, cost/resource management at scale, or independent case studies validating throughput gains at team scale; missing for 10: quantified concurrency limits/scaling guidance, independent evidence of large-scale parallel usage, and cost/governance controls for many simultaneous sessions.

   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “create sessions, manage playbooks and knowledge, set up schedules, and more”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   Docs describe an architecture (tasks/sessions, multi-repository workspaces, remote daemons on cloud VMs, and a CLI 'automation run' for CI/cron/scripts) that could support running many agent tasks in parallel, and 'Advanced Metrics' track usage/productivity across an org. However, tutorials and guides are framed around running one task/session at a time, and there is no explicit documentation or evidence of concurrent multi-task orchestration, throughput dashboards, or scaling guarantees for many simultaneous agents. Missing for 10: explicit multi-session concurrency docs, evidence of parallel task orchestration at scale, and independent confirmation of throughput gains.

   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”

   proof ↗⚑ flag

2. engineering-leadCreate agent sessions on behalf of other users in my organization

   weight 2 · round to Devin
   Devin✓fullclaimed8/10

   Devin's API explicitly supports creating sessions on behalf of any user in the organization via the create_as_user_id parameter, directly matching the engineering-lead use case for org-wide session delegation. Missing for 10: independent/hands-on corroboration of this specific feature and details on permission scoping/audit controls for delegated sessions.

   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   The evidence describes tasks, sessions, integrations, and org-wide metrics access but never mentions an engineering-lead or admin creating/assigning agent sessions on behalf of another named user in the organization. No account-delegation, impersonation, or 'assign session to teammate' capability is documented.

   ⚑ flag

Deployment flexibility

1. developerUse a managed cloud offering to run agents without operating my own backend infrastructure

   weight 2 · round to Devin
   Devin✓fullclaimed7/10

   Devin is delivered as a cloud-hosted agent (web app, API, session creation on behalf of users) so developers don't need to run their own backend/agent infrastructure — sessions run on Devin's managed cloud with its own configured environment, VPN access, and parallel execution. missing for 10: independent/hands-on confirmation of the managed infrastructure model, and explicit documentation of uptime/scaling/ops guarantees for the hosted service.

   • [claimed-docs] “using /handoff to send longer tasks to cloud Devin”
   • [claimed-docs] “Delegating a more complex task via the web application and taking over in Devin’s IDE once it gives you a good first draft.”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Run multiple Devins in parallel: Carve out independent tasks and run them simultaneously.”
   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “Configure it once, and every session boots into that known-good state.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   HumanLayer offers a cloud control plane (app.humanlayer.com, automation sessions marked 'Cloud-visible', advanced metrics for paid plans) that lets users monitor and manage agent runs remotely, suggesting a managed service layer. However, docs explicitly state the actual agent execution still runs on a host that the user must select and operate—'a cloud VM, workstation, or private-network machine'—meaning the developer still needs to provision/manage backend compute for the daemon rather than HumanLayer fully hosting execution. Missing for 10: evidence of a fully HumanLayer-operated compute backend (no user-managed VM/daemon required), and independent confirmation of a true zero-ops managed runtime.

   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”

   proof ↗⚑ flag

2. engineering-leadSelf-host agent infrastructure locally, in containers, or on my own VMs

   weight 2 · round to HumanLayer
   Devin—none0/10

   Devin is delivered as a cloud-hosted SaaS agent platform; all evidence points to cloud sessions, API access, VPN connectivity into customer networks, and a CLI/terminal client for local interaction, but nothing indicates the core agent infrastructure itself can be self-hosted on-prem, in customer-managed containers, or on customer VMs. Missing for 10: any documentation of a self-hosted/on-prem deployment option, container images for the agent runtime, or VM-based installation of Devin's backend.

   • [claimed-docs] “Devin can connect to a VPN from inside its workspace, so sessions can reach internal services such as package registries, databases, and int…”
   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “Running Devin for Terminal in your local environment for quick fixes, code exploration, and interactive coding right from the command line”

   proof ↗⚑ flag

   HumanLayer~partialclaimed7/10

   Docs describe running the remote daemon on a cloud VM, workstation, or private-network machine that you control (humanlayer-docs-7), plus automation sessions from CI/cron/server environments (humanlayer-docs-12) and launch tokens for bootstrap scripts (humanlayer-docs-13), showing self-hosted deployment flexibility. However there is no explicit mention of container/Docker deployment or an official container image, and no independent verification of self-hosted setups at scale. Missing for 10: explicit container/Docker packaging docs, independent hands-on confirmation of self-hosted deployments.

   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

Headless automation

1. developerRun an agent headlessly inside CI/CD pipelines and shell scripts

   weight 2 · round to HumanLayer
   Devin~partialprobed6/10

   Devin's API is explicitly designed to 'integrate Devin into your applications, automate workflows, and build powerful tools' and supports creating sessions on behalf of users, which enables scripted/headless invocation, and a CLI is also documented. However, there is no explicit CI/CD pipeline example, exit-code/status handling, or GitHub Actions/shell-script integration guide in the evidence — the closest CI-related capability shown is 'Auto-Fix' reacting to CI failures rather than Devin running as a CI step itself. missing for 10: explicit CI/CD pipeline integration docs (e.g., GitHub Actions/shell script examples), documented headless/non-interactive CLI invocation with exit codes.

   • [claimed-docs] “The Devin API enables you to integrate Devin into your applications, automate workflows, and build powerful tools.”
   • [claimed-docs] “you can create sessions on behalf of any user in your organization using the create_as_user_id parameter”
   • [claimed-docs] “Enable Devin Review with Auto-Fix so Devin automatically responds to code review comments, fixes flagged bugs, and iterates on CI failures”
   • [probe] “official CLI documented at https://docs.devin.ai/cli/adaptive”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed8/10

   Docs explicitly describe `humanlayer automation run` for running Cloud-visible coding sessions from CI jobs, cron machines, or scripts, plus launch tokens for non-interactive/non-PTY execution suited to headless environments. This directly matches running an agent headlessly in CI/CD and shell scripts. Missing for 10: independent/hands-on verification of CI usage and concrete pipeline examples (e.g. GitHub Actions config).

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”

   proof ↗⚑ flag