Codegen vs HumanLayer

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Codegen
   Codegen✓fullprobed9/10

   A probe confirms Codegen's docs site serves a working llms.txt at https://docs.codegen.com/llms.txt (HTTP 200) listing structured doc links, directly enabling an agent to be pointed at agent-oriented docs; this is corroborated by extensive first-party documentation of agent-friendly interfaces (SDK, CLI, API). Missing for 10: independent third-party confirmation of agents actually consuming the llms.txt in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.codegen.com/llms.txt # Codegen ## Docs - [Codegen](https://docs.codegen.com/introduction/overview…”
   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   A direct probe for llms.txt returned 404, and no evidence pack item mentions agent-oriented documentation formats or an alternative machine-readable docs endpoint.

   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round drawn
   Codegen✓fullprobed8/10

   Codegen explicitly supports headless/automated usage via a Python SDK for programmatic agent invocation, a REST API with explicit CI/CD trigger support ('Trigger agents from CI/CD pipelines when builds fail'), and a CLI for scripted interactions and telemetry — all first-party documented. Missing for 10: independent/hands-on confirmation of CI usage in the wild and a working OpenAPI spec (probe found 404s), so it's vendor-doc-only evidence.

   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”

   proof ↗⚑ flag

   HumanLayer✓fullprobed8/10

   Docs explicitly describe running HumanLayer headlessly via `humanlayer automation run` from CI jobs, cron machines, or scripts, plus launch tokens for non-interactive/non-PTY environments, and remote daemon hosts on cloud VMs or servers, directly matching the CI/automation story. Missing for 10: independent/hands-on confirmation of CI usage and more detail on auth/config specifics for automated pipelines.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to Codegen
   Codegen~partialclaimed6/10

   Codegen's docs explicitly mention connecting to Slack, Linear, Figma, databases and extending capabilities 'with custom MCP tools', and separately reference provisioning MCP servers across an org — direct evidence of MCP client integration. However, the evidence is a single brief doc mention with no detailed setup guide, supported server list, or independent/hands-on confirmation of the feature working. Missing for 10: detailed MCP integration docs/tutorial, list of supported MCP servers, independent verification of the feature in practice.

   • [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
   • [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   No evidence anywhere in the pack mentions MCP servers or the ability to plug external tool servers into HumanLayer; integrations mentioned (Jira, Slack, GitHub, Linear) are task-source connectors, not MCP tool servers. Missing for 10: any mention of MCP protocol support, MCP server configuration, or tool-plugin mechanism.

   ⚑ flag

4. ai-native userUse an official CLI

   weight 2 · round to Codegen
   Codegen✓fullprobed8/10

   Codegen documents an official CLI (docs.codegen.com/introduction/cli) with concrete capabilities: viewing/pulling agent work, creating new agents, running Claude Code with telemetry, and managing organizations/repositories — matching an AI-native agentic workflow tool. missing for 10: independent/hands-on corroboration beyond first-party docs, and no details on installation/versioning maturity.

   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
   • [claimed-docs] “Manage organizations - Switch between organizations and configure repositories”
   • [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”

   proof ↗⚑ flag

   HumanLayer✓fullprobed7/10

   HumanLayer documents an official CLI (e.g. `humanlayer automation run`, launch tokens, remote-daemon control) used for CI, cron, and scripted agentic workflows, confirmed by a dedicated CLI doc page. missing for 10: no independent/hands-on verification of the CLI, no full command reference, and no evidence of broader CLI feature parity with the app.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”

   proof ↗⚑ flag

5. ai-native userDrive the product through a documented public API

   weight 3 · round to Codegen
   Codegen✓fullprobed8/10

   Codegen documents a full API reference (agent creation, traces, CI/CD triggers), a Python SDK for programmatic agent control, and a CLI for managing agents/orgs, giving AI-native users multiple documented ways to drive the product programmatically. Missing for 10: a discoverable machine-readable OpenAPI/swagger spec (probe returned 404s) and independent third-party corroboration of the API's reliability.

   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
   • [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”

   proof ↗⚑ flag

   HumanLayer~partialprobed4/10

   HumanLayer documents a CLI (`humanlayer automation run`, launch tokens, remote daemons) that lets automation environments drive sessions programmatically, which is a form of documented programmatic control, but there is no evidence of a documented public REST/OpenAPI API — probes explicitly found openapi.json/swagger.json/llms.txt all 404. missing for 10: a documented HTTP/OpenAPI public API spec, SDK/client library docs, and independent confirmation of API usage.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”

   proof ↗⚑ flag

6. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to HumanLayer
   Codegen—none0/10

   Codegen's SDK/API uses a simple org_id + token authentication (codegen-docs-15) with no documentation of scoped, role-based, or least-privilege credential issuance, token permissions, or granular access controls. Missing for 10: any mention of scoped API keys, permission levels, or restricted-access tokens for agents.

   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

   HumanLayer~partialclaimed3/10

   The only relevant evidence is a mention of a 'launch token' scoped to a single non-interactive command, which implies some least-privilege token issuance, but there is no documentation of a broader credential/permission model, scopes, or API key management for agents. missing for 10: explicit least-privilege credential scoping model, permission granularity, revocation/rotation mechanisms, and any independent corroboration.

   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”

   proof ↗⚑ flag

7. ai-native userBuild against official SDKs

   weight 2 · round to Codegen
   Codegen✓fullprobed8/10

   Codegen documents an official Python SDK for programmatic agent control (agent.run, org_id/token auth), a full API reference with traces and CI/CD triggers, and a companion CLI — all first-party docs. missing for 10: independent/hands-on corroboration beyond docs, and no public OpenAPI spec was found (probe returned 404s).

   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
   • [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”

   proof ↗⚑ flag

   HumanLayer—none0/10

   The evidence pack covers HumanLayer's CLI, workspace config, and third-party integrations (Jira, Slack, GitHub, Linear) but contains no mention of an official SDK (Python, TypeScript, etc.) for building against HumanLayer programmatically. Probes for openapi.json and llms.txt both returned 404, further suggesting no discoverable API/SDK surface.

   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

8. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Codegen—none0/10

   No evidence in the pack mentions webhooks or event-subscription mechanisms; only SDK/API polling, CI/CD triggers, and integrations (Slack, Linear, Jira) are documented, none of which describe webhook subscriptions.

   ⚑ flag

   HumanLayer—none0/10

   No evidence pack item mentions webhooks or event subscription mechanisms; integrations described (Slack, Jira, Linear, GitHub) are inbound task-creation connectors, not outbound webhook events, and API/OpenAPI probes returned 404s.

   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Codegen
   Codegen~partialclaimed5/10

   Codegen generates AI-driven insights specifically around code artifacts—summarizing PR diffs, running security/secrets scans, checking test coverage, and suggesting changes/comments on PRs and issues—which counts as data-driven insight generation within its coding-agent context. However, these insights are narrowly scoped to code/repo/PR data rather than broader 'your data' (e.g., databases, business metrics) despite a mention of database connectivity via MCP with no insight-generation detail there. Missing for 10: evidence of insights/suggestions generated from non-code data sources (databases, analytics), and no independent/hands-on corroboration of insight quality.

   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed4/10

   HumanLayer's agents do generate task artifacts, draft PRs, and comments derived from a user's codebase/tickets, and 'Advanced Metrics' surfaces usage/cost/productivity data, which loosely resembles data-derived insight. However there is no evidence of dedicated AI-generated analytical insights or proactive suggestions distinct from executing assigned coding tasks. Missing for 10: explicit insight/analytics generation from user data, proactive suggestion features beyond task execution, and any first-party or community evidence of an 'insights' capability.

   • [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”
   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round drawn
   Codegen✓fullclaimed7/10

   Codegen documents agents that trigger autonomously from CI/CD failures, Slack/Linear/Jira events, and PR activity (auto-review, auto-comment), plus SDK/CLI for programmatic background automation — this matches the story of autonomous background automations. Missing for 10: independent/hands-on verification of reliability, and no explicit scheduled/cron-trigger mechanism documented.

   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed7/10

   Docs explicitly describe `humanlayer automation run` for running Cloud-visible coding sessions from CI jobs, cron machines, or scripts, plus remote daemons and launch tokens for non-interactive/background execution contexts, directly supporting autonomous background automations. Missing for 10: independent/hands-on verification of long-running background automations, native scheduling UI, and clarity on how human-approval gates interact with continuous autonomous runs.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”

   proof ↗⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round to Codegen
   Codegen✓fullclaimed8/10

   Codegen's documentation clearly describes delegating coding tasks to a built-in AI agent that analyzes requirements, implements features, fixes bugs, writes tests, and reports progress via prompts, accessible through chat, Slack, CLI, and SDK. Missing for 10: independent/hands-on verification of the assistant's actual task-completion quality beyond vendor docs.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed7/10

   HumanLayer's core product model is task delegation to AI coding agents (Claude Code, Codex, Bedrock-backed Claude) via 'sessions', with workflow modes (Oneshot, RPI, PRD-Oriented, Freeform) and automation runs from CI/cron, directly matching 'delegate tasks to a built-in AI assistant'. missing for 10: independent/hands-on verification of the delegation experience beyond vendor docs, and clarity on how autonomous vs supervised the assistant is in practice.

   • [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”

   proof ↗⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round to Codegen
   Codegen✓fullclaimed8/10

   Codegen is fundamentally natural-language driven: docs show prompt-based task execution ('Leave a review on PR #123'), Slack/Jira/Linear interactions, and SDK agent.run(prompt=...) all operate via NL commands across code, PRs, and issue trackers. This spans chat, CLI, and SDK surfaces, consistently prompt/NL-driven. Missing for 10: independent/hands-on corroboration of NL command interpretation quality and edge cases beyond first-party docs.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed7/10

   HumanLayer's core interaction model is giving natural-language instructions to agent sessions (Claude Code, Codex) to create tasks, configure workspaces, and choose workflow phases, as shown by the example NL workspace-config prompt and workflow-selection docs. Missing for 10: independent/hands-on corroboration of NL command robustness and no evidence of a broader NL command surface beyond task/workflow setup.

   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Codegen—none0/10

   Codegen has an API reference page (overview, SDK docs) but no evidence of an interactive reference with runnable/executable examples; the OpenAPI/spec probe returned 404 for all candidate paths, suggesting no interactive API explorer exists.

   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”

   proof ↗⚑ flag

   HumanLayer—none0/10

   No evidence of an interactive API reference or runnable examples; probes explicitly show no OpenAPI/swagger spec and no llms.txt found, and docs are guide/tutorial style rather than an API reference sandbox.

   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   Codegen—none0/10

   Codegen has an API reference (docs-16/17/18) but explicit probes for machine-readable spec files (openapi.json, swagger.json, etc.) all returned 404, indicating no downloadable OpenAPI/machine-readable spec is available.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

   HumanLayer—none0/10

   A direct probe for OpenAPI/swagger specs at all standard locations returned 404s, and no evidence pack item shows a downloadable machine-readable API spec being offered.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Codegen
   Codegen~partialclaimed5/10

   Codegen docs state that code changes run in 'robust isolated environments' to safely execute code and test changes, which implies sandboxed testing separate from production. However, there's no explicit description of how production data is excluded or protected, no detail on the sandbox's scope/persistence, and no independent verification of this isolation claim. missing for 10: explicit production-data isolation guarantees, technical detail on sandbox architecture, independent/hands-on confirmation.

   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's docs describe remote daemons, workspaces, and automation sessions, but there is no mention of a sandbox environment, staging/test data isolation, or any mechanism to separate test runs from production data. missing for 10: explicit sandbox/staging environment documentation, data isolation guarantees, evidence of test-vs-production separation.

   ⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Codegen—none0/10

   There's an API reference and SDK docs, but no evidence of API versioning scheme or a documented deprecation policy; the openapi probe even returned 404s, suggesting no formal spec surfacing version info.

   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”

   proof ↗⚑ flag

   HumanLayer—none0/10

   No evidence of API versioning scheme or a documented deprecation policy; probes for openapi.json/llms.txt returned 404s and no API reference or changelog covering versioning/deprecation is present.

   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Codegen
   Codegen~partialclaimed5/10

   The Python SDK and API allow programmatic creation/monitoring of agents and triggering from CI/CD, which could be scripted in a loop to perform bulk operations, but there is no documented batch/bulk endpoint or explicit guidance for operating across many items at once. missing for 10: dedicated bulk/batch API or CLI command, documented examples of multi-item automation, and evidence of scale/performance handling for bulk workloads.

   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's documentation consistently frames work as single tasks/sessions ('one small task', 'one task on a remote machine', 'one Cloud-visible coding session') with per-task review and approval workflows; there is no mention of batch/bulk operations spanning many items at once. missing for 10: any documented bulk-action API/CLI flag, batch approval mechanism, or multi-item automation workflow.

   • [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to HumanLayer
   Codegen~partialclaimed5/10

   Codegen supports event-driven automation via CI/CD triggers (docs-18) and PR review triggers on failure (docs-20), plus SDK/API for programmatic agent invocation, but there's no evidence of a general user-facing rules engine for defining arbitrary trigger-condition-action automations across events. missing for 10: a documented rules/automation builder UI, broader event types beyond CI/CD failures and PR diffs, and independent confirmation of custom trigger logic.

   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer integrations (GitHub, Jira, Linear) create tasks automatically from external events like new issues/tickets, and `humanlayer automation run` lets sessions be triggered from CI jobs, cron, or scripts — both are forms of event-driven automation. However, there's no evidence of a general-purpose rules/conditions engine letting users define arbitrary trigger-condition-action logic; the automation is limited to fixed integration hooks and script-based invocation. Missing for 10: a documented rule-definition interface (conditions, filters, custom triggers) beyond fixed platform integrations, and independent confirmation these event-triggers work reliably in practice.

   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round to HumanLayer
   Codegen—none0/10

   No evidence of any scheduling, cron-like, or recurring job/workflow capability; the docs cover SDK/API triggers, CI/CD triggers, and manual agent runs, but nothing about time-based or recurring automation.

   ⚑ flag

   HumanLayer~partialclaimed4/10

   HumanLayer's `automation run` CLI lets you trigger a Cloud-visible coding session from a cron machine or CI job, implying recurring runs are possible via external schedulers, but there is no documented native scheduling/recurrence feature (no cron syntax, interval config, or job queue) inside HumanLayer itself. missing for 10: built-in recurring scheduler, interval/cron configuration options, evidence of persistent recurring workflow management.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round to HumanLayer
   Codegen—none0/10

   Evidence covers PR/code review, commits, branches, and traces/telemetry, but there is no mention of versioning automations themselves, reviewing automation history, or rolling back agent configurations/workflows. missing for 10: automation versioning system, rollback mechanism for agent workflows, audit/history UI for automations.

   ⚑ flag

   HumanLayer~partialclaimed5/10

   HumanLayer's task model provides review (comments, PR draft creation) and history that persists across sessions, giving some review/audit capability, but there is no explicit documentation of versioning workflow definitions or rolling back an automation to a prior version. missing for 10: explicit automation versioning/history diffing, a documented rollback mechanism, and independent confirmation these review features extend to full automation lifecycle management.

   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”

   proof ↗⚑ flag

End to end feature delivery

1. ai-native userHave an agent automatically generate and run tests to validate its own code changes before proposing them

   weight 2 · round to Codegen
   Codegen~partialclaimed6/10

   Docs state the agent writes tests as part of implementing features and can run code/install dependencies/test changes in isolated environments, and a PR-review flow checks test coverage before flagging issues — together implying self-validation via generated tests. However, there is no explicit end-to-end walkthrough showing the agent generating tests, running them, and iterating/fixing before proposing a PR, nor independent/hands-on confirmation. missing for 10: explicit documented workflow tying test generation+execution to pre-PR validation, independent corroboration of this loop working in practice.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”

   proof ↗⚑ flag

   HumanLayer—none0/10

   No evidence in the pack describes HumanLayer agents automatically writing or executing tests to self-validate code changes; docs cover task/session management, integrations, and workflow phases but not test generation/execution as a validation step.

   ⚑ flag

2. developerHave an agent autonomously diagnose and fix a reported bug

   weight 3 · round drawn
   Codegen~partialclaimed6/10

   Docs explicitly claim the agent can 'analyze requirements, implement features, fix bugs, write tests' and can be triggered from issue trackers/CI to autonomously act, plus run in isolated environments to test changes — directly supporting autonomous bug diagnosis and fix. However, all evidence is first-party marketing/docs with no independent or hands-on validation of end-to-end autonomous diagnosis-to-fix workflows, and no detailed walkthrough of the diagnosis step itself (e.g., root-cause analysis) beyond generic 'fix bugs' language. Missing for 10: independent/hands-on verification of successful autonomous bug fixes, detailed diagnostic reasoning process, and success-rate or case-study evidence.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

   HumanLayer~partialcommunity6/10

   HumanLayer supports creating tasks directly from GitHub/Jira/Linear issues and then running an agent session (Claude Code/Codex) against the linked repo, with an 'Oneshot' workflow phase designed for small, clear changes — a plausible bug-fix pipeline. However, the product's core design is human-in-the-loop with approval gates rather than fully autonomous action, and there's no end-to-end documented example of an agent independently diagnosing a bug from a ticket and shipping a fix without human review. Missing for 10: a concrete autonomous bug-diagnosis-and-fix walkthrough, and clarity on how much human approval is required mid-flow.

   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”

   proof ↗⚑ flag

3. product-managerGo from a mockup or design to a working implementation without an engineering handoff

   weight 2 · round to Codegen
   Codegen~partialclaimed4/10

   Codegen's docs show relevant building blocks — a Figma connector, Slack/Linear interfaces that let non-engineers prompt the agent, and full PR/branch creation — implying a PM could describe a design and get code without direct engineering handoff. However, there is no concrete workflow, case study, or example showing a mockup/design being turned into a working implementation end-to-end. Missing for 10: an explicit design-to-code workflow example, evidence of Figma-to-PR pipeline, and PM-specific case studies or testimonials showing the handoff-free process actually works.

   • [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's evidence is entirely about developer-facing workflows: running coding sessions, connecting Jira/Linear/GitHub/Slack, managing remote daemons, and CLI automation for engineers overseeing coding agents. Nothing in the pack shows a mockup/design import capability, a no-code interface, or any path for a non-engineer product manager to turn a design into a working implementation without engineering involvement — in fact the workflow language (RPI, PRD-oriented, Oneshot) and workspace configs assume an engineering operator. Missing for evidence of delivering this story: mockup/design ingestion, PM-oriented no-code UI, and any case study of a non-engineer shipping code end-to-end.

   • [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”

   proof ↗⚑ flag

4. developerHave an agent implement a requested feature end-to-end, including writing tests

   weight 3 · round to Codegen
   Codegen✓fullclaimed8/10

   Docs explicitly describe agents analyzing requirements, implementing features, writing tests, and running/testing changes in isolated environments, plus PR/branch management for end-to-end delivery. Missing for 10: independent/hands-on validation of end-to-end feature delivery and no concrete example transcript showing a full feature+test cycle.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   HumanLayer clearly supports end-to-end autonomous coding sessions (Oneshot, RPI, PRD-Oriented workflows) that implement tasks using agents like Claude Code and Codex, including structured phases and sub-agents for research/plan/implement, which implies substantial feature work can be delegated (humanlayer-docs-6, humanlayer-docs-9, humanlayer-docs-10, humanlayer-docs-14). However, no evidence explicitly confirms the agent writes or runs tests as part of the workflow, and no hands-on account demonstrates a full feature-plus-tests delivery. Missing for 10: explicit documentation or example showing test generation/execution as part of the implementation flow, and independent verification of end-to-end feature completion including tests.

   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

Environment setup

1. developerHave an agent automatically clone the repo, install dependencies, and configure its own working environment

   weight 2 · round to Codegen
   Codegen✓fullclaimed8/10

   Docs describe agents that access repos via GitHub App integration, run in isolated environments, and 'safely run code, install dependencies, and test changes' automatically without complex setup, matching the story of autonomous environment setup. Missing for 10: independent/hands-on verification of environment provisioning behavior and more detail on how dependency installation is auto-configured beyond the brief doc mention.

   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   Docs describe workspace configuration (workspace.json, multi-repo setups) and remote hosts that must have access to code/tools/credentials, and one example prompt asks the agent about 'setup commands or local files to copy,' implying some environment configuration ability. However, there is no explicit description of the agent autonomously cloning a repo or installing dependencies end-to-end without human setup of the host/workspace first. Missing for 10: explicit documentation of automatic repo cloning, dependency installation steps, and end-to-end environment bootstrap without prior manual host/workspace configuration.

   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

Interactive takeover

1. developerTake over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work

   weight 2 · round to HumanLayer
   Codegen~partialclaimed5/10

   Codegen supports interacting with agents across multiple surfaces (Slack notifications/clarification requests, CLI to pull agent branches locally, Python SDK to run/monitor agents) and lets you download in-progress work to your local environment, which implies some ability to pick up and continue work. However there is no explicit documentation of pausing, redirecting, or steering a task mid-execution from editor/terminal/browser as a unified handoff flow. Missing for 10: explicit 'takeover' or redirect mid-task feature, editor/IDE integration for live task control, and evidence of seamless cross-surface handoff (Slack->CLI->browser) of the same in-progress task.

   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed7/10

   Docs describe tasks with session/history continuity 'across agents and workstations', remote control from app.humanlayer.com on any machine or phone, and CLI-driven remote daemons for terminal/server contexts, all pointing to genuine hand-off of in-progress work between editor (Claude Code), terminal (remote daemon/CLI), and browser (web app). Live multiplayer draft recovery further shows shared/continuable session state. missing for 10: independent/hands-on confirmation of a live takeover mid-task, and explicit description of an in-editor (IDE plugin) takeover UI rather than just CLI/app/web.

   • [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”

   proof ↗⚑ flag

2. developerSend follow-up instructions to an active agent session to steer its work without restarting

   weight 2 · round to HumanLayer
   Codegen—none0/10

   No evidence describes sending follow-up instructions to an already-running agent session to redirect its work; docs only describe initiating agents via prompts, Slack notifications, or SDK task creation, not mid-session steering. Missing for 10: any documentation of a 'continue/steer/update' API, chat-like follow-up in an active run, or session-resume mechanism.

   ⚑ flag

   HumanLayer~partialclaimed4/10

   HumanLayer's task/session model implies ongoing interaction with agents across a task's lifetime (docs-5) and release notes mention live prompt collaboration during sessions (docs-18), suggesting some capacity to interact with an active session, but there is no explicit documentation of sending mid-session follow-up instructions to steer a running agent without restarting it. missing for 10: explicit docs on injecting new instructions into a live/running session, confirmation the agent incorporates such input without restart, and independent/hands-on verification of this steering behavior.

   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”

   proof ↗⚑ flag

Sandbox execution

1. developerHave an agent safely execute code and install dependencies inside an isolated sandbox

   weight 3 · round to Codegen
   Codegen✓fullclaimed7/10

   First-party docs explicitly state the agent can 'Safely run code, install dependencies, and test changes in robust isolated environments,' directly matching the story. missing for 10: independent/hands-on corroboration of sandbox isolation, technical details on sandbox implementation (e.g., container tech, resource limits) beyond a single doc line.

   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer docs describe running sessions on remote hosts (cloud VM, workstation, private-network machine) and automation environments, but there is no mention of an isolated/sandboxed execution environment for running code or installing dependencies safely — the host selection is about access/credentials, not isolation guarantees. Missing for full/partial: any explicit sandbox, container, or isolation mechanism; no evidence of dependency-install safety controls.

   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

Approval controls

1. developerConfigure an agent to auto-approve all its actions instead of confirming each one

   weight 2 · round to HumanLayer
   Codegen—none0/10

   The evidence describes Codegen's autonomous capabilities (running code, committing, creating PRs, Slack/Jira integration) but never mentions any per-action confirmation mode or an auto-approve toggle/setting for oversight control. Missing for 10: any documentation of an approval/confirmation workflow, a setting to enable/disable auto-approval, or human-in-the-loop gating that this story presupposes.

   ⚑ flag

   HumanLayer~partialclaimed4/10

   HumanLayer's core premise is human-in-the-loop approval, and docs mention workflow phases like 'Oneshot' for low-risk changes and automation sessions (humanlayer automation run) that run non-interactively without confirmation, implying some auto-approve capability exists, but no explicit documentation of a configurable 'auto-approve all actions' toggle or setting is shown. missing for 10: explicit config/flag to disable per-action confirmation entirely, documentation confirming automation sessions skip all human review rather than just running unattended, and independent confirmation this works as intended.

   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

2. product-managerApprove key agent decisions from my phone while agents continue working

   weight 1 · round to HumanLayer
   Codegen~partialclaimed3/10

   Codegen supports Slack-based notifications and agent interaction (docs-2, docs-9), which could technically be used from a phone via the Slack app, but there is no explicit mobile app, push-approval workflow, or documented 'approve on the go' feature for key agent decisions. missing for 10: dedicated mobile/approval UI, evidence of approving specific agent decisions (e.g., merge/deploy gates) from a phone, and any hands-on confirmation of this workflow.

   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”

   proof ↗⚑ flag

   HumanLayer✓fullcommunity8/10

   Docs explicitly describe controlling and continuing a running agent task from a phone via app.humanlayer.com, with tasks providing a shared review/comment history across devices, directly matching the phone-approval-while-agent-continues story; community sentiment corroborates the human-approval-of-agent-actions use case. Missing for 10: a hands-on/independent account specifically confirming the mobile approval UI in practice, and explicit documentation of an 'approve/deny decision' action (vs. general task control) on mobile.

   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”

   proof ↗⚑ flag

3. engineering-leadSet tiered autonomy levels controlling what an agent can do without manual confirmation

   weight 3 · round to HumanLayer
   Codegen—none0/10

   No evidence describes configurable autonomy tiers, approval gates, or permission levels for agent actions; the docs describe what agents can do (code, PRs, Slack, Jira/Linear) but nothing about setting graduated confirmation thresholds for an engineering lead to control autonomy.

   ⚑ flag

   HumanLayer~partialclaimed5/10

   HumanLayer's workflow-phases doc shows tiered approaches (Oneshot for low-risk changes vs. more review for unclear/risky changes) and skills-workflows lets users select Oneshot/RPI/PRD-Oriented/Freeform modes, which map to different levels of autonomy vs. oversight. However, there's no explicit documentation of a formal 'autonomy level' setting per agent/task with configurable confirmation thresholds, and no independent evidence confirming this tiered control works as an oversight mechanism in practice. missing for 10: explicit named autonomy-tier configuration (e.g., low/medium/high) tied to confirmation gating, evidence of engineering-lead-level policy controls across a team, and independent/hands-on validation that these workflow phases actually reduce unnecessary confirmations without sacrificing safety.

   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”

   proof ↗⚑ flag

Model control

1. ai-native userHave each task prompt automatically routed to the most suitable underlying model

   weight 2 · round drawn
   Codegen—none0/10

   No evidence anywhere in the pack of automatic model routing per task/prompt; Codegen's docs describe agent capabilities, integrations, CLI/SDK, but nothing about selecting or routing between underlying LLMs based on task type.

   ⚑ flag

   HumanLayer—none0/10

   Evidence shows HumanLayer lets users manually select or configure which model/backend to use (Claude via Bedrock, Codex, RPI sub-agents) but there is no evidence of automatic routing of a task prompt to the 'most suitable' model based on task characteristics.

   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
   • [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
   • [claimed-docs] “HumanLayer sessions can run Claude through Amazon Bedrock instead of the Anthropic API.”

   proof ↗⚑ flag

2. engineering-leadSwitch away from automatic model selection to a specific model of my choice

   weight 1 · round to HumanLayer
   Codegen—none0/10

   No evidence in the pack mentions model selection, choice of underlying LLM, or any setting to override automatic model routing; the docs cover integrations, CLI, SDK, and agent workflows but never reference model choice.

   ⚑ flag

   HumanLayer~partialclaimed6/10

   Docs show explicit model/backend selection — choosing Codex ("install, authenticate, select, and check Codex") or running Claude via Amazon Bedrock instead of the Anthropic API, plus subagent model registration — indicating an engineering lead can pick a specific model rather than a default. However, there is no explicit documentation of an 'automatic' default-selection mode being toggled off, so the framing of 'switching away from automatic' isn't directly evidenced. Missing for 10: explicit mention of an automatic/default model-selection setting and a documented UI/CLI flag to override it, independent confirmation of model-switching behavior.

   • [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
   • [claimed-docs] “HumanLayer sessions can run Claude through Amazon Bedrock instead of the Anthropic API.”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”

   proof ↗⚑ flag

Visibility monitoring

1. developerWatch what a running agent is doing in real time, including its current status

   weight 3 · round drawn
   Codegen~partialclaimed6/10

   Codegen's CLI lets developers view agents and run them with 'full telemetry and monitoring,' and agents can 'report progress' via Slack, giving some real-time visibility into agent activity and status. However, there's no explicit documentation of a live dashboard/status stream showing step-by-step actions as they happen, and the closest 'trace' feature is framed as post-hoc analysis/debugging rather than live watching. Missing for 10: a documented real-time UI/dashboard showing live agent status, explicit streaming of in-progress actions, and independent/hands-on confirmation of live monitoring accuracy.

   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer's docs describe remote-daemon control from app.humanlayer.com (including from a phone), live multiplayer session viewing with presence/read-only viewing, and task/session history — all suggesting real-time visibility into agent activity. However, there's no explicit documentation of a dedicated live status/log stream or dashboard showing granular agent state (e.g., current tool call, progress bar) beyond session/task views. missing for 10: explicit real-time status/log streaming documentation, independent hands-on confirmation of live monitoring UX.

   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

2. developerGet notified when an agent completes a task or needs my input

   weight 2 · round to Codegen
   Codegen✓fullclaimed8/10

   Codegen explicitly supports sending notifications, asking for clarification, and reporting progress directly in Slack channels, plus updating statuses/comments in Jira, Linear, Clickup, and Monday.com, covering both completion and input-needed scenarios. missing for 10: independent/hands-on confirmation of real-time notification delivery, and no explicit mention of email/other channel notification options.

   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”

   proof ↗⚑ flag

   HumanLayer~partialcommunity6/10

   HumanLayer supports Slack/GitHub/Jira/Linear integrations that push task artifact updates and human-in-the-loop approvals, implying notification when tasks progress or need input, and its core design centers on human oversight of agent work. However, there is no explicit documentation of a dedicated 'task complete' or 'needs input' notification/alert mechanism (e.g., push notification, email, or webhook triggered specifically on completion/approval-required events) beyond generic artifact updates in Slack. missing for 10: explicit completion/needs-input notification docs, evidence of notification latency/reliability, independent/hands-on confirmation of notification delivery.

   • [claimed-docs] “Connect Slack to send HumanLayer task artifact updates into the channels where your team already works.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”

   proof ↗⚑ flag

Natural language task intake

1. developerDescribe a feature or bug in plain language and have it automatically turned into a scoped implementation task

   weight 3 · round to Codegen
   Codegen✓fullclaimed7/10

   Codegen's docs describe taking natural-language prompts and turning them into scoped agent tasks: analyzing requirements, implementing features, fixing bugs, writing tests, and running agents from prompts via SDK/CLI/Slack/issue trackers. This directly matches the intent-to-spec story of describing a feature/bug and having it become an actionable implementation task. Missing for 10: no independent/hands-on evidence showing the quality or accuracy of the generated scoped task (e.g. how well it decomposes ambiguous requests), and no detail on how it handles clarification/scoping edge cases beyond a general 'ask for clarification' mention.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer's task model (docs-5) and workflow-phase selection (docs-6, docs-10) show that a task is created and can be routed through 'Oneshot' for small clear changes or heavier RPI/PRD-oriented flows for ambiguous work, and RPI sub-agents (docs-9) imply a research→plan→implement pipeline that turns a description into a scoped plan. However, there is no explicit walkthrough showing a raw plain-language bug/feature description being automatically parsed into a scoped implementation task end-to-end, and integrations (Jira/Linear/GitHub) mostly create tasks from existing tickets rather than free-form language input. Missing for 10: a concrete example or tutorial of plain-language-to-scoped-task conversion, and independent/hands-on confirmation that this pipeline works as described.

   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”

   proof ↗⚑ flag

2. product-managerConvert user feedback submissions into structured tasks with proposed scope

   weight 2 · round to HumanLayer
   Codegen~partialclaimed3/10

   Codegen can analyze requirements and create/link tasks in Jira, Linear, Clickup, Monday.com, which could theoretically be used to turn feedback into tasks, but there is no evidence of a feedback-intake feature or automatic scope proposal tailored for PM workflows. Missing for 10: dedicated feedback ingestion mechanism, structured scope/estimate output, PM-specific workflow examples or case studies.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed4/10

   HumanLayer supports creating tasks from external issue trackers (Jira, Linear, GitHub) which could serve as a proxy for user feedback submissions, and tasks include shared files/scope info, but there's no evidence of a dedicated feature for ingesting raw user feedback (e.g., support tickets, survey responses) and auto-structuring it into a task with a proposed scope specifically tailored for PM workflows. missing for 10: dedicated feedback-ingestion mechanism, explicit 'proposed scope' generation from unstructured feedback text, PM-specific workflow templates, and any hands-on/community validation of this specific use case.

   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

3. developerAttach a marked-up screenshot or mockup to a task so the agent implements the correct visual change

   weight 2 · round to HumanLayer
   Codegen—none0/10

   No evidence in the pack mentions attaching screenshots, images, or marked-up mockups to a task/prompt; Codegen's docs cover text prompts, Slack/Jira/Linear integrations, CLI/SDK, and PR review but nothing about visual/image input for implementing UI changes. Figma is mentioned only as a connectable integration, not as an image-annotation workflow.

   • [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed4/10

   Docs confirm images can be pasted into the new task composer as attachments (humanlayer-docs-22), which supports attaching a screenshot to a task, but there is no evidence of markup/annotation tooling or of the agent parsing visual annotations to implement a corresponding UI change. Missing for 10: annotation/markup capability for screenshots, evidence the agent interprets visual markup into a specific implementation, and any hands-on example of this workflow succeeding.

   • [claimed-docs] “Image paste in new tasks — Paste images straight into the new task composer as attachments.”

   proof ↗⚑ flag

Plan approval

1. developerReview and approve an agent's implementation plan before any code changes are made

   weight 3 · round to HumanLayer
   Codegen—none0/10

   No evidence in the pack of a plan-preview or approval step prior to code changes; agents appear to run prompts directly to implementation (create branches, commit code, open PRs) without a documented plan-review gate. missing for 10: any mention of a plan/preview mode, approval gate, or dry-run step before code execution.

   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer's workflow-phases and RPI sub-agent docs describe planning phases (e.g., 'a change with unclear behavior or code shape needs more review before implementation') and PRD-Oriented/RPI workflows imply a plan stage before code changes, with tasks providing 'one place for comments and review.' However, no evidence explicitly shows a dedicated plan-approval gate/UI step where a developer reviews and approves a plan artifact before implementation begins. missing for 10: explicit documentation of a plan-approval step/UI, first-party example of blocking implementation until plan is approved, independent/hands-on confirmation of this specific gate.

   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

2. engineering-leadApprove a task's scope and contract before an agent is allowed to modify the repository

   weight 2 · round to HumanLayer
   Codegen—none0/10

   Evidence shows Codegen agents can be given prompts and run autonomously with PR review/creation, but there is no mention of any scope/contract approval gate or engineering-lead sign-off step required before an agent is allowed to modify a repository.

   ⚑ flag

   HumanLayer~partialcommunity6/10

   HumanLayer's workflow-phases doc explicitly supports scoping review before implementation (e.g., 'a change with unclear behavior or code shape needs more review before implementation'), and tasks/sessions provide a structured place for comments and review prior to agent execution, plus human-in-the-loop approval is core to the product's value prop per community discussion. However, there's no explicit documented feature for an engineering-lead specifically approving a 'scope and contract' artifact as a gating step before repo modification — it's inferred from general workflow-phase and review mechanics rather than a dedicated scope-approval gate. missing for 10: explicit documentation of a formal scope/contract approval step tied to lead sign-off, evidence of blocking repo writes until such approval, and independent/hands-on confirmation this gate works as intended.

   • [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
   • [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”

   proof ↗⚑ flag

Ticket driven tasking

1. developerAssign a coding task to an agent directly from an existing issue or ticket

   weight 3 · round to HumanLayer
   Codegen✓fullclaimed7/10

   Docs show native integrations with Jira, Linear, ClickUp and Monday.com allowing the agent to update issue statuses, add comments, link PRs and create tasks, and specifically state connecting a Linear workspace 'enables agent interactions with your issues,' which directly supports assigning work from a ticket. Missing for 10: a concrete end-to-end example of assigning a task from an issue/ticket, and independent/hands-on corroboration beyond vendor docs.

   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed8/10

   Docs explicitly describe connecting GitHub, Jira, and Linear so HumanLayer creates tasks directly from issues/tickets and links artifacts back to the source, directly matching the story of assigning agent work from an existing ticket. Missing for 10: independent/hands-on confirmation that this ticket-to-task flow works reliably in practice, and more detail on the actual assignment UX.

   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Codegen
   Codegen~partialprobed5/10

   Codegen offers a Python SDK, CLI, and API to create/run agents, pull work, and monitor executions, showing solid programmatic control (codegen-docs-11,15,16,17,18). However, there's no evidence of a complete OpenAPI/API surface matching every UI capability (e.g., Slack/Jira/Linear integration management, PR review workflows) — the OpenAPI probe returned 404s, suggesting no public full API spec. missing for 10: OpenAPI reference confirming full UI parity, evidence that integration configs and review workflows are API-accessible, independent confirmation of API completeness.

   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”

   proof ↗⚑ flag

   HumanLayer~partialprobed3/10

   HumanLayer offers a CLI (`humanlayer automation run`) and remote daemon controls that let you launch and manage sessions outside the UI, but there is no documented public API/OpenAPI spec (both openapi.json and llms.txt probes 404), and UI-only features like live multiplayer drafts, keyboard navigation, and image paste have no CLI/API equivalent documented. missing for 10: a documented REST/GraphQL API or OpenAPI spec, confirmation that all UI actions (draft PRs, multiplayer editing, metrics views) are exposed programmatically, and independent verification of API-UI parity.

   • [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
   • [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”
   • [claimed-docs] “Keyboard navigation for changed files — Move through the PR changes tree with J/K, N/P, G shortcuts, and Enter.”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   Codegen—none0/10

   No evidence of a bulk data-export feature, open-format export of agent histories/traces, or account-portability tooling; only partial building blocks exist (pulling agent branches via CLI, retrieving traces via API) but nothing documenting a full data export/exit path.

   • [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”

   proof ↗⚑ flag

   HumanLayer—none0/10

   No evidence of data export functionality or open-format data portability; docs focus on tasks, workflows, and integrations, with no mention of exporting user data or account deletion/leaving.

   ⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Codegen—none0/10

   No evidence of an open-license source repository or public codebase for Codegen; all evidence relates to product features, docs, CLI, and SDK usage, not source availability or licensing.

   ⚑ flag

   HumanLayer—none0/10

   No evidence pack item mentions an open-source license or public source repository for HumanLayer; all evidence is docs, community discussion, or probes about APIs/CLI. Absence of evidence for this applicable axis yields none.

   ⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   Codegen—none0/10

   Codegen is presented as a cloud-hosted SaaS agent platform (GitHub App, Slack App, cloud CLI/SDK, hosted API); no evidence of any self-hosted deployment option, on-prem package, or open-source core repository. Absence of evidence for this applicable capability yields 'none'.

   ⚑ flag

   HumanLayer—none0/10

   Evidence shows HumanLayer's daemon/agent execution can run on a user-controlled host (cloud VM, workstation, private network), but the core control plane is explicitly tied to the hosted app.humanlayer.com service — no docs describe self-hosting that core product. missing for 10: no self-hosted control-plane/server option, no on-prem deployment guide, no Docker/Helm chart or license for running the full stack independently.

   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

Enterprise licensing

1. engineering-leadLicense an enterprise deployment with SSO and commercial support for organization-wide rollout

   weight 2 · round drawn
   Codegen—none0/10

   No evidence in the pack mentions SSO, enterprise licensing, commercial support tiers, or org-wide rollout plans; only feature-level docs and API/CLI references are provided. Missing for 10: SSO/SAML support, enterprise plan/pricing tier documentation, commercial support SLA, org-wide admin/rollout tooling.

   ⚑ flag

   HumanLayer—none0/10

   No evidence pack item mentions SSO, enterprise licensing, org-wide rollout, or commercial support tiers; docs cover tutorials, integrations, and workflow features only, and community discussion focuses on pricing/skepticism unrelated to enterprise licensing terms.

   ⚑ flag

Model flexibility

1. engineering-leadBring my own LLM or API key so agents run on the model of my choice

   weight 2 · round to HumanLayer
   Codegen—none0/10

   No evidence pack item mentions BYO-LLM, custom API key configuration, or model selection; Codegen's docs focus on its own agents, integrations, and SDK/CLI without any option to swap the underlying model provider.

   ⚑ flag

   HumanLayer~partialclaimed5/10

   Docs show HumanLayer sessions can use different backends/models — Claude Code, OpenAI Codex, and Claude via Amazon Bedrock instead of the Anthropic API — indicating some flexibility in model/provider choice, which implies bringing your own credentials for these paths. However, there is no explicit doc describing a generic 'bring your own API key' mechanism for arbitrary LLMs or a pricing-tier note tying this to cost savings for engineering leads. Missing for 10: explicit BYO-API-key configuration docs, support for arbitrary/third-party model providers beyond Claude/Codex/Bedrock, and any pricing-related messaging about cost control via own keys.

   • [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
   • [claimed-docs] “HumanLayer sessions can run Claude through Amazon Bedrock instead of the Anthropic API.”
   • [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”

   proof ↗⚑ flag

Usage quotas

1. engineering-leadSee and manage plan-based daily task and concurrency limits for agent workflows

   weight 2 · round drawn
   Codegen—none0/10

   No evidence pack item mentions plan-based daily task limits, concurrency limits, or any admin dashboard/controls for engineering leads to monitor or manage usage quotas; documentation covers agent capabilities, integrations, CLI, and SDK but nothing on pricing tiers or limit management.

   ⚑ flag

   HumanLayer—none0/10

   The evidence pack has no mention of plan-based daily task/concurrency limits or any admin controls for managing such limits; only a vague reference to 'Advanced Metrics for all paid plans' which covers usage/cost/productivity viewing, not concurrency or daily task limits management.

   • [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   Codegen—none0/10

   No evidence in the pack mentions data residency, region selection, or storage location options for Codegen; the docs cover integrations, CLI, SDK, and agent workflows but nothing about data storage location controls.

   ⚑ flag

   HumanLayer—none0/10

   No evidence in the pack addresses data region/residency selection or storage location controls; docs cover hosts, integrations, and workflows but never data residency options.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Codegen—none0/10

   No evidence in the pack addresses data-training opt-out, privacy controls, or policies about AI model training usage; all citations concern feature functionality (agents, CLI, SDK, integrations) rather than privacy posture.

   ⚑ flag

   HumanLayer—none0/10

   No evidence in the pack addresses data usage for AI model training, opt-out policies, or privacy commitments regarding training data; the docs focus on task workflows, integrations, and remote sessions.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Codegen—none0/10

   No evidence pack items discuss data retention policies, deletion controls, or privacy/compliance settings; the docs cover functionality (agents, integrations, CLI, SDK) but nothing about controlling or deleting stored data.

   ⚑ flag

   HumanLayer—none0/10

   No evidence pack items mention data retention policies, deletion controls, or privacy settings for HumanLayer; the docs focus on task/session workflows, integrations, and remote daemons rather than data governance.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Codegen—none0/10

   No evidence pack item mentions a telemetry opt-out setting or privacy controls; in fact docs-12 explicitly notes the CLI runs Claude Code 'with full telemetry and monitoring,' with no mention of a way to disable it.

   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   No evidence in the pack addresses telemetry, usage tracking, or opt-out controls for HumanLayer; docs cover integrations, workflows, and CLI usage but nothing about privacy/telemetry settings. Missing for 10: any mention of telemetry collection, opt-out mechanism, privacy policy, or data-handling documentation.

   ⚑ flag

Chat integration

1. developerTag an agent in a chat thread to discuss and delegate a bug or task

   weight 2 · round to Codegen
   Codegen~partialclaimed6/10

   Codegen supports Slack-based chat interactions where users can message/tag the agent to report progress, ask for clarification, and delegate tasks (codegen-docs-2, codegen-docs-9), and Linear/Jira integration allows commenting and task creation to delegate bugs (codegen-docs-3, codegen-docs-10). However, evidence doesn't explicitly confirm an '@mention' tagging mechanism within an existing chat thread to delegate a specific bug — it's inferred from general Slack/Linear interaction docs rather than a documented tagging workflow. Missing for 10: explicit documentation of @-mention/tagging syntax in threads, confirmation of thread-level context retention when delegating, and independent/hands-on verification of this specific workflow.

   • [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
   • [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's Slack integration only pushes task-artifact updates into channels (docs-4) and other integrations (GitHub, Jira, Linear) create tasks from tickets/issues, not from tagging an agent inside a chat thread. There is no evidence of a chat-native @mention or in-thread delegation workflow for discussing/assigning tasks to an agent.

   • [claimed-docs] “Connect Slack to send HumanLayer task artifact updates into the channels where your team already works.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”

   proof ↗⚑ flag

Knowledge context

1. developerAdd a context file describing my codebase conventions so agents generate more relevant plans and code

   weight 3 · round drawn
   Codegen—none0/10

   The evidence pack covers Codegen's integrations, CLI, SDK, and PR/issue automation, but nowhere mentions a repo-level context or rules file (e.g., conventions file) that developers can add to steer agent plans/code generation. missing for 10: any mention of a context/config file for codebase conventions, documentation on custom instructions, or repo-level rules ingestion.

   ⚑ flag

   HumanLayer—none0/10

   The docs describe workspace-level config files (workspace.json/workspace.local.json) for team/machine settings and multi-repo setup, but there is no evidence of a dedicated context file for describing codebase conventions to improve agent-generated plans/code.

   • [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”

   proof ↗⚑ flag

Project management integration

1. product-managerConnect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly

   weight 3 · round drawn
   Codegen~partialclaimed6/10

   Docs explicitly claim support for Jira, Linear, ClickUp, and Monday.com (update statuses, add comments, link PRs, create tasks), but detailed setup instructions are only shown for Linear (codegen-docs-10); Jira, ClickUp, and Monday.com lack dedicated connection/integration docs equivalent to Linear or Slack. Missing for 10: explicit setup guides or independent verification for Jira/ClickUp/Monday.com, and hands-on confirmation beyond first-party docs.

   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
   • [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer documents native Jira Cloud and Linear integrations that create tasks from tickets and sync/link status back to source issues, directly matching the story for those trackers. However, there is no evidence of ClickUp or Monday.com integrations, so the story is only partially delivered. Missing for 10: ClickUp integration docs, Monday.com integration docs, independent/hands-on verification of ticket sync working in practice.

   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”

   proof ↗⚑ flag

Version control integration

1. developerConnect a GitHub repository so an agent can access the code and open pull requests against it

   weight 3 · round drawn
   Codegen✓fullclaimed8/10

   Codegen provides a GitHub App install for repo access with no complex setup, and docs describe the agent creating branches, committing code, reviewing/opening PRs, and managing repositories. missing for 10: independent/hands-on confirmation beyond vendor docs, and explicit detail on multi-repo/org-level permission scoping.

   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed8/10

   Docs describe connecting GitHub to create tasks from issues and link artifacts back (humanlayer-docs-16), plus agent sessions can access code via configured workspaces/repos (humanlayer-docs-8, humanlayer-docs-11) and open draft PRs directly from the session UI (humanlayer-docs-19). missing for 10: independent/hands-on corroboration of the GitHub connection flow and PR-opening working end-to-end beyond first-party docs.

   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”

   proof ↗⚑ flag

2. developerGrant an agent access to my repositories with a one-click install, without complex setup

   weight 2 · round to Codegen
   Codegen✓fullclaimed8/10

   Docs explicitly state installing the GitHub App grants repo access with 'no complex setup required', directly matching the story, and this is corroborated by broader repo-integration capabilities (PR review, branch/commit management). Missing for 10: independent/hands-on confirmation of the one-click flow beyond first-party docs.

   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   Docs describe GitHub/Jira/Linear integrations for creating tasks from issues, but connecting a repo requires selecting a host, configuring workspace.json/workspace.local.json, and setting up remote daemons or multi-repo workspaces — none of this is framed as a one-click, no-setup install. No evidence pack item claims or demonstrates a one-click repo access flow.

   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”

   proof ↗⚑ flag

Ci remediation

1. engineering-leadHave failed CI workflows automatically diagnosed and fixed with a proposed pull request

   weight 3 · round to Codegen
   Codegen~partialclaimed6/10

   Docs explicitly mention triggering agents from CI/CD pipelines when builds fail and agents that fix bugs/write code and open PRs, directly supporting automated diagnosis-and-fix-PR workflows for failed CI. However, there is no dedicated end-to-end CI-failure-diagnosis feature description, no example of the proposed PR format for a failing build, and no independent/hands-on confirmation of this specific workflow. missing for 10: dedicated CI-failure-diagnosis workflow documentation, concrete example of a CI-triggered fix PR, independent verification of this specific capability.

   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   HumanLayer supports running automation sessions from CI (`humanlayer automation run`) and can open draft PRs from a session's diff/GitHub tab, and GitHub integration links tasks to issues—so the building blocks for a CI-triggered fix-and-PR flow exist. However there is no direct evidence of automatic diagnosis of failed CI logs/errors or a documented end-to-end 'CI failure → agent diagnosis → PR' pipeline. Missing for 10: explicit CI-failure-detection/diagnosis workflow docs, example of a failing pipeline auto-triggering a session, and confirmation the resulting PR addresses the CI failure specifically.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”

   proof ↗⚑ flag

2. developerTrigger an agent from CI/CD pipelines to fix a broken build or failing test

   weight 2 · round to Codegen
   Codegen✓fullclaimed8/10

   Docs explicitly state agents can be 'Triggered from CI/CD pipelines when builds fail' and support fixing bugs/tests via API/SDK with programmatic control, which directly matches the story. Missing for 10: independent/hands-on confirmation of a real CI/CD integration example (e.g., GitHub Actions snippet) and details on how failing test context is passed to the agent.

   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”

   proof ↗⚑ flag

   HumanLayer~partialclaimed6/10

   HumanLayer docs explicitly describe `humanlayer automation run` for running a Cloud-visible coding session 'from any automation environment — a CI job, a cron machine, or a script on a server,' plus launch tokens for non-interactive command execution, directly supporting CI/CD-triggered agent runs. However, there is no explicit example or integration guide showing a CI failure (broken build/failing test) triggering the agent to diagnose and fix it, nor independent/hands-on confirmation of this workflow. missing for 10: a concrete CI/CD pipeline example tied to build/test failures, evidence of automatic failure detection triggering the agent, and independent verification of this automation flow working in practice.

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”

   proof ↗⚑ flag

Diff review

1. developerConfigure an agent to automatically open a pull request when its task completes

   weight 2 · round to Codegen
   Codegen✓fullclaimed7/10

   Codegen's agent creates branches, commits code, and manages repositories via GitHub App integration, and can be run programmatically via SDK/CLI/API to complete tasks; PR creation is a core documented capability of the agent workflow. Missing for 10: explicit first-party documentation/example showing a config flag or setting to 'auto-open PR on task completion' and independent/hands-on confirmation of this exact behavior.

   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

   HumanLayer~partialclaimed4/10

   Docs confirm HumanLayer can open a draft pull request from a session (humanlayer-docs-19) and integrates with GitHub for issue-linked tasks (humanlayer-docs-16), but the evidence describes PR creation as a manual 'ask the session agent' action from the UI, not an automatic trigger fired upon task completion. Missing for 10: explicit configuration option/workflow setting for automatic PR creation on task completion, and any evidence of it happening without a manual prompt.

   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”

   proof ↗⚑ flag

2. developerReview a diff of an agent's changes and approve it before it becomes a pull request

   weight 3 · round to HumanLayer
   Codegen~partialclaimed5/10

   Codegen agents create branches, commit code, and open PRs, and the agent itself can review PRs and post comments, but the evidence pack does not show a dedicated human-in-the-loop diff review/approval gate before a PR is created — the CLI lets you 'pull agent work' locally which could support manual review, but no explicit approval workflow is documented. missing for 10: explicit pre-PR diff approval UI/workflow, evidence of a human approve/reject gate, independent confirmation that PR creation is blocked pending review.

   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”

   proof ↗⚑ flag

   HumanLayer~partialcommunity7/10

   Release notes explicitly reference a diff view and 'draft PR creation' workflow (docs-19, docs-21), and tasks include 'One place for comments and review' (docs-5), indicating a review-before-PR mechanism. However, there's no detailed documentation of an explicit approve/reject gate tied specifically to diff review prior to PR creation, and community discussion raises concerns about automation bias in approvals (comm-3) without disputing the core capability. Missing for 10: explicit step-by-step approval workflow docs, independent hands-on verification of the diff-review-then-approve flow, and clarity on how rejection/edits are handled before PR creation.

   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Keyboard navigation for changed files — Move through the PR changes tree with J/K, N/P, G shortcuts, and Enter.”
   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”
   • [community] “User raised concern about automation bias: once an agent proves reliable, humans may rubber-stamp approvals without careful review, letting …”

   proof ↗⚑ flag

Pr review automation

1. ai-native userHave incoming issues automatically triaged with severity suggested and routed to the right owner

   weight 2 · round drawn
   Codegen—none0/10

   Evidence shows Codegen integrates with issue trackers (Jira, Linear, etc.) to update statuses, add comments, and create tasks, but there is no mention of automatic severity assessment or routing incoming issues to the correct owner. Missing for 10: automatic severity classification, owner/routing logic, any triage workflow evidence.

   • [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
   • [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer connects issue trackers (Jira, GitHub, Linear) to create tasks from tickets, but there is no evidence of automatic triage, severity classification, or routing to an owner — integrations only create/link tasks, not assess or assign severity/ownership.

   • [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
   • [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”

   proof ↗⚑ flag

2. engineering-leadHave every pull request automatically reviewed with AI-generated inline comments

   weight 3 · round to Codegen
   Codegen✓fullclaimed7/10

   Docs explicitly describe automated PR review functionality: reading the diff, summarizing changes, running security/secrets scans, checking test coverage, and posting a review comment, plus general 'Review PRs, suggest changes, comment on issues' capability tied to GitHub App integration for repo access. This matches the engineering-lead story of automatic AI-reviewed PRs with comments. Missing for 10: explicit confirmation that comments are inline (per-line) rather than a single summary comment, and independent/hands-on evidence corroborating this in production use.

   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's evidence covers task/session management, human-in-the-loop approvals, draft PR creation, and a PR diff-viewing UI, but there is no mention of an automated review process that generates inline review comments on every pull request. Missing for 10: no evidence of automatic PR review triggers, no mention of AI-generated inline comments, no review-quality-gate CI integration for PRs.

   • [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
   • [claimed-docs] “Keyboard navigation for changed files — Move through the PR changes tree with J/K, N/P, G shortcuts, and Enter.”
   • [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”

   proof ↗⚑ flag

Readiness checks

1. engineering-leadRun a readiness report that evaluates how ready my repository is for autonomous agents

   weight 2 · round drawn
   Codegen—none0/10

   The evidence pack shows PR review, security scanning, and CI/CD triggers, but there is no mention of a dedicated 'readiness report' feature that evaluates overall repository readiness for autonomous agents. Missing for 10: any documentation of a repo-wide readiness/scorecard report, criteria for agent-readiness, or a command/API that produces such an assessment.

   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”

   proof ↗⚑ flag

   HumanLayer—none0/10

   No evidence of any readiness-report or repository-readiness-assessment feature; HumanLayer's docs focus on task/session management, integrations, and human-in-the-loop workflows, not a diagnostic scan of a repo's agent-readiness.

   ⚑ flag

Security remediation

1. engineering-leadHave security alerts automatically validated and remediated with an opened pull request

   weight 2 · round to Codegen
   Codegen~partialclaimed5/10

   Codegen documents a security/secrets scan that posts a review comment on failure (codegen-docs-20) and general bug-fixing/PR-creation capabilities plus CI/CD-triggered agents (codegen-docs-1, codegen-docs-4, codegen-docs-18), suggesting it could act on a flagged issue and open a fix PR. However, there is no explicit evidence of ingesting external security alerts (e.g., Dependabot/CodeQL/Snyk), validating them as true positives, or automatically opening a remediation PR end-to-end — the documented scan flow stops at commenting, not remediating. missing for 10: explicit security-alert ingestion/validation workflow, evidence of an opened remediation PR tied to a specific alert, independent confirmation of this flow working in practice.

   • [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
   • [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”

   proof ↗⚑ flag

   HumanLayer—none0/10

   HumanLayer's docs show generic task creation from GitHub/Jira/Linear issues and draft PR creation from agent sessions, but there is no evidence of any security-alert-specific validation or automated vulnerability remediation workflow (no CVE, dependency-alert, or security-scanner integration mentioned).

   ⚑ flag

Concurrent execution

1. engineering-leadRun many agent tasks concurrently to scale delivery throughput

   weight 3 · round to Codegen
   Codegen~partialclaimed6/10

   Codegen's SDK/API let you programmatically create and run multiple agents (agent.run, API 'full programmatic control over execution and monitoring') and CLI can view multiple running agents, implying support for parallel task execution, and CI/CD triggers suggest scaling automation into pipelines. However, there is no explicit documentation of concurrency limits, throughput guarantees, or dashboards for managing many simultaneous tasks at scale. Missing for 10: explicit concurrency/parallelism claims, scaling limits or throughput benchmarks, and independent evidence of running many agents simultaneously in practice.

   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   Docs describe an architecture (tasks/sessions, multi-repository workspaces, remote daemons on cloud VMs, and a CLI 'automation run' for CI/cron/scripts) that could support running many agent tasks in parallel, and 'Advanced Metrics' track usage/productivity across an org. However, tutorials and guides are framed around running one task/session at a time, and there is no explicit documentation or evidence of concurrent multi-task orchestration, throughput dashboards, or scaling guarantees for many simultaneous agents. Missing for 10: explicit multi-session concurrency docs, evidence of parallel task orchestration at scale, and independent confirmation of throughput gains.

   • [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”

   proof ↗⚑ flag

2. engineering-leadCreate agent sessions on behalf of other users in my organization

   weight 2 · round to Codegen
   Codegen~partialclaimed4/10

   Codegen's org-level SDK/CLI/API (org_id+token, 'manage organizations', programmatic agent creation) implies an engineering lead with org credentials could spin up agents at scale, but no docs explicitly describe creating or assigning a session to another named user in the org. Missing for 10: explicit user-impersonation or 'create on behalf of' API/UI feature, role-based permissions for delegating agent sessions, and any admin console for assigning sessions to teammates.

   • [claimed-docs] “Manage organizations - Switch between organizations and configure repositories”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

   HumanLayer—none0/10

   The evidence describes tasks, sessions, integrations, and org-wide metrics access but never mentions an engineering-lead or admin creating/assigning agent sessions on behalf of another named user in the organization. No account-delegation, impersonation, or 'assign session to teammate' capability is documented.

   ⚑ flag

Deployment flexibility

1. developerUse a managed cloud offering to run agents without operating my own backend infrastructure

   weight 2 · round to Codegen
   Codegen✓fullclaimed7/10

   Codegen offers cloud-run agents with isolated execution environments, API/SDK/CLI access, and no need to self-host infrastructure (codegen-docs-5, codegen-docs-11, codegen-docs-15, codegen-docs-16). This directly matches running agents without operating own backend. Missing for 10: independent/hands-on verification of the managed infrastructure's reliability at scale, and explicit pricing/SLA details for the managed cloud offering.

   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
   • [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”

   proof ↗⚑ flag

   HumanLayer~partialclaimed5/10

   HumanLayer offers a cloud control plane (app.humanlayer.com, automation sessions marked 'Cloud-visible', advanced metrics for paid plans) that lets users monitor and manage agent runs remotely, suggesting a managed service layer. However, docs explicitly state the actual agent execution still runs on a host that the user must select and operate—'a cloud VM, workstation, or private-network machine'—meaning the developer still needs to provision/manage backend compute for the daemon rather than HumanLayer fully hosting execution. Missing for 10: evidence of a fully HumanLayer-operated compute backend (no user-managed VM/daemon required), and independent confirmation of a true zero-ops managed runtime.

   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”

   proof ↗⚑ flag

2. engineering-leadSelf-host agent infrastructure locally, in containers, or on my own VMs

   weight 2 · round to HumanLayer
   Codegen—none0/10

   Codegen is presented entirely as a cloud/SaaS agent service (GitHub App, Slack App, API, SDK, isolated cloud environments) with no evidence of a self-hostable runtime, container image, or on-prem/VM deployment option; the CLI logs local Claude Code sessions to Codegen's own cloud rather than hosting infrastructure locally.

   • [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
   • [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”
   • [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”

   proof ↗⚑ flag

   HumanLayer~partialclaimed7/10

   Docs describe running the remote daemon on a cloud VM, workstation, or private-network machine that you control (humanlayer-docs-7), plus automation sessions from CI/cron/server environments (humanlayer-docs-12) and launch tokens for bootstrap scripts (humanlayer-docs-13), showing self-hosted deployment flexibility. However there is no explicit mention of container/Docker deployment or an official container image, and no independent verification of self-hosted setups at scale. Missing for 10: explicit container/Docker packaging docs, independent hands-on confirmation of self-hosted deployments.

   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
   • [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”

   proof ↗⚑ flag

Headless automation

1. developerRun an agent headlessly inside CI/CD pipelines and shell scripts

   weight 2 · round to HumanLayer
   Codegen~partialprobed6/10

   Codegen explicitly documents triggering agents from CI/CD pipelines (e.g., on build failures) and offers a Python SDK plus CLI for programmatic/scripted invocation, which supports headless usage in pipelines and scripts. However, there's no dedicated CI/CD integration guide, no example pipeline configs (e.g., GitHub Actions YAML), and no independent/hands-on evidence confirming real-world headless CI usage. missing for 10: concrete CI/CD pipeline examples or config snippets, exit-code/status handling for scripts, independent verification of headless operation in automated pipelines.

   • [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
   • [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
   • [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
   • [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
   • [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”

   proof ↗⚑ flag

   HumanLayer✓fullclaimed8/10

   Docs explicitly describe `humanlayer automation run` for running Cloud-visible coding sessions from CI jobs, cron machines, or scripts, plus launch tokens for non-interactive/non-PTY execution suited to headless environments. This directly matches running an agent headlessly in CI/CD and shell scripts. Missing for 10: independent/hands-on verification of CI usage and concrete pipeline examples (e.g. GitHub Actions config).

   • [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
   • [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
   • [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”

   proof ↗⚑ flag