Foreloop vs HumanLayer
HumanLayer wins · 17–24 (24 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round drawnForeloopnone0/10No evidence pack item mentions llms.txt or an agent-oriented docs endpoint that agents can be pointed at; docs reference CLI, MCP endpoints, and skills but not an llms.txt-style discoverable doc file.
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to HumanLayerForeloop ships a global CLI with a stable, additive `--json` output on every command, and docs explicitly state it can be run 'from anything that can run one, including coding agents and cron,' which is direct evidence of headless/CI-style automation support. However, there's no CI-specific documentation (e.g. GitHub Actions example, env-based auth for non-interactive contexts) and the approval-gate design ('nothing runs until a person approves a contract') implies a human-in-the-loop checkpoint that could block fully unattended pipelines. missing for 10: CI/pipeline-specific setup docs or examples, confirmation that approval gates can be bypassed or pre-authorized for headless runs, independent/hands-on verification of CI usage.
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [probe] “official CLI documented at https://foreloop.com/install”
Docs explicitly describe running HumanLayer headlessly via `humanlayer automation run` from CI jobs, cron machines, or scripts, plus launch tokens for non-interactive/non-PTY environments, and remote daemon hosts on cloud VMs or servers, directly matching the CI/automation story. Missing for 10: independent/hands-on confirmation of CI usage and more detail on auth/config specifics for automated pipelines.
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round drawnForeloopnone0/10All MCP-related evidence describes Foreloop exposing its own MCP server endpoint (/api/app/public/v1/mcp) so that external MCP clients (agents) can call into it — this is the reverse of the story, which asks whether the user can plug external MCP servers into Foreloop so it can use their tools. No evidence shows Foreloop acting as an MCP client that consumes other servers' tools.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
HumanLayernone0/10No evidence anywhere in the pack mentions MCP servers or the ability to plug external tool servers into HumanLayer; integrations mentioned (Jira, Slack, GitHub, Linear) are task-source connectors, not MCP tool servers. Missing for 10: any mention of MCP protocol support, MCP server configuration, or tool-plugin mechanism.
ai-native userConnect an agent via an official MCP server
weight 3 · round to ForeloopForeloop documents a first-party MCP server endpoint (/api/app/public/v1/mcp over streamable HTTP) that any MCP client can connect to with just a URL and API key, explicitly enabling agents to claim tasks and work in repos. This is first-party documentation only, with no independent/hands-on corroboration of the MCP connection working in practice. Missing for 10: independent verification that an external MCP client successfully connects and operates against this endpoint.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
HumanLayernone0/10HumanLayer is a platform/control-plane for running and overseeing coding-agent sessions (Claude Code, Codex) rather than itself being an agent, so an official MCP server is a fair, applicable axis. The evidence pack documents many integrations (Jira, Slack, GitHub, Linear), a CLI, and remote daemons, but no MCP server offering is mentioned anywhere, and API/OpenAPI probes returned 404s. missing for 10: any first-party MCP server documentation, endpoint, or 'mcp serve' style capability.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
- [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
ai-native userUse an official CLI
weight 2 · round to ForeloopForeloop ships an official globally-installable CLI (npm i -g foreloop) that manages projects, loops, intentions, stories, and tasks, supports scripting via `--json` with a stable additive schema, and is explicitly designed to be run 'from anything that can run one, including coding agents and cron,' making it clearly AI-native/agentic-workflow friendly. missing for 10: independent/hands-on corroboration beyond vendor docs, and more detail on full command surface.
- [claimed-docs] “npm i -g foreloop # installs globally, so the CLI works in any directory”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [probe] “official CLI documented at https://foreloop.com/install”
HumanLayer documents an official CLI (e.g. `humanlayer automation run`, launch tokens, remote-daemon control) used for CI, cron, and scripted agentic workflows, confirmed by a dedicated CLI doc page. missing for 10: no independent/hands-on verification of the CLI, no full command reference, and no evidence of broader CLI feature parity with the app.
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”
ai-native userDrive the product through a documented public API
weight 3 · round to ForeloopForeloop documents a public MCP endpoint (/api/app/public/v1/mcp over streamable HTTP with just a URL and API key) plus a CLI with stable, additive `--json` output usable from any scriptable context including cron and coding agents, giving AI-native users two documented programmatic surfaces. missing for 10: no independent/third-party corroboration of the API's stability or real-world agent integration beyond first-party docs, and no formal OpenAPI/REST reference is shown alongside the MCP/CLI docs.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [probe] “official CLI documented at https://foreloop.com/install”
HumanLayer documents a CLI (`humanlayer automation run`, launch tokens, remote daemons) that lets automation environments drive sessions programmatically, which is a form of documented programmatic control, but there is no evidence of a documented public REST/OpenAPI API — probes explicitly found openapi.json/swagger.json/llms.txt all 404. missing for 10: a documented HTTP/OpenAPI public API spec, SDK/client library docs, and independent confirmation of API usage.
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
- [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
- [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round drawnThe docs mention a 'write-only' API key type (flpk_YOUR_WRITE_ONLY_KEY) which implies some credential scoping, and the MCP endpoint for agents requires 'an API key', but there is no explicit documentation of issuing least-privilege or role-scoped credentials specifically for agent access to repos/tasks. missing for 10: explicit agent-scoped API key creation/management flow, documented permission levels (read/write/task-only), and evidence of per-agent credential rotation or revocation.
- [claimed-docs] “await initForeloopFeedback({ apiKey: "flpk_YOUR_WRITE_ONLY_KEY", reporterName: signedInUser?.email, });”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
The only relevant evidence is a mention of a 'launch token' scoped to a single non-interactive command, which implies some least-privilege token issuance, but there is no documentation of a broader credential/permission model, scopes, or API key management for agents. missing for 10: explicit least-privilege credential scoping model, permission granularity, revocation/rotation mechanisms, and any independent corroboration.
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
ai-native userBuild against official SDKs
weight 2 · round to ForeloopForeloop exposes an official CLI, a JS feedback SDK (initForeloopFeedback), and an MCP endpoint for agent clients, giving AI-native builders official integration surfaces beyond just docs. However, this is all first-party vendor documentation with no independent corroboration or SDK versioning/language-coverage details (e.g. only JS snippet shown, no Python/other language SDKs). Missing for 10: independent/hands-on verification of SDK usage, multi-language SDK coverage, and evidence of SDK stability/versioning beyond the JS snippet.
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “await initForeloopFeedback({ apiKey: "flpk_YOUR_WRITE_ONLY_KEY", reporterName: signedInUser?.email, });”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [probe] “official CLI documented at https://foreloop.com/install”
HumanLayernone0/10The evidence pack covers HumanLayer's CLI, workspace config, and third-party integrations (Jira, Slack, GitHub, Linear) but contains no mention of an official SDK (Python, TypeScript, etc.) for building against HumanLayer programmatically. Probes for openapi.json and llms.txt both returned 404, further suggesting no discoverable API/SDK surface.
- [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
ai-native userSubscribe to events via webhooks
weight 2 · round drawnForeloopnone0/10The evidence pack documents MCP connectivity, CLI, and feedback widget integrations, but nowhere mentions webhooks or event subscriptions that an AI-native user could subscribe to; no push/callback mechanism is described.
HumanLayernone0/10No evidence pack item mentions webhooks or event subscription mechanisms; integrations described (Slack, Jira, Linear, GitHub) are inbound task-creation connectors, not outbound webhook events, and API/OpenAPI probes returned 404s.
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round to ForeloopForeloop's agent reads submitted feedback/reports and proposes intentions and tasks from that data (foreloop-docs-12, foreloop-docs-4), which is a form of AI-generated suggestion from collected data, but this is limited to feedback-to-task triage rather than broader analytics or insight generation across product data. Missing for 10: evidence of AI summarizing trends/patterns across historical data, dashboards or reports surfacing insights beyond individual feedback items, and independent corroboration of insight quality.
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
HumanLayer's agents do generate task artifacts, draft PRs, and comments derived from a user's codebase/tickets, and 'Advanced Metrics' surfaces usage/cost/productivity data, which loosely resembles data-derived insight. However there is no evidence of dedicated AI-generated analytical insights or proactive suggestions distinct from executing assigned coding tasks. Missing for 10: explicit insight/analytics generation from user data, proactive suggestion features beyond task execution, and any first-party or community evidence of an 'insights' capability.
- [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”
- [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
ai-native userSet up automations that run autonomously in the background
weight 2 · round to HumanLayerForeloop supports background-style automation via its CLI (usable from cron or other schedulers) and agents that autonomously claim and work tasks in a repo, but it explicitly gates all agent actions behind a mandatory human approval step ('Nothing runs until a person approves a contract'), meaning true unattended autonomy is limited. Missing for 10: concrete documentation of scheduled/recurring automation setups, evidence of fully unattended runs without approval gating, and details on monitoring/alerting for background tasks.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Approve the big calls from your phone.”
Docs explicitly describe `humanlayer automation run` for running Cloud-visible coding sessions from CI jobs, cron machines, or scripts, plus remote daemons and launch tokens for non-interactive/background execution contexts, directly supporting autonomous background automations. Missing for 10: independent/hands-on verification of long-running background automations, native scheduling UI, and clarity on how human-approval gates interact with continuous autonomous runs.
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round to HumanLayerForeloopnone0/10Foreloop's documented model relies on external agents (Claude Code or Codex) running on the user's own machine and connecting via MCP to claim tasks — there is no built-in AI assistant shipped inside Foreloop itself that a user delegates tasks to directly.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
HumanLayer's core product model is task delegation to AI coding agents (Claude Code, Codex, Bedrock-backed Claude) via 'sessions', with workflow modes (Oneshot, RPI, PRD-Oriented, Freeform) and automation runs from CI/cron, directly matching 'delegate tasks to a built-in AI assistant'. missing for 10: independent/hands-on verification of the delegation experience beyond vendor docs, and clarity on how autonomous vs supervised the assistant is in practice.
- [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
- [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
- [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
- [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
ai-native userOperate the product with natural-language commands
weight 2 · round to HumanLayerForeloop lets users describe problems in plain language (e.g., "Users can't tell which invoices are overdue") to kick off an intention that an agent turns into tasks, showing some natural-language entry point, but its actual operating surface is a structured CLI (with --json flags) and an MCP interface for agents, not a general natural-language command interface for the product as a whole. Missing for 10: evidence of NL commands controlling arbitrary product actions (not just initial feedback/intention creation), and any hands-on/independent confirmation of this NL flow working.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
HumanLayer's core interaction model is giving natural-language instructions to agent sessions (Claude Code, Codex) to create tasks, configure workspaces, and choose workflow phases, as shown by the example NL workspace-config prompt and workflow-selection docs. Missing for 10: independent/hands-on corroboration of NL command robustness and no evidence of a broader NL command surface beyond task/workflow setup.
- [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
- [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
- [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
- [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnForeloopnone0/10The evidence pack shows CLI docs, MCP endpoint info, and install snippets, but nowhere describes an interactive API reference (e.g., Swagger/OpenAPI explorer, try-it-now console) with runnable examples.
HumanLayernone0/10No evidence of an interactive API reference or runnable examples; probes explicitly show no OpenAPI/swagger spec and no llms.txt found, and docs are guide/tutorial style rather than an API reference sandbox.
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round drawnForeloopnone0/10Evidence shows Foreloop exposes an MCP endpoint and CLI with --json output, but there is no mention of a downloadable OpenAPI spec or any machine-readable API specification document for its REST/API surface.
HumanLayernone0/10A direct probe for OpenAPI/swagger specs at all standard locations returned 404s, and no evidence pack item shows a downloadable machine-readable API spec being offered.
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round to ForeloopThe docs mention a stability guarantee for CLI `--json` output ('has a stable shape. Additive changes only') and the MCP/API endpoint path includes a 'v1' segment, implying some versioning discipline, but there is no explicit documented deprecation policy, versioning scheme, or migration guidance for the API itself. missing for 10: explicit API version numbering scheme, documented deprecation timeline/process, changelog or migration guides, evidence beyond CLI JSON stability.
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
HumanLayernone0/10No evidence of API versioning scheme or a documented deprecation policy; probes for openapi.json/llms.txt returned 404s and no API reference or changelog covering versioning/deprecation is present.
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round drawnForeloopnone0/10Foreloop's docs describe per-task workflows (claim one task, agent does the work, opens one PR) and a CLI/MCP interface, but nothing in the evidence describes performing an operation across many items/tasks simultaneously (e.g., batch approve, bulk edit, multi-task claim). The axis is plausible for a CLI/automation tool but no evidence of bulk operation support exists.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
HumanLayernone0/10HumanLayer's documentation consistently frames work as single tasks/sessions ('one small task', 'one task on a remote machine', 'one Cloud-visible coding session') with per-task review and approval workflows; there is no mention of batch/bulk operations spanning many items at once. missing for 10: any documented bulk-action API/CLI flag, batch approval mechanism, or multi-item automation workflow.
- [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
- [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to HumanLayerForeloopnone0/10Foreloop's docs describe human-approved task workflows (agents claim tasks, but 'nothing runs until a person approves a contract'), not a rules engine for automatic event-triggered actions; the closest mention is CLI usability from cron, which is not evidence of a native rules/trigger system. missing for 10: any documented rule-definition syntax, event-trigger configuration, or automation-without-approval capability.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
HumanLayer integrations (GitHub, Jira, Linear) create tasks automatically from external events like new issues/tickets, and `humanlayer automation run` lets sessions be triggered from CI jobs, cron, or scripts — both are forms of event-driven automation. However, there's no evidence of a general-purpose rules/conditions engine letting users define arbitrary trigger-condition-action logic; the automation is limited to fixed integration hooks and script-based invocation. Missing for 10: a documented rule-definition interface (conditions, filters, custom triggers) beyond fixed platform integrations, and independent confirmation these event-triggers work reliably in practice.
- [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
- [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
ai-native userSchedule recurring jobs or workflows
weight 2 · round to HumanLayerThe only relevant evidence is a single line noting the CLI can be run 'from anything that can run one, including coding agents and cron,' implying users could wire up their own cron-based scheduling — but Foreloop itself documents no native recurring-job/workflow scheduler, retry policy, or schedule management UI. Missing for 10: a first-party scheduling feature, documented recurring-workflow configuration, and evidence of built-in cron/interval management within the product.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
HumanLayer's `automation run` CLI lets you trigger a Cloud-visible coding session from a cron machine or CI job, implying recurring runs are possible via external schedulers, but there is no documented native scheduling/recurrence feature (no cron syntax, interval config, or job queue) inside HumanLayer itself. missing for 10: built-in recurring scheduler, interval/cron configuration options, evidence of persistent recurring workflow management.
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
ai-native userVersion, review, and roll back my automations
weight 1 · round to HumanLayerForeloop's workflow requires human approval before any agent-driven change runs (foreloop-docs-7) and changes are delivered as GitHub pull requests (foreloop-docs-3, foreloop-docs-15), which inherently gives code-level versioning and review via git. However, there is no documentation of versioning, reviewing, or rolling back the automations/tasks/loops themselves (as opposed to the resulting code diffs) — no rollback command, task history, or automation-versioning feature is evidenced. Missing for 10: explicit automation/task versioning, a documented rollback mechanism for tasks or loops, and audit/history views beyond the live agent status list.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
HumanLayer's task model provides review (comments, PR draft creation) and history that persists across sessions, giving some review/audit capability, but there is no explicit documentation of versioning workflow definitions or rolling back an automation to a prior version. missing for 10: explicit automation versioning/history diffing, a documented rollback mechanism, and independent confirmation these review features extend to full automation lifecycle management.
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
- [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
- [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionAutonomous implementation
End-to-end implementation by the agent — multi-file changes, task completion
End to end feature delivery
ai-native userHave an agent automatically generate and run tests to validate its own code changes before proposing them
weight 2 · round drawnForeloopnone0/10No evidence describes Foreloop's agents generating or running tests to validate code changes before proposing them; the docs describe task claiming, PR opening, and human approval gates but never mention automated test generation/execution as a validation step.
developerHave an agent autonomously diagnose and fix a reported bug
weight 3 · round to ForeloopForeloop's documented workflow directly supports this: a bug report captured via the feedback widget (with page, metadata, screenshot) becomes an intention/task that an agent (Claude Code or Codex) claims, works on in a repo checkout, and resolves by opening a PR, without the reporter needing to know which files change. Human approval gates task initiation but the diagnosis-and-fix work itself is agent-driven end to end. Missing for 10: independent/hands-on evidence that agents actually diagnose root causes correctly, and more detail on how the agent investigates ambiguous bugs beyond the general task pipeline description.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
HumanLayer supports creating tasks directly from GitHub/Jira/Linear issues and then running an agent session (Claude Code/Codex) against the linked repo, with an 'Oneshot' workflow phase designed for small, clear changes — a plausible bug-fix pipeline. However, the product's core design is human-in-the-loop with approval gates rather than fully autonomous action, and there's no end-to-end documented example of an agent independently diagnosing a bug from a ticket and shipping a fix without human review. Missing for 10: a concrete autonomous bug-diagnosis-and-fix walkthrough, and clarity on how much human approval is required mid-flow.
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
- [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
- [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”
product-managerGo from a mockup or design to a working implementation without an engineering handoff
weight 2 · round to ForeloopForeloop lets a non-engineer describe a need in plain language (e.g. "Users can't tell which invoices are overdue") or flag a page via a feedback widget with screenshot, and an agent turns that into tasks and a PR without the PM needing to specify files — this covers a 'no engineering handoff' description-to-implementation flow. However, the evidence never shows ingesting an actual mockup/design file (e.g. Figma) as the starting artifact, and a human still must approve the contract before code changes run, so it's not a fully autonomous design-to-code path. Missing for 10: direct mockup/design-file import as an input, evidence of end-to-end completion without any review gate, and a demonstrated design-to-code case study.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
HumanLayernone0/10HumanLayer's evidence is entirely about developer-facing workflows: running coding sessions, connecting Jira/Linear/GitHub/Slack, managing remote daemons, and CLI automation for engineers overseeing coding agents. Nothing in the pack shows a mockup/design import capability, a no-code interface, or any path for a non-engineer product manager to turn a design into a working implementation without engineering involvement — in fact the workflow language (RPI, PRD-oriented, Oneshot) and workspace configs assume an engineering operator. Missing for evidence of delivering this story: mockup/design ingestion, PM-oriented no-code UI, and any case study of a non-engineer shipping code end-to-end.
- [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
- [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
- [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
developerHave an agent implement a requested feature end-to-end, including writing tests
weight 3 · round to ForeloopForeloop orchestrates tasks and lets an agent (Claude Code/Codex) claim a task, do the work, and open a PR in the user's repo, which covers end-to-end feature implementation via an external coding agent — but Foreloop itself doesn't write code or tests; it delegates that entirely to the connected agent, and there's no explicit mention of test-writing behavior or verification. missing for 10: explicit evidence that generated PRs include tests, independent/hands-on confirmation of successful end-to-end feature delivery, and detail on how test coverage is verified before approval.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
HumanLayer clearly supports end-to-end autonomous coding sessions (Oneshot, RPI, PRD-Oriented workflows) that implement tasks using agents like Claude Code and Codex, including structured phases and sub-agents for research/plan/implement, which implies substantial feature work can be delegated (humanlayer-docs-6, humanlayer-docs-9, humanlayer-docs-10, humanlayer-docs-14). However, no evidence explicitly confirms the agent writes or runs tests as part of the workflow, and no hands-on account demonstrates a full feature-plus-tests delivery. Missing for 10: explicit documentation or example showing test generation/execution as part of the implementation flow, and independent verification of end-to-end feature completion including tests.
- [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
- [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
- [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
- [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
Environment setup
developerHave an agent automatically clone the repo, install dependencies, and configure its own working environment
weight 2 · round to HumanLayerForeloop's docs describe the developer running an agent 'in a checkout of your repository on your own machine' and using `foreloop init` to install a task-executor skill into that existing checkout — implying the human, not the agent, does the initial clone/setup, with the agent only claiming and working tasks afterward. There's no documentation of the agent autonomously cloning the repo, installing dependencies, or provisioning its own environment from scratch. Missing for 10: evidence of agent-initiated repo cloning, dependency installation steps, and autonomous environment configuration (e.g., a sandboxed or cloud checkout flow) rather than reliance on a pre-existing local checkout.
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
Docs describe workspace configuration (workspace.json, multi-repo setups) and remote hosts that must have access to code/tools/credentials, and one example prompt asks the agent about 'setup commands or local files to copy,' implying some environment configuration ability. However, there is no explicit description of the agent autonomously cloning a repo or installing dependencies end-to-end without human setup of the host/workspace first. Missing for 10: explicit documentation of automatic repo cloning, dependency installation steps, and end-to-end environment bootstrap without prior manual host/workspace configuration.
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
- [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
Interactive takeover
developerTake over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work
weight 2 · round to HumanLayerForeloop lets agents run locally in a checkout and claim/report tasks, and humans can approve contracts or view agent status via a dashboard row per worker, but there's no evidence of a live handoff mechanism to pause, take over, or redirect an in-progress task from editor/terminal/browser mid-execution. missing for 10: explicit takeover/interrupt UX, editor-integrated control of a running agent, terminal command to seize an active task, and any hands-on confirmation of mid-task redirection.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “Approve the big calls from your phone.”
Docs describe tasks with session/history continuity 'across agents and workstations', remote control from app.humanlayer.com on any machine or phone, and CLI-driven remote daemons for terminal/server contexts, all pointing to genuine hand-off of in-progress work between editor (Claude Code), terminal (remote daemon/CLI), and browser (web app). Live multiplayer draft recovery further shows shared/continuable session state. missing for 10: independent/hands-on confirmation of a live takeover mid-task, and explicit description of an in-editor (IDE plugin) takeover UI rather than just CLI/app/web.
- [claimed-docs] “This tutorial teaches you how to create and run one small task in HumanLayer. You will use the macOS app and Claude Code.”
- [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
- [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”
developerSend follow-up instructions to an active agent session to steer its work without restarting
weight 2 · round to HumanLayerForeloopnone0/10The evidence describes agents claiming tasks, opening PRs, and reporting status, plus approval gates and skill-writing to correct recurring mistakes — but nothing describes sending live follow-up/steering instructions to an already-running agent session mid-task without restarting it.
HumanLayer's task/session model implies ongoing interaction with agents across a task's lifetime (docs-5) and release notes mention live prompt collaboration during sessions (docs-18), suggesting some capacity to interact with an active session, but there is no explicit documentation of sending mid-session follow-up instructions to steer a running agent without restarting it. missing for 10: explicit docs on injecting new instructions into a live/running session, confirmation the agent incorporates such input without restart, and independent/hands-on verification of this steering behavior.
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
- [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”
Sandbox execution
developerHave an agent safely execute code and install dependencies inside an isolated sandbox
weight 3 · round drawnForeloopnone0/10Foreloop's docs explicitly describe agents running 'in a checkout of your repository on your own machine' (foreloop-docs-3), with no mention of sandboxing, containerization, or isolated execution environments for code/dependency installation. Missing for 10: any documentation of sandbox/container isolation, dependency install safety, or resource/network restrictions during agent execution.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
HumanLayernone0/10HumanLayer docs describe running sessions on remote hosts (cloud VM, workstation, private-network machine) and automation environments, but there is no mention of an isolated/sandboxed execution environment for running code or installing dependencies safely — the host selection is about access/credentials, not isolation guarantees. Missing for full/partial: any explicit sandbox, container, or isolation mechanism; no evidence of dependency-install safety controls.
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsHuman oversight
Keeping a human in the loop — approvals, checkpoints, interrupts
Approval controls
developerConfigure an agent to auto-approve all its actions instead of confirming each one
weight 2 · round to HumanLayerForeloopnone0/10Foreloop's docs explicitly state that human approval of a contract is a deliberate, mandatory gate before any agent action runs ('Nothing runs until a person approves a contract. Approval is the only gate'), with no mention of a setting to bypass or auto-approve this gate. This indicates the product does not offer an auto-approve configuration, so the story's capability is unsupported.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Approve the big calls from your phone.”
HumanLayer's core premise is human-in-the-loop approval, and docs mention workflow phases like 'Oneshot' for low-risk changes and automation sessions (humanlayer automation run) that run non-interactively without confirmation, implying some auto-approve capability exists, but no explicit documentation of a configurable 'auto-approve all actions' toggle or setting is shown. missing for 10: explicit config/flag to disable per-action confirmation entirely, documentation confirming automation sessions skip all human review rather than just running unattended, and independent confirmation this works as intended.
- [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
product-managerApprove key agent decisions from my phone while agents continue working
weight 1 · round to HumanLayerForeloop explicitly markets phone-based approval of 'big calls' and documents that no agent work runs until a human approves a contract, which matches the core ask. However, evidence is limited to marketing taglines with no detail on the mobile UI/notification flow, how approvals interact with agents continuing to work in parallel, or independent corroboration. Missing for 10: concrete mobile-app/notification workflow details, evidence of multi-agent concurrent operation while awaiting approval, and third-party confirmation.
- [claimed-docs] “Approve the big calls from your phone.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
Docs explicitly describe controlling and continuing a running agent task from a phone via app.humanlayer.com, with tasks providing a shared review/comment history across devices, directly matching the phone-approval-while-agent-continues story; community sentiment corroborates the human-approval-of-agent-actions use case. Missing for 10: a hands-on/independent account specifically confirming the mobile approval UI in practice, and explicit documentation of an 'approve/deny decision' action (vs. general task control) on mobile.
- [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”
engineering-leadSet tiered autonomy levels controlling what an agent can do without manual confirmation
weight 3 · round to HumanLayerForeloopnone0/10Foreloop documents a single approval gate before any agent work starts ('Nothing runs until a person approves a contract') and lets you approve from your phone, but there is no evidence of configurable tiered autonomy levels (e.g., auto-approving certain task types while requiring confirmation for others).
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Approve the big calls from your phone.”
HumanLayer's workflow-phases doc shows tiered approaches (Oneshot for low-risk changes vs. more review for unclear/risky changes) and skills-workflows lets users select Oneshot/RPI/PRD-Oriented/Freeform modes, which map to different levels of autonomy vs. oversight. However, there's no explicit documentation of a formal 'autonomy level' setting per agent/task with configurable confirmation thresholds, and no independent evidence confirming this tiered control works as an oversight mechanism in practice. missing for 10: explicit named autonomy-tier configuration (e.g., low/medium/high) tied to confirmation gating, evidence of engineering-lead-level policy controls across a team, and independent/hands-on validation that these workflow phases actually reduce unnecessary confirmations without sacrificing safety.
- [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
- [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
- [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
Visibility monitoring
developerWatch what a running agent is doing in real time, including its current status
weight 3 · round drawnThe docs describe a Development/Agents view listing each worker's agent identity, checkout name, current task, and last message — a real-time-ish status view (foreloop-docs-8). However, there's no evidence of live streaming updates, logs, or granular in-progress activity beyond a status line, and no independent/hands-on confirmation of real-time refresh behavior. missing for 10: evidence of live/streaming updates or detailed activity logs, independent confirmation of real-time refresh, granularity beyond a single status line per agent.
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
HumanLayer's docs describe remote-daemon control from app.humanlayer.com (including from a phone), live multiplayer session viewing with presence/read-only viewing, and task/session history — all suggesting real-time visibility into agent activity. However, there's no explicit documentation of a dedicated live status/log stream or dashboard showing granular agent state (e.g., current tool call, progress bar) beyond session/task views. missing for 10: explicit real-time status/log streaming documentation, independent hands-on confirmation of live monitoring UX.
- [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
- [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
developerGet notified when an agent completes a task or needs my input
weight 2 · round to HumanLayerForeloop provides a dashboard showing agent status ('last thing it said') and mentions approving 'big calls from your phone', implying some notification/approval flow, but there is no explicit documentation of push/email/Slack notifications when a task completes or needs input. missing for 10: explicit notification channel (email/Slack/push) documentation, confirmation of real-time alerts rather than manual dashboard polling, evidence of notification content for 'needs input' events.
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “Approve the big calls from your phone.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
HumanLayer supports Slack/GitHub/Jira/Linear integrations that push task artifact updates and human-in-the-loop approvals, implying notification when tasks progress or need input, and its core design centers on human oversight of agent work. However, there is no explicit documentation of a dedicated 'task complete' or 'needs input' notification/alert mechanism (e.g., push notification, email, or webhook triggered specifically on completion/approval-required events) beyond generic artifact updates in Slack. missing for 10: explicit completion/needs-input notification docs, evidence of notification latency/reliability, independent/hands-on confirmation of notification delivery.
- [claimed-docs] “Connect Slack to send HumanLayer task artifact updates into the channels where your team already works.”
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
- [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
- [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
- [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”
Intent to spec — stories about intent to spec in this arenaIntent to spec
Stories about intent to spec in this arena
Natural language task intake
developerDescribe a feature or bug in plain language and have it automatically turned into a scoped implementation task
weight 3 · round to ForeloopDocs describe plain-language input ('Users can't tell which invoices are overdue') turning into scoped, approvable tasks via 'intentions' that an agent fills in and proposes as tasks, gated by human approval before an agent acts on the repo. This directly matches intent-to-spec workflow with first-party documentation across multiple pages. Missing for 10: independent/hands-on verification of the resulting task's scoping quality and no example of the full generated task artifact.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
HumanLayer's task model (docs-5) and workflow-phase selection (docs-6, docs-10) show that a task is created and can be routed through 'Oneshot' for small clear changes or heavier RPI/PRD-oriented flows for ambiguous work, and RPI sub-agents (docs-9) imply a research→plan→implement pipeline that turns a description into a scoped plan. However, there is no explicit walkthrough showing a raw plain-language bug/feature description being automatically parsed into a scoped implementation task end-to-end, and integrations (Jira/Linear/GitHub) mostly create tasks from existing tickets rather than free-form language input. Missing for 10: a concrete example or tutorial of plain-language-to-scoped-task conversion, and independent/hands-on confirmation that this pipeline works as described.
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
- [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
- [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
- [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
product-managerConvert user feedback submissions into structured tasks with proposed scope
weight 2 · round to ForeloopDocs show a feedback-collection widget that captures user submissions, and an agent-driven 'intention' workflow that reads the submission, fills in an intention, and proposes tasks (foreloop-docs-12), with human approval gating execution (foreloop-docs-7). This directly matches converting feedback into structured tasks with proposed scope, though evidence is entirely vendor docs with no independent/hands-on corroboration of output quality or scope structure. Missing for 10: independent/hands-on verification of the feedback-to-task pipeline, and concrete example of the 'proposed scope' structure/fields.
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
HumanLayer supports creating tasks from external issue trackers (Jira, Linear, GitHub) which could serve as a proxy for user feedback submissions, and tasks include shared files/scope info, but there's no evidence of a dedicated feature for ingesting raw user feedback (e.g., support tickets, survey responses) and auto-structuring it into a task with a proposed scope specifically tailored for PM workflows. missing for 10: dedicated feedback-ingestion mechanism, explicit 'proposed scope' generation from unstructured feedback text, PM-specific workflow templates, and any hands-on/community validation of this specific use case.
- [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
- [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
developerAttach a marked-up screenshot or mockup to a task so the agent implements the correct visual change
weight 2 · round drawnThe feedback widget lets a customer point at an element and attach an optional screenshot with metadata so 'the model fixes the right thing' (foreloop-docs-21), which is adjacent to the story but is customer-bug-report flow, not a developer attaching an annotated mockup directly to a task. There's no documentation of markup/annotation tools, mockup uploads, or a developer-facing task-attachment workflow for visual specs. Missing for 10: developer-facing screenshot/mockup attachment to tasks, annotation/markup capability, and evidence the agent parses visual markup to implement precise UI changes.
- [claimed-docs] “Customers point at the element. The widget adds the page, its metadata, and an optional screenshot, so the model fixes the right thing.”
Docs confirm images can be pasted into the new task composer as attachments (humanlayer-docs-22), which supports attaching a screenshot to a task, but there is no evidence of markup/annotation tooling or of the agent parsing visual annotations to implement a corresponding UI change. Missing for 10: annotation/markup capability for screenshots, evidence the agent interprets visual markup into a specific implementation, and any hands-on example of this workflow succeeding.
- [claimed-docs] “Image paste in new tasks — Paste images straight into the new task composer as attachments.”
Plan approval
developerReview and approve an agent's implementation plan before any code changes are made
weight 3 · round to ForeloopDocs explicitly describe a 'contract' approval gate before any agent touches the repository ('Nothing runs until a person approves a contract... an agent is about to change your repository'), and mention approving 'big calls' from a phone, indicating a review/approve workflow prior to code changes. Missing for 10: independent/hands-on confirmation of the approval UI showing the actual implementation plan content and evidence of what happens if a plan is rejected/revised.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Approve the big calls from your phone.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
HumanLayer's workflow-phases and RPI sub-agent docs describe planning phases (e.g., 'a change with unclear behavior or code shape needs more review before implementation') and PRD-Oriented/RPI workflows imply a plan stage before code changes, with tasks providing 'one place for comments and review.' However, no evidence explicitly shows a dedicated plan-approval gate/UI step where a developer reviews and approves a plan artifact before implementation begins. missing for 10: explicit documentation of a plan-approval step/UI, first-party example of blocking implementation until plan is approved, independent/hands-on confirmation of this specific gate.
- [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
- [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
- [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
engineering-leadApprove a task's scope and contract before an agent is allowed to modify the repository
weight 2 · round to ForeloopForeloop's docs explicitly state approval of a contract is a mandatory gate before any agent touches the repository ('Nothing runs until a person approves a contract... an agent is about to change your repository'), and agents only claim 'approved tasks' from the repo. This directly matches the engineering-lead approval-gate story, reinforced by mobile approval flow. Missing for 10: independent/hands-on verification that the gate is enforced in practice (not just documented) and detail on what 'scope and contract' specifically contains.
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Approve the big calls from your phone.”
HumanLayer's workflow-phases doc explicitly supports scoping review before implementation (e.g., 'a change with unclear behavior or code shape needs more review before implementation'), and tasks/sessions provide a structured place for comments and review prior to agent execution, plus human-in-the-loop approval is core to the product's value prop per community discussion. However, there's no explicit documented feature for an engineering-lead specifically approving a 'scope and contract' artifact as a gating step before repo modification — it's inferred from general workflow-phase and review mechanics rather than a dedicated scope-approval gate. missing for 10: explicit documentation of a formal scope/contract approval step tied to lead sign-off, evidence of blocking repo writes until such approval, and independent/hands-on confirmation this gate works as intended.
- [claimed-docs] “Use the shortest path that controls the main risk. A clear and small change can use Oneshot. A change with unclear behavior or code shape ne…”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
- [claimed-docs] “Select **Oneshot**, **RPI**, **PRD-Oriented**, or **Freeform**.”
- [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”
Ticket driven tasking
developerAssign a coding task to an agent directly from an existing issue or ticket
weight 3 · round to HumanLayerForeloopnone0/10Foreloop's workflow starts from user feedback reports or free-text intentions that get turned into tasks an agent claims (foreloop-docs-4, foreloop-docs-12), but there is no evidence of importing or linking an existing GitHub issue/Jira ticket as the task source for an agent to pick up.
Docs explicitly describe connecting GitHub, Jira, and Linear so HumanLayer creates tasks directly from issues/tickets and links artifacts back to the source, directly matching the story of assigning agent work from an existing ticket. Missing for 10: independent/hands-on confirmation that this ticket-to-task flow works reliably in practice, and more detail on the actual assignment UX.
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
- [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
- [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to ForeloopForeloop ships a CLI covering projects, loops, intentions, stories and tasks with a stable `--json` output on every command, plus an MCP endpoint for agent clients, suggesting broad API/CLI parity with the UI. However, there's no explicit statement or independent confirmation that literally every UI action (e.g., approvals, feedback widget config, agent monitoring dashboard views) is exposed via API/CLI. missing for 10: explicit UI-to-API parity documentation or audit, independent/hands-on verification, coverage of approval workflow and dashboard views via API.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “--json works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
- [probe] “official CLI documented at https://foreloop.com/install”
HumanLayer offers a CLI (`humanlayer automation run`) and remote daemon controls that let you launch and manage sessions outside the UI, but there is no documented public API/OpenAPI spec (both openapi.json and llms.txt probes 404), and UI-only features like live multiplayer drafts, keyboard navigation, and image paste have no CLI/API equivalent documented. missing for 10: a documented REST/GraphQL API or OpenAPI spec, confirmation that all UI actions (draft PRs, multiplayer editing, metrics views) are exposed programmatically, and independent verification of API-UI parity.
- [probe] “official CLI documented at https://docs.humanlayer.com/guide/remote-daemons”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
- [probe] “PROBE llms.txt: HTTP 404 at https://docs.humanlayer.com/llms.txt”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.humanlayer.com/openapi.json, https://docs.humanlayer.com/swagger.json, https://docs.hum…”
- [claimed-docs] “Live multiplayer response drafts — Write prompts together with shared text, named cursors, presence, read-only viewing, and draft recovery.”
- [claimed-docs] “Keyboard navigation for changed files — Move through the PR changes tree with J/K, N/P, G shortcuts, and Enter.”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnForeloopnone0/10No evidence of a bulk data export feature or open-format export of projects/tasks/intentions for migration away from Foreloop; the closest hints (--json CLI output, MCP API access) are for programmatic access, not a documented full-data export/leave capability.
ai-native userRead the product's source under an open license
weight 2 · round drawnForeloopnone0/10No evidence of an open-source license or public source code repository; Foreloop is presented as a closed SaaS product with a CLI/widget/MCP server, but nothing indicates the source is available under an open license.
ai-native userSelf-host the core product
weight 3 · round drawnForeloopnone0/10All evidence describes Foreloop as a hosted SaaS (with a CLI/MCP endpoint at foreloop.com/api) that agents connect to remotely; there is no mention of a self-hostable core server, on-prem deployment, or open-source release. missing for 10: any documentation of a self-hosted deployment option, container/server image, or license permitting self-hosting.
HumanLayernone0/10Evidence shows HumanLayer's daemon/agent execution can run on a user-controlled host (cloud VM, workstation, private network), but the core control plane is explicitly tied to the hosted app.humanlayer.com service — no docs describe self-hosting that core product. missing for 10: no self-hosted control-plane/server option, no on-prem deployment guide, no Docker/Helm chart or license for running the full stack independently.
- [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
Enterprise licensing
engineering-leadLicense an enterprise deployment with SSO and commercial support for organization-wide rollout
weight 2 · round drawnForeloopnone0/10No evidence in the pack mentions enterprise licensing, SSO, or commercial support tiers; all citations focus on CLI usage, MCP integration, and agent workflows. This is a plausible axis for a SaaS product like Foreloop (organizations often need SSO/enterprise support), so absence of evidence yields 'none' rather than 'na'.
Model flexibility
engineering-leadBring my own LLM or API key so agents run on the model of my choice
weight 2 · round to HumanLayerForeloopnone0/10The docs describe running third-party agents (Claude Code or Codex) locally against Foreloop's task API, but there's no evidence Foreloop lets a user configure or supply their own LLM/API key for Foreloop's own model usage (e.g., intention generation, feedback triage). Missing for 10: any documentation of model/provider selection, API key configuration UI, or support for alternate LLM backends.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
Docs show HumanLayer sessions can use different backends/models — Claude Code, OpenAI Codex, and Claude via Amazon Bedrock instead of the Anthropic API — indicating some flexibility in model/provider choice, which implies bringing your own credentials for these paths. However, there is no explicit doc describing a generic 'bring your own API key' mechanism for arbitrary LLMs or a pricing-tier note tying this to cost savings for engineering leads. Missing for 10: explicit BYO-API-key configuration docs, support for arbitrary/third-party model providers beyond Claude/Codex/Bedrock, and any pricing-related messaging about cost control via own keys.
- [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
- [claimed-docs] “HumanLayer sessions can run Claude through Amazon Bedrock instead of the Anthropic API.”
- [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
Usage quotas
engineering-leadSee and manage plan-based daily task and concurrency limits for agent workflows
weight 2 · round drawnForeloopnone0/10No evidence in the pack mentions plan-based daily task limits, concurrency limits, or any pricing/plan management dashboard for engineering leads; the docs focus on agent workflow mechanics (task claiming, approvals, MCP) with no reference to usage caps or plan tiers.
HumanLayernone0/10The evidence pack has no mention of plan-based daily task/concurrency limits or any admin controls for managing such limits; only a vague reference to 'Advanced Metrics for all paid plans' which covers usage/cost/productivity viewing, not concurrency or daily task limits management.
- [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnForeloopnone0/10No evidence anywhere in the pack mentions data residency, region selection, or storage location options; Foreloop's docs cover CLI, MCP, agents, and feedback widgets but never address where data is stored or any regional controls.
ai-native userControl data retention and deletion
weight 2 · round drawnForeloopnone0/10No evidence pack items mention data retention policies, deletion controls, data export, or privacy settings for user/feedback data collected by Foreloop; all evidence focuses on agent workflows, CLI usage, and MCP integration.
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnForeloopnone0/10No evidence in the pack mentions telemetry, usage tracking, analytics collection, or an opt-out mechanism for Foreloop; the docs cover CLI usage, agent workflows, and feedback widgets but never address data collection or privacy controls.
HumanLayernone0/10No evidence in the pack addresses telemetry, usage tracking, or opt-out controls for HumanLayer; docs cover integrations, workflows, and CLI usage but nothing about privacy/telemetry settings. Missing for 10: any mention of telemetry collection, opt-out mechanism, privacy policy, or data-handling documentation.
Repo integration — stories about repo integration in this arenaRepo integration
Stories about repo integration in this arena
Chat integration
developerTag an agent in a chat thread to discuss and delegate a bug or task
weight 2 · round drawnForeloopnone0/10Foreloop's workflow is built around agents polling for approved tasks/contracts and reporting status via a dashboard (docs-3, docs-7, docs-8), not around tagging or chatting with agents in a thread. No evidence describes a chat-thread interface or an @-mention mechanism to delegate work conversationally.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
HumanLayernone0/10HumanLayer's Slack integration only pushes task-artifact updates into channels (docs-4) and other integrations (GitHub, Jira, Linear) create tasks from tickets/issues, not from tagging an agent inside a chat thread. There is no evidence of a chat-native @mention or in-thread delegation workflow for discussing/assigning tasks to an agent.
- [claimed-docs] “Connect Slack to send HumanLayer task artifact updates into the channels where your team already works.”
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
- [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
- [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
Knowledge context
developerAdd a context file describing my codebase conventions so agents generate more relevant plans and code
weight 3 · round to ForeloopForeloop's docs describe a 'skill' mechanism ('Write a skill when you notice yourself giving an agent the same correction twice') and an installed task-executor skill in the checkout, which loosely maps to encoding project-specific conventions for agents, but there's no explicit documentation of a dedicated context file (e.g., conventions.md) for describing codebase style/conventions. Missing for 10: explicit context-file feature, guidance on structuring codebase conventions, evidence that agents consume such a file specifically for improving plan/code relevance.
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “Write a skill when you notice yourself giving an agent the same correction twice.”
HumanLayernone0/10The docs describe workspace-level config files (workspace.json/workspace.local.json) for team/machine settings and multi-repo setup, but there is no evidence of a dedicated context file for describing codebase conventions to improve agent-generated plans/code.
- [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
- [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
Project management integration
product-managerConnect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly
weight 3 · round to HumanLayerForeloopnone0/10Foreloop's evidence covers GitHub integration, its own task/loop tracking, MCP server, and feedback widgets, but there is no mention of connecting to Jira, Linear, ClickUp, or Monday.com at all — Foreloop appears to use its own internal issue/task system rather than integrating external issue trackers.
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
HumanLayer documents native Jira Cloud and Linear integrations that create tasks from tickets and sync/link status back to source issues, directly matching the story for those trackers. However, there is no evidence of ClickUp or Monday.com integrations, so the story is only partially delivered. Missing for 10: ClickUp integration docs, Monday.com integration docs, independent/hands-on verification of ticket sync working in practice.
- [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
- [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
Version control integration
developerConnect a GitHub repository so an agent can access the code and open pull requests against it
weight 3 · round to HumanLayerDocs explicitly describe connecting GitHub so Foreloop can read code and agents can open PRs against it, with a full workflow (agent claims task in a checkout, does work, opens a PR from the user's GitHub account, gated by human approval). Missing for 10: independent/hands-on verification of the GitHub connection flow and no screenshots or third-party confirmation that PRs are actually opened successfully.
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
Docs describe connecting GitHub to create tasks from issues and link artifacts back (humanlayer-docs-16), plus agent sessions can access code via configured workspaces/repos (humanlayer-docs-8, humanlayer-docs-11) and open draft PRs directly from the session UI (humanlayer-docs-19). missing for 10: independent/hands-on corroboration of the GitHub connection flow and PR-opening working end-to-end beyond first-party docs.
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
- [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
- [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
- [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
developerGrant an agent access to my repositories with a one-click install, without complex setup
weight 2 · round to ForeloopDocs confirm connecting GitHub gives Foreloop read/PR access to repos (foreloop-docs-15) and that agents run against a checkout, but achieving this requires installing a global CLI, running foreloop init, configuring MCP with an API key, and running an agent locally — multiple manual steps rather than a true one-click flow. missing for 10: explicit one-click OAuth install flow, evidence of a simple GitHub App install button, independent confirmation of setup simplicity.
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “npm i -g foreloop # installs globally, so the CLI works in any directory”
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else.”
HumanLayernone0/10Docs describe GitHub/Jira/Linear integrations for creating tasks from issues, but connecting a repo requires selecting a host, configuring workspace.json/workspace.local.json, and setting up remote daemons or multi-repo workspaces — none of this is framed as a one-click, no-setup install. No evidence pack item claims or demonstrates a one-click repo access flow.
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [claimed-docs] “Use `.humanlayer/workspace.json` for shared repository or team configuration. Use `.humanlayer/workspace.local.json` for optional user or ma…”
- [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
Review quality gates — quality gates on changes — review flow, required checks, merge protectionReview quality gates
Quality gates on changes — review flow, required checks, merge protection
Ci remediation
engineering-leadHave failed CI workflows automatically diagnosed and fixed with a proposed pull request
weight 3 · round to HumanLayerForeloopnone0/10Foreloop's documented workflow is task/intention-based (feedback → task → agent → PR), with no evidence of CI pipeline integration, failed-build detection, or automated diagnosis-and-fix triggered by CI failures. The axis is plausible for an agent-driven PR tool, but nothing in the evidence pack shows Foreloop watching or reacting to CI status.
HumanLayer supports running automation sessions from CI (`humanlayer automation run`) and can open draft PRs from a session's diff/GitHub tab, and GitHub integration links tasks to issues—so the building blocks for a CI-triggered fix-and-PR flow exist. However there is no direct evidence of automatic diagnosis of failed CI logs/errors or a documented end-to-end 'CI failure → agent diagnosis → PR' pipeline. Missing for 10: explicit CI-failure-detection/diagnosis workflow docs, example of a failing pipeline auto-triggering a session, and confirmation the resulting PR addresses the CI failure specifically.
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
developerTrigger an agent from CI/CD pipelines to fix a broken build or failing test
weight 2 · round to HumanLayerForeloopnone0/10Foreloop's docs describe agents claiming human-approved tasks and opening PRs from a local checkout, with a CLI usable from "anything that can run one, including coding agents and cron" (foreloop-docs-16), but there is no evidence of CI/CD pipeline integration or of the agent being triggered specifically to fix broken builds or failing tests — its workflow centers on approved feature/task work, not automated CI failure remediation.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
HumanLayer docs explicitly describe `humanlayer automation run` for running a Cloud-visible coding session 'from any automation environment — a CI job, a cron machine, or a script on a server,' plus launch tokens for non-interactive command execution, directly supporting CI/CD-triggered agent runs. However, there is no explicit example or integration guide showing a CI failure (broken build/failing test) triggering the agent to diagnose and fix it, nor independent/hands-on confirmation of this workflow. missing for 10: a concrete CI/CD pipeline example tied to build/test failures, evidence of automatic failure detection triggering the agent, and independent verification of this automation flow working in practice.
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
Diff review
developerConfigure an agent to automatically open a pull request when its task completes
weight 2 · round to ForeloopDocs clearly describe the agent workflow: it claims a task, does the work, and opens a pull request from the user's GitHub account, with GitHub connection enabling PR creation, gated by human approval of the contract before work starts. Missing for 10: independent/hands-on confirmation that PRs are reliably opened on task completion, and no detail on configuring PR-specific settings (e.g., target branch, draft PRs).
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
Docs confirm HumanLayer can open a draft pull request from a session (humanlayer-docs-19) and integrates with GitHub for issue-linked tasks (humanlayer-docs-16), but the evidence describes PR creation as a manual 'ask the session agent' action from the UI, not an automatic trigger fired upon task completion. Missing for 10: explicit configuration option/workflow setting for automatic PR creation on task completion, and any evidence of it happening without a manual prompt.
- [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
developerReview a diff of an agent's changes and approve it before it becomes a pull request
weight 3 · round to HumanLayerForeloopnone0/10Foreloop's 'approval' gate (foreloop-docs-7) is a pre-work contract approval, not a diff review after the agent finishes changes — the agent then 'opens a pull request from your GitHub account' directly (foreloop-docs-3), with no documented step where a person reviews the diff before the PR is created. No evidence describes a diff-preview/approve-before-PR workflow.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
Release notes explicitly reference a diff view and 'draft PR creation' workflow (docs-19, docs-21), and tasks include 'One place for comments and review' (docs-5), indicating a review-before-PR mechanism. However, there's no detailed documentation of an explicit approve/reject gate tied specifically to diff review prior to PR creation, and community discussion raises concerns about automation bias in approvals (comm-3) without disputing the core capability. Missing for 10: explicit step-by-step approval workflow docs, independent hands-on verification of the diff-review-then-approve flow, and clarity on how rejection/edits are handled before PR creation.
- [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
- [claimed-docs] “Keyboard navigation for changed files — Move through the PR changes tree with J/K, N/P, G shortcuts, and Enter.”
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
- [community] “"I feel much more comfortable with senior developers approving AI changes to our codebase, then letting loose an autonomous agent with no hu…”
- [community] “User raised concern about automation bias: once an agent proves reliable, humans may rubber-stamp approvals without careful review, letting …”
Pr review automation
ai-native userHave incoming issues automatically triaged with severity suggested and routed to the right owner
weight 2 · round drawnForeloopnone0/10Foreloop's docs describe turning user feedback into intentions/tasks that agents pick up and claim (foreloop-docs-12, foreloop-docs-4), but there is no evidence of automatic severity classification or routing to a specific human owner — tasks are self-claimed by agents rather than triaged and assigned. missing for 10: severity scoring, owner/routing logic, any triage workflow evidence.
- [claimed-docs] “"Users can't tell which invoices are overdue" is enough to start; you do not need to know which files change.”
- [claimed-docs] “An intention starts from the report, and its agent reads the submission, fills in the intention, and proposes the tasks”
HumanLayernone0/10HumanLayer connects issue trackers (Jira, GitHub, Linear) to create tasks from tickets, but there is no evidence of automatic triage, severity classification, or routing to an owner — integrations only create/link tasks, not assess or assign severity/ownership.
- [claimed-docs] “Connect Jira Cloud so HumanLayer can create tasks from Jira tickets and keep ticket-driven agent work connected to its source.”
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
- [claimed-docs] “Connect Linear so HumanLayer can create tasks from Linear issues, sync issue status, and link HumanLayer artifacts back to the source issue.”
Readiness checks
engineering-leadRun a readiness report that evaluates how ready my repository is for autonomous agents
weight 2 · round drawnForeloopnone0/10No evidence in the pack describes a 'readiness report' or any repository-readiness scoring/assessment feature for autonomous agents; Foreloop's docs cover task claiming, MCP connectivity, feedback widgets, and approval gates, but nothing about evaluating repo readiness.
Security remediation
engineering-leadHave security alerts automatically validated and remediated with an opened pull request
weight 2 · round drawnForeloopnone0/10Foreloop's evidence describes a general feedback/task-to-PR agent workflow (bug reports, feature requests) with human approval gates, but nothing in the pack mentions ingesting security alerts (e.g., from SAST/SCA/dependency scanners), validating them, or triggering remediation PRs specifically for vulnerabilities. The axis is plausible for a task-automation platform, but no evidence shows this specific capability.
HumanLayernone0/10HumanLayer's docs show generic task creation from GitHub/Jira/Linear issues and draft PR creation from agent sessions, but there is no evidence of any security-alert-specific validation or automated vulnerability remediation workflow (no CVE, dependency-alert, or security-scanner integration mentioned).
Scale parallelism — running many jobs at once — concurrency, fleets, queueingScale parallelism
Running many jobs at once — concurrency, fleets, queueing
Concurrent execution
engineering-leadRun many agent tasks concurrently to scale delivery throughput
weight 3 · round to HumanLayerThe docs show that multiple agent workers can each claim tasks and run in parallel checkouts, with a dashboard listing one row per worker (foreloop-docs-8), implying some capacity for concurrent agent execution across a team. However, there is no explicit product feature for orchestrating, scaling, or managing many concurrent tasks at once, no throughput/scale claims, and no evidence of built-in concurrency limits or fleet management tooling. missing for 10: explicit support/documentation for running many agents simultaneously at scale, concurrency controls, throughput metrics, or a scheduler/queue for parallel task dispatch.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
Docs describe an architecture (tasks/sessions, multi-repository workspaces, remote daemons on cloud VMs, and a CLI 'automation run' for CI/cron/scripts) that could support running many agent tasks in parallel, and 'Advanced Metrics' track usage/productivity across an org. However, tutorials and guides are framed around running one task/session at a time, and there is no explicit documentation or evidence of concurrent multi-task orchestration, throughput dashboards, or scaling guarantees for many simultaneous agents. Missing for 10: explicit multi-session concurrency docs, evidence of parallel task orchestration at scale, and independent confirmation of throughput gains.
- [claimed-docs] “A HumanLayer task gives one piece of work: A group of related sessions, A shared set of task files, One place for comments and review, A his…”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [claimed-docs] “Configure a multi-repository workspace with this repository, ../api, and ../web. Make ../web the primary repository. Ask me before you choos…”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”
engineering-leadCreate agent sessions on behalf of other users in my organization
weight 2 · round drawnForeloopnone0/10The evidence describes agents run locally by an individual user in their own repo checkout, claiming tasks and opening PRs from their own GitHub account, but nothing describes an engineering-lead provisioning or launching agent sessions on behalf of other org members. Missing for 10: any admin/org-level session creation API or UI, delegated-session documentation, or multi-user session management evidence.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
HumanLayernone0/10The evidence describes tasks, sessions, integrations, and org-wide metrics access but never mentions an engineering-lead or admin creating/assigning agent sessions on behalf of another named user in the organization. No account-delegation, impersonation, or 'assign session to teammate' capability is documented.
Deployment flexibility
developerUse a managed cloud offering to run agents without operating my own backend infrastructure
weight 2 · round to HumanLayerForeloopnone0/10Foreloop's own docs state the opposite of this story: agents (Claude Code or Codex) run 'in a checkout of your repository on your own machine,' with Foreloop only coordinating tasks and approvals — there is no evidence of a managed cloud runtime that executes agents on Foreloop's infrastructure.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Development, under Agents, lists one row per worker: which agent it is running as, the name that checkout gave itself, the task it holds, an…”
HumanLayer offers a cloud control plane (app.humanlayer.com, automation sessions marked 'Cloud-visible', advanced metrics for paid plans) that lets users monitor and manage agent runs remotely, suggesting a managed service layer. However, docs explicitly state the actual agent execution still runs on a host that the user must select and operate—'a cloud VM, workstation, or private-network machine'—meaning the developer still needs to provision/manage backend compute for the daemon rather than HumanLayer fully hosting execution. Missing for 10: evidence of a fully HumanLayer-operated compute backend (no user-managed VM/daemon required), and independent confirmation of a true zero-ops managed runtime.
- [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
- [claimed-docs] “Advanced Metrics for all paid plans — View usage, cost, and productivity metrics, with optional access for every organization member.”
Headless automation
developerRun an agent headlessly inside CI/CD pipelines and shell scripts
weight 2 · round to HumanLayerDocs mention the CLI is invocable from 'anything that can run one, including coding agents and cron' and support scriptable --json output, suggesting headless invocation is possible (foreloop-docs-16, foreloop-docs-9). However, other docs state that 'nothing runs until a person approves a contract' (foreloop-docs-7), implying a human-in-the-loop gate that complicates fully unattended CI/CD execution, and there is no explicit CI/CD pipeline example or corroboration. Missing for 10: explicit CI/CD/pipeline example, evidence of non-interactive approval flow, independent confirmation of headless operation.
- [claimed-docs] “Projects, loops, intentions, stories and tasks from a terminal — and from anything that can run one, including coding agents and cron.”
- [claimed-docs] “`--json` works on any command and has a stable shape. Additive changes only.”
- [claimed-docs] “Nothing runs until a person approves a contract. Approval is the only gate, and the gate is deliberate: an agent is about to change your rep…”
Docs explicitly describe `humanlayer automation run` for running Cloud-visible coding sessions from CI jobs, cron machines, or scripts, plus launch tokens for non-interactive/non-PTY execution suited to headless environments. This directly matches running an agent headlessly in CI/CD and shell scripts. Missing for 10: independent/hands-on verification of CI usage and concrete pipeline examples (e.g. GitHub Actions config).
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
Not comparable on these axes
ai-native userTest against a sandbox environment without touching production data
weight 1 · not comparableForeloopn/aForeloop is a project/task-management and agent-orchestration tool for coding work (tasks, PRs, feedback widgets); it has no concept of a sandbox testing environment vs production data. This is a category error for its product type — the axis of 'sandbox vs production data testing' doesn't apply.
HumanLayernone0/10HumanLayer's docs describe remote daemons, workspaces, and automation sessions, but there is no mention of a sandbox environment, staging/test data isolation, or any mechanism to separate test runs from production data. missing for 10: explicit sandbox/staging environment documentation, data isolation guarantees, evidence of test-vs-production separation.
ai-native userHave each task prompt automatically routed to the most suitable underlying model
weight 2 · not comparableForeloopn/aForeloop is a task-management/agent-orchestration layer that hands work to whatever coding agent (Claude Code, Codex) the developer runs locally — it does not itself route prompts across multiple underlying LLMs. Automatic model routing per task is a wrong axis for this product's category (it's a task/PM layer, not a multi-model gateway).
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “foreloop init installs the task-executor skill into the checkout, which is what lets an agent working in that directory pick up the project'…”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
HumanLayernone0/10Evidence shows HumanLayer lets users manually select or configure which model/backend to use (Claude via Bedrock, Codex, RPI sub-agents) but there is no evidence of automatic routing of a task prompt to the 'most suitable' model based on task characteristics.
- [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
- [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
- [claimed-docs] “HumanLayer sessions can run Claude through Amazon Bedrock instead of the Anthropic API.”
engineering-leadSwitch away from automatic model selection to a specific model of my choice
weight 1 · not comparableForeloopn/aForeloop orchestrates tasks/agents (Claude Code, Codex) run by the user rather than offering its own model-selection setting; there is no evidence of an internal model picker, so this axis is a category error for this product's role as an agent-orchestration/task platform.
Docs show explicit model/backend selection — choosing Codex ("install, authenticate, select, and check Codex") or running Claude via Amazon Bedrock instead of the Anthropic API, plus subagent model registration — indicating an engineering lead can pick a specific model rather than a default. However, there is no explicit documentation of an 'automatic' default-selection mode being toggled off, so the framing of 'switching away from automatic' isn't directly evidenced. Missing for 10: explicit mention of an automatic/default model-selection setting and a documented UI/CLI flag to override it, independent confirmation of model-switching behavior.
- [claimed-docs] “This guide shows you how to install, authenticate, select, and check Codex for HumanLayer sessions.”
- [claimed-docs] “HumanLayer sessions can run Claude through Amazon Bedrock instead of the Anthropic API.”
- [claimed-docs] “HumanLayer registers these RPI sub-agents for Claude Code sessions”
ai-native userPrevent my data from being used to train AI models
weight 3 · not comparableForeloopn/aForeloop is a task/agent orchestration and feedback-collection tool for software repos, not a data-processing or AI-model-training service; the evidence pack contains no data-usage or model-training policy at all, and this axis is a category error for this product type.
developerQuery generated documentation for any public or private repository
weight 1 · not comparableForeloopn/aForeloop is a task/agent orchestration and feedback-collection tool that connects to GitHub repos to let agents claim tasks and open PRs; it does not generate or expose queryable documentation for repositories. This story targets a documentation-generation/query product category, which is a different axis from Foreloop's core offering.
engineering-leadHave every pull request automatically reviewed with AI-generated inline comments
weight 3 · not comparableForeloopn/aForeloop is a task-management/agent-orchestration tool that lets agents claim tasks and open PRs from feedback, not a PR-review/code-review product; there's no evidence it reviews pull requests or adds inline review comments — this is a different product category (task orchestration vs. review-quality-gates).
HumanLayernone0/10HumanLayer's evidence covers task/session management, human-in-the-loop approvals, draft PR creation, and a PR diff-viewing UI, but there is no mention of an automated review process that generates inline review comments on every pull request. Missing for 10: no evidence of automatic PR review triggers, no mention of AI-generated inline comments, no review-quality-gate CI integration for PRs.
- [claimed-docs] “Draft PR creation — Ask the session agent to open a draft pull request from the GitHub tab or diff view.”
- [claimed-docs] “Keyboard navigation for changed files — Move through the PR changes tree with J/K, N/P, G shortcuts, and Enter.”
- [claimed-docs] “Connect GitHub to create HumanLayer tasks from issues and link task artifacts back to the source issue.”
engineering-leadAutomatically fix failing agent-readiness criteria in my repository
weight 1 · not comparableForeloopnone0/10Foreloop's docs describe a task/feedback pipeline where agents claim approved tasks and open PRs, but nothing in the evidence pack mentions detecting or automatically remediating 'agent-readiness criteria' (e.g., AGENTS.md, tool-permission manifests, readiness scoring) in a repository. Missing for 10: any readiness-criteria scanning/grading feature, evidence of automatic detection of failing criteria, and evidence of automated fixes tied to such criteria.
HumanLayern/aHumanLayer is a human-in-the-loop agent orchestration/approval tool for running coding tasks, not a repository readiness/compliance scanner with auto-remediation of 'agent-readiness criteria'. This axis is a category error for this product type — no evidence pack content relates to detecting or auto-fixing repo readiness criteria.
engineering-leadSelf-host agent infrastructure locally, in containers, or on my own VMs
weight 2 · not comparableForeloopn/aForeloop is a hosted SaaS orchestration layer (agents run locally in your repo checkout and talk to Foreloop's cloud API/MCP endpoint) — there is no evidence of a self-hostable server/backend component; the product's core value is the hosted service coordinating tasks, so 'self-hosting agent infrastructure' is a category mismatch rather than a missing feature.
- [claimed-docs] “You run an agent — Claude Code or Codex — in a checkout of your repository on your own machine. That agent asks Foreloop for approved tasks …”
- [claimed-docs] “Foreloop speaks MCP at /api/app/public/v1/mcp over streamable HTTP. An MCP client needs that URL and an API key and nothing else”
- [claimed-docs] “Connecting GitHub is how Foreloop reads your code, and how an agent opens a pull request against it.”
Docs describe running the remote daemon on a cloud VM, workstation, or private-network machine that you control (humanlayer-docs-7), plus automation sessions from CI/cron/server environments (humanlayer-docs-12) and launch tokens for bootstrap scripts (humanlayer-docs-13), showing self-hosted deployment flexibility. However there is no explicit mention of container/Docker deployment or an official container image, and no independent verification of self-hosted setups at scale. Missing for 10: explicit container/Docker packaging docs, independent hands-on confirmation of self-hosted deployments.
- [claimed-docs] “The host can be a cloud VM, workstation, or private-network machine. Select a host that can access the code, tools, credentials, and private…”
- [claimed-docs] “This tutorial teaches you how to run one task on a remote machine. You will control the task from app.humanlayer.com on a machine or phone.”
- [claimed-docs] “Use `humanlayer automation run` to run one Cloud-visible coding session from any automation environment — a CI job, a cron machine, or a scr…”
- [claimed-docs] “Use a launch token for one non-interactive command. Examples include a shell without an interactive PTY and a one-time remote bootstrap scri…”