Factory vs Codegen
Codegen wins · 15–24 (32 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to CodegenFactory hosts an actual llms.txt at docs.factory.ai/llms.txt (HTTP 200) confirmed by direct probe, and its docs describe an agent-native platform with structured agent-oriented documentation (droid-cli, agent-readiness, missions) that an agent could be pointed at. missing for 10: independent/hands-on confirmation that an agent successfully consumes llms.txt in practice, and no explicit vendor statement encouraging users to point agents at llms.txt.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.factory.ai/llms.txt # Factory Documentation > Documentation for Factory, the agent-native software…”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [claimed-docs] “Run the /readiness-report slash command in a Factory App web or desktop session, or in the Droid CLI, to evaluate a repository's readiness l…”
A probe confirms Codegen's docs site serves a working llms.txt at https://docs.codegen.com/llms.txt (HTTP 200) listing structured doc links, directly enabling an agent to be pointed at agent-oriented docs; this is corroborated by extensive first-party documentation of agent-friendly interfaces (SDK, CLI, API). Missing for 10: independent third-party confirmation of agents actually consuming the llms.txt in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.codegen.com/llms.txt # Codegen ## Docs - [Codegen](https://docs.codegen.com/introduction/overview…”
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round drawnFactory documents droid exec as a one-shot CLI command explicitly designed for CI/CD pipelines, shell scripts, and batch processing, with tiered autonomy controls for unattended operation. This directly satisfies headless/CI automation. Missing for 10: independent/hands-on third-party verification of CI usage and more detail on exit codes/output formats for pipeline integration.
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [probe] “official CLI documented at https://docs.factory.ai/droid-cli/quickstart”
Codegen explicitly supports headless/automated usage via a Python SDK for programmatic agent invocation, a REST API with explicit CI/CD trigger support ('Trigger agents from CI/CD pipelines when builds fail'), and a CLI for scripted interactions and telemetry — all first-party documented. Missing for 10: independent/hands-on confirmation of CI usage in the wild and a working OpenAPI spec (probe found 404s), so it's vendor-doc-only evidence.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round drawnDocs explicitly state Droid CLI can connect MCP tools alongside Jira/Notion/Slack/Linear/PagerDuty integrations, confirming MCP client support. However, there is no detail on setup/configuration process, supported transport types, or independent/hands-on confirmation of MCP tool usage in practice. missing for 10: configuration details for adding MCP servers, examples of MCP tools being invoked, independent verification of functionality.
- [claimed-docs] “Connect Jira, Notion, Slack, Linear, PagerDuty, and MCP tools to keep development synchronized with team systems”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
Codegen's docs explicitly mention connecting to Slack, Linear, Figma, databases and extending capabilities 'with custom MCP tools', and separately reference provisioning MCP servers across an org — direct evidence of MCP client integration. However, the evidence is a single brief doc mention with no detailed setup guide, supported server list, or independent/hands-on confirmation of the feature working. Missing for 10: detailed MCP integration docs/tutorial, list of supported MCP servers, independent verification of the feature in practice.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
- [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”
ai-native userUse an official CLI
weight 2 · round to FactoryFactory ships an official Droid CLI with documented quickstart, terminal/editor/Git integration, bash mode, one-shot exec for CI/CD, and slash-command features—clearly an official CLI for AI-native workflows. Missing for 10: independent/hands-on third-party corroboration beyond vendor docs.
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [claimed-docs] “Press ! to toggle bash mode and run shell commands directly without AI interpretation.”
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [probe] “official CLI documented at https://docs.factory.ai/droid-cli/quickstart”
Codegen documents an official CLI (docs.codegen.com/introduction/cli) with concrete capabilities: viewing/pulling agent work, creating new agents, running Claude Code with telemetry, and managing organizations/repositories — matching an AI-native agentic workflow tool. missing for 10: independent/hands-on corroboration beyond first-party docs, and no details on installation/versioning maturity.
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “Manage organizations - Switch between organizations and configure repositories”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
ai-native userDrive the product through a documented public API
weight 3 · round to CodegenFactory documents a public API for creating and driving Droid sessions (factory-docs-13) plus CLI/exec automation paths (droid exec, CI/CD) that give programmatic control, but there's no discoverable OpenAPI/swagger spec (probe found only 404s), suggesting the API reference is thinner or less formally specified than a full REST API story would require. missing for 10: a published OpenAPI/swagger schema, broader API endpoint coverage beyond sessions, independent third-party confirmation of API usage.
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.factory.ai/openapi.json, https://docs.factory.ai/swagger.json, https://docs.factory.ai/…”
Codegen documents a full API reference (agent creation, traces, CI/CD triggers), a Python SDK for programmatic agent control, and a CLI for managing agents/orgs, giving AI-native users multiple documented ways to drive the product programmatically. Missing for 10: a discoverable machine-readable OpenAPI/swagger spec (probe returned 404s) and independent third-party corroboration of the API's reliability.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round drawnFactorynone0/10No evidence of scoped or least-privilege API credential/token issuance for agents; docs mention API sessions and integrations (Jira, Slack, MCP) but nothing about credential scoping, permission tiers for API keys, or least-privilege access control. Tiered autonomy (factory-docs-7) governs action confirmation, not credential scoping.
Codegennone0/10Codegen's SDK/API uses a simple org_id + token authentication (codegen-docs-15) with no documentation of scoped, role-based, or least-privilege credential issuance, token permissions, or granular access controls. Missing for 10: any mention of scoped API keys, permission levels, or restricted-access tokens for agents.
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userBuild against official SDKs
weight 2 · round to CodegenFactory documents an API reference for creating/driving Droid sessions (factory-docs-13) and MCP/CLI integration points, suggesting some programmatic access, but there is no evidence of an official language SDK (Python/JS/etc.) and the OpenAPI spec probe returned 404s, indicating thin or unpublished API tooling. missing for 10: no named official SDK packages, no OpenAPI/schema availability, no code samples showing SDK usage.
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.factory.ai/openapi.json, https://docs.factory.ai/swagger.json, https://docs.factory.ai/…”
- [probe] “official MCP server documented at https://docs.factory.ai/harness/mcp”
Codegen documents an official Python SDK for programmatic agent control (agent.run, org_id/token auth), a full API reference with traces and CI/CD triggers, and a companion CLI — all first-party docs. missing for 10: independent/hands-on corroboration beyond docs, and no public OpenAPI spec was found (probe returned 404s).
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
ai-native userSubscribe to events via webhooks
weight 2 · round drawnFactorynone0/10No evidence of webhook subscription support; docs mention API sessions, MCP tools, and integrations (Jira, Slack, etc.) but nothing about webhooks for event notifications. missing for 10: any webhook documentation, event types, subscription endpoints, or third-party confirmation of webhook support.
Agentic features
ai-native userSet up automations that run autonomously in the background
weight 2 · round to Codegendroid exec supports one-shot autonomous runs with tiered autonomy suited for CI/CD, scripts, and batch automation, and the API supports creating/driving Droid sessions programmatically, which enables background automation setups. However, there's no evidence of a scheduling/trigger system (e.g., cron-like or event-driven automations) or a dedicated 'automations' dashboard for persistent background jobs. Missing for 10: native scheduling/triggers for autonomous background runs, independent hands-on confirmation of unattended long-running automations, and a dedicated automations management UI.
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
- [claimed-docs] “Use Factory Missions to plan and execute large, multi-feature projects with structured orchestration.”
Codegen documents agents that trigger autonomously from CI/CD failures, Slack/Linear/Jira events, and PR activity (auto-review, auto-comment), plus SDK/CLI for programmatic background automation — this matches the story of autonomous background automations. Missing for 10: independent/hands-on verification of reliability, and no explicit scheduled/cron-trigger mechanism documented.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round drawnFactory's core product is built around delegating tasks to Droid agents via App, CLI, or exec, with clear docs on task delegation, custom droids, and session management. missing for 10: independent/hands-on corroboration beyond vendor docs, and quantified reliability of delegated task completion.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [claimed-docs] “delegate scoped tasks to Custom Droids via /droids, and package reusable procedures as Skills with /skills or /create-skill”
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
Codegen's documentation clearly describes delegating coding tasks to a built-in AI agent that analyzes requirements, implements features, fixes bugs, writes tests, and reports progress via prompts, accessible through chat, Slack, CLI, and SDK. Missing for 10: independent/hands-on verification of the assistant's actual task-completion quality beyond vendor docs.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userOperate the product with natural-language commands
weight 2 · round drawnFactory's Droid CLI and App let users delegate tasks, run commands, and manage workflows via natural-language prompts, with bash-mode fallback and slash commands (/droids, /skills, /readiness-report) layered on top of NL interaction, showing deep agentic natural-language operation across CLI, exec, and Missions. missing for 10: independent/hands-on user reports confirming NL command quality and reliability outside vendor docs.
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [claimed-docs] “Press ! to toggle bash mode and run shell commands directly without AI interpretation.”
- [claimed-docs] “delegate scoped tasks to Custom Droids via /droids, and package reusable procedures as Skills with /skills or /create-skill”
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Use Factory Missions to plan and execute large, multi-feature projects with structured orchestration.”
- [probe] “official CLI documented at https://docs.factory.ai/droid-cli/quickstart”
Codegen is fundamentally natural-language driven: docs show prompt-based task execution ('Leave a review on PR #123'), Slack/Jira/Linear interactions, and SDK agent.run(prompt=...) all operate via NL commands across code, PRs, and issue trackers. This spans chat, CLI, and SDK surfaces, consistently prompt/NL-driven. Missing for 10: independent/hands-on corroboration of NL command interpretation quality and edge cases beyond first-party docs.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnFactorynone0/10Factory does have an API reference section (docs.factory.ai/api-reference/sessions) but there's no evidence of an interactive, runnable-example explorer — the probe for standard OpenAPI/Swagger interactive docs (openapi.json, swagger.json, etc.) returned 404 at all candidate paths, suggesting no live 'try it' interface exists.
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.factory.ai/openapi.json, https://docs.factory.ai/swagger.json, https://docs.factory.ai/…”
Codegennone0/10Codegen has an API reference page (overview, SDK docs) but no evidence of an interactive reference with runnable/executable examples; the OpenAPI/spec probe returned 404 for all candidate paths, suggesting no interactive API explorer exists.
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round drawnFactorynone0/10Factory has an API reference (sessions endpoints) but probes for standard OpenAPI/swagger spec locations all returned 404, indicating no downloadable machine-readable spec is published; missing for 10: any accessible OpenAPI/swagger JSON file or equivalent machine-readable spec.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.factory.ai/openapi.json, https://docs.factory.ai/swagger.json, https://docs.factory.ai/…”
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
Codegennone0/10Codegen has an API reference (docs-16/17/18) but explicit probes for machine-readable spec files (openapi.json, swagger.json, etc.) all returned 404, indicating no downloadable OpenAPI/machine-readable spec is available.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to CodegenFactorynone0/10Factory's evidence covers coding agent workflows (CLI, sessions, MCP, readiness reports) but nothing addresses a sandbox environment for testing separate from production data. No mention of sandbox mode, staging environment, or data isolation guarantees.
Codegen docs state that code changes run in 'robust isolated environments' to safely execute code and test changes, which implies sandboxed testing separate from production. However, there's no explicit description of how production data is excluded or protected, no detail on the sandbox's scope/persistence, and no independent verification of this isolation claim. missing for 10: explicit production-data isolation guarantees, technical detail on sandbox architecture, independent/hands-on confirmation.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnFactorynone0/10There is an API reference (sessions) but no evidence of API versioning scheme or a documented deprecation policy; the OpenAPI spec probe even returned 404s across candidate paths, suggesting no formal versioned spec is exposed.
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.factory.ai/openapi.json, https://docs.factory.ai/swagger.json, https://docs.factory.ai/…”
Codegennone0/10There's an API reference and SDK docs, but no evidence of API versioning scheme or a documented deprecation policy; the openapi probe even returned 404s, suggesting no formal spec surfacing version info.
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to Factorydroid exec is explicitly documented as a one-shot command 'ideal for CI/CD pipelines, shell scripts, and batch processing,' and the API lets users programmatically create/drive many Droid sessions, both enabling bulk automation across items. However, there's no explicit example, docs, or tooling demonstrating a native 'run across many files/repos/tickets at once' bulk operation feature — it relies on the user scripting droid exec in loops rather than a built-in bulk-operation UI/command. Missing for 10: a dedicated bulk-operation command or documented multi-item batch workflow example, and independent/hands-on evidence of it working at scale.
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
- [claimed-docs] “Use Factory Missions to plan and execute large, multi-feature projects with structured orchestration.”
The Python SDK and API allow programmatic creation/monitoring of agents and triggering from CI/CD, which could be scripted in a loop to perform bulk operations, but there is no documented batch/bulk endpoint or explicit guidance for operating across many items at once. missing for 10: dedicated bulk/batch API or CLI command, documented examples of multi-item automation, and evidence of scale/performance handling for bulk workloads.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to CodegenFactory's droid exec is explicitly designed to run as a one-shot command in CI/CD pipelines, shell scripts, and batch jobs, which implies it can be wired to external events (e.g., git hooks, CI triggers) with tiered autonomy controlling what runs unattended — but this is an execution mode, not a native rule/trigger definition system where a user declares 'on event X, do Y'. Missing for 10: explicit rule/trigger authoring UI or config, built-in event listeners (e.g., webhook triggers, issue-created triggers), and any documented automation-rules engine beyond CI invocation.
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
- [claimed-docs] “Open Software Factory to see your delivery lifecycle as an automation coverage map.”
Codegen supports event-driven automation via CI/CD triggers (docs-18) and PR review triggers on failure (docs-20), plus SDK/API for programmatic agent invocation, but there's no evidence of a general user-facing rules engine for defining arbitrary trigger-condition-action automations across events. missing for 10: a documented rules/automation builder UI, broader event types beyond CI/CD failures and PR diffs, and independent confirmation of custom trigger logic.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
ai-native userSchedule recurring jobs or workflows
weight 2 · round drawnFactorynone0/10Evidence shows droid exec for CI/CD one-shot automation and API session management, but no documentation of scheduling or recurring/cron-style job execution exists anywhere in the pack. missing for 10: any mention of scheduling, cron, recurring triggers, or timed/repeated workflow execution.
ai-native userVersion, review, and roll back my automations
weight 1 · round to FactoryFactory supports reviewing diffs and merging via git workflow (factory-docs-1, factory-docs-2), and packages automations as reusable Custom Droids/Skills (factory-docs-5), which implies some git-based versioning, but there is no explicit documentation of a version-history feature for Droids/Skills nor an explicit rollback mechanism for automations themselves. missing for 10: explicit versioning UI/history for Skills/Droids, dedicated rollback command or feature distinct from generic git revert.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [claimed-docs] “delegate scoped tasks to Custom Droids via /droids, and package reusable procedures as Skills with /skills or /create-skill”
Codegennone0/10Evidence covers PR/code review, commits, branches, and traces/telemetry, but there is no mention of versioning automations themselves, reviewing automation history, or rolling back agent configurations/workflows. missing for 10: automation versioning system, rollback mechanism for agent workflows, audit/history UI for automations.
Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionAutonomous implementation
End-to-end implementation by the agent — multi-file changes, task completion
End to end feature delivery
ai-native userHave an agent automatically generate and run tests to validate its own code changes before proposing them
weight 2 · round to CodegenFactorynone0/10Docs mention integration with 'tests' as part of Git workflow and bash-mode shell execution, plus readiness-report/readiness-fix commands, but none of this describes an agent autonomously generating and running tests to validate its own code changes before proposing a diff. Missing for 10: explicit documentation of automated test generation, self-validation loop, or evidence droid runs tests as a pre-proposal gate.
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [claimed-docs] “Press ! to toggle bash mode and run shell commands directly without AI interpretation.”
- [claimed-docs] “Run the /readiness-report slash command in a Factory App web or desktop session, or in the Droid CLI, to evaluate a repository's readiness l…”
- [claimed-docs] “Automatically fix failing criteria with the /readiness-fix slash command in a Factory App web or desktop session, or in the Droid CLI”
Docs state the agent writes tests as part of implementing features and can run code/install dependencies/test changes in isolated environments, and a PR-review flow checks test coverage before flagging issues — together implying self-validation via generated tests. However, there is no explicit end-to-end walkthrough showing the agent generating tests, running them, and iterating/fixing before proposing a PR, nor independent/hands-on confirmation. missing for 10: explicit documented workflow tying test generation+execution to pre-PR validation, independent corroboration of this loop working in practice.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
developerHave an agent autonomously diagnose and fix a reported bug
weight 3 · round drawnFactory's Droid CLI/exec and delegated task workflow support autonomous code changes (diff review and merge) and integrations like Jira/Linear/PagerDuty for bug tracking, plus tiered autonomy for unattended operation, which together plausibly support autonomous bug diagnosis and fixing. However, no evidence explicitly describes an end-to-end 'diagnose a reported bug from ticket to verified fix' workflow or hands-on validation of bug-fixing accuracy. Missing for 10: explicit bug-diagnosis workflow documentation, independent/hands-on evidence of successful autonomous bug fixes, and details on root-cause diagnosis capability.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
- [claimed-docs] “Connect Jira, Notion, Slack, Linear, PagerDuty, and MCP tools to keep development synchronized with team systems”
- [claimed-docs] “Automatically fix failing criteria with the /readiness-fix slash command in a Factory App web or desktop session, or in the Droid CLI”
Docs explicitly claim the agent can 'analyze requirements, implement features, fix bugs, write tests' and can be triggered from issue trackers/CI to autonomously act, plus run in isolated environments to test changes — directly supporting autonomous bug diagnosis and fix. However, all evidence is first-party marketing/docs with no independent or hands-on validation of end-to-end autonomous diagnosis-to-fix workflows, and no detailed walkthrough of the diagnosis step itself (e.g., root-cause analysis) beyond generic 'fix bugs' language. Missing for 10: independent/hands-on verification of successful autonomous bug fixes, detailed diagnostic reasoning process, and success-rate or case-study evidence.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
product-managerGo from a mockup or design to a working implementation without an engineering handoff
weight 2 · round to FactoryFactory's agent-readiness docs explicitly describe the exact scenario: "A designer shares a mockup, and the system implements it without handoffs" (factory-docs-8), backed by delegation/review/merge workflow (factory-docs-1) and readiness tooling (factory-docs-9, factory-docs-10) that lets non-engineers trigger and validate implementation. Missing for 10: independent or hands-on corroboration of a PM-specific end-to-end mockup-to-merge case study, and detail on how a non-technical PM reviews/approves the diff without engineering involvement.
- [claimed-docs] “A designer shares a mockup, and the system implements it without handoffs.”
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “Run the /readiness-report slash command in a Factory App web or desktop session, or in the Droid CLI, to evaluate a repository's readiness l…”
- [claimed-docs] “Automatically fix failing criteria with the /readiness-fix slash command in a Factory App web or desktop session, or in the Droid CLI”
Codegen's docs show relevant building blocks — a Figma connector, Slack/Linear interfaces that let non-engineers prompt the agent, and full PR/branch creation — implying a PM could describe a design and get code without direct engineering handoff. However, there is no concrete workflow, case study, or example showing a mockup/design being turned into a working implementation end-to-end. Missing for 10: an explicit design-to-code workflow example, evidence of Figma-to-PR pipeline, and PM-specific case studies or testimonials showing the handoff-free process actually works.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
developerHave an agent implement a requested feature end-to-end, including writing tests
weight 3 · round to CodegenFactory's docs show agents can be delegated end-to-end feature tasks (delegate, diff review, merge), run in terminal/CI with test execution and git workflow, and orchestrate multi-feature projects via Missions, but no explicit first-party evidence confirms the agent autonomously writes tests as part of implementation. missing for 10: explicit documentation of test-writing behavior, independent/hands-on verification of end-to-end feature delivery.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Use Factory Missions to plan and execute large, multi-feature projects with structured orchestration.”
Docs explicitly describe agents analyzing requirements, implementing features, writing tests, and running/testing changes in isolated environments, plus PR/branch management for end-to-end delivery. Missing for 10: independent/hands-on validation of end-to-end feature delivery and no concrete example transcript showing a full feature+test cycle.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
Environment setup
developerHave an agent automatically clone the repo, install dependencies, and configure its own working environment
weight 2 · round to CodegenFactory's Droid CLI/exec can run shell commands autonomously (bash mode, tiered autonomy for unconfirmed operations) and operates within a repo's terminal/Git workflow, which implies it could run clone/install commands, but no evidence explicitly describes an agent autonomously cloning a repo or bootstrapping its own dev environment from scratch. missing for 10: explicit documentation of automated repo cloning, dependency installation, or environment provisioning as a first-class capability, and any hands-on example showing this workflow.
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [claimed-docs] “Press ! to toggle bash mode and run shell commands directly without AI interpretation.”
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
Docs describe agents that access repos via GitHub App integration, run in isolated environments, and 'safely run code, install dependencies, and test changes' automatically without complex setup, matching the story of autonomous environment setup. Missing for 10: independent/hands-on verification of environment provisioning behavior and more detail on how dependency installation is auto-configured beyond the brief doc mention.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
Interactive takeover
developerTake over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work
weight 2 · round to FactoryFactory explicitly supports multi-surface access (App/web, terminal via Droid CLI, browser) and delegating tasks, reviewing diffs, and merging from any of these surfaces, which implies continuity across surfaces. However, there's no explicit documentation of a 'takeover mid-task' handoff flow (e.g., pausing an in-progress session in one surface and resuming/redirecting it live in another) — the evidence shows task delegation and review/merge but not explicit interactive takeover semantics. Missing for 10: explicit documentation of resuming/redirecting an in-progress session across surfaces, and independent/hands-on confirmation of this handoff working smoothly.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
Codegen supports interacting with agents across multiple surfaces (Slack notifications/clarification requests, CLI to pull agent branches locally, Python SDK to run/monitor agents) and lets you download in-progress work to your local environment, which implies some ability to pick up and continue work. However there is no explicit documentation of pausing, redirecting, or steering a task mid-execution from editor/terminal/browser as a unified handoff flow. Missing for 10: explicit 'takeover' or redirect mid-task feature, editor/IDE integration for live task control, and evidence of seamless cross-surface handoff (Slack->CLI->browser) of the same in-progress task.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
developerSend follow-up instructions to an active agent session to steer its work without restarting
weight 2 · round to FactoryFactory's API reference explicitly supports creating and driving Droid sessions including sending messages within an active session (factory-docs-13), and the CLI is interactive by nature, implying follow-up prompts are possible without restarting. However, there is no explicit documentation describing mid-task interruption/steering while the agent is actively executing a long-running task (e.g., Missions or droid exec), so the steering-while-running behavior is not directly evidenced. Missing for 10: explicit docs on interrupting/redirecting an in-progress autonomous run, and independent/hands-on confirmation that follow-up messages actually steer ongoing work rather than queue for the next turn.
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [claimed-docs] “Use Factory Missions to plan and execute large, multi-feature projects with structured orchestration.”
Codegennone0/10No evidence describes sending follow-up instructions to an already-running agent session to redirect its work; docs only describe initiating agents via prompts, Slack notifications, or SDK task creation, not mid-session steering. Missing for 10: any documentation of a 'continue/steer/update' API, chat-like follow-up in an active run, or session-resume mechanism.
Sandbox execution
developerHave an agent safely execute code and install dependencies inside an isolated sandbox
weight 3 · round to CodegenFactorynone0/10Evidence describes tiered autonomy, bash mode, and CI/CD execution (droid exec) but never mentions an isolated sandbox environment for code execution or dependency installation; no container/VM isolation is documented.
First-party docs explicitly state the agent can 'Safely run code, install dependencies, and test changes in robust isolated environments,' directly matching the story. missing for 10: independent/hands-on corroboration of sandbox isolation, technical details on sandbox implementation (e.g., container tech, resource limits) beyond a single doc line.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsHuman oversight
Keeping a human in the loop — approvals, checkpoints, interrupts
Approval controls
developerConfigure an agent to auto-approve all its actions instead of confirming each one
weight 2 · round to FactoryFactory's docs confirm 'tiered autonomy' in droid exec that controls what operations run without manual confirmation, implying a configurable auto-approve mode for CI/batch use, but there's no explicit documentation of a full 'auto-approve all actions' toggle or its exact configuration options/flags. missing for 10: explicit config syntax/flag for full auto-approval, independent confirmation of behavior, coverage of auto-approve in interactive (non-exec) sessions.
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
Codegennone0/10The evidence describes Codegen's autonomous capabilities (running code, committing, creating PRs, Slack/Jira integration) but never mentions any per-action confirmation mode or an auto-approve toggle/setting for oversight control. Missing for 10: any documentation of an approval/confirmation workflow, a setting to enable/disable auto-approval, or human-in-the-loop gating that this story presupposes.
product-managerApprove key agent decisions from my phone while agents continue working
weight 1 · round drawnDocs mention delegating tasks, reviewing diffs, and merging 'from the App or your terminal' and tiered autonomy that gates operations needing confirmation, implying some human-in-the-loop review outside the terminal, but there is no explicit evidence of a mobile/phone app or of approving in-flight agent decisions remotely while agents keep working. Missing for 10: explicit mobile app/phone interface, evidence of asynchronous approval while agent continues running, and independent confirmation of this workflow.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
Codegen supports Slack-based notifications and agent interaction (docs-2, docs-9), which could technically be used from a phone via the Slack app, but there is no explicit mobile app, push-approval workflow, or documented 'approve on the go' feature for key agent decisions. missing for 10: dedicated mobile/approval UI, evidence of approving specific agent decisions (e.g., merge/deploy gates) from a phone, and any hands-on confirmation of this workflow.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
engineering-leadSet tiered autonomy levels controlling what an agent can do without manual confirmation
weight 3 · round to FactoryFactory explicitly documents 'tiered autonomy' in droid exec to control which operations run without manual confirmation, directly matching the story, but this is scoped to the CI/CD-oriented droid exec mode rather than a broader, configurable set of autonomy tiers across all agent surfaces. missing for 10: detail on specific tier levels/permissions, configuration UI or granular controls, and evidence this applies uniformly across App/CLI sessions, not just droid exec.
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
Codegennone0/10No evidence describes configurable autonomy tiers, approval gates, or permission levels for agent actions; the docs describe what agents can do (code, PRs, Slack, Jira/Linear) but nothing about setting graduated confirmation thresholds for an engineering lead to control autonomy.
Model control
ai-native userHave each task prompt automatically routed to the most suitable underlying model
weight 2 · round drawnFactorynone0/10No evidence describes automatic routing of prompts to the most suitable underlying model; docs mention model-agnostic droid workflows, custom droids, and orchestration but nothing about auto-selecting models per task.
engineering-leadSwitch away from automatic model selection to a specific model of my choice
weight 1 · round drawnFactorynone0/10No evidence in the pack mentions model selection, automatic model routing, or the ability to choose a specific model over an automatic default; the docs cover CLI usage, integrations, missions, and readiness reports but nothing about model choice controls.
Visibility monitoring
developerWatch what a running agent is doing in real time, including its current status
weight 3 · round to CodegenFactory's session API lets you create and manage Droid sessions—including lifecycle, settings, and messages—implying some ability to track a running agent's state, and the App/CLI let you review diffs as work progresses. However, there is no explicit documentation of a live status dashboard, streaming logs, or real-time progress view of an in-flight agent. Missing for 10: dedicated real-time monitoring UI/stream, explicit 'live status' feature documentation, independent confirmation of live tracking.
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
Codegen's CLI lets developers view agents and run them with 'full telemetry and monitoring,' and agents can 'report progress' via Slack, giving some real-time visibility into agent activity and status. However, there's no explicit documentation of a live dashboard/status stream showing step-by-step actions as they happen, and the closest 'trace' feature is framed as post-hoc analysis/debugging rather than live watching. Missing for 10: a documented real-time UI/dashboard showing live agent status, explicit streaming of in-progress actions, and independent/hands-on confirmation of live monitoring accuracy.
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
developerGet notified when an agent completes a task or needs my input
weight 2 · round to CodegenFactory integrates with Slack and PagerDuty and supports tiered autonomy that pauses for manual confirmation, implying some mechanism for alerting developers when input is needed, but there's no explicit documentation of a notification feature for task completion or input requests. missing for 10: explicit notification/alert documentation, evidence of completion pings, confirmation of Slack/PagerDuty being used specifically for task-status alerts.
- [claimed-docs] “Connect Jira, Notion, Slack, Linear, PagerDuty, and MCP tools to keep development synchronized with team systems”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
Codegen explicitly supports sending notifications, asking for clarification, and reporting progress directly in Slack channels, plus updating statuses/comments in Jira, Linear, Clickup, and Monday.com, covering both completion and input-needed scenarios. missing for 10: independent/hands-on confirmation of real-time notification delivery, and no explicit mention of email/other channel notification options.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
Intent to spec — stories about intent to spec in this arenaIntent to spec
Stories about intent to spec in this arena
Natural language task intake
developerDescribe a feature or bug in plain language and have it automatically turned into a scoped implementation task
weight 3 · round to CodegenFactory supports delegating tasks described in plain language (via droid CLI, App, or Missions) which get turned into scoped, executable work with diffs to review and merge, and readiness tooling helps scope repo work automatically. However, there's no explicit documented workflow for turning a raw bug/feature description into a formal 'scoped implementation task' spec artifact (e.g., structured ticket/spec generation before execution) distinct from just running the agent. missing for 10: explicit intent-to-spec artifact generation/preview step, independent/hands-on evidence of accurate scoping from vague input, and detail on how ambiguous requests are clarified before execution.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “delegate scoped tasks to Custom Droids via /droids, and package reusable procedures as Skills with /skills or /create-skill”
- [claimed-docs] “A designer shares a mockup, and the system implements it without handoffs.”
- [claimed-docs] “Run the /readiness-report slash command in a Factory App web or desktop session, or in the Droid CLI, to evaluate a repository's readiness l…”
- [claimed-docs] “Use Factory Missions to plan and execute large, multi-feature projects with structured orchestration.”
Codegen's docs describe taking natural-language prompts and turning them into scoped agent tasks: analyzing requirements, implementing features, fixing bugs, writing tests, and running agents from prompts via SDK/CLI/Slack/issue trackers. This directly matches the intent-to-spec story of describing a feature/bug and having it become an actionable implementation task. Missing for 10: no independent/hands-on evidence showing the quality or accuracy of the generated scoped task (e.g. how well it decomposes ambiguous requests), and no detail on how it handles clarification/scoping edge cases beyond a general 'ask for clarification' mention.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
product-managerConvert user feedback submissions into structured tasks with proposed scope
weight 2 · round to CodegenFactorynone0/10Factory documents task delegation, Missions planning, and integrations with Jira/Linear/Notion, but nothing in the evidence describes ingesting user feedback submissions and outputting a structured task with proposed scope — this specific PM workflow is not documented.
Codegen can analyze requirements and create/link tasks in Jira, Linear, Clickup, Monday.com, which could theoretically be used to turn feedback into tasks, but there is no evidence of a feedback-intake feature or automatic scope proposal tailored for PM workflows. Missing for 10: dedicated feedback ingestion mechanism, structured scope/estimate output, PM-specific workflow examples or case studies.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
developerAttach a marked-up screenshot or mockup to a task so the agent implements the correct visual change
weight 2 · round drawnFactorynone0/10Evidence only mentions a designer sharing a mockup conceptually (factory-docs-8), with no documented capability to attach an image/screenshot with markup annotations to a task or session, nor any multimodal image-input feature described.
- [claimed-docs] “A designer shares a mockup, and the system implements it without handoffs.”
Codegennone0/10No evidence in the pack mentions attaching screenshots, images, or marked-up mockups to a task/prompt; Codegen's docs cover text prompts, Slack/Jira/Linear integrations, CLI/SDK, and PR review but nothing about visual/image input for implementing UI changes. Figma is mentioned only as a connectable integration, not as an image-annotation workflow.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
Plan approval
developerReview and approve an agent's implementation plan before any code changes are made
weight 3 · round to FactoryFactory's tiered autonomy in droid exec implies some operations require manual confirmation before running, and Missions offer 'plan and execute' orchestration, suggesting a planning phase exists, but there is no explicit documentation of a discrete implementation-plan artifact that a developer reviews and approves before any code changes are made — the described workflow (docs-1) instead centers on reviewing the diff/output after changes. missing for 10: explicit plan-approval UI/step description, evidence of a pre-execution plan artifact, confirmation that no code is touched until plan is approved.
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
- [claimed-docs] “Use Factory Missions to plan and execute large, multi-feature projects with structured orchestration.”
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
Codegennone0/10No evidence in the pack of a plan-preview or approval step prior to code changes; agents appear to run prompts directly to implementation (create branches, commit code, open PRs) without a documented plan-review gate. missing for 10: any mention of a plan/preview mode, approval gate, or dry-run step before code execution.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
engineering-leadApprove a task's scope and contract before an agent is allowed to modify the repository
weight 2 · round to FactoryFactory documents tiered autonomy that limits what operations run without manual confirmation and diff review before merge, which implies some human-gate mechanics, but there's no explicit evidence of an engineering-lead approving a task's defined scope/contract *before* the agent is permitted to touch the repository — reviews are framed as post-hoc diff review rather than pre-execution scope sign-off. Missing for 10: explicit scope/contract definition step, an approval gate that blocks agent execution until lead sign-off, and evidence this is lead-specific rather than generic autonomy tiering.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
Ticket driven tasking
developerAssign a coding task to an agent directly from an existing issue or ticket
weight 3 · round to CodegenFactory documents integrations with Jira and Linear ('Connect Jira, Notion, Slack, Linear, PagerDuty, and MCP tools to keep development synchronized with team systems'), implying tickets can feed into agent workflows, but there is no explicit documentation of assigning/triggering a droid task directly from an issue or ticket UI. missing for 10: explicit workflow/screenshot showing task creation from a Jira/Linear ticket, confirmation that ticket assignment auto-triggers an agent session, independent corroboration.
- [claimed-docs] “Connect Jira, Notion, Slack, Linear, PagerDuty, and MCP tools to keep development synchronized with team systems”
Docs show native integrations with Jira, Linear, ClickUp and Monday.com allowing the agent to update issue statuses, add comments, link PRs and create tasks, and specifically state connecting a Linear workspace 'enables agent interactions with your issues,' which directly supports assigning work from a ticket. Missing for 10: a concrete end-to-end example of assigning a task from an issue/ticket, and independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round drawnFactory documents a real API (session creation/lifecycle) alongside its CLI and UI, showing some functional overlap, but there's no OpenAPI spec (404s on all candidate URLs) and no evidence that UI-exclusive features like Missions, Software Factory's coverage map, or readiness-report/fix slash commands are exposed via the API. missing for 10: documented API parity for Missions, Software Factory, and readiness commands; a public OpenAPI/schema confirming full surface coverage.
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
- [claimed-docs] “Use Factory Missions to plan and execute large, multi-feature projects with structured orchestration.”
- [claimed-docs] “Open Software Factory to see your delivery lifecycle as an automation coverage map.”
- [claimed-docs] “Run the /readiness-report slash command in a Factory App web or desktop session, or in the Droid CLI, to evaluate a repository's readiness l…”
- [claimed-docs] “Automatically fix failing criteria with the /readiness-fix slash command in a Factory App web or desktop session, or in the Droid CLI”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.factory.ai/openapi.json, https://docs.factory.ai/swagger.json, https://docs.factory.ai/…”
Codegen offers a Python SDK, CLI, and API to create/run agents, pull work, and monitor executions, showing solid programmatic control (codegen-docs-11,15,16,17,18). However, there's no evidence of a complete OpenAPI/API surface matching every UI capability (e.g., Slack/Jira/Linear integration management, PR review workflows) — the OpenAPI probe returned 404s, suggesting no public full API spec. missing for 10: OpenAPI reference confirming full UI parity, evidence that integration configs and review workflows are API-accessible, independent confirmation of API completeness.
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.codegen.com/openapi.json, https://docs.codegen.com/swagger.json, https://docs.codegen.c…”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnFactorynone0/10No evidence addresses data export or portability in open formats, nor any account-deletion/data-takeout mechanism; the docs focus on session management, CLI, and integrations, not exporting user data to leave the platform.
Codegennone0/10No evidence of a bulk data-export feature, open-format export of agent histories/traces, or account-portability tooling; only partial building blocks exist (pulling agent branches via CLI, retrieving traces via API) but nothing documenting a full data export/exit path.
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
ai-native userRead the product's source under an open license
weight 2 · round drawnFactorynone0/10No evidence of an open-source license or publicly available source code for Factory/Droid; all evidence points to closed docs and a proprietary CLI/platform. missing for 10: any license file, GitHub repo, or open-source statement covering the product's source code.
ai-native userSelf-host the core product
weight 3 · round drawnFactorynone0/10Factory is presented as a cloud-hosted platform (Factory App, Droid CLI connecting to hosted services, API sessions) with no evidence of a self-hostable core server or on-prem deployment option anywhere in the docs or probes.
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
Enterprise licensing
engineering-leadLicense an enterprise deployment with SSO and commercial support for organization-wide rollout
weight 2 · round drawnFactorynone0/10No evidence pack items mention enterprise licensing, SSO, or commercial support offerings; documentation only covers product features like Droid CLI, MCP integration, and readiness reports. Missing for 10: any mention of SSO/SAML, enterprise tier, commercial support SLAs, or org-wide licensing terms.
Codegennone0/10No evidence in the pack mentions SSO, enterprise licensing, commercial support tiers, or org-wide rollout plans; only feature-level docs and API/CLI references are provided. Missing for 10: SSO/SAML support, enterprise plan/pricing tier documentation, commercial support SLA, org-wide admin/rollout tooling.
Model flexibility
engineering-leadBring my own LLM or API key so agents run on the model of my choice
weight 2 · round drawnFactorynone0/10No evidence pack item mentions bringing your own LLM, custom API keys, or model selection/configuration options; all docs focus on CLI, integrations, and workflow features.
Usage quotas
engineering-leadSee and manage plan-based daily task and concurrency limits for agent workflows
weight 2 · round drawnFactorynone0/10No evidence of plan-based daily task limits, concurrency caps, or admin controls for managing such limits; docs cover CLI, sessions, missions, and integrations but nothing on usage/plan limit visibility or management.
Codegennone0/10No evidence pack item mentions plan-based daily task limits, concurrency limits, or any admin dashboard/controls for engineering leads to monitor or manage usage quotas; documentation covers agent capabilities, integrations, CLI, and SDK but nothing on pricing tiers or limit management.
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnFactorynone0/10No evidence in the pack addresses data residency, regional storage options, or compliance/data-location controls; all evidence covers agent workflows, CLI, and integrations unrelated to data storage location.
ai-native userPrevent my data from being used to train AI models
weight 3 · round drawnFactorynone0/10No evidence in the pack addresses data usage/training opt-out, privacy policy, or data retention controls; the docs focus entirely on product features like CLI, missions, and integrations.
ai-native userControl data retention and deletion
weight 2 · round drawnFactorynone0/10No evidence pack items address data retention policies, deletion controls, or privacy settings for user data; all citations relate to product features like CLI, MCP integrations, and agent workflows rather than privacy posture.
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnFactorynone0/10No evidence in the pack mentions telemetry, usage tracking, data collection, or any opt-out/privacy settings for Factory or the Droid CLI.
Codegennone0/10No evidence pack item mentions a telemetry opt-out setting or privacy controls; in fact docs-12 explicitly notes the CLI runs Claude Code 'with full telemetry and monitoring,' with no mention of a way to disable it.
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
Repo integration — stories about repo integration in this arenaRepo integration
Stories about repo integration in this arena
Chat integration
developerTag an agent in a chat thread to discuss and delegate a bug or task
weight 2 · round to CodegenFactory's docs mention connecting Slack as an integration to 'keep development synchronized with team systems,' implying some chat-based interaction, but there is no explicit evidence of an @-mention/tagging mechanism in a chat thread to discuss or delegate a specific bug/task to the agent. Missing for 10: explicit Slack @droid tagging workflow, thread-based task delegation UI, and confirmation that discussion happens inline in chat rather than just triggering external actions.
- [claimed-docs] “Connect Jira, Notion, Slack, Linear, PagerDuty, and MCP tools to keep development synchronized with team systems”
Codegen supports Slack-based chat interactions where users can message/tag the agent to report progress, ask for clarification, and delegate tasks (codegen-docs-2, codegen-docs-9), and Linear/Jira integration allows commenting and task creation to delegate bugs (codegen-docs-3, codegen-docs-10). However, evidence doesn't explicitly confirm an '@mention' tagging mechanism within an existing chat thread to delegate a specific bug — it's inferred from general Slack/Linear interaction docs rather than a documented tagging workflow. Missing for 10: explicit documentation of @-mention/tagging syntax in threads, confirmation of thread-level context retention when delegating, and independent/hands-on verification of this specific workflow.
- [claimed-docs] “Send notifications, ask for clarification, report progress, and interact directly with your team in Slack channels.”
- [claimed-docs] “Add the Codegen Slack App to communicate with the agent directly in your workspace.”
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
Knowledge context
developerAdd a context file describing my codebase conventions so agents generate more relevant plans and code
weight 3 · round drawnFactorynone0/10The evidence pack does not mention any context file mechanism (e.g., AGENTS.md, .factory config, or similar) for describing codebase conventions to guide agent behavior; it covers CLI usage, integrations, readiness reports, and missions but nothing about persistent repo-convention context files.
Codegennone0/10The evidence pack covers Codegen's integrations, CLI, SDK, and PR/issue automation, but nowhere mentions a repo-level context or rules file (e.g., conventions file) that developers can add to steer agent plans/code generation. missing for 10: any mention of a context/config file for codebase conventions, documentation on custom instructions, or repo-level rules ingestion.
developerQuery generated documentation for any public or private repository
weight 1 · round drawnFactorynone0/10The evidence pack covers Factory's CLI, agent-readiness reports, missions, and integrations, but nothing describes generating or querying documentation for a repository's codebase (e.g., an auto-generated repo wiki/docs feature). This is a plausible capability for a repo-integrated dev tool, so the axis applies, but no supporting evidence exists.
Codegennone0/10Evidence shows Codegen can 'improve documentation' as part of coding tasks, but there is no evidence of a feature to generate and then query documentation for a repository (public or private) in a Q&A style. No docs, API, or CLI reference describe a documentation-query capability.
Project management integration
product-managerConnect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly
weight 3 · round to CodegenDocs confirm Jira and Linear integration explicitly (factory-docs-4), but ClickUp and Monday.com are not mentioned anywhere in the evidence, and there's no detail on ticket management workflows (creating/updating tickets) beyond 'connecting' systems to sync development. Missing for 10: ClickUp integration, Monday.com integration, concrete ticket-management/CRUD workflows via these connectors, independent corroboration.
- [claimed-docs] “Connect Jira, Notion, Slack, Linear, PagerDuty, and MCP tools to keep development synchronized with team systems”
Docs explicitly claim support for Jira, Linear, ClickUp, and Monday.com (update statuses, add comments, link PRs, create tasks), but detailed setup instructions are only shown for Linear (codegen-docs-10); Jira, ClickUp, and Monday.com lack dedicated connection/integration docs equivalent to Linear or Slack. Missing for 10: explicit setup guides or independent verification for Jira/ClickUp/Monday.com, and hands-on confirmation beyond first-party docs.
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
Version control integration
developerConnect a GitHub repository so an agent can access the code and open pull requests against it
weight 3 · round to CodegenDocs indicate Factory works within Git workflows (delegate tasks, review diffs, merge from App/terminal) and can run in CI/CD via droid exec, implying repo access and PR-opening capability, but there is no explicit documentation describing connecting/authorizing a GitHub repository or an explicit PR-creation feature. Missing for 10: explicit GitHub repo connection/auth flow documentation, explicit 'open pull request' feature description, and independent/hands-on confirmation of PR creation.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
Codegen provides a GitHub App install for repo access with no complex setup, and docs describe the agent creating branches, committing code, reviewing/opening PRs, and managing repositories. missing for 10: independent/hands-on confirmation beyond vendor docs, and explicit detail on multi-repo/org-level permission scoping.
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
developerGrant an agent access to my repositories with a one-click install, without complex setup
weight 2 · round to CodegenFactorynone0/10Evidence shows Factory integrates with Git workflows, CLI, and external tools like Jira/Slack/MCP, but there is no mention of a one-click GitHub/repo install or simplified OAuth-based repo authorization flow. Setup appears to require CLI installation and configuration rather than a one-click grant.
Docs explicitly state installing the GitHub App grants repo access with 'no complex setup required', directly matching the story, and this is corroborated by broader repo-integration capabilities (PR review, branch/commit management). Missing for 10: independent/hands-on confirmation of the one-click flow beyond first-party docs.
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
Review quality gates — quality gates on changes — review flow, required checks, merge protectionReview quality gates
Quality gates on changes — review flow, required checks, merge protection
Ci remediation
engineering-leadHave failed CI workflows automatically diagnosed and fixed with a proposed pull request
weight 3 · round to CodegenFactory's droid exec is explicitly designed to run in CI/CD pipelines as a one-shot task with tiered autonomy, and Droid CLI integrates with Git workflows, suggesting the building blocks exist for automating CI fixes. However, there is no direct evidence of a dedicated feature that detects a failed CI workflow, diagnoses the failure, and automatically opens a proposed pull request end-to-end. missing for 10: explicit CI-failure detection/trigger integration, automatic diagnosis-to-PR workflow documentation, and any hands-on/independent proof of this specific use case.
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
Docs explicitly mention triggering agents from CI/CD pipelines when builds fail and agents that fix bugs/write code and open PRs, directly supporting automated diagnosis-and-fix-PR workflows for failed CI. However, there is no dedicated end-to-end CI-failure-diagnosis feature description, no example of the proposed PR format for a failing build, and no independent/hands-on confirmation of this specific workflow. missing for 10: dedicated CI-failure-diagnosis workflow documentation, concrete example of a CI-triggered fix PR, independent verification of this specific capability.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
developerTrigger an agent from CI/CD pipelines to fix a broken build or failing test
weight 2 · round drawnFactory explicitly documents `droid exec` as a one-shot CLI command designed for CI/CD pipelines, shell scripts, and batch processing, with tiered autonomy controls for unattended operation — directly enabling triggering an agent from CI to fix builds/tests. Missing for 10: a concrete worked example of a CI pipeline invoking droid exec on a failing test/build, and independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
- [probe] “official CLI documented at https://docs.factory.ai/droid-cli/quickstart”
Docs explicitly state agents can be 'Triggered from CI/CD pipelines when builds fail' and support fixing bugs/tests via API/SDK with programmatic control, which directly matches the story. Missing for 10: independent/hands-on confirmation of a real CI/CD integration example (e.g., GitHub Actions snippet) and details on how failing test context is passed to the agent.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
Diff review
developerConfigure an agent to automatically open a pull request when its task completes
weight 2 · round to CodegenDocs show git workflow integration (droid-cli git workflow, delegate task then review diff and merge, droid exec for CI/CD pipelines) which implies PR-related automation, but there is no explicit documentation of a feature to automatically open a pull request when a task completes. Missing for 10: explicit config/flag for auto-PR creation, first-party example of a droid opening a PR on completion, independent/hands-on confirmation.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
Codegen's agent creates branches, commits code, and manages repositories via GitHub App integration, and can be run programmatically via SDK/CLI/API to complete tasks; PR creation is a core documented capability of the agent workflow. Missing for 10: explicit first-party documentation/example showing a config flag or setting to 'auto-open PR on task completion' and independent/hands-on confirmation of this exact behavior.
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
developerReview a diff of an agent's changes and approve it before it becomes a pull request
weight 3 · round to FactoryDocs explicitly describe the delegate-review-merge workflow ('Delegate a task, review the diff, and merge from the App or your terminal') and tiered autonomy controls that gate what runs without confirmation, supporting a review-before-PR gate. However, there is no detailed documentation of the diff review UI itself, approval mechanics, or independent/hands-on confirmation of this exact flow. Missing for 10: dedicated diff-review UI documentation, explicit approval-step mechanics, and independent verification of the review-before-PR gate.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
Codegen agents create branches, commit code, and open PRs, and the agent itself can review PRs and post comments, but the evidence pack does not show a dedicated human-in-the-loop diff review/approval gate before a PR is created — the CLI lets you 'pull agent work' locally which could support manual review, but no explicit approval workflow is documented. missing for 10: explicit pre-PR diff approval UI/workflow, evidence of a human approve/reject gate, independent confirmation that PR creation is blocked pending review.
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Pull agent work - Download branches and code changes created by agents directly to your local environment”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
Pr review automation
ai-native userHave incoming issues automatically triaged with severity suggested and routed to the right owner
weight 2 · round drawnFactorynone0/10Factory documents integrations with issue trackers like Jira, Linear, and PagerDuty (factory-docs-4) and generic custom droid/skill automation (factory-docs-5), but there is no evidence of an automated triage workflow that suggests severity or routes issues to an owner. missing for 10: any documented triage feature, severity classification logic, or owner-routing capability.
- [claimed-docs] “Connect Jira, Notion, Slack, Linear, PagerDuty, and MCP tools to keep development synchronized with team systems”
- [claimed-docs] “delegate scoped tasks to Custom Droids via /droids, and package reusable procedures as Skills with /skills or /create-skill”
Codegennone0/10Evidence shows Codegen integrates with issue trackers (Jira, Linear, etc.) to update statuses, add comments, and create tasks, but there is no mention of automatic severity assessment or routing incoming issues to the correct owner. Missing for 10: automatic severity classification, owner/routing logic, any triage workflow evidence.
- [claimed-docs] “Update statuses, add comments, link PRs to issues, and create new tasks. Support for Jira, Linear, Clickup and Monday.com.”
- [claimed-docs] “Connect your Linear workspace to enable agent interactions with your issues.”
engineering-leadHave every pull request automatically reviewed with AI-generated inline comments
weight 3 · round to CodegenFactorynone0/10Evidence shows Factory's droid CLI/exec can review diffs, be triggered in CI/CD, and connect to Git workflows, but nothing in the pack describes automatic PR review with AI-generated inline comments posted to pull requests. Missing for 10: any documentation of automated PR-triggered review, inline comment generation on PRs, or GitHub/GitLab PR integration specifics.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
Docs explicitly describe automated PR review functionality: reading the diff, summarizing changes, running security/secrets scans, checking test coverage, and posting a review comment, plus general 'Review PRs, suggest changes, comment on issues' capability tied to GitHub App integration for repo access. This matches the engineering-lead story of automatic AI-reviewed PRs with comments. Missing for 10: explicit confirmation that comments are inline (per-line) rather than a single summary comment, and independent/hands-on evidence corroborating this in production use.
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
Readiness checks
engineering-leadAutomatically fix failing agent-readiness criteria in my repository
weight 1 · round to FactoryFactory explicitly documents a /readiness-fix slash command that automatically fixes failing agent-readiness criteria, usable from the Factory App or Droid CLI, complementing the /readiness-report diagnostic command. This directly matches the story's ask, though only first-party docs support it with no independent/hands-on corroboration. Missing for 10: independent or hands-on verification that /readiness-fix reliably resolves criteria, and detail on which criteria types it can/cannot auto-fix.
- [claimed-docs] “Run the /readiness-report slash command in a Factory App web or desktop session, or in the Droid CLI, to evaluate a repository's readiness l…”
- [claimed-docs] “Automatically fix failing criteria with the /readiness-fix slash command in a Factory App web or desktop session, or in the Droid CLI”
Codegen agents can fix bugs, write tests, and run CI/CD-triggered fixes (codegen-docs-1, codegen-docs-18, codegen-docs-20), which could be applied to remediate failing agent-readiness checks, but there is no explicit documentation of an 'agent-readiness criteria' concept, scoring rubric, or automated fix workflow tied specifically to such criteria. missing for 10: explicit agent-readiness criteria/checklist, automated detection-to-fix pipeline for those criteria, evidence of hands-on success fixing readiness gates.
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
engineering-leadRun a readiness report that evaluates how ready my repository is for autonomous agents
weight 2 · round to FactoryFactory explicitly documents a `/readiness-report` slash command (in App or Droid CLI) that evaluates a repository's agent-readiness level, plus a companion `/readiness-fix` command to auto-remediate failing criteria, directly matching the engineering-lead's story. Missing for 10: independent/hands-on validation of report contents and no sample report output shown.
- [claimed-docs] “Run the /readiness-report slash command in a Factory App web or desktop session, or in the Droid CLI, to evaluate a repository's readiness l…”
- [claimed-docs] “Automatically fix failing criteria with the /readiness-fix slash command in a Factory App web or desktop session, or in the Droid CLI”
- [claimed-docs] “A designer shares a mockup, and the system implements it without handoffs.”
Codegennone0/10The evidence pack shows PR review, security scanning, and CI/CD triggers, but there is no mention of a dedicated 'readiness report' feature that evaluates overall repository readiness for autonomous agents. Missing for 10: any documentation of a repo-wide readiness/scorecard report, criteria for agent-readiness, or a command/API that produces such an assessment.
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
Security remediation
engineering-leadHave security alerts automatically validated and remediated with an opened pull request
weight 2 · round to CodegenFactorynone0/10Evidence covers task delegation, CLI automation, readiness reports/fixes, and CI/CD execution, but nothing describes ingesting security alerts (e.g., from a scanner like Dependabot/Snyk) and automatically validating and remediating them via an opened PR. No security-alert-specific workflow or integration is documented.
Codegen documents a security/secrets scan that posts a review comment on failure (codegen-docs-20) and general bug-fixing/PR-creation capabilities plus CI/CD-triggered agents (codegen-docs-1, codegen-docs-4, codegen-docs-18), suggesting it could act on a flagged issue and open a fix PR. However, there is no explicit evidence of ingesting external security alerts (e.g., Dependabot/CodeQL/Snyk), validating them as true positives, or automatically opening a remediation PR end-to-end — the documented scan flow stops at commenting, not remediating. missing for 10: explicit security-alert ingestion/validation workflow, evidence of an opened remediation PR tied to a specific alert, independent confirmation of this flow working in practice.
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Analyze requirements, implement features, fix bugs, write tests, and improve documentation based on your prompts.”
Scale parallelism — running many jobs at once — concurrency, fleets, queueingScale parallelism
Running many jobs at once — concurrency, fleets, queueing
Concurrent execution
engineering-leadRun many agent tasks concurrently to scale delivery throughput
weight 3 · round to FactoryFactory provides several building blocks for scaling agent work: droid exec is designed for one-shot, CI/CD, and batch processing use cases, the Sessions API lets you create and manage multiple Droid sessions programmatically, and Missions support orchestrating large multi-feature projects with structured coordination. Together these imply the ability to run many concurrent tasks, but no evidence explicitly states a documented concurrency limit, dashboard for tracking many simultaneous droids, or independent case study proving throughput scaling. Missing for 10: explicit concurrency/parallelism guarantees or limits, a multi-task monitoring UI description, and independent/hands-on validation of running many tasks simultaneously.
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Use Factory Missions to plan and execute large, multi-feature projects with structured orchestration.”
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
- [claimed-docs] “Open Software Factory to see your delivery lifecycle as an automation coverage map.”
Codegen's SDK/API let you programmatically create and run multiple agents (agent.run, API 'full programmatic control over execution and monitoring') and CLI can view multiple running agents, implying support for parallel task execution, and CI/CD triggers suggest scaling automation into pipelines. However, there is no explicit documentation of concurrency limits, throughput guarantees, or dashboards for managing many simultaneous tasks at scale. Missing for 10: explicit concurrency/parallelism claims, scaling limits or throughput benchmarks, and independent evidence of running many agents simultaneously in practice.
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
engineering-leadCreate agent sessions on behalf of other users in my organization
weight 2 · round to CodegenFactorynone0/10The API reference (factory-docs-13) shows session creation/management exists, but nothing in the evidence indicates an org-admin or lead can create/manage sessions on behalf of other specific users within an organization. missing for 10: evidence of org-level user impersonation, admin controls for delegating sessions to teammates, or role-based session ownership across users.
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
Codegen's org-level SDK/CLI/API (org_id+token, 'manage organizations', programmatic agent creation) implies an engineering lead with org credentials could spin up agents at scale, but no docs explicitly describe creating or assigning a session to another named user in the org. Missing for 10: explicit user-impersonation or 'create on behalf of' API/UI feature, role-based permissions for delegating agent sessions, and any admin console for assigning sessions to teammates.
- [claimed-docs] “Manage organizations - Switch between organizations and configure repositories”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
Deployment flexibility
developerUse a managed cloud offering to run agents without operating my own backend infrastructure
weight 2 · round drawnFactory's docs describe a hosted 'Factory App' (web/desktop) where users delegate tasks and review diffs without local setup, plus an API for creating and managing Droid sessions server-side, indicating a managed backend rather than self-hosted infra. Missing for 10: independent/hands-on confirmation of the cloud service's scalability and infrastructure details beyond first-party docs.
- [claimed-docs] “Delegate a task, review the diff, and merge from the App or your terminal.”
- [claimed-docs] “Run the /readiness-report slash command in a Factory App web or desktop session, or in the Droid CLI, to evaluate a repository's readiness l…”
- [claimed-docs] “Automatically fix failing criteria with the /readiness-fix slash command in a Factory App web or desktop session, or in the Droid CLI”
- [claimed-docs] “Create and drive Droid sessions: manage their lifecycle, settings, and messages.”
Codegen offers cloud-run agents with isolated execution environments, API/SDK/CLI access, and no need to self-host infrastructure (codegen-docs-5, codegen-docs-11, codegen-docs-15, codegen-docs-16). This directly matches running agents without operating own backend. Missing for 10: independent/hands-on verification of the managed infrastructure's reliability at scale, and explicit pricing/SLA details for the managed cloud offering.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
- [claimed-docs] “programmatically retrieve detailed agent traces for analysis and debugging”
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
engineering-leadSelf-host agent infrastructure locally, in containers, or on my own VMs
weight 2 · round drawnFactorynone0/10No evidence describes self-hosting Factory's agent infrastructure locally, in containers, or on customer-owned VMs; all evidence points to Factory's own cloud-hosted App, CLI, and API rather than a deployable/self-hosted backend.
Codegennone0/10Codegen is presented entirely as a cloud/SaaS agent service (GitHub App, Slack App, API, SDK, isolated cloud environments) with no evidence of a self-hostable runtime, container image, or on-prem/VM deployment option; the CLI logs local Claude Code sessions to Codegen's own cloud rather than hosting infrastructure locally.
- [claimed-docs] “Safely run code, install dependencies, and test changes in robust isolated environments”
- [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”
- [claimed-docs] “Install the GitHub App to grant the agent access to your repositories. No complex setup required.”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [claimed-docs] “Create and manage AI agents that can write code, fix bugs, and handle development tasks across your repositories with full programmatic cont…”
Headless automation
developerRun an agent headlessly inside CI/CD pipelines and shell scripts
weight 2 · round to FactoryFactory explicitly documents droid exec as a one-shot headless command designed for CI/CD pipelines, shell scripts, and batch processing, with tiered autonomy controls for unattended operation. Missing for 10: independent/hands-on confirmation of real-world CI pipeline usage beyond first-party docs.
- [claimed-docs] “droid exec runs as a one-shot command that completes a task and exits, making it ideal for CI/CD pipelines, shell scripts, and batch process…”
- [claimed-docs] “Droid exec uses tiered autonomy to control what operations can run without manual confirmation.”
- [claimed-docs] “Use the Droid CLI to run Factory from your terminal, editor, tests, and Git workflow.”
Codegen explicitly documents triggering agents from CI/CD pipelines (e.g., on build failures) and offers a Python SDK plus CLI for programmatic/scripted invocation, which supports headless usage in pipelines and scripts. However, there's no dedicated CI/CD integration guide, no example pipeline configs (e.g., GitHub Actions YAML), and no independent/hands-on evidence confirming real-world headless CI usage. missing for 10: concrete CI/CD pipeline examples or config snippets, exit-code/status handling for scripts, independent verification of headless operation in automated pipelines.
- [claimed-docs] “Trigger agents from CI/CD pipelines when builds fail”
- [claimed-docs] “Programmatically interact with agents using the Python SDK for advanced automation.”
- [claimed-docs] “agent = Agent(org_id="...", token="...") # Run an agent with a prompt task = agent.run(prompt="Leave a review on PR #123")”
- [claimed-docs] “Use it to view agents, pull their work, create new agents, and run Claude Code with full telemetry and monitoring.”
- [probe] “official CLI documented at https://docs.codegen.com/introduction/cli”
Not comparable on these axes
ai-native userConnect an agent via an official MCP server
weight 3 · not comparableFactory is primarily a coding agent (Droid), and factory-docs-4 shows it can connect OUT to MCP tools (client-side), which wouldn't count under the na rule — but factory-probe-3 explicitly indicates an official MCP server is documented at docs.factory.ai/harness/mcp, flipping this to the applicable server-side capability. However, the evidence pack only confirms the page's existence via a probe hit with no detail on what the server exposes or how other agents connect to it. Missing for 10: detailed documentation of the MCP server's tool surface, setup instructions for external agents, and independent confirmation of a working connection.
- [probe] “official MCP server documented at https://docs.factory.ai/harness/mcp”
- [claimed-docs] “Connect Jira, Notion, Slack, Linear, PagerDuty, and MCP tools to keep development synchronized with team systems”
Codegenn/aCodegen is itself an AI coding agent, so serving as an MCP server for other agents to connect to is a different product role (client vs server) per the category rule. Evidence only shows Codegen acting as an MCP client (connecting to Slack/Linear/Figma/custom MCP tools) or provisioning MCP servers for its own org's Claude Code instances, not exposing itself as an official MCP server endpoint for external agents to connect to.
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”
- [claimed-docs] “Log local Claude Code instances to the cloud and provision MCP servers across your org.”
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · not comparableFactoryn/aFactory is an agentic coding platform focused on delegating software development tasks (code diffs, PRs, readiness reports), not a data analytics or BI product that surfaces AI-generated insights/suggestions from a user's own data. This story is a category error for this product type.
Codegen generates AI-driven insights specifically around code artifacts—summarizing PR diffs, running security/secrets scans, checking test coverage, and suggesting changes/comments on PRs and issues—which counts as data-driven insight generation within its coding-agent context. However, these insights are narrowly scoped to code/repo/PR data rather than broader 'your data' (e.g., databases, business metrics) despite a mention of database connectivity via MCP with no insight-generation detail there. Missing for 10: evidence of insights/suggestions generated from non-code data sources (databases, analytics), and no independent/hands-on corroboration of insight quality.
- [claimed-docs] “Reads the PR diff Summarizes changes across files Runs security and secrets scan Checks test coverage Posts a review comment on failure”
- [claimed-docs] “Review PRs, suggest changes, comment on issues, create branches, commit code, and manage repositories.”
- [claimed-docs] “Connect with Slack, Linear, Figma, databases, and extend capabilities with custom MCP tools.”