🏭Software Factory arenaSoftware Factory

Agent access

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🤖Agenticness — how well agents can access and operate the productPoint an agent at llms.txt or agent-oriented docs	ai-native	—none 0/10	✓fullT 8/10	✓fullT 7/10	✓fullT 9/10	✓fullT 9/10	—none 0/10	✓fullT 8/10	—none 0/10	—none 0/10
🤖Agenticness — how well agents can access and operate the productRun the product headlessly / in CI for automation	ai-native	~partialT 6/10	✓fullT 8/10	✓fullC 8/10	✓fullT 8/10	✓fullT 8/10	✓fullX 7/10	~partialT 6/10	✓fullT 7/10	✓fullT 8/10
🤖Agenticness — how well agents can access and operate the productPlug MCP servers into this product so it can use their tools	ai-native	—none 0/10	~partialC 6/10	✓fullT 9/10	—none 0/10	~partialC 6/10	—none 0/10	✓fullC 7/10	—none 0/10	—none 0/10
🤖Agenticness — how well agents can access and operate the productConnect an agent via an official MCP server	ai-native	✓fullC 8/10	~partialT 5/10	✓fullC 8/10	n/a	n/a	n/a	—none 0/10	—none 0/10	—none 0/10
🤖Agenticness — how well agents can access and operate the productUse an official CLI	ai-native	✓fullT 8/10	✓fullT 9/10	✓fullT 8/10	✓fullT 8/10	✓fullT 8/10	—none 0/10	✓fullT 7/10	✓fullT 8/10	✓fullT 7/10
🤖Agenticness — how well agents can access and operate the productDrive the product through a documented public API	ai-native	✓fullT 8/10	~partialT 6/10	✓fullT 8/10	✓fullT 8/10	✓fullT 8/10	✓fullT 8/10	✓fullT 8/10	~partialT 5/10	~partialT 4/10
🤖Agenticness — how well agents can access and operate the productIssue scoped/least-privilege API credentials for an agent	ai-native	~partialC 3/10	—none 0/10	~partialC 3/10	—none 0/10	—none 0/10	—none 0/10	~partialC 5/10	n/a	~partialC 3/10
🤖Agenticness — how well agents can access and operate the productBuild against official SDKs	ai-native	~partialT 6/10	~partialT 4/10	✓fullT 8/10	✓fullT 8/10	✓fullT 8/10	✓fullT 7/10	~partialT 6/10	—none 0/10	—none 0/10
🤖Agenticness — how well agents can access and operate the productSubscribe to events via webhooks	ai-native	—none 0/10	—none 0/10	—none 0/10	~partialC 5/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10

Agentic features

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🤖Agenticness — how well agents can access and operate the productGet AI-generated insights and suggestions from my data inside the product	ai-native	~partialC 5/10	n/a	~partialC 6/10	✓fullC 7/10	~partialC 5/10	~partialX 6/10	—none 0/10	—none 0/10	~partialC 4/10
🤖Agenticness — how well agents can access and operate the productSet up automations that run autonomously in the background	ai-native	~partialC 5/10	~partialC 6/10	✓fullC 8/10	✓fullC 8/10	✓fullC 7/10	~partialX 6/10	~partialC 6/10	~partialC 7/10	✓fullC 7/10
🤖Agenticness — how well agents can access and operate the productDelegate tasks to a built-in AI assistant inside the product	ai-native	—none 0/10	✓fullC 8/10	✓fullX 8/10	✓fullT 8/10	✓fullC 8/10	✓fullX 8/10	✓fullX 8/10	~partialC 4/10	✓fullC 7/10
🤖Agenticness — how well agents can access and operate the productOperate the product with natural-language commands	ai-native	~partialC 5/10	✓fullT 8/10	✓fullX 8/10	✓fullT 8/10	✓fullC 8/10	✓fullX 8/10	✓fullC 7/10	—none 0/10	✓fullC 7/10

Api quality

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🤖Agenticness — how well agents can access and operate the productExplore an interactive API reference with runnable examples	ai-native	—none 0/10	—none 0/10	—none 0/10	~partialT 3/10	—none 0/10	—none 0/10	~partialT 4/10	—none 0/10	—none 0/10
🤖Agenticness — how well agents can access and operate the productDownload a machine-readable API spec (OpenAPI or equivalent)	ai-native	—none 0/10	—none 0/10	—none 0/10	✓fullT 9/10	—none 0/10	—none 0/10	✓fullT 9/10	—none 0/10	—none 0/10
🤖Agenticness — how well agents can access and operate the productTest against a sandbox environment without touching production data	ai-native	n/a	—none 0/10	~partialC 3/10	~partialC 4/10	~partialC 5/10	~partialC 5/10	n/a	✓fullC 7/10	—none 0/10
🤖Agenticness — how well agents can access and operate the productRely on versioned APIs with a documented deprecation policy	ai-native	~partialC 3/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10

End to end feature delivery

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🤖Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionHave an agent automatically generate and run tests to validate its own code changes before proposing them	ai-native	—none 0/10	—none 0/10	~partialC 6/10	—none 0/10	~partialC 6/10	~partialX 4/10	n/a	—none 0/10	—none 0/10
🤖Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionHave an agent autonomously diagnose and fix a reported bug	developer	✓fullC 7/10	~partialC 6/10	!disputedD 5/10	✓fullC 8/10	~partialC 6/10	✓fullX 8/10	—none 0/10	~partialC 5/10	~partialX 6/10
🤖Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionGo from a mockup or design to a working implementation without an engineering handoff	product-manager	~partialC 6/10	✓fullC 7/10	~partialC 5/10	—none 0/10	~partialC 4/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10
🤖Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionHave an agent implement a requested feature end-to-end, including writing tests	developer	~partialC 6/10	~partialC 7/10	!disputedD 5/10	~partialC 5/10	✓fullC 8/10	~partialX 6/10	~partialX 3/10	~partialC 5/10	~partialC 5/10

Environment setup

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🤖Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionHave an agent automatically clone the repo, install dependencies, and configure its own working environment	developer	~partialC 4/10	~partialC 4/10	✓fullC 8/10	~partialC 4/10	✓fullC 8/10	✓fullX 8/10	—none 0/10	~partialT 7/10	~partialC 5/10

Interactive takeover

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🤖Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionTake over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work	developer	~partialC 4/10	~partialC 6/10	✓fullC 8/10	~partialC 5/10	~partialC 5/10	~partialX 5/10	✓fullX 8/10	~partialC 4/10	✓fullC 7/10
🤖Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionSend follow-up instructions to an active agent session to steer its work without restarting	developer	—none 0/10	~partialC 6/10	~partialC 5/10	~partialT 5/10	—none 0/10	✓fullX 7/10	✓fullC 8/10	—none 0/10	~partialC 4/10

Sandbox execution

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🤖Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionHave an agent safely execute code and install dependencies inside an isolated sandbox	developer	—none 0/10	—none 0/10	~partialC 6/10	~partialC 6/10	✓fullC 7/10	✓fullX 8/10	—none 0/10	—none 0/10	—none 0/10

Approval controls

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🤖Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsConfigure an agent to auto-approve all its actions instead of confirming each one	developer	—none 0/10	~partialC 6/10	—none 0/10	✓fullT 8/10	—none 0/10	~partialC 4/10	~partialC 6/10	—none 0/10	~partialC 4/10
🤖Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsApprove key agent decisions from my phone while agents continue working	product-manager	~partialC 5/10	~partialC 3/10	—none 0/10	~partialC 4/10	~partialC 3/10	~partialX 6/10	✓fullX 7/10	—none 0/10	✓fullX 8/10
🤖Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsSet tiered autonomy levels controlling what an agent can do without manual confirmation	engineering-lead	—none 0/10	~partialC 6/10	~partialC 4/10	~partialC 4/10	—none 0/10	~partialC 4/10	~partialC 6/10	—none 0/10	~partialC 5/10

Model control

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🤖Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsHave each task prompt automatically routed to the most suitable underlying model	ai-native	n/a	—none 0/10	✓fullT 8/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10
🤖Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsSwitch away from automatic model selection to a specific model of my choice	engineering-lead	n/a	—none 0/10	✓fullT 8/10	~partialC 5/10	—none 0/10	—none 0/10	~partialC 5/10	~partialC 5/10	~partialC 6/10

Visibility monitoring

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🤖Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsWatch what a running agent is doing in real time, including its current status	developer	~partialC 6/10	~partialC 5/10	✓fullC 7/10	~partialT 4/10	~partialC 6/10	~partialX 5/10	✓fullX 8/10	~partialC 4/10	~partialC 6/10
🤖Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsGet notified when an agent completes a task or needs my input	developer	~partialC 3/10	~partialC 4/10	~partialC 6/10	~partialC 6/10	✓fullC 8/10	✓fullX 8/10	✓fullX 8/10	~partialC 5/10	~partialX 6/10

Natural language task intake

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
💻Intent to spec — stories about intent to spec in this arenaDescribe a feature or bug in plain language and have it automatically turned into a scoped implementation task	developer	✓fullC 8/10	~partialC 6/10	✓fullX 7/10	~partialC 5/10	✓fullC 7/10	~partialX 6/10	—none 0/10	~partialC 6/10	~partialC 6/10
💻Intent to spec — stories about intent to spec in this arenaConvert user feedback submissions into structured tasks with proposed scope	product-manager	✓fullC 7/10	—none 0/10	~partialC 5/10	—none 0/10	~partialC 3/10	—none 0/10	n/a	~partialT 4/10	~partialC 4/10
💻Intent to spec — stories about intent to spec in this arenaAttach a marked-up screenshot or mockup to a task so the agent implements the correct visual change	developer	~partialC 4/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	~partialC 4/10	n/a	~partialC 4/10

Plan approval

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
💻Intent to spec — stories about intent to spec in this arenaReview and approve an agent's implementation plan before any code changes are made	developer	✓fullC 8/10	~partialC 4/10	—none 0/10	—none 0/10	—none 0/10	✓fullC 8/10	~partialX 4/10	—none 0/10	~partialC 6/10
💻Intent to spec — stories about intent to spec in this arenaApprove a task's scope and contract before an agent is allowed to modify the repository	engineering-lead	✓fullC 8/10	~partialC 4/10	—none 0/10	—none 0/10	—none 0/10	~partialC 6/10	~partialC 4/10	~partialC 4/10	~partialX 6/10

Ticket driven tasking

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
💻Intent to spec — stories about intent to spec in this arenaAssign a coding task to an agent directly from an existing issue or ticket	developer	—none 0/10	~partialC 4/10	✓fullC 8/10	~partialC 5/10	✓fullC 7/10	✓fullX 8/10	—none 0/10	~partialC 6/10	✓fullC 8/10

Enterprise licensing

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
💲Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payLicense an enterprise deployment with SSO and commercial support for organization-wide rollout	engineering-lead	—none 0/10	—none 0/10	—none 0/10	~partialC 6/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10

Model flexibility

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
💲Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payBring my own LLM or API key so agents run on the model of my choice	engineering-lead	—none 0/10	—none 0/10	—none 0/10	✓fullC 8/10	—none 0/10	—none 0/10	✓fullC 8/10	—none 0/10	~partialC 5/10

Usage quotas

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
💲Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to paySee and manage plan-based daily task and concurrency limits for agent workflows	engineering-lead	—none 0/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	~partialX 4/10	—none 0/10	n/a	—none 0/10

Chat integration

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
💻Repo integration — stories about repo integration in this arenaTag an agent in a chat thread to discuss and delegate a bug or task	developer	—none 0/10	~partialC 3/10	✓fullC 8/10	~partialC 6/10	~partialC 6/10	~partialX 4/10	~partialC 4/10	n/a	—none 0/10

Knowledge context

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
💻Repo integration — stories about repo integration in this arenaAdd a context file describing my codebase conventions so agents generate more relevant plans and code	developer	~partialC 4/10	—none 0/10	✓fullC 9/10	—none 0/10	—none 0/10	✓fullC 7/10	~partialC 3/10	—none 0/10	—none 0/10
💻Repo integration — stories about repo integration in this arenaQuery generated documentation for any public or private repository	developer	n/a	—none 0/10	✓fullC 9/10	—none 0/10	—none 0/10	—none 0/10	n/a	n/a	n/a

Project management integration

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
💻Repo integration — stories about repo integration in this arenaConnect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly	product-manager	—none 0/10	~partialC 5/10	~partialC 6/10	~partialC 4/10	~partialC 6/10	—none 0/10	—none 0/10	—none 0/10	~partialC 6/10

Version control integration

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
💻Repo integration — stories about repo integration in this arenaConnect a GitHub repository so an agent can access the code and open pull requests against it	developer	✓fullC 7/10	~partialC 5/10	~partialC 6/10	~partialC 6/10	✓fullC 8/10	✓fullX 9/10	—none 0/10	—none 0/10	✓fullC 8/10
💻Repo integration — stories about repo integration in this arenaGrant an agent access to my repositories with a one-click install, without complex setup	developer	~partialC 5/10	—none 0/10	~partialC 5/10	~partialC 4/10	✓fullC 8/10	~partialX 6/10	!disputedD 3/10	—none 0/10	—none 0/10

Ci remediation

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🧪Review quality gates — quality gates on changes — review flow, required checks, merge protectionHave failed CI workflows automatically diagnosed and fixed with a proposed pull request	engineering-lead	—none 0/10	~partialC 4/10	✓fullC 7/10	✓fullC 7/10	~partialC 6/10	~partialC 3/10	n/a	—none 0/10	~partialC 5/10
🧪Review quality gates — quality gates on changes — review flow, required checks, merge protectionTrigger an agent from CI/CD pipelines to fix a broken build or failing test	developer	—none 0/10	✓fullT 8/10	~partialC 6/10	~partialC 7/10	✓fullC 8/10	~partialX 5/10	—none 0/10	—none 0/10	~partialC 6/10

Diff review

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🧪Review quality gates — quality gates on changes — review flow, required checks, merge protectionConfigure an agent to automatically open a pull request when its task completes	developer	✓fullC 8/10	~partialC 3/10	~partialC 5/10	~partialC 6/10	✓fullC 7/10	✓fullX 8/10	—none 0/10	—none 0/10	~partialC 4/10
🧪Review quality gates — quality gates on changes — review flow, required checks, merge protectionReview a diff of an agent's changes and approve it before it becomes a pull request	developer	—none 0/10	~partialC 6/10	~partialC 4/10	~partialC 4/10	~partialC 5/10	✓fullX 9/10	—none 0/10	—none 0/10	~partialX 7/10

Pr review automation

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🧪Review quality gates — quality gates on changes — review flow, required checks, merge protectionHave incoming issues automatically triaged with severity suggested and routed to the right owner	ai-native	—none 0/10	—none 0/10	—none 0/10	✓fullC 8/10	—none 0/10	—none 0/10	n/a	—none 0/10	—none 0/10
🧪Review quality gates — quality gates on changes — review flow, required checks, merge protectionHave every pull request automatically reviewed with AI-generated inline comments	engineering-lead	n/a	—none 0/10	~partialC 5/10	~partialC 6/10	✓fullC 7/10	—none 0/10	n/a	—none 0/10	—none 0/10

Readiness checks

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🧪Review quality gates — quality gates on changes — review flow, required checks, merge protectionAutomatically fix failing agent-readiness criteria in my repository	engineering-lead	—none 0/10	✓fullC 8/10	~partialC 6/10	—none 0/10	~partialC 5/10	—none 0/10	n/a	—none 0/10	n/a
🧪Review quality gates — quality gates on changes — review flow, required checks, merge protectionRun a readiness report that evaluates how ready my repository is for autonomous agents	engineering-lead	—none 0/10	✓fullC 8/10	—none 0/10	—none 0/10	—none 0/10	—none 0/10	n/a	~partialC 4/10	—none 0/10

Security remediation

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
🧪Review quality gates — quality gates on changes — review flow, required checks, merge protectionHave security alerts automatically validated and remediated with an opened pull request	engineering-lead	—none 0/10	—none 0/10	~partialC 5/10	✓fullC 8/10	~partialC 5/10	—none 0/10	n/a	n/a	—none 0/10

Concurrent execution

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
⚡Scale parallelism — running many jobs at once — concurrency, fleets, queueingRun many agent tasks concurrently to scale delivery throughput	engineering-lead	~partialC 4/10	~partialC 7/10	~partialC 7/10	~partialC 6/10	~partialC 6/10	~partialX 6/10	~partialC 5/10	~partialC 6/10	~partialC 5/10
⚡Scale parallelism — running many jobs at once — concurrency, fleets, queueingCreate agent sessions on behalf of other users in my organization	engineering-lead	—none 0/10	—none 0/10	✓fullC 8/10	—none 0/10	~partialC 4/10	—none 0/10	~partialC 4/10	n/a	—none 0/10

Deployment flexibility

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
⚡Scale parallelism — running many jobs at once — concurrency, fleets, queueingUse a managed cloud offering to run agents without operating my own backend infrastructure	developer	—none 0/10	✓fullC 7/10	✓fullC 7/10	✓fullC 8/10	✓fullC 7/10	✓fullX 8/10	~partialT 5/10	n/a	~partialC 5/10
⚡Scale parallelism — running many jobs at once — concurrency, fleets, queueingSelf-host agent infrastructure locally, in containers, or on my own VMs	engineering-lead	n/a	—none 0/10	—none 0/10	✓fullC 8/10	—none 0/10	—none 0/10	✓fullC 7/10	~partialT 5/10	~partialC 7/10

Headless automation

Story	Persona	Foreloop	Factory	Devin	OpenHands	Codegen	Jules	Omnara	YYLO	HumanLayer
⚡Scale parallelism — running many jobs at once — concurrency, fleets, queueingRun an agent headlessly inside CI/CD pipelines and shell scripts	developer	~partialC 5/10	✓fullC 9/10	~partialT 6/10	✓fullT 7/10	~partialT 6/10	~partialX 5/10	~partialT 4/10	~partialT 6/10	✓fullC 8/10