Software Factory arenaBuyer checklist
Every requirement we judge software factory products against, as a ready-to-send RFP checklist — with each item's priority, why it matters, and how the top-ranked products score on it today.
73 requirements · 11 themes · verdicts for 9 products · updated 2026-09-16 · priorities mirror the story weights our scoring uses (methodology)
Show the markdown export
# Software Factory — buyer checklist (RFP) Derived from ProductArena's evidence-graded user-story taxonomy for Software Factory: 73 judged requirements. Priorities mirror story weights (3 = must-have, 2 = should-have, 1 = nice-to-have). ## Agenticness - [ ] **[must-have]** Plug MCP servers into this product so it can use their tools - [ ] **[must-have]** Connect an agent via an official MCP server - [ ] **[must-have]** Drive the product through a documented public API - [ ] **[must-have]** Delegate tasks to a built-in AI assistant inside the product - [ ] **[should-have]** Point an agent at llms.txt or agent-oriented docs - [ ] **[should-have]** Run the product headlessly / in CI for automation - [ ] **[should-have]** Use an official CLI - [ ] **[should-have]** Issue scoped/least-privilege API credentials for an agent - [ ] **[should-have]** Build against official SDKs - [ ] **[should-have]** Subscribe to events via webhooks - [ ] **[should-have]** Get AI-generated insights and suggestions from my data inside the product - [ ] **[should-have]** Set up automations that run autonomously in the background - [ ] **[should-have]** Operate the product with natural-language commands - [ ] **[should-have]** Explore an interactive API reference with runnable examples - [ ] **[should-have]** Download a machine-readable API spec (OpenAPI or equivalent) - [ ] **[should-have]** Rely on versioned APIs with a documented deprecation policy - [ ] **[nice-to-have]** Test against a sandbox environment without touching production data ## Automation depth - [ ] **[must-have]** Define rules that trigger actions automatically on events - [ ] **[should-have]** Perform bulk operations across many items at once - [ ] **[should-have]** Schedule recurring jobs or workflows - [ ] **[nice-to-have]** Version, review, and roll back my automations ## Autonomous implementation - [ ] **[must-have]** Have an agent autonomously diagnose and fix a reported bug - [ ] **[must-have]** Have an agent implement a requested feature end-to-end, including writing tests - [ ] **[must-have]** Have an agent safely execute code and install dependencies inside an isolated sandbox - [ ] **[should-have]** Have an agent automatically generate and run tests to validate its own code changes before proposing them - [ ] **[should-have]** Go from a mockup or design to a working implementation without an engineering handoff - [ ] **[should-have]** Have an agent automatically clone the repo, install dependencies, and configure its own working environment - [ ] **[should-have]** Take over an in-progress agent task in my editor, terminal, or browser to finish or redirect the work - [ ] **[should-have]** Send follow-up instructions to an active agent session to steer its work without restarting ## Human oversight - [ ] **[must-have]** Set tiered autonomy levels controlling what an agent can do without manual confirmation - [ ] **[must-have]** Watch what a running agent is doing in real time, including its current status - [ ] **[should-have]** Configure an agent to auto-approve all its actions instead of confirming each one - [ ] **[should-have]** Have each task prompt automatically routed to the most suitable underlying model - [ ] **[should-have]** Get notified when an agent completes a task or needs my input - [ ] **[nice-to-have]** Approve key agent decisions from my phone while agents continue working - [ ] **[nice-to-have]** Switch away from automatic model selection to a specific model of my choice ## Intent to spec - [ ] **[must-have]** Describe a feature or bug in plain language and have it automatically turned into a scoped implementation task - [ ] **[must-have]** Review and approve an agent's implementation plan before any code changes are made - [ ] **[must-have]** Assign a coding task to an agent directly from an existing issue or ticket - [ ] **[should-have]** Convert user feedback submissions into structured tasks with proposed scope - [ ] **[should-have]** Attach a marked-up screenshot or mockup to a task so the agent implements the correct visual change - [ ] **[should-have]** Approve a task's scope and contract before an agent is allowed to modify the repository ## Openness - [ ] **[must-have]** Export all of my data in open formats and leave - [ ] **[must-have]** Self-host the core product - [ ] **[should-have]** Do everything through the API that I can do in the UI - [ ] **[should-have]** Read the product's source under an open license ## Pricing limits - [ ] **[should-have]** License an enterprise deployment with SSO and commercial support for organization-wide rollout - [ ] **[should-have]** Bring my own LLM or API key so agents run on the model of my choice - [ ] **[should-have]** See and manage plan-based daily task and concurrency limits for agent workflows ## Privacy posture - [ ] **[must-have]** Prevent my data from being used to train AI models - [ ] **[should-have]** Choose where my data is stored (region/residency) - [ ] **[should-have]** Control data retention and deletion - [ ] **[should-have]** Opt out of telemetry and usage tracking ## Repo integration - [ ] **[must-have]** Add a context file describing my codebase conventions so agents generate more relevant plans and code - [ ] **[must-have]** Connect issue trackers like Jira, Linear, ClickUp, or Monday.com so agents can manage tickets directly - [ ] **[must-have]** Connect a GitHub repository so an agent can access the code and open pull requests against it - [ ] **[should-have]** Tag an agent in a chat thread to discuss and delegate a bug or task - [ ] **[should-have]** Grant an agent access to my repositories with a one-click install, without complex setup - [ ] **[nice-to-have]** Query generated documentation for any public or private repository ## Review quality gates - [ ] **[must-have]** Have failed CI workflows automatically diagnosed and fixed with a proposed pull request - [ ] **[must-have]** Review a diff of an agent's changes and approve it before it becomes a pull request - [ ] **[must-have]** Have every pull request automatically reviewed with AI-generated inline comments - [ ] **[should-have]** Trigger an agent from CI/CD pipelines to fix a broken build or failing test - [ ] **[should-have]** Configure an agent to automatically open a pull request when its task completes - [ ] **[should-have]** Have incoming issues automatically triaged with severity suggested and routed to the right owner - [ ] **[should-have]** Run a readiness report that evaluates how ready my repository is for autonomous agents - [ ] **[should-have]** Have security alerts automatically validated and remediated with an opened pull request - [ ] **[nice-to-have]** Automatically fix failing agent-readiness criteria in my repository ## Scale parallelism - [ ] **[must-have]** Run many agent tasks concurrently to scale delivery throughput - [ ] **[should-have]** Create agent sessions on behalf of other users in my organization - [ ] **[should-have]** Use a managed cloud offering to run agents without operating my own backend infrastructure - [ ] **[should-have]** Self-host agent infrastructure locally, in containers, or on my own VMs - [ ] **[should-have]** Run an agent headlessly inside CI/CD pipelines and shell scripts --- Source: https://ultrametric.ai/productarena/arena/software-factory (evidence-graded verdicts for 9 products) · methodology: https://ultrametric.ai/productarena/methodology
Chips show the top 5 ranked products' current verdict on each requirement — ✓ full · ~ partial · ! disputed · — none · n/a not applicable.
Agenticness — how well agents can access and operate the productAgenticness· 17 items
How well agents can access and operate the product
Automation depth — how much of the product can run unattendedAutomation depth· 4 items
How much of the product can run unattended
Autonomous implementation — end-to-end implementation by the agent — multi-file changes, task completionAutonomous implementation· 8 items
End-to-end implementation by the agent — multi-file changes, task completion
Human oversight — keeping a human in the loop — approvals, checkpoints, interruptsHuman oversight· 7 items
Keeping a human in the loop — approvals, checkpoints, interrupts
Intent to spec — stories about intent to spec in this arenaIntent to spec· 6 items
Stories about intent to spec in this arena
Openness — open source, data portability, and self-hosting storiesOpenness· 4 items
Open source, data portability, and self-hosting stories
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits· 3 items
Free-tier ceilings, usage caps, and rate limits before you have to pay
Privacy posture — data-handling and privacy storiesPrivacy posture· 4 items
Data-handling and privacy stories
Repo integration — stories about repo integration in this arenaRepo integration· 6 items
Stories about repo integration in this arena
Review quality gates — quality gates on changes — review flow, required checks, merge protectionReview quality gates· 9 items
Quality gates on changes — review flow, required checks, merge protection
Scale parallelism — running many jobs at once — concurrency, fleets, queueingScale parallelism· 5 items
Running many jobs at once — concurrency, fleets, queueing
Full evidence behind every verdict lives on the arena page and each product page — chips above deep-link straight to the judged story.