AI Coding Agents — procurement report
ProductArena · rankings as of 2026-09-16 · evidence as of 2026-09-16 · 13 products · 74 judged requirements · 962 judged cells
Methodology: Every product is judged against a shared taxonomy of user stories using cited evidence — hands-on probes > repository code > independent community sources > vendor claims — never opinion. Full writeup: https://ultrametric.ai/productarena/methodology
Leaderboard
| # | Product | PA Score | Coverage score | Applicable cells | Confidence |
|---|---|---|---|---|---|
| 1 | OpenCode | 47.4 | 40.7 | 72/74 | C |
| 2 | Claude Code | 40.4 | 48.6 | 74/74 | C |
| 3 | Cline | 39.8 | 40.2 | 71/74 | A |
| 4 | GitHub Copilot | 37.4 | 45.5 | 74/74 | C |
| 5 | Google Antigravity | 36.8 | 36.8 | 71/74 | C |
| 6 | Codex | 32.7 | 39.9 | 72/74 | B |
| 7 | Aider | 30.5 | 28.6 | 63/74 | C |
| 8 | Devin | 30.2 | 37.7 | 72/74 | C |
| 9 | Conductor | 29.8 | 32.8 | 70/74 | C |
| 10 | cubic | 26.8 | 25.2 | 63/74 | B |
| 11 | Cursor | 26.7 | 34.0 | 71/74 | C |
| 12 | Gemini CLI | 21.2 | 21.9 | 71/74 | B |
| 13 | Slate | 15.0 | 18.0 | 67/74 | C |
PA Score = agent-readiness blend (see methodology). Coverage score = weighted share of judged requirements met. Confidence = how much of the score rests on tested vs claimed evidence (A–D).
Uncertainty note
The current #1/#2 gap in this arena is not close enough to qualify for the multi-judge uncertainty pass — no extra caveat applies beyond the per-product confidence grades above.
Buyer checklist (RFP)
The arena's 74 judged user stories as requirements, grouped by theme. Priorities mirror the story weights our scoring uses (3 = must-have, 2 = should-have, 1 = nice-to-have). Interactive version with per-requirement verdicts for the top products: /arena/ai-coding/checklist
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
- ai-native userPlug MCP servers into this product so it can use their toolsmust-have
- ai-native userConnect an agent via an official MCP servermust-have
- ai-native userDrive the product through a documented public APImust-have
- ai-native userDelegate tasks to a built-in AI assistant inside the productmust-have
- ai-native userPoint an agent at llms.txt or agent-oriented docsshould-have
- ai-native userRun the product headlessly / in CI for automationshould-have
- ai-native userUse an official CLIshould-have
- ai-native userIssue scoped/least-privilege API credentials for an agentshould-have
- ai-native userBuild against official SDKsshould-have
- ai-native userSubscribe to events via webhooksshould-have
- ai-native userGet AI-generated insights and suggestions from my data inside the productshould-have
- ai-native userSet up automations that run autonomously in the backgroundshould-have
- ai-native userOperate the product with natural-language commandsshould-have
- ai-native userExplore an interactive API reference with runnable examplesshould-have
- ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)should-have
- ai-native userRely on versioned APIs with a documented deprecation policyshould-have
- ai-native userTest against a sandbox environment without touching production datanice-to-have
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
- ai-native userDefine rules that trigger actions automatically on eventsmust-have
- ai-native userPerform bulk operations across many items at onceshould-have
- ai-native userSchedule recurring jobs or workflowsshould-have
- ai-native userVersion, review, and roll back my automationsnice-to-have
Autonomy agents — stories about autonomy agents in this arenaAutonomy agents
Stories about autonomy agents in this arena
- developerDelegate longer-running coding tasks to run in the background in an isolated cloud environmentmust-have
- ai-native userHave a cloud agent build, test, and demo a feature end-to-end for my reviewshould-have
- developerConfigure a reproducible cloud environment with the dependencies and setup steps my repository needsshould-have
- ai-native userLaunch fleets of autonomous agents that work in parallel on different tasks for hours or daysshould-have
- ai-native userSet up always-on agents that run on schedules or triggers to maintain and fix my software autonomouslyshould-have
- developerRun several task attempts in parallel and compare results before choosing onenice-to-have
Code generation — quality of generated code — correctness, style, fit to the codebaseCode generation
Quality of generated code — correctness, style, fit to the codebase
- developerReceive inline code completions and next-edit suggestions as I typemust-have
- developerTurn a tracked issue into a complete pull request end-to-endmust-have
- developerDescribe a feature or bug in plain language and have the agent implement or fix it across multiple filesmust-have
- developerHave the agent write tests, fix lint errors, resolve merge conflicts, and update dependencies for memust-have
- developerDebug issues and troubleshoot using natural-language queriesshould-have
- ai-native userGenerate a working app from a sketch, image, or PDF designshould-have
- developerDebug a live running web application directly from my coding assistantnice-to-have
Codebase understanding — how deeply the tool maps your repo — cross-file context, architecture awareness, historyCodebase understanding
How deeply the tool maps your repo — cross-file context, architecture awareness, history
- developerUnderstand how a codebase fits together to find where to start making changesmust-have
- developerHave the agent map and explain an entire unfamiliar codebase without manually selecting context filesmust-have
- developerAdd a project instructions file to set coding standards and conventions the agent followsmust-have
- developerReproduce issues, narrow down root causes, and verify fixesmust-have
- developerHave the agent build and recall memory automatically across sessionsshould-have
- developerInclude multiple project directories in a single session for broader contextshould-have
Ecosystem — integrations, plugins, and third-party ecosystem storiesEcosystem
Integrations, plugins, and third-party ecosystem stories
- developerConnect the agent to workflow tools like Jira, Slack, and Google Drive to extend its contextmust-have
- developerKick off agent tasks directly from GitHub, GitLab, Linear, or Slackshould-have
- developerEquip the agent with custom skills to perform specialized tasksnice-to-have
- engineering-leadIntegrate third-party partner-built agent apps into my workflowsnice-to-have
- engineering-leadCreate a shared workspace from my docs and repos as a common source of truth for the teamnice-to-have
Ide terminal integration — meeting you in the IDE and terminal — extensions, inline flows, contextIde terminal integration
Meeting you in the IDE and terminal — extensions, inline flows, context
- developerChat with the coding assistant directly inside my IDE for contextual helpmust-have
- developerRun a coding agent locally from my terminalmust-have
- developerStart a task on one device and continue it later from another device or browsershould-have
- developerReview diffs visually and run multiple sessions side by side in a desktop appshould-have
- engineering-leadManage multiple agent-driven coding sessions from one unified workspaceshould-have
- developerRun the agent non-interactively in scripts for workflow automationshould-have
- developerView interactive diffs and share selected code as context from within my JetBrains IDEnice-to-have
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
- ai-native userExport all of my data in open formats and leavemust-have
- ai-native userSelf-host the core productmust-have
- ai-native userDo everything through the API that I can do in the UIshould-have
- ai-native userRead the product's source under an open licenseshould-have
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
- developerAuthenticate with an API key instead of an account loginshould-have
- engineering-leadAuthenticate through an enterprise identity or cloud platform for compliance and scalabilityshould-have
- developerSign in with my existing product subscription plan to use the coding agentshould-have
- developerChoose which underlying AI model powers my session from multiple providersshould-have
- developerSign in with a personal account to get free-tier access without managing API keysnice-to-have
- developerLet the tool automatically pick the best model for each tasknice-to-have
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
- ai-native userPrevent my data from being used to train AI modelsmust-have
- ai-native userChoose where my data is stored (region/residency)should-have
- ai-native userControl data retention and deletionshould-have
- ai-native userOpt out of telemetry and usage trackingshould-have
Review safety — keeping generated changes safe — diffs, approvals, guardrailsReview safety
Keeping generated changes safe — diffs, approvals, guardrails
- developerHave the agent stage changes, write commit messages, create branches, and open pull requestsmust-have
- developerGet automatic code review with contextual feedback on every pull requestmust-have
- developerInspect diffs and run checks to catch problems before mergingmust-have
- engineering-leadControl which external tools and integrations the agent is allowed to accessshould-have
- engineering-leadHave the agent operate inside a sandbox when interacting with code, tools, and network resourcesshould-have
- developerGet contextual explanations and automatic fixes for security vulnerabilitiesshould-have
- engineering-leadOpt out of having my code and prompts used for AI model trainingnice-to-have
- engineering-leadSee license and public-code matching references for AI-suggested codenice-to-have
Appendix: recorded probes
Hands-on probe recordings — transcripts/videos a human can replay, the strongest evidence tier. Watch them at https://ultrametric.ai/productarena/proofs
- Google Antigravity
curl -s --compressed https://antigravity.google/docs/getting-started.md | head -6terminal · recorded 2026-09-14 · exit 0 - Google Antigravity
curl -s https://antigravity.google/llms.txt | head -4terminal · recorded 2026-09-14 · exit 0 - Claude Code
claude --versionterminal · recorded 2026-09-03 · exit 0 - Claude Code
echo '<jsonrpc initialize>' | claude mcp serveterminal · recorded 2026-09-03 · exit 0 - Codex
codex --versionterminal · recorded 2026-09-03 · exit 0 - Codex
codex exec --helpterminal · recorded 2026-09-03 · exit 0 - Codex
echo '<jsonrpc initialize>' | codex mcp-serverterminal · recorded 2026-09-03 · exit 0 - Conductor
curl -s https://www.conductor.build/docs/api/mcp.md | head -8terminal · recorded 2026-09-14 · exit 0 - Conductor
curl -s https://www.conductor.build/llms.txt | head -4terminal · recorded 2026-09-14 · exit 0 - Conductor
curl -si -X POST https://api.conductor.build/mcp -H 'Content-Type: application/json' -d '<jsonrpc initialize>'terminal · recorded 2026-09-14 · exit 0 - cubic
curl -s https://docs.cubic.dev/llms.txt | head -6terminal · recorded 2026-09-10 · exit 0 - cubic
curl -si -X POST https://www.cubic.dev/api/mcp -H 'Content-Type: application/json' -d '<jsonrpc initialize>'terminal · recorded 2026-09-10 · exit 0 - Slate
curl -s https://docs.randomlabs.ai/en/getting-started/quickstart.md | head -8terminal · recorded 2026-09-14 · exit 0 - Slate
curl -s https://docs.randomlabs.ai/llms.txt | head -4terminal · recorded 2026-09-14 · exit 0 - Slate
curl -s https://registry.npmjs.org/@randomlabs/slate/latest | head -c 400terminal · recorded 2026-09-14 · exit 0
Cite as: ProductArena by Ultrametric Inc, AI Coding Agents arena, rankings as of 2026-09-16 — https://ultrametric.ai/productarena/arena/ai-coding
License: © 2026 Ultrametric Inc. Brief quotation of individual verdicts, scores, or evidence excerpts is permitted with attribution to "ProductArena by Ultrametric Inc (ultrametric.ai/productarena)", as is use of the data to evaluate, contest, or contribute corrections. Bulk copying, redistribution, or use to build competing datasets requires prior written permission (see DATA-LICENSE in the repository).
No liability: rankings, verdicts, and scores are research outputs derived from the cited evidence at a point in time, provided "as is", without warranties. Ultrametric Inc accepts no responsibility for procurement, purchasing, or other decisions made in reliance on them — verify against the cited evidence before acting (https://ultrametric.ai/productarena/terms).