Skip to content
ProductArena

AI Code Review Arena

AI Code Review — procurement report

ProductArena · rankings as of 2026-09-16 · evidence as of 2026-09-16 · 6 products · 53 judged requirements · 318 judged cells

Methodology: Every product is judged against a shared taxonomy of user stories using cited evidence — hands-on probes > repository code > independent community sources > vendor claims — never opinion. Full writeup: https://ultrametric.ai/productarena/methodology

Leaderboard

#ProductPA ScoreCoverage scoreApplicable cellsConfidence
1CodeRabbit36.648.953/53C
2Qodo28.439.752/53B
3cubic27.844.551/53B
4Greptile27.135.953/53C
5Graphite25.230.553/53C
6Cursor Bugbot23.528.542/53C

PA Score = agent-readiness blend (see methodology). Coverage score = weighted share of judged requirements met. Confidence = how much of the score rests on tested vs claimed evidence (A–D).

Uncertainty note

The current #1/#2 gap in this arena is not close enough to qualify for the multi-judge uncertainty pass (or the pass has not covered it yet) — no extra caveat applies beyond the per-product confidence grades above.

Buyer checklist (RFP)

The arena's 53 judged user stories as requirements, grouped by theme. Priorities mirror the story weights our scoring uses (3 = must-have, 2 = should-have, 1 = nice-to-have). Interactive version with per-requirement verdicts for the top products: /arena/ai-code-review/checklist

Agenticness — how well agents can access and operate the productAgenticness

How well agents can access and operate the product

  • ai-native userPlug MCP servers into this product so it can use their toolsmust-have
  • ai-native userConnect an agent via an official MCP servermust-have
  • ai-native userDrive the product through a documented public APImust-have
  • ai-native userDelegate tasks to a built-in AI assistant inside the productmust-have
  • ai-native userPoint an agent at llms.txt or agent-oriented docsshould-have
  • ai-native userRun the product headlessly / in CI for automationshould-have
  • ai-native userUse an official CLIshould-have
  • ai-native userIssue scoped/least-privilege API credentials for an agentshould-have
  • ai-native userBuild against official SDKsshould-have
  • ai-native userSubscribe to events via webhooksshould-have
  • ai-native userGet AI-generated insights and suggestions from my data inside the productshould-have
  • ai-native userSet up automations that run autonomously in the backgroundshould-have
  • ai-native userOperate the product with natural-language commandsshould-have
  • ai-native userExplore an interactive API reference with runnable examplesshould-have
  • ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)should-have
  • ai-native userRely on versioned APIs with a documented deprecation policyshould-have
  • ai-native userTest against a sandbox environment without touching production datanice-to-have

Autofix agents — stories about autofix agents in this arenaAutofix agents

Stories about autofix agents in this arena

  • developerI turn a review finding into an applied fix — a committed patch or an agent-generated follow-up — without leaving the PRmust-have
  • ai-native userThe reviewer holds the line on AI-generated PRs — it verifies agent-authored code at a volume no human team could reviewshould-have
  • ai-native userI define custom agentic pre-merge checks in plain language — 'docs updated', 'tests cover new paths' — that run on every PRshould-have
  • ai-native userReview findings hand off cleanly to my coding agent — copyable fix prompts or direct integration with Claude Code, Cursor, or Codexshould-have

Automation depth — how much of the product can run unattendedAutomation depth

How much of the product can run unattended

  • ai-native userDefine rules that trigger actions automatically on eventsmust-have
  • ai-native userPerform bulk operations across many items at onceshould-have
  • ai-native userSchedule recurring jobs or workflowsshould-have
  • ai-native userVersion, review, and roll back my automationsnice-to-have

Codebase understanding — how deeply the tool maps your repo — cross-file context, architecture awareness, historyCodebase understanding

How deeply the tool maps your repo — cross-file context, architecture awareness, history

  • developerReview comments reflect the whole repository — call sites, related modules, existing conventions — not just the changed hunksmust-have
  • engineering leadThe reviewer understands changes that span multiple repositories or a large monorepo and reviews them coherentlyshould-have
  • ai-native userThe reviewer builds a persistent memory of my team's conventions and past review decisions and applies it to future PRsshould-have

Interaction — how you steer it — commands, replies, review conversations, configurability in the loopInteraction

How you steer it — commands, replies, review conversations, configurability in the loop

  • developerI reply to the reviewer in the PR thread to ask questions, get explanations, or issue commands — and it answers in contextshould-have
  • developerI control when reviews run — skip drafts, trigger on demand, filter by branch or label — so the bot shows up only when wantednice-to-have

Openness — open source, data portability, and self-hosting storiesOpenness

Open source, data portability, and self-hosting stories

  • ai-native userExport all of my data in open formats and leavemust-have
  • ai-native userSelf-host the core productmust-have
  • ai-native userDo everything through the API that I can do in the UIshould-have
  • ai-native userRead the product's source under an open licenseshould-have

Pr integration — stories about pr integration in this arenaPr integration

Stories about pr integration in this arena

  • developerThe reviewer installs as a GitHub/GitLab app and posts reviews as native inline comments on my pull requests within minutesmust-have
  • developerReview comments include committable suggested diffs I can apply with one clickmust-have
  • developerEvery PR gets an auto-generated summary and change walkthrough so human reviewers orient fastshould-have
  • developerPushing new commits triggers an incremental re-review that tracks what was fixed instead of repeating old commentsshould-have

Privacy posture — data-handling and privacy storiesPrivacy posture

Data-handling and privacy stories

  • ai-native userPrevent my data from being used to train AI modelsmust-have
  • ai-native userChoose where my data is stored (region/residency)should-have
  • ai-native userControl data retention and deletionshould-have
  • ai-native userOpt out of telemetry and usage trackingshould-have

Quality gates — stories about quality gates in this arenaQuality gates

Stories about quality gates in this arena

  • engineering leadThe reviewer can gate merges — a required status check or blocking review that enforces resolution of critical findingsshould-have
  • engineering leadI see dashboards of findings, acceptance rates, and review coverage across my orgnice-to-have

Review accuracy — stories about review accuracy in this arenaReview accuracy

Stories about review accuracy in this arena

  • developerThe reviewer catches real bugs in my PR — logic errors, race conditions, broken edge cases — not just style nitsmust-have
  • engineering leadThe reviewer keeps noise low — few false positives, deduplicated comments, severity labels — so my team doesn't tune it outmust-have
  • developerPush back on a bad review comment and the reviewer learns — it stops repeating the same rejected feedbackshould-have
  • security engineerReviews flag security problems in the diff — injection risks, leaked secrets, insecure patterns — alongside functional bugsshould-have

Surfaces — where it meets your workflow — IDE, CLI, web, PR comments, CI checksSurfaces

Where it meets your workflow — IDE, CLI, web, PR comments, CI checks

  • developerI run reviews from a CLI against local diffs or in CI scripts, with machine-readable output my tooling can consumeshould-have
  • developerI get the same review inside my IDE before I push, catching issues while the code is still in my editorshould-have

Workflow config — stories about workflow config in this arenaWorkflow config

Stories about workflow config in this arena

  • engineering leadI configure the reviewer with a versioned config file in my repo — path filters, per-path instructions, review profilesmust-have
  • engineering leadI encode my team's own review guidelines — natural-language rules, AST patterns, or linked style guides — and the reviewer enforces themshould-have
  • engineering leadI roll out org-level review defaults across hundreds of repos and manage exceptions centrallynice-to-have

Appendix: recorded probes

Hands-on probe recordings — transcripts/videos a human can replay, the strongest evidence tier. Watch them at https://ultrametric.ai/productarena/proofs

  • CodeRabbitcurl -s https://docs.coderabbit.ai/llms.txt | head -6terminal · recorded 2026-09-10 · exit 0
  • CodeRabbitcurl -sL https://docs.coderabbit.ai/getting-started/quickstart.md | head -12terminal · recorded 2026-09-10 · exit 0
  • cubicnpx -y @cubic-dev-ai/cli --versionterminal · recorded 2026-09-10 · exit 0
  • cubiccurl -si -X POST https://www.cubic.dev/api/mcp -H 'Content-Type: application/json' -d '<jsonrpc initialize>' | grep -iE '^HTTP|^www-authenticate'terminal · recorded 2026-09-10 · exit 0
  • cubiccurl -s https://docs.cubic.dev/llms.txt | head -6terminal · recorded 2026-09-10 · exit 0
  • cubiccurl -sL https://docs.cubic.dev/ai-review/quickstart.md | head -12terminal · recorded 2026-09-10 · exit 0
  • Cursor Bugbotcurl -s https://cursor.com/llms.txt | head -6terminal · recorded 2026-09-10 · exit 0
  • Cursor Bugbotcurl -sL https://cursor.com/docs/bugbot.md | head -6terminal · recorded 2026-09-10 · exit 0
  • Graphitenpx -y @withgraphite/graphite-cli --versionterminal · recorded 2026-09-10 · exit 0
  • Graphitecurl -s https://graphite.com/docs/llms.txt | head -6terminal · recorded 2026-09-10 · exit 0
  • Graphitecurl -sL https://graphite.com/docs/ai-reviews.md | head -12terminal · recorded 2026-09-10 · exit 0
  • Greptilecurl -s -X POST https://api.greptile.com/v2/repositories -H 'Content-Type: application/json' -d '{}'terminal · recorded 2026-09-10 · exit 0
  • Greptilenpx -y greptile --versionterminal · recorded 2026-09-10 · exit 0
  • Greptilecurl -s -X POST https://api.greptile.com/mcp -H 'Content-Type: application/json' -d '<jsonrpc initialize>' # FULL keyless handshaketerminal · recorded 2026-09-10 · exit 0
  • Greptilecurl -s https://www.greptile.com/docs/llms.txt | head -6terminal · recorded 2026-09-10 · exit 0
  • Greptilecurl -sL https://www.greptile.com/docs/quickstart.md | head -12terminal · recorded 2026-09-10 · exit 0
  • Qodocurl -si -X POST https://sdk.qodo.ai/v1/tools/mcp/ -H 'Content-Type: application/json' -d '<jsonrpc initialize>'terminal · recorded 2026-09-10 · exit 0
  • Qodocurl -s https://docs.qodo.ai/llms.txt | head -6terminal · recorded 2026-09-10 · exit 0
  • Qodocurl -sL https://docs.qodo.ai/code-review/overview.md | head -12terminal · recorded 2026-09-10 · exit 0

Cite as: ProductArena by Ultrametric Inc, AI Code Review arena, rankings as of 2026-09-16 — https://ultrametric.ai/productarena/arena/ai-code-review

License: © 2026 Ultrametric Inc. Brief quotation of individual verdicts, scores, or evidence excerpts is permitted with attribution to "ProductArena by Ultrametric Inc (ultrametric.ai/productarena)", as is use of the data to evaluate, contest, or contribute corrections. Bulk copying, redistribution, or use to build competing datasets requires prior written permission (see DATA-LICENSE in the repository).

No liability: rankings, verdicts, and scores are research outputs derived from the cited evidence at a point in time, provided "as is", without warranties. Ultrametric Inc accepts no responsibility for procurement, purchasing, or other decisions made in reliance on them — verify against the cited evidence before acting (https://ultrametric.ai/productarena/terms).