AI Code Review arenaBuyer checklist

Every requirement we judge ai code review products against, as a ready-to-send RFP checklist — with each item's priority, why it matters, and how the top-ranked products score on it today.

53 requirements · 12 themes · verdicts for 6 products · updated 2026-09-16 · priorities mirror the story weights our scoring uses (methodology)

Procurement report →

Show the markdown export

# AI Code Review — buyer checklist (RFP)

Derived from ProductArena's evidence-graded user-story taxonomy for AI Code Review: 53 judged requirements. Priorities mirror story weights (3 = must-have, 2 = should-have, 1 = nice-to-have).

## Agenticness

- [ ] **[must-have]** Plug MCP servers into this product so it can use their tools
- [ ] **[must-have]** Connect an agent via an official MCP server
- [ ] **[must-have]** Drive the product through a documented public API
- [ ] **[must-have]** Delegate tasks to a built-in AI assistant inside the product
- [ ] **[should-have]** Point an agent at llms.txt or agent-oriented docs
- [ ] **[should-have]** Run the product headlessly / in CI for automation
- [ ] **[should-have]** Use an official CLI
- [ ] **[should-have]** Issue scoped/least-privilege API credentials for an agent
- [ ] **[should-have]** Build against official SDKs
- [ ] **[should-have]** Subscribe to events via webhooks
- [ ] **[should-have]** Get AI-generated insights and suggestions from my data inside the product
- [ ] **[should-have]** Set up automations that run autonomously in the background
- [ ] **[should-have]** Operate the product with natural-language commands
- [ ] **[should-have]** Explore an interactive API reference with runnable examples
- [ ] **[should-have]** Download a machine-readable API spec (OpenAPI or equivalent)
- [ ] **[should-have]** Rely on versioned APIs with a documented deprecation policy
- [ ] **[nice-to-have]** Test against a sandbox environment without touching production data

## Autofix agents

- [ ] **[must-have]** I turn a review finding into an applied fix — a committed patch or an agent-generated follow-up — without leaving the PR
- [ ] **[should-have]** The reviewer holds the line on AI-generated PRs — it verifies agent-authored code at a volume no human team could review
- [ ] **[should-have]** I define custom agentic pre-merge checks in plain language — 'docs updated', 'tests cover new paths' — that run on every PR
- [ ] **[should-have]** Review findings hand off cleanly to my coding agent — copyable fix prompts or direct integration with Claude Code, Cursor, or Codex

## Automation depth

- [ ] **[must-have]** Define rules that trigger actions automatically on events
- [ ] **[should-have]** Perform bulk operations across many items at once
- [ ] **[should-have]** Schedule recurring jobs or workflows
- [ ] **[nice-to-have]** Version, review, and roll back my automations

## Codebase understanding

- [ ] **[must-have]** Review comments reflect the whole repository — call sites, related modules, existing conventions — not just the changed hunks
- [ ] **[should-have]** The reviewer understands changes that span multiple repositories or a large monorepo and reviews them coherently
- [ ] **[should-have]** The reviewer builds a persistent memory of my team's conventions and past review decisions and applies it to future PRs

## Interaction

- [ ] **[should-have]** I reply to the reviewer in the PR thread to ask questions, get explanations, or issue commands — and it answers in context
- [ ] **[nice-to-have]** I control when reviews run — skip drafts, trigger on demand, filter by branch or label — so the bot shows up only when wanted

## Openness

- [ ] **[must-have]** Export all of my data in open formats and leave
- [ ] **[must-have]** Self-host the core product
- [ ] **[should-have]** Do everything through the API that I can do in the UI
- [ ] **[should-have]** Read the product's source under an open license

## Pr integration

- [ ] **[must-have]** The reviewer installs as a GitHub/GitLab app and posts reviews as native inline comments on my pull requests within minutes
- [ ] **[must-have]** Review comments include committable suggested diffs I can apply with one click
- [ ] **[should-have]** Every PR gets an auto-generated summary and change walkthrough so human reviewers orient fast
- [ ] **[should-have]** Pushing new commits triggers an incremental re-review that tracks what was fixed instead of repeating old comments

## Privacy posture

- [ ] **[must-have]** Prevent my data from being used to train AI models
- [ ] **[should-have]** Choose where my data is stored (region/residency)
- [ ] **[should-have]** Control data retention and deletion
- [ ] **[should-have]** Opt out of telemetry and usage tracking

## Quality gates

- [ ] **[should-have]** The reviewer can gate merges — a required status check or blocking review that enforces resolution of critical findings
- [ ] **[nice-to-have]** I see dashboards of findings, acceptance rates, and review coverage across my org

## Review accuracy

- [ ] **[must-have]** The reviewer catches real bugs in my PR — logic errors, race conditions, broken edge cases — not just style nits
- [ ] **[must-have]** The reviewer keeps noise low — few false positives, deduplicated comments, severity labels — so my team doesn't tune it out
- [ ] **[should-have]** Push back on a bad review comment and the reviewer learns — it stops repeating the same rejected feedback
- [ ] **[should-have]** Reviews flag security problems in the diff — injection risks, leaked secrets, insecure patterns — alongside functional bugs

## Surfaces

- [ ] **[should-have]** I run reviews from a CLI against local diffs or in CI scripts, with machine-readable output my tooling can consume
- [ ] **[should-have]** I get the same review inside my IDE before I push, catching issues while the code is still in my editor

## Workflow config

- [ ] **[must-have]** I configure the reviewer with a versioned config file in my repo — path filters, per-path instructions, review profiles
- [ ] **[should-have]** I encode my team's own review guidelines — natural-language rules, AST patterns, or linked style guides — and the reviewer enforces them
- [ ] **[nice-to-have]** I roll out org-level review defaults across hundreds of repos and manage exceptions centrally

---

Source: https://ultrametric.ai/productarena/arena/ai-code-review (evidence-graded verdicts for 6 products) · methodology: https://ultrametric.ai/productarena/methodology

Chips show the top 5 ranked products' current verdict on each requirement — ✓ full · ~ partial · ! disputed · — none · n/a not applicable.

Agenticness — how well agents can access and operate the productAgenticness· 17 items

How well agents can access and operate the product

ai-native userPlug MCP servers into this product so it can use their tools
Core requirement — weighs 3× in arena scoring · 1 of 5 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
must-have
ai-native userConnect an agent via an official MCP server
Core requirement — weighs 3× in arena scoring · 4 of 5 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
must-have
ai-native userDrive the product through a documented public API
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
must-have
ai-native userDelegate tasks to a built-in AI assistant inside the product
Core requirement — weighs 3× in arena scoring · 4 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
must-have
ai-native userPoint an agent at llms.txt or agent-oriented docs
Important, not disqualifying — weighs 2× in arena scoring · 5 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userRun the product headlessly / in CI for automation
Important, not disqualifying — weighs 2× in arena scoring · 2 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userUse an official CLI
Important, not disqualifying — weighs 2× in arena scoring · all 5 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userIssue scoped/least-privilege API credentials for an agent
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userBuild against official SDKs
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userSubscribe to events via webhooks
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userGet AI-generated insights and suggestions from my data inside the product
Important, not disqualifying — weighs 2× in arena scoring · 5 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userSet up automations that run autonomously in the background
Important, not disqualifying — weighs 2× in arena scoring · 4 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userOperate the product with natural-language commands
Important, not disqualifying — weighs 2× in arena scoring · 3 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userExplore an interactive API reference with runnable examples
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userRely on versioned APIs with a documented deprecation policy
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userTest against a sandbox environment without touching production data
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 1 of 3 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
nice-to-have

Autofix agents — stories about autofix agents in this arenaAutofix agents· 4 items

Stories about autofix agents in this arena

developerI turn a review finding into an applied fix — a committed patch or an agent-generated follow-up — without leaving the PR
Core requirement — weighs 3× in arena scoring · 5 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
must-have
ai-native userThe reviewer holds the line on AI-generated PRs — it verifies agent-authored code at a volume no human team could review
Important, not disqualifying — weighs 2× in arena scoring · 2 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userI define custom agentic pre-merge checks in plain language — 'docs updated', 'tests cover new paths' — that run on every PR
Important, not disqualifying — weighs 2× in arena scoring · 1 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userReview findings hand off cleanly to my coding agent — copyable fix prompts or direct integration with Claude Code, Cursor, or Codex
Important, not disqualifying — weighs 2× in arena scoring · 3 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have

Automation depth — how much of the product can run unattendedAutomation depth· 4 items

How much of the product can run unattended

ai-native userDefine rules that trigger actions automatically on events
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
must-have
ai-native userPerform bulk operations across many items at once
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userSchedule recurring jobs or workflows
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userVersion, review, and roll back my automations
Differentiator, not a dealbreaker — weighs 1× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
nice-to-have

Codebase understanding — how deeply the tool maps your repo — cross-file context, architecture awareness, historyCodebase understanding· 3 items

How deeply the tool maps your repo — cross-file context, architecture awareness, history

developerReview comments reflect the whole repository — call sites, related modules, existing conventions — not just the changed hunks
Core requirement — weighs 3× in arena scoring · 3 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
must-have
engineering leadThe reviewer understands changes that span multiple repositories or a large monorepo and reviews them coherently
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userThe reviewer builds a persistent memory of my team's conventions and past review decisions and applies it to future PRs
Important, not disqualifying — weighs 2× in arena scoring · 4 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have

Interaction — how you steer it — commands, replies, review conversations, configurability in the loopInteraction· 2 items

How you steer it — commands, replies, review conversations, configurability in the loop

developerI reply to the reviewer in the PR thread to ask questions, get explanations, or issue commands — and it answers in context
Important, not disqualifying — weighs 2× in arena scoring · 4 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
developerI control when reviews run — skip drafts, trigger on demand, filter by branch or label — so the bot shows up only when wanted
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 1 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
nice-to-have

Openness — open source, data portability, and self-hosting storiesOpenness· 4 items

Open source, data portability, and self-hosting stories

ai-native userExport all of my data in open formats and leave
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
must-have
ai-native userSelf-host the core product
Core requirement — weighs 3× in arena scoring · 3 of 5 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
must-have
ai-native userDo everything through the API that I can do in the UI
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userRead the product's source under an open license
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have

Pr integration — stories about pr integration in this arenaPr integration· 4 items

Stories about pr integration in this arena

developerThe reviewer installs as a GitHub/GitLab app and posts reviews as native inline comments on my pull requests within minutes
Core requirement — weighs 3× in arena scoring · 4 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
must-have
developerReview comments include committable suggested diffs I can apply with one click
Core requirement — weighs 3× in arena scoring · 4 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
must-have
developerEvery PR gets an auto-generated summary and change walkthrough so human reviewers orient fast
Important, not disqualifying — weighs 2× in arena scoring · 4 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
developerPushing new commits triggers an incremental re-review that tracks what was fixed instead of repeating old comments
Important, not disqualifying — weighs 2× in arena scoring · 2 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have

Privacy posture — data-handling and privacy storiesPrivacy posture· 4 items

Data-handling and privacy stories

ai-native userPrevent my data from being used to train AI models
Core requirement — weighs 3× in arena scoring · 4 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
must-have
ai-native userChoose where my data is stored (region/residency)
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userControl data retention and deletion
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
ai-native userOpt out of telemetry and usage tracking
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have

Quality gates — stories about quality gates in this arenaQuality gates· 2 items

Stories about quality gates in this arena

engineering leadThe reviewer can gate merges — a required status check or blocking review that enforces resolution of critical findings
Important, not disqualifying — weighs 2× in arena scoring · 1 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
engineering leadI see dashboards of findings, acceptance rates, and review coverage across my org
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 2 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
nice-to-have

Review accuracy — stories about review accuracy in this arenaReview accuracy· 4 items

Stories about review accuracy in this arena

developerThe reviewer catches real bugs in my PR — logic errors, race conditions, broken edge cases — not just style nits
Core requirement — weighs 3× in arena scoring · 1 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
must-have
engineering leadThe reviewer keeps noise low — few false positives, deduplicated comments, severity labels — so my team doesn't tune it out
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
must-have
developerPush back on a bad review comment and the reviewer learns — it stops repeating the same rejected feedback
Important, not disqualifying — weighs 2× in arena scoring · 1 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
security engineerReviews flag security problems in the diff — injection risks, leaked secrets, insecure patterns — alongside functional bugs
Important, not disqualifying — weighs 2× in arena scoring · 2 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have

Surfaces — where it meets your workflow — IDE, CLI, web, PR comments, CI checksSurfaces· 2 items

Where it meets your workflow — IDE, CLI, web, PR comments, CI checks

developerI run reviews from a CLI against local diffs or in CI scripts, with machine-readable output my tooling can consume
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
CodeRabbit Qodo cubic Greptile Graphite
should-have
developerI get the same review inside my IDE before I push, catching issues while the code is still in my editor
Important, not disqualifying — weighs 2× in arena scoring · 2 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have

Workflow config — stories about workflow config in this arenaWorkflow config· 3 items

Stories about workflow config in this arena

engineering leadI configure the reviewer with a versioned config file in my repo — path filters, per-path instructions, review profiles
Core requirement — weighs 3× in arena scoring · 2 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
must-have
engineering leadI encode my team's own review guidelines — natural-language rules, AST patterns, or linked style guides — and the reviewer enforces them
Important, not disqualifying — weighs 2× in arena scoring · 3 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
should-have
engineering leadI roll out org-level review defaults across hundreds of repos and manage exceptions centrally
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 1 of 6 products fully deliver this today
CodeRabbit Qodo cubic Greptile Graphite
nice-to-have

Full evidence behind every verdict lives on the arena page and each product page — chips above deep-link straight to the judged story.

🕵️AI Code Review arenaBuyer checklist

🤖Agenticness — how well agents can access and operate the productAgenticness· 17 items

🤖Autofix agents — stories about autofix agents in this arenaAutofix agents· 4 items

⚙️Automation depth — how much of the product can run unattendedAutomation depth· 4 items

🧭Codebase understanding — how deeply the tool maps your repo — cross-file context, architecture awareness, historyCodebase understanding· 3 items

💬Interaction — how you steer it — commands, replies, review conversations, configurability in the loopInteraction· 2 items

🔓Openness — open source, data portability, and self-hosting storiesOpenness· 4 items

🔌Pr integration — stories about pr integration in this arenaPr integration· 4 items

🔒Privacy posture — data-handling and privacy storiesPrivacy posture· 4 items

🧪Quality gates — stories about quality gates in this arenaQuality gates· 2 items

🧪Review accuracy — stories about review accuracy in this arenaReview accuracy· 4 items

🧩Surfaces — where it meets your workflow — IDE, CLI, web, PR comments, CI checksSurfaces· 2 items

⚙️Workflow config — stories about workflow config in this arenaWorkflow config· 3 items