Methodology
How ProductArena scores products
Evidence in, rankings out. Every claim traces back to a cited evidence item — vendor docs, GitHub, an independent community source, or a hands-on probe — and an LLM judge scores every (product, story) cell from that evidence alone, never outside knowledge.
Evidence tiers
probe (tested) > github (code) > community (independent) > claimed-docs (vendor claim)
⚿ auth-gated: a probe reached the live endpoint and hit a vendor sign-in wall (HTTP 401/403, OAuth) — verified reachable, untestable keylessly. Proof of life, never evidence of absence; the marker never changes a verdict or score.
Verdicts
full / partial / disputed / none / na, each with a 0–10 quality score and cited evidence ids.
Scoring
score = story.weight × quality × verdictFactor, summed over applicable (non-na) cells only.
Bias disclosure
The judge model is made by Anthropic; the ai-coding arena includes its own product. Full adversarial audit writeup on GitHub.
Popularity (not scored)
GitHub stars/npm/PyPI downloads, shown as a survival signal only — measures adoption, not AI-readiness, so it's never part of the PA Score or rankings.
Pricing tiers (not scored)
Delivered stories (full/partial) carry a free / paid / enterprise chip when the verdict's cited evidence or the vendor's own pricing evidence states which plan the capability needs — each chip quotes that gating evidence (e.g. “SSO on Enterprise plan only”).
unknown means the evidence never states gating — we never infer a tier from reputation or vibes, and silence is never rendered as “free”. An annotation layer only: it never moves a verdict, quality, or any score.
The PA Score
Formerly displayed as the "Arena Score" (and before that the "AI-Era Index") — same formula, new name.
Every leaderboard entry carries a PA Score (0–100, aiEra internally) — a weighted, renormalized blend of five components:
| agent-ready | ×0.30 — can an agent reach the product |
| API quality | ×0.20 — how good is that API surface |
| openness | ×0.20 — self-host, export, read the source |
| agentic app | ×0.15 — does the product act agentically itself |
| automation | ×0.15 — depth of rules/scheduling/bulk primitives |
n/a components are excluded and weights renormalized over the rest. See the full methodology (link above) for the exact formula and the case for leading with this number over raw coverage.
Confidence grades (A–D)
The letter next to a PA Score says how much of that score rests on evidence we tested ourselves versus evidence the vendor merely claims. It grades the receipts, not the product — a great product can carry a C simply because we haven't probed it deeply yet.
| A | broad story coverage and a high share of probe/community-tested verdicts |
| B | solid coverage, mostly tested — a few cells still rest on vendor docs alone |
| C | meaningful gaps: thin coverage or verdicts leaning on claimed docs |
| D | treat the score as provisional — little tested evidence behind it yet |
Computed from story coverage × tested-evidence share (lib/confidence.ts). Grades move as probes land — the fastest way to raise one is to submit reproducible evidence.
Claims integrity
One number per product for "does the vendor's website deliver what it promises?": we extract capability claims from the vendor's own docs/GitHub materials, map each onto this arena's stories, and reconcile them against our judge's independent verdicts.
testable = verified + unverified + contradicted
integrity = 100 × max(0, verified − 2 × contradicted) / testable
| verified | claim maps to a story with a probed/community-backed full or partial verdict — counts fully |
| unverified | full/partial verdict, but only the vendor's own claim backs it — inflates the denominator only |
| contradicted | our judge found disputed/none/na — each one cancels two verified claims (overpromising is worse than staying silent); the score is clamped at 0 |
| untestable | outside this arena's story taxonomy — excluded from both numerator and denominator (a taxonomy gap is never a mark for or against the product) |
Products with no testable claims are unscored (null, never a fabricated 0) and sort last in the claims-vs-reality ranking. Computed in lib/claimsIntegrity.ts; each product page's "Claims vs evidence" section shows the claim-by-claim breakdown.
Rankings, verdicts, and scores are research outputs derived from cited evidence at a point in time, provided "as is" — verify against the evidence before acting on them. See terms of use.