LLM Evals & Observability Arena
LLM Evals & Observability — procurement report
ProductArena · rankings as of 2026-09-14 · evidence as of 2026-09-14 · 8 products · 52 judged requirements · 416 judged cells
Methodology: Every product is judged against a shared taxonomy of user stories using cited evidence — hands-on probes > repository code > independent community sources > vendor claims — never opinion. Full writeup: https://ultrametric.ai/productarena/methodology
Leaderboard
| # | Product | PA Score | Coverage score | Applicable cells | Confidence |
|---|---|---|---|---|---|
| 1 | Braintrust | 37.7 | 43.8 | 52/52 | A |
| 2 | Cekura | 36.4 | 31.7 | 51/52 | B |
| 3 | W&B Weave | 35.2 | 41.8 | 49/52 | B |
| 4 | Helicone | 34.2 | 32.4 | 46/52 | B |
| 5 | Langfuse | 33.8 | 49.4 | 51/52 | B |
| 6 | Arize Phoenix | 31.6 | 43.5 | 51/52 | B |
| 7 | LangSmith | 22.9 | 38.6 | 51/52 | B |
| 8 | Galileo | 18.5 | 24.8 | 50/52 | B |
PA Score = agent-readiness blend (see methodology). Coverage score = weighted share of judged requirements met. Confidence = how much of the score rests on tested vs claimed evidence (A–D).
Uncertainty note
This arena is currently a close race: Braintrust (37.7) vs Cekura (36.4), a gap of 1.3 PA Score. The ordering was re-checked with extra judge samples: 34 decisive cells were triple-judged and 5 came back unstable. Treat the #1/#2 ordering as contested — shortlist both.
Buyer checklist (RFP)
The arena's 52 judged user stories as requirements, grouped by theme. Priorities mirror the story weights our scoring uses (3 = must-have, 2 = should-have, 1 = nice-to-have). Interactive version with per-requirement verdicts for the top products: /arena/llm-evals-observability/checklist
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
- ai-native userPlug MCP servers into this product so it can use their toolsmust-have
- ai-native userConnect an agent via an official MCP servermust-have
- ai-native userDrive the product through a documented public APImust-have
- ai-native userDelegate tasks to a built-in AI assistant inside the productmust-have
- ai-native userPoint an agent at llms.txt or agent-oriented docsshould-have
- ai-native userRun the product headlessly / in CI for automationshould-have
- ai-native userUse an official CLIshould-have
- ai-native userIssue scoped/least-privilege API credentials for an agentshould-have
- ai-native userBuild against official SDKsshould-have
- ai-native userSubscribe to events via webhooksshould-have
- ai-native userGet AI-generated insights and suggestions from my data inside the productshould-have
- ai-native userSet up automations that run autonomously in the backgroundshould-have
- ai-native userOperate the product with natural-language commandsshould-have
- ai-native userExplore an interactive API reference with runnable examplesshould-have
- ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)should-have
- ai-native userRely on versioned APIs with a documented deprecation policyshould-have
- ai-native userTest against a sandbox environment without touching production datanice-to-have
Alerting dashboards — stories about alerting dashboards in this arenaAlerting dashboards
Stories about alerting dashboards in this arena
- ml engineerBuild custom dashboards over latency, error, cost, and eval-score metricsshould-have
- developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or emailshould-have
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
- ai-native userDefine rules that trigger actions automatically on eventsmust-have
- ai-native userPerform bulk operations across many items at onceshould-have
- ai-native userSchedule recurring jobs or workflowsshould-have
- ai-native userVersion, review, and roll back my automationsnice-to-have
Cost monitoring — stories about cost monitoring in this arenaCost monitoring
Stories about cost monitoring in this arena
- developerSee cost and token usage per request, model, and time period in dashboardsmust-have
- developerAttribute cost and usage to users, sessions, and features via custom metadatashould-have
Data access export — stories about data access export in this arenaData access export
Stories about data access export in this arena
- developerBulk-export traces and datasets to blob storage or my data warehouseshould-have
Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets
Measuring quality — datasets, eval runs, regression tracking
- ml engineerCompare eval runs side by side to catch regressions between prompt or model versionsmust-have
- ml engineerScore outputs with configurable LLM-as-a-judge evaluatorsmust-have
- ml engineerCurate datasets from production traces and run offline evaluations against themmust-have
- ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the resultsshould-have
- ml engineerRoute outputs to human annotation queues for review and labelingshould-have
- developerRun evals in CI and gate deployments on their resultsshould-have
- ml engineerWrite custom code-based scorers and metrics for my evaluationsshould-have
- ml engineerRun evaluators continuously on live production traffic, not just offline datasetsshould-have
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
- ai-native userExport all of my data in open formats and leavemust-have
- ai-native userSelf-host the core productmust-have
- ai-native userDo everything through the API that I can do in the UIshould-have
- ai-native userRead the product's source under an open licenseshould-have
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
- ai-native userPrevent my data from being used to train AI modelsmust-have
- ai-native userChoose where my data is stored (region/residency)should-have
- ai-native userControl data retention and deletionshould-have
- ai-native userOpt out of telemetry and usage trackingshould-have
Prompt management — stories about prompt management in this arenaPrompt management
Stories about prompt management in this arena
- developerVersion prompts and deploy changes to production without shipping codemust-have
- developerIterate on prompts in a playground against real models and variablesshould-have
Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation
Instrumenting code and tracing requests end to end
- ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my appmust-have
- developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary formatmust-have
- developerCapture traces of my LLM calls with inputs, outputs, latency, and token usagemust-have
- developerMask or redact sensitive data before it is stored in tracesshould-have
- developerInstrument apps in both Python and JS/TS with officially supported SDKsshould-have
- developerTrace multi-step agent runs as nested spans grouped into sessions or threadsshould-have
- developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDKshould-have
- developerCapture multimodal payloads (images, audio, files) inside my tracesnice-to-have
Appendix: recorded probes
Hands-on probe recordings — transcripts/videos a human can replay, the strongest evidence tier. Watch them at https://ultrametric.ai/productarena/proofs
- Cekura
uvx --from 'cekura[cli]' cekura --helpterminal · recorded 2026-09-14 · exit 0 - Cekura
curl -s https://docs.cekura.ai/llms.txt | head -6terminal · recorded 2026-09-14 · exit 0 - Cekura
curl -si -X POST https://api.cekura.ai/mcp -H 'Content-Type: application/json' -d '<jsonrpc initialize>'terminal · recorded 2026-09-14 · exit 0
Cite as: ProductArena by Ultrametric Inc, LLM Evals & Observability arena, rankings as of 2026-09-14 — https://ultrametric.ai/productarena/arena/llm-evals-observability
License: © 2026 Ultrametric Inc. Brief quotation of individual verdicts, scores, or evidence excerpts is permitted with attribution to "ProductArena by Ultrametric Inc (ultrametric.ai/productarena)", as is use of the data to evaluate, contest, or contribute corrections. Bulk copying, redistribution, or use to build competing datasets requires prior written permission (see DATA-LICENSE in the repository).
No liability: rankings, verdicts, and scores are research outputs derived from the cited evidence at a point in time, provided "as is", without warranties. Ultrametric Inc accepts no responsibility for procurement, purchasing, or other decisions made in reliance on them — verify against the cited evidence before acting (https://ultrametric.ai/productarena/terms).