Braintrust vs Arize Phoenix

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Arize Phoenix
   Braintrust~partialprobed6/10

   A live probe confirms llms.txt exists at braintrust.dev/llms.txt returning 200 with a concise agent-oriented description, directly satisfying the story's core requirement. However, docs.md and OpenAPI/machine-readable doc endpoints return 404, indicating incomplete agent-oriented documentation coverage beyond the single llms.txt file. missing for 10: full docs-as-markdown or OpenAPI endpoints for deeper agent consumption, independent confirmation that agents successfully use llms.txt in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://www.braintrust.dev/llms.txt # Braintrust > Braintrust is the active observability platform for instrume…”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.braintrust.dev/docs.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Arize Phoenix✓fullprobed9/10

   Direct probe evidence confirms llms.txt returns HTTP 200 with agent-oriented content, and docs pages are also available in markdown form (.md) with a documentation index, both explicitly designed for agent/LLM consumption. Missing for 10: no independent/community confirmation of an agent actually consuming these files successfully.

   • [probe] “PROBE llms.txt: HTTP 200 at https://arize.com/llms.txt # Arize — AI & Agent Engineering > Arize is the AI engineering platform for teams bu…”
   • [probe] “PROBE docs-md: HTTP 200 at https://arize.com/docs/phoenix.md > ## Documentation Index > Fetch the complete documentation index at: https://a…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust supports headless/CI usage via code-based Eval() runs, an API for programmatic control, a CLI (`bt`) for auth/eval/log operations from the terminal, and explicit documentation of CI/CD integration to run evals on every pull request. This combination clearly enables automated, non-interactive operation in pipelines. Missing for 10: independent hands-on confirmation of CI pipeline usage beyond vendor docs.

   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed7/10

   Phoenix ships a CLI (`uvx arize-phoenix serve`), a Docker image pinned for production deployments, and a Python SDK with dataset evaluators explicitly described as forming 'an evaluation harness similar to a unit test suite,' all of which support headless/automated use in CI-like pipelines. However, there is no explicit CI/CD example (e.g., GitHub Actions workflow) or documentation of running evaluations as an automated CI gate. Missing for 10: explicit CI pipeline integration examples/docs, evidence of automated test-gating in CI, independent confirmation of headless CI usage.

   • [claimed-docs] “$ uvx arize-phoenix serve”
   • [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
   • [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
   • [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”

   proof ↗⚑ flag

3. ai-native userConnect an agent via an official MCP server

   weight 3 · round drawn
   Braintrust✓fullprobed8/10

   Braintrust documents an official MCP server that lets Claude Code, Cursor, Codex, VS Code, and other MCP-compatible clients query logs, author prompts/scorers, configure monitoring, and run evals — a clear server-side MCP offering that agents can connect to. Missing for 10: independent/hands-on third-party confirmation that the MCP server works as documented (only first-party docs and a probe confirm its existence).

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   Arize Phoenix✓fullprobed8/10

   Phoenix documents an official remote MCP server exposing a `/mcp` endpoint that MCP-compatible clients (Claude Code, Cursor, VS Code, etc.) can connect to for searching and operating on projects, traces, datasets, experiments, prompts, and annotations. This is first-party documentation confirming the exact capability. Missing for 10: independent/hands-on community corroboration of the MCP server working in practice, and details on authentication/setup robustness.

   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
   • [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”

   proof ↗⚑ flag

4. ai-native userUse an official CLI

   weight 2 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust ships an official `bt` CLI, documented with a dedicated quickstart, allowing authentication, tracing coding-agent sessions, running evals, browsing/querying logs, syncing data, and managing functions from the terminal — directly matching AI-native/agentic CLI usage. Missing for 10: independent/hands-on community corroboration of the CLI itself (community evidence is generic, not CLI-specific).

   • [claimed-docs] “Install the `bt` CLI to set up, instrument, and run Braintrust with your coding agent”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed6/10

   Phoenix ships an official CLI (`@arizeai/phoenix-cli` via `npx -y @arizeai/phoenix-cli setup`) for auto-instrumentation setup, plus a `uvx arize-phoenix serve` command to launch the server, confirming a genuine first-party CLI exists for agentic/automated workflows. However, the CLI's documented scope is narrow (setup + serve) rather than covering the full product surface (datasets, evals, prompts, experiments), and there's no independent/community confirmation of its use. Missing for 10: broader CLI command coverage across Phoenix's feature set, independent hands-on verification of the CLI in practice.

   • [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
   • [claimed-docs] “$ uvx arize-phoenix serve”
   • [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app's root directory.”
   • [claimed-docs] “start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app’s root directory”

   proof ↗⚑ flag

5. ai-native userDrive the product through a documented public API

   weight 3 · round to Braintrust
   Braintrust✓fullprobed8/10

   Braintrust documents a full public API reference for managing projects, experiments, datasets, prompts, scorers, and logging programmatically (braintrust-docs-11, -34), backed by SDKs (braintrust-gh-1/2), a CLI (braintrust-docs-33), and MCP server for agent-driven access (braintrust-docs-9, -25). Missing for 10: no discoverable OpenAPI/swagger spec (braintrust-probe-3 shows 404s) and no independent third-party corroboration of API robustness beyond docs.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Arize Phoenix✓fullprobed7/10

   Phoenix documents a full SDK API reference (tracing decorators, dataset/experiment APIs, prompt sync SDKs), OTLP-based trace ingestion, a CLI (phoenix-cli), and an official remote MCP endpoint that lets AI-native clients search/query/operate on projects, traces, datasets, experiments, and prompts—together constituting a documented, programmatically-drivable public API surface. The openapi.json probe returned 404s at standard paths, meaning there's no discoverable formal REST/OpenAPI spec, so the API surface is SDK/CLI/MCP-centric rather than a single canonical REST API. Missing for 10: a published OpenAPI/REST spec, independent third-party confirmation of API robustness.

   • [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
   • [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
   • [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
   • [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
   • [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
   • [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
   • [claimed-docs] “$ uvx arize-phoenix serve”

   proof ↗⚑ flag

6. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   Braintrust—none0/10

   The evidence describes Braintrust's general API, CLI, and MCP integrations but contains no mention of scoped, role-based, or least-privilege API key/credential issuance for agents; the only security-related item is a breach report telling customers to rotate keys, which does not demonstrate a scoping/least-privilege capability.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [community] “AI evaluation startup Braintrust confirms breach, tells every customer to rotate sensitive keys.”

   proof ↗⚑ flag

   Arize Phoenix—none0/10

   No evidence in the pack describes scoped or least-privilege API key/credential issuance for agents; Phoenix's docs cover tracing, evaluation, prompt management, and an MCP endpoint, but nothing about credential scoping or access control granularity.

   ⚑ flag

7. ai-native userBuild against official SDKs

   weight 2 · round drawn
   Braintrust✓fullprobed8/10

   Braintrust ships official JS SDK (github examples using Eval()), an API reference for programmatic access, a CLI (bt), and an MCP server, giving AI-native developers multiple official, documented ways to build against the platform. One community comment calls docs 'sparse' but this is general skepticism, not a concrete contradiction of SDK functionality. missing for 10: independent hands-on validation of the SDK itself (beyond vendor docs/examples) and a public OpenAPI spec (probe found 404s).

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”
   • [community] “Braintrust: Interesting approach to evaluation but feels like an early-stage product. Documentation is sparse and integration options are li…”

   proof ↗⚑ flag

   Arize Phoenix✓fullclaimed8/10

   Phoenix ships an official, modular Python SDK plus client SDKs for tracing, prompts, evaluations, and datasets, with OTLP-based tracing decorators, auto-instrumentation, and dedicated SDK/API reference docs. Coverage spans core agentic workflows (tracing, evals, prompt management, datasets/experiments) and is corroborated by GitHub repo and docs pages, though independent hands-on SDK usage reports are thin. Missing for 10: broader independent/third-party validation of SDK usage beyond official docs, and explicit multi-language SDK support details beyond Python/JS CLI.

   • [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
   • [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
   • [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
   • [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
   • [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “Prompts in Code - Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
   • [github] “Datasets - Create versioned datasets of examples for experimen”

   proof ↗⚑ flag

8. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Braintrust—none0/10

   No evidence in the pack mentions webhooks or any event-subscription mechanism; Braintrust's documented interfaces are API, CLI, MCP server, and UI, none of which are shown to support webhook subscriptions.

   ⚑ flag

   Arize Phoenix—none0/10

   No evidence anywhere in the pack of a webhook subscription mechanism; Phoenix's integration surface is OTLP tracing ingestion, an MCP server, and SDKs, but nothing about outbound event webhooks for subscribing to Phoenix events.

   ⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust ships Loop, an AI agent that investigates logs via natural language and can build scorers/datasets/dashboards, plus Patterns which proactively scans trace backlogs for recurring issues and suggests fixes, and Topics which auto-clusters/classifies traces against business dimensions — all directly generating AI insights/suggestions inside the product. Missing for 10: independent/hands-on validation of Loop/Patterns quality (evidence is vendor docs only) and no third-party review confirming suggestion usefulness.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Define the dimensions that matter to your business, like use case, customer segment, compliance, or tone. Topics continuously clusters every…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review. Each trace is analyzed by facets that extract sho”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed5/10

   Phoenix offers LLM-as-judge evaluators that return explanations and reasoning alongside scores, and lets users explore traces to spot systematic issues, which counts as AI-generated signal on the data — but this is evaluator-driven scoring, not proactive, unprompted insights/suggestions surfaced automatically inside the product UI. missing for 10: evidence of autonomous/proactive insight generation (e.g., auto-detected anomalies, recommended fixes, natural-language summaries of trends) rather than only judge-model scoring/explanations tied to evals a user configures.

   • [claimed-docs] “Built-in Explanations — all Phoenix LLM evaluations return explanations by default for better results and richer signals”
   • [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model's full reasoning, the final scores, a…”
   • [claimed-docs] “Use Phoenix's trace viewer to explore evaluation traces, identify systematic biases, and continuously improve evaluator performance.”
   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust supports autonomous background automation via 'Patterns', which runs Loop (its AI agent) on a schedule against the trace backlog to surface recurring issues and suggested fixes, plus online scoring that runs asynchronously on production traces and CI/CD-triggered evals. However, this is narrower than general-purpose autonomous automations — it's scoped to trace analysis/scoring rather than arbitrary user-defined background tasks, and there's no independent/hands-on corroboration of Patterns actually running unattended. Missing for 10: broader/customizable scheduled automation beyond Patterns and online scoring, independent evidence of reliability, and detail on failure handling or configurability of these background jobs.

   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”

   proof ↗⚑ flag

   Arize Phoenix—none0/10

   Phoenix's evidence covers tracing, evaluation, prompt management, and datasets, but nothing describes scheduled or autonomous background automations (e.g., recurring eval jobs, alerting rules, or triggers) that run without user initiation. The evaluation/experiment features described are invoked by the user (via SDK/UI), not autonomous background processes.

   ⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed7/10

   Braintrust ships "Loop," a built-in AI agent that can investigate logs via natural language, build scorers/datasets/dashboards, and "Patterns" which runs Loop automatically on trace backlogs to surface issues and fixes — a clear built-in AI assistant delegated to perform tasks within the product. missing for 10: independent/hands-on user validation of Loop's task delegation (community evidence is thin/mixed and doesn't specifically address Loop), and more detail on the scope/limits of tasks it can autonomously complete.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Loop is Braintrust's AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”

   proof ↗⚑ flag

   Arize Phoenix—none0/10

   Phoenix is an observability/evaluation platform; the evidence describes tracing, evals, prompt management, datasets, and an MCP server that lets *external* agents (Claude Code, Cursor, etc.) operate on Phoenix data — not a built-in AI assistant living inside Phoenix that users delegate tasks to. No docs mention any embedded assistant/chat agent within the Phoenix UI itself.

   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
   • [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app's root directory.”
   • [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”

   proof ↗⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round to Braintrust
   Braintrust✓fullprobed8/10

   Braintrust's Loop agent lets users investigate logs and build scorers/datasets/dashboards via natural language instead of writing queries, and this is exposed through an official MCP server so natural-language commands work from Claude Code, Cursor, Codex, etc. Missing for 10: independent hands-on validation of natural-language command accuracy/reliability and broader coverage beyond Loop's stated scope.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Loop is Braintrust's AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   Arize Phoenix✓fullprobed7/10

   Phoenix documents a dedicated remote MCP integration letting any MCP-compatible client (Claude Code, Cursor, VS Code, etc.) operate on projects, traces, datasets, experiments, prompts, and annotations via natural-language driven agents, and also supports agent-driven CLI setup. This directly satisfies AI-native natural-language operability. Missing for 10: independent/hands-on validation of the MCP command experience and more detail on the breadth/reliability of natural-language operations beyond the docs claim.

   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
   • [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
   • [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Braintrust—none0/10

   Braintrust has an API reference page (braintrust-docs-11/34) but the evidence pack shows no interactive or runnable-example capability — probes for an OpenAPI/swagger spec (needed for typical interactive API explorers) all returned 404, and there's no mention of a try-it-now console or embedded runnable snippets in the API reference itself. missing for 10: evidence of an interactive API console, runnable code snippets embedded in the reference, or a machine-readable OpenAPI spec powering such interactivity.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.braintrust.dev/docs.md”

   proof ↗⚑ flag

   Arize Phoenix—none0/10

   Docs mention an 'sdk-api-reference' page listing decorators and SDK features, but there is no evidence of an interactive, runnable API reference (e.g., a Swagger/OpenAPI explorer or live code sandbox); a direct probe for OpenAPI/swagger specs returned 404 on all candidate paths, indicating no such interactive reference is discoverable.

   • [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
   • [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
   • [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
   • [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   Braintrust—none0/10

   Braintrust documents a REST API (api-reference) but explicit probes for OpenAPI/swagger specs at all standard paths returned 404, and no docs mention a downloadable machine-readable spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Arize Phoenix—none0/10

   A direct probe for OpenAPI/swagger endpoints at Phoenix's domain returned 404 for all candidate paths, and no docs mention a downloadable machine-readable API spec. Missing for 10: any OpenAPI/Swagger spec, documented REST API reference, or SDK-generated schema.

   • [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Arize Phoenix
   Braintrust~partialclaimed4/10

   Braintrust supports evals/experiments/datasets and separate projects that could function as a sandbox for testing without hitting production logs, and self-hosting offers infrastructure isolation, but there is no explicit documentation of a dedicated 'sandbox environment' or isolation guarantee preventing test runs from touching production data. missing for 10: explicit sandbox/staging environment concept, documentation on isolating test evals from production datasets/logs, and independent confirmation that eval runs never write to production stores.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed5/10

   Phoenix supports self-hosting with full data isolation (data never leaves your infra) and lets you organize traces into separate 'projects', collect data 'from production, staging, evaluations, and even manually' into datasets, and run experiments/evaluations against those datasets rather than live traffic — all of which let a user test without touching real production data. However, there's no explicit documented 'sandbox environment' feature, no isolated test-instance provisioning, and no guidance on separating staging from prod deployments beyond generic self-hosting flexibility. Missing for 10: explicit sandbox/staging environment tooling, isolation guarantees beyond self-hosting, and independent confirmation that experiments truly never touch production data.

   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
   • [claimed-docs] “Projects Organize traces into separate projects for different applications”
   • [claimed-docs] “Organize traces into separate projects for different applications”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Braintrust—none0/10

   There is an API reference documenting functionality, but no evidence of API versioning scheme or any documented deprecation policy; probes even show no OpenAPI spec discoverable at standard locations. Missing for 10: versioning scheme documentation, explicit deprecation policy, changelog/migration guides.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Arize Phoenix—none0/10

   Evidence shows only generic container/image version pinning (e.g., 'version-8.0.0' Docker tags) but no documented API versioning scheme or deprecation policy for Phoenix's SDK/API; an OpenAPI probe also returned 404s, finding no formal API spec to review versioning against.

   • [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”

   proof ↗⚑ flag

Monitoring

1. ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust captures latency, token/cost, and eval-score metrics via tracing and experiments (braintrust-docs-1, braintrust-docs-27), and one doc line explicitly says you can 'ask Braintrust's AI agent to investigate your data and build scorers, datasets, and dashboards' (braintrust-docs-18), implying custom dashboard creation. However there is no dedicated dashboard-building documentation, no detail on customizing charts/widgets, and no explicit mention of error-rate metrics or alerting tied to dashboards. Missing for 10: dedicated dashboard-configuration docs, explicit error-metric support, and independent/hands-on evidence of building a custom dashboard.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”

   proof ↗⚑ flag

   Arize Phoenix—none0/10

   Phoenix's docs describe tracing, span-level latency/cost/token data, and eval scoring, but nowhere is there evidence of a dashboard-building feature (custom charts, widgets, or dashboard layouts) that lets an ML engineer aggregate these metrics into a dashboard view. The evidence only covers per-trace exploration, filtering spans, and projects/sessions organization, not dashboard construction or alerting.

   ⚑ flag

2. developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email

   weight 2 · round drawn
   Braintrust—none0/10

   The evidence pack covers tracing, evals, online scoring, Loop/Patterns/Topics analysis, and self-hosting, but contains no mention of configurable alerts, thresholds, or notification integrations (Slack, PagerDuty, email) for error rates, cost spikes, or eval-score drops.

   ⚑ flag

   Arize Phoenix—none0/10

   The evidence pack covers tracing, evaluation, prompt management, and dataset/experiment features but contains no mention of alerting, notification channels (Slack/PagerDuty/email), or threshold-based triggers for error rates, cost, or eval-score drops.

   ⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed7/10

   Braintrust evals run against entire datasets in one operation, Patterns scans the full trace backlog on a schedule, Topics classifies every trace automatically, and the CLI/API support bulk sync ('bt sync pull'), bulk log queries, and programmatic management of experiments/datasets/prompts at scale. Missing for 10: explicit documentation of bulk edit/delete/update operations on datasets or logs, and independent (non-vendor) confirmation of bulk-operation performance at scale.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed5/10

   Phoenix supports bulk-style operations through datasets (grouping many traces/examples), running experiments and evaluations across thousands of items with built-in concurrency/retry handling, and MCP-based querying/operating on projects, traces, datasets, and prompts. However, there is no explicit documentation of a bulk UI action (e.g., multi-select delete/tag/export) or a dedicated batch API for arbitrary bulk edits across items. missing for 10: explicit bulk UI operations (multi-select edit/delete/export), a documented batch API for arbitrary item updates, and independent/hands-on confirmation of large-scale bulk actions beyond evaluation/experiment runs.

   • [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust supports several automatic, event-driven behaviors — online scoring runs automatically on every new production trace, evals run automatically in CI/CD on pull requests, and Patterns runs Loop against traces on a schedule — but these are fixed built-in automations rather than a general user-defined 'if event then action' rules engine (e.g., no evidence of custom triggers/webhooks/alerts tied to arbitrary conditions). Missing for 10: a documented rules/alerting framework letting users define arbitrary trigger conditions and custom actions, and independent confirmation of this automation depth.

   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”

   proof ↗⚑ flag

   Arize Phoenix—none0/10

   Phoenix's evidence covers tracing, evaluation, datasets, prompt management, and MCP integration, but nothing describes a rules/triggers engine that automatically fires actions on events (e.g., alerting, auto-remediation, webhooks on thresholds). The axis is plausible for an observability platform, but no such capability is evidenced.

   ⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust documents 'Patterns,' which runs Loop on the trace backlog 'on a schedule' to find recurring issues, and separately supports triggering evals via CI/CD — both are forms of recurring automation. However, there's no evidence of general-purpose, user-configurable recurring job/workflow scheduling (e.g., cron-like scheduling for arbitrary evals, datasets, or pipelines) beyond this specific built-in feature. Missing for 10: user-defined scheduling API/UI for arbitrary evals or workflows, documentation of schedule configuration options, and independent confirmation of reliability.

   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”

   proof ↗⚑ flag

   Arize Phoenix—none0/10

   Phoenix is an observability/evaluation platform for LLM apps (tracing, evals, prompt management, datasets) but no evidence describes any scheduler, cron-like job runner, or recurring workflow automation feature.

   ⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round drawn
   Braintrust~partialclaimed5/10

   Braintrust documents versioned datasets and immutable, comparable experiment records, plus CLI/API management of prompts, scorers, and functions, which gives some version/review capability for its automation artifacts (evals, scorers, prompts). However, there is no explicit documentation of a rollback mechanism for automations (e.g., reverting a prompt, scorer, or online-scoring rule to a prior version) or a review/approval workflow before rollback. Missing for 10: explicit rollback functionality, version history UI/CLI commands, and independent confirmation that rollback works in practice.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed5/10

   Phoenix offers prompt versioning ('Version, store, and deploy prompts', 'you can version prompts, test prompt variants... before rolling them out') and dataset/experiment comparisons across app versions, which cover review and some rollback-like workflow for prompts. However, Phoenix is an observability/eval platform, not an automation/workflow builder, so there's no evidence of versioning or rolling back full agentic automations/pipelines themselves — only prompts and datasets. Missing for 10: explicit rollback mechanism for automations/workflows (not just prompts), audit trail of automation changes, and evidence of reverting a live automation to a prior version.

   • [claimed-docs] “Prompt Management — Version, store, and deploy prompts”
   • [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
   • [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
   • [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”

   proof ↗⚑ flag

Cost tracking

1. developerAttribute cost and usage to users, sessions, and features via custom metadata

   weight 2 · round drawn
   Braintrust~partialclaimed4/10

   Braintrust's tracing captures token usage and custom metadata for every request, which is the raw ingredient for cost attribution, but no evidence describes dashboards, filters, or reports that actually roll up cost/usage by user, session, or feature tags. missing for 10: explicit cost-attribution UI/reporting by custom metadata dimensions, user/session/feature breakdown examples, and independent confirmation this works in practice.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed4/10

   Phoenix documents token-usage breakdowns per LLM call and lets you organize traces into projects and group them into sessions/conversations, which provides some basis for usage attribution by session/application. However, there is no evidence of a general custom-metadata/tagging mechanism for arbitrary attribution to users or features, nor any cost aggregation/dashboard that rolls up spend by user, session, or feature tag. missing for 10: explicit custom metadata/tagging API for attributing spans to users or features, cost-specific aggregation/dashboard views by user/session/feature, documentation showing cost roll-ups beyond raw token counts.

   • [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
   • [claimed-docs] “Projects Organize traces into separate projects for different applications”
   • [claimed-docs] “Sessions Group related traces into conversations and user sessions”
   • [claimed-docs] “Organize traces into separate projects for different applications”
   • [claimed-docs] “Group related traces into conversations and user sessions”

   proof ↗⚑ flag

2. developerSee cost and token usage per request, model, and time period in dashboards

   weight 3 · round drawn
   Braintrust~partialclaimed5/10

   Braintrust's tracing explicitly captures token usage per request (braintrust-docs-1) and traces/spans can be browsed in the UI or terminal (braintrust-docs-14/19/35), which supports basic usage visibility, but the evidence never mentions a cost metric, cost calculation, or a dedicated dashboard aggregating cost/tokens by model or time period. Missing for 10: explicit cost tracking/computation, dashboard views aggregating usage by model, dashboard views aggregating usage by time period, and any independent confirmation of these dashboard features.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`]”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed5/10

   Phoenix provides a token usage breakdown per LLM call/span (docs-46) and organizes traces into projects and sessions (docs-15/16/48/49), which supports some per-model and per-time grouping, but there is no evidence of aggregated cost dashboards, dollar-cost calculations, or dedicated per-model/time-period cost views — only per-span token counts. missing for 10: dedicated cost dashboards showing dollar cost (not just tokens), aggregated views by model and by time period, and any UI screenshot/independent confirmation of such dashboards.

   • [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
   • [claimed-docs] “Projects Organize traces into separate projects for different applications”
   • [claimed-docs] “Sessions Group related traces into conversations and user sessions”
   • [claimed-docs] “Organize traces into separate projects for different applications”
   • [claimed-docs] “Group related traces into conversations and user sessions”

   proof ↗⚑ flag

Data export

1. developerBulk-export traces and datasets to blob storage or my data warehouse

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust supports exporting logs/traces via CLI (`bt sync pull`, download as CSV/JSON) and a full API for programmatic access to traces, datasets, and experiments, which developers could pipe into blob storage or a warehouse themselves. However, there is no documented native connector or bulk-export feature specifically targeting blob storage (S3/GCS) or data warehouses (Snowflake/BigQuery) — missing for 10: dedicated warehouse/blob-storage export integrations, scheduled/automated bulk export pipelines, and independent confirmation of large-scale export performance.

   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”

   proof ↗⚑ flag

   Arize Phoenix—none0/10

   The evidence pack covers tracing, datasets, evaluation, prompt management, and self-hosting, but contains no mention of bulk-exporting traces or datasets to blob storage (S3, GCS) or a data warehouse (Snowflake, BigQuery, etc.). The closest evidence is generic dataset/SDK querying (docs-25, docs-30) which is not the same as a bulk export/warehouse-sync feature.

   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”

   proof ↗⚑ flag

Ai eval ops

1. ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust provides programmatic APIs and MCP/CLI tooling that let an agent create datasets, trigger evals, and read back results: the API supports creating/managing datasets and experiments, the SDK's Eval() function runs evals programmatically, and the MCP server/CLI explicitly let agents like Claude Code/Cursor 'run evals' and 'query logs' from within their workflow. missing for 10: a concrete end-to-end example showing an agent specifically creating a dataset via API/MCP and then reading back eval results in one flow, and independent (non-vendor) confirmation of this specific workflow.

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed6/10

   Phoenix has a Python/SDK-driven dataset creation, experiment/eval-run triggering, and result retrieval (datasets, dataset evaluators, experiments, MCP endpoint letting agents 'operate on datasets, experiments'), which supports an agent programmatically creating datasets, running evals, and reading back scores. However the evidence is all docs-level description without a concrete end-to-end code example or independent confirmation of an agent performing this full loop. Missing for 10: a worked code/API example showing dataset creation -> eval trigger -> result read-back in one flow, and independent/hands-on verification beyond vendor docs.

   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
   • [github] “Datasets - Create versioned datasets of examples for experimen”

   proof ↗⚑ flag

Human review

1. ml engineerRoute outputs to human annotation queues for review and labeling

   weight 2 · round to Arize Phoenix
   Braintrust~partialclaimed5/10

   Braintrust supports building custom annotation interfaces for human review and labeling (docs-12) and datasets can be built/curated from production logs and manual curation (docs-5, docs-31), implying human review workflows, but there's no explicit evidence of a dedicated 'routing' mechanism, queue assignment, reviewer task distribution, or workflow status tracking for human annotators. missing for 10: explicit queue/routing mechanism to assign outputs to specific human reviewers, task assignment/status tracking, and independent corroboration of the annotation workflow in practice.

   • [claimed-docs] “Build annotation interfaces that match your team's workflow. Review support conversations differently than co”
   • [claimed-docs] “Build datasets from production logs, user feedback, manual curation, or generate them with Loop.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed6/10

   Phoenix supports attaching human annotations/ground-truth labels directly in the UI on traces/spans, which enables human review and labeling of outputs, and scoring can combine human labels with automated evals. However, there's no evidence of a dedicated 'annotation queue' routing mechanism, task assignment to reviewers, or workflow management for distributing items to human labelers. missing for 10: dedicated annotation queue/task routing to specific reviewers, workflow/assignment management, evidence of multi-user labeling workflows beyond UI annotation.

   • [claimed-docs] “Human annotations — Attach ground truth labels directly in the UI”
   • [claimed-docs] “Attach ground truth labels directly in the UI”
   • [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”

   proof ↗⚑ flag

Offline evals

1. developerRun evals in CI and gate deployments on their results

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Docs explicitly describe running evals via code/CLI, integrating into CI/CD to catch regressions, and gating on experiment results ('Automate in CI/CD — Run evals on every pull request to catch regressions'; 'integrate into CI/CD to catch regressions before they reach production'), backed by SDK eval code examples. Missing for 10: independent/hands-on confirmation of actual CI gating in production and details on failure-threshold/gate configuration.

   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed5/10

   Phoenix explicitly frames dataset evaluators as forming 'an evaluation harness similar to a unit test suite' and provides SDK/Python-based experiment running with automatic scoring, which are the building blocks for CI-based evals. However, there is no direct documentation of CI pipeline integration (e.g., GitHub Actions examples), exit-code/pass-fail thresholds, or deployment-gating mechanics tied to eval results. Missing for 10: explicit CI/CD integration docs, pass/fail gating logic, deployment-blocking examples.

   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
   • [claimed-docs] “Dataset Evaluators serve as **test cases** that automatically score outputs when running experiments—forming an evaluation harness similar t…”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”

   proof ↗⚑ flag

2. ml engineerWrite custom code-based scorers and metrics for my evaluations

   weight 2 · round drawn
   Braintrust✓fullclaimed8/10

   Braintrust's SDK examples show Eval() calls with custom `scores` arrays (e.g. LevenshteinScorer), demonstrating code-based custom scorers, and docs describe managing/authoring scorers via API, CLI, and playgrounds. Missing for 10: independent hands-on validation of writing a fully custom scorer function beyond the built-in Levenshtein example, and more detail on scorer authoring API/signature.

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”

   proof ↗⚑ flag

   Arize Phoenix✓fullclaimed8/10

   Docs explicitly state Phoenix supports deterministic code-based evaluators (exact match, regex, custom heuristics) alongside LLM-as-a-judge, and Dataset Evaluators act as test cases scoring outputs automatically in experiments similar to a unit test suite, which is exactly code-based custom scoring for evals. Missing for 10: no independent/hands-on corroboration of writing custom Python scorer code, and no detailed API reference/example showing custom scorer implementation.

   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators, where a sec…”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
   • [claimed-docs] “Dataset Evaluators serve as **test cases** that automatically score outputs when running experiments—forming an evaluation harness similar t…”
   • [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”

   proof ↗⚑ flag

3. ml engineerCompare eval runs side by side to catch regressions between prompt or model versions

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed9/10

   Braintrust's Experiments feature is explicitly described as an immutable, comparable record of eval runs that tracks progress over time and integrates into CI/CD to catch regressions, and Playgrounds let users run full evaluations and compare results side-by-side across prompt/model configurations. This directly matches the story of comparing eval runs to catch regressions between versions. Missing for 10: independent/hands-on corroboration of the side-by-side comparison UI beyond first-party docs.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”

   proof ↗⚑ flag

   Arize Phoenix✓fullclaimed8/10

   Phoenix's experiments feature explicitly groups traces into datasets, reruns them through different app/prompt/model versions, and compares evaluation results to confirm whether a change improved performance, with automatic scoring via configured evaluators — directly matching the side-by-side regression-catching story. Missing for 10: no independent/hands-on example of an actual side-by-side UI diff view or third-party corroboration beyond vendor docs.

   • [claimed-docs] “optimize your app with experiments that compare changes on the same inputs”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results to confirm whe…”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [github] “Datasets - Create versioned datasets of examples for experimen”

   proof ↗⚑ flag

4. ml engineerScore outputs with configurable LLM-as-a-judge evaluators

   weight 3 · round to Arize Phoenix
   Braintrust✓fullclaimed8/10

   Braintrust's docs describe configurable scorers used within evals and playgrounds (LLM-as-a-judge style scorers), online scoring of production traces, and code examples show custom scorer configuration (e.g., LevenshteinScorer swapped in evals). Docs explicitly mention 'author scorers' via CLI/MCP and building scorers with Loop, supporting configurability. Missing for 10: explicit documentation naming 'LLM-as-a-judge' scorer templates/library and independent hands-on validation of judge configurability beyond vendor docs.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Arize Phoenix✓fullclaimed9/10

   Docs explicitly describe LLM-as-a-judge evaluators configurable via model-agnostic adapters (OpenAI, LiteLLM, LangChain, AI SDK), with built-in explanations, automatic scoring of experiment results in the UI or via SDK, and full traceability of judge prompts/reasoning/scores. This is a first-party documented core feature well beyond a single mention. Missing for 10: independent/hands-on corroboration beyond vendor docs.

   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Model Agnostic via adapters (for OpenAI, LiteLLM, LangChain, AI SDK, and more) — so you can easily switch judge models”
   • [claimed-docs] “Built-in Explanations — all Phoenix LLM evaluations return explanations by default for better results and richer signals”
   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators, where a sec…”
   • [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model’s full reasoning, the final scores, a…”
   • [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model's full reasoning, the final scores, a…”

   proof ↗⚑ flag

5. ml engineerCurate datasets from production traces and run offline evaluations against them

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed9/10

   Docs explicitly describe building versioned datasets from production logs/traces (docs-5, docs-31) and running offline evaluations against them via Experiments, code, or Playgrounds (docs-27, docs-23, gh-1/gh-2 SDK examples), with CI/CD integration for regression testing (docs-2). Missing for 10: independent/hands-on third-party validation of the end-to-end curate-from-production-to-offline-eval workflow beyond first-party docs.

   • [claimed-docs] “Build datasets from production logs, user feedback, manual curation, or generate them with Loop.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Arize Phoenix✓fullclaimed8/10

   Phoenix docs describe grouping production traces into versioned datasets (docs-10, docs-25, gh-1), attaching evaluators (code-based and LLM-as-judge) to datasets and scoring experiment results automatically as an offline eval harness (docs-17, docs-18, docs-26, docs-34), and comparing experiment results across app versions (docs-41). This directly matches curating datasets from traces and running offline evals. missing for 10: independent/hands-on corroboration beyond vendor docs, and concrete example of the full curate-to-evaluate workflow output.

   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [github] “Datasets - Create versioned datasets of examples for experimen”
   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results to confirm whe…”

   proof ↗⚑ flag

Online evals

1. ml engineerRun evaluators continuously on live production traffic, not just offline datasets

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust explicitly supports 'online scoring' that evaluates production traces automatically as they're logged asynchronously with no latency impact, plus continuous background analysis via Patterns/Topics scanning trace backlogs on a schedule for issues no scorer already catches. This directly matches continuous evaluation on live traffic beyond offline datasets. Missing for 10: independent/hands-on verification of online scoring behavior in production and more detail on configuring which evaluators run continuously.

   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Define the dimensions that matter to your business, like use case, customer segment, compliance, or tone. Topics continuously clusters every…”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed5/10

   Docs confirm Phoenix ingests live production traces (OTLP) and can score traces & spans with LLM, code-based, or human evaluators (docs-39, docs-25), suggesting evaluation can be applied beyond static datasets. However, the more detailed evaluation workflow described is dataset/experiment-centric (attach evaluators to datasets, score experiment results) rather than an explicit always-on/scheduled evaluator pipeline running automatically on incoming production traffic. Missing for 10: explicit documentation of a continuous/scheduled evaluation job or streaming eval pipeline that runs on live traces without manual dataset curation, and independent confirmation of this in production use.

   • [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”
   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Braintrust
   Braintrust~partialprobed7/10

   Braintrust offers a comprehensive API/SDK ('interact with all aspects of the Braintrust platform programmatically'), a full-featured CLI (bt) for auth, tracing, evals, log queries, and function management, and an MCP server letting agents query logs, author scorers, configure monitoring, and run evals — covering most UI functionality outside the terminal. However, some UI-only features like Playgrounds (explicitly 'no-code workspace') and annotation-interface building appear UI-centric without clear API/CLI parity, and no public OpenAPI spec was found to confirm full API coverage. missing for 10: confirmed OpenAPI/spec proving 1:1 API-UI parity, explicit API/CLI equivalent for Playgrounds and custom annotation UI workflows, independent verification of completeness.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Arize Phoenix~partialprobed5/10

   Phoenix's Python/SDK APIs cover most workflows (tracing, datasets, experiments, evals, prompts) matching many UI features, and human annotations/UI-configured evaluators are documented, but there's no comprehensive API/CLI reference confirming full parity with UI-only actions like human annotation labeling or UI-based evaluator configuration, and no public OpenAPI spec was found (404s). missing for 10: a documented OpenAPI/REST spec confirming full UI-to-API parity, explicit API equivalents for UI-only actions like human annotation attachment, and independent verification that every UI action has an API counterpart.

   • [claimed-docs] “Human annotations — Attach ground truth labels directly in the UI”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
   • [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
   • [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to Arize Phoenix
   Braintrust~partialclaimed5/10

   Braintrust provides data export via `bt sync pull`, `bt view logs` downloads as CSV/JSON, and a full API to programmatically extract projects, experiments, datasets, traces, and prompts, plus a self-hosting option for data control. However, there's no explicit documentation of full-account export/portability guarantees, no mention of open/standard interchange formats beyond CSV/JSON logs, and no independent confirmation of a clean 'leave' workflow. Missing for 10: comprehensive account-level export documentation, open-format guarantees for all entity types (prompts, scorers, configs), and third-party validation of export completeness.

   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed6/10

   Phoenix is open-source and self-hostable with data staying entirely in the user's infrastructure, and it ingests/exports traces via the open OpenTelemetry (OTLP) standard rather than a proprietary format, which strongly supports data portability and no lock-in. However, there is no explicit documentation of a dedicated 'export all my data' feature (e.g., bulk dataset/trace export to CSV/Parquet or a migration tool) beyond OTLP ingestion and self-hosting. Missing for 10: explicit bulk data export tooling/documentation, independent confirmation of successful full data migration/export by a user.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
   • [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [github] “Datasets - Create versioned datasets of examples for experimen”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round to Arize Phoenix
   Braintrust—none0/10

   Evidence shows only client SDK repos on GitHub (braintrust-sdk-javascript) and self-hosted deployment of infrastructure, but the core platform/UI is explicitly described as Braintrust-managed and not released under an open license; no evidence of the product's full source being open-licensed.

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed5/10

   Phoenix's docs state it is 'built by Arize AI and the open-source community' and a public GitHub repo (Arize-ai/phoenix) is referenced, indicating the source is publicly readable. However, no evidence pack item specifies the actual license terms (e.g., MIT, Apache, ELv2) or confirms it is a permissive open-source license. Missing for 10: explicit license identification/text, independent confirmation of license terms, and any discussion of license restrictions.

   • [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
   • [github] “Datasets - Create versioned datasets of examples for experimen”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round to Arize Phoenix
   Braintrust~partialclaimed6/10

   Braintrust documents an official self-hosted deployment option, but it explicitly separates data storage (self-hosted) from platform management (Braintrust-managed UI, auth, updates), meaning it's not a fully self-hostable core product. Missing for 10: evidence of fully self-hosting the entire platform (UI, control plane, auth) independent of Braintrust's managed services, and independent/community confirmation of self-hosting experience.

   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”

   proof ↗⚑ flag

   Arize Phoenix✓fullclaimed9/10

   Docs explicitly state Phoenix is free to self-host with no feature limitations, data stays fully within infrastructure and can be air-gapped, with a simple CLI command (uvx arize-phoenix serve) and Docker image versioning for production deployments. Missing for 10: independent/hands-on confirmation of self-hosting experience beyond vendor docs.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
   • [claimed-docs] “$ uvx arize-phoenix serve”
   • [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   Braintrust~partialclaimed6/10

   Braintrust's self-hosting docs explicitly state it's designed to meet data residency requirements by keeping customer data within your own cloud account and region, giving users region/residency control — but only via full self-hosted deployment, not a simple region-selection toggle in the managed cloud offering. Missing for 10: evidence of region choice within the standard hosted/SaaS plan, list of specific supported regions, and compliance certifications tied to residency.

   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed6/10

   Phoenix documents that it is free to self-host with data staying entirely within your own infrastructure and can be fully air-gapped, which lets users choose exactly where their data lives (docs-27/56). However, there is no explicit mention of a managed/cloud offering with selectable regions or residency guarantees, so the 'choose region' capability is only implicit via self-hosting. Missing for 10: explicit region-selection UI/settings, documented data residency options for any hosted/cloud tier, and independent confirmation of deployment flexibility.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
   • [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”

   proof ↗⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round to Arize Phoenix
   Braintrust—none0/10

   No evidence pack item addresses data-training opt-out, a no-training policy, or any statement about whether customer data is used to train AI models. Self-hosting docs address data residency/compliance but do not mention model training use. missing for 10: any explicit no-training-on-customer-data policy, opt-out controls, or terms-of-service statement about AI training use.

   ⚑ flag

   Arize Phoenix~partialclaimed6/10

   Phoenix's self-hosting docs state that data stays entirely within the user's infrastructure and 'nothing is sent to Arize' and can be fully air-gapped, which strongly implies no data is available to Arize for model training when self-hosted. However, there is no explicit statement of a data-training policy (e.g., for any hosted/cloud offering) or a formal privacy/DPA commitment on this point. Missing for 10: explicit no-training-data policy statement, coverage of hosted/cloud deployment privacy terms, independent verification.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”

   proof ↗⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round to Arize Phoenix
   Braintrust~partialclaimed4/10

   Braintrust's self-hosted deployment option lets customers keep all data within their own cloud account/region for compliance and residency, giving some control over where data lives, and the API/CLI allow programmatic management of logs and datasets. However, there is no explicit documentation of a retention policy, TTL settings, or a delete/purge API for logs, traces, or datasets on the managed (non-self-hosted) offering. Missing for 10: explicit retention window configuration, an in-platform delete/purge mechanism for traces or datasets, and confirmation that these controls exist outside of self-hosting.

   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed5/10

   Phoenix's self-hosting model gives users full control over where data lives ('free to self-host... data stays entirely within your infrastructure... nothing is sent to Arize and can be fully air-gapped'), which implicitly enables retention/deletion control since the operator owns the storage. However, there is no explicit documentation of retention policies, TTL settings, or data-deletion APIs/commands within Phoenix itself. Missing for 10: explicit retention configuration options, documented deletion/purge commands or APIs, and any cloud-hosted retention controls for non-self-hosted deployments.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
   • [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”

   proof ↗⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round to Arize Phoenix
   Braintrust—none0/10

   No evidence of any telemetry opt-out or usage-tracking control setting for Braintrust itself as a tool; the evidence covers Braintrust's own tracing/observability product features (for user's AI apps), not the vendor's own telemetry collection practices or an opt-out mechanism. Self-hosting controls where customer AI data resides but says nothing about product telemetry/usage tracking opt-out.

   ⚑ flag

   Arize Phoenix~partialclaimed6/10

   Phoenix's self-hosting docs state that when self-hosted, 'nothing is sent to Arize and can be fully air-gapped,' implying no forced telemetry to the vendor, which directly supports a privacy-conscious opt-out posture. However, there is no explicit documentation of a dedicated telemetry/usage-tracking toggle (e.g., an env var or CLI flag to disable analytics) for the hosted or default install path. Missing for 10: explicit opt-out mechanism/config flag for anonymous usage analytics, confirmation of what telemetry (if any) is collected by default in non-air-gapped mode.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”

   proof ↗⚑ flag

Prompt workflow

1. developerIterate on prompts in a playground against real models and variables

   weight 2 · round drawn
   Braintrust✓fullclaimed8/10

   Docs explicitly describe Playgrounds as a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets, running full evaluations in real-time against real models with variable inputs, and comparing results side-by-side (braintrust-docs-3, braintrust-docs-23, braintrust-docs-17). This directly matches the story of iterating on prompts in a playground against real models and variables. Missing for 10: independent/hands-on user corroboration of the playground experience specifically (community evidence is generic/mixed) and no detail on variable-substitution mechanics beyond the general description.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Measure and improve AI application quality with playgrounds and evals”

   proof ↗⚑ flag

   Arize Phoenix✓fullclaimed8/10

   Docs explicitly describe a Prompt Playground where developers can 'play with prompts, models, invocation parameters' and track progress via tracing/experiments, plus use of real production examples/datasets for iteration. This directly matches the story of iterating on prompts against real models and variables. Missing for 10: independent/hands-on user reports specifically about the playground UX (only vendor docs, no third-party corroboration of the playground itself).

   • [claimed-docs] “Play with prompts, models, invocation parameters and track your progress via tracing and experiments”
   • [claimed-docs] “Prompt Playground - Play with prompts, models, invocation parameters and track your progress via tracing and experiments”
   • [claimed-docs] “iterate on your prompts using real production examples”
   • [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
   • [claimed-docs] “Replay the invocation of an LLM. Whether it's an LLM step in an LLM workflow or a router query, you can step into the LLM invocation and see…”
   • [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”

   proof ↗⚑ flag

2. developerVersion prompts and deploy changes to production without shipping code

   weight 3 · round to Arize Phoenix
   Braintrust~partialclaimed6/10

   Braintrust supports managing prompts via playgrounds, API, and CLI ("Manage prompts, tools, and scorers"), and datasets/experiments are versioned, suggesting prompt versioning is possible; however, there's no explicit evidence of a distinct 'deploy to production without code' workflow (e.g., prompt promotion/publish/rollback mechanism) separate from logging or evals. missing for 10: explicit prompt versioning/history UI, a documented deploy/publish/rollback mechanism for prompts, and independent confirmation that production apps can swap prompts without redeploying code.

   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”

   proof ↗⚑ flag

   Arize Phoenix✓fullclaimed8/10

   Phoenix has explicit Prompt Management features supporting versioning, storage, and deployment of prompts, plus 'Prompts in Code' SDKs to sync prompts across environments so changes can be pushed to production without code changes. Docs show a UI-based prompt playground and versioning workflow decoupled from application code deploys. Missing for 10: independent/hands-on verification beyond vendor docs of production rollback/rollout mechanics and no explicit mention of environment-specific staged rollout controls.

   • [claimed-docs] “Prompt Management — Version, store, and deploy prompts”
   • [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
   • [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “Prompts in Code - Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”

   proof ↗⚑ flag

Ai observability

1. ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app

   weight 3 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust documents a first-party MCP server enabling MCP-compatible agents (Claude Code, Cursor, Codex, etc.) to query logs, run evals, author scorers, and configure monitoring, plus a full programmatic API for traces/experiments/datasets, and Loop, Braintrust's own AI agent, for natural-language investigation of logs and eval data. This directly matches the story of an agent querying traces, metrics, and eval results via API or MCP to debug an app. Missing for 10: independent hands-on corroboration of the MCP server working end-to-end (only vendor docs/probe confirm existence, no third-party review of actual debugging workflow).

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   Arize Phoenix✓fullprobed8/10

   Phoenix documents a first-party MCP server (/mcp endpoint) that lets MCP-compatible clients (Claude Code, Cursor, VS Code, etc.) search, query, and operate on projects, traces, datasets, experiments, prompts, and annotations, directly enabling an agent to query traces/metrics/eval results for debugging. This is backed by an official docs page and confirmed by probe evidence. Missing for 10: independent/hands-on third-party validation of the MCP server in practice and a documented general-purpose REST/GraphQL API spec (openapi probes 404) for non-MCP API access.

   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
   • [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
   • [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”

   proof ↗⚑ flag

Data controls

1. developerMask or redact sensitive data before it is stored in traces

   weight 2 · round drawn
   Braintrust—none0/10

   No evidence in the pack describes any PII masking, redaction, or scrubbing feature for traces before storage; documentation covers instrumentation, tracing, evals, and observability but never mentions data masking or redaction capabilities. This is a plausible and common axis for a tracing/observability platform, so absence of evidence yields 'none' rather than 'na'.

   ⚑ flag

   Arize Phoenix—none0/10

   No evidence pack item mentions masking, redaction, PII scrubbing, or any data-sanitization capability for traces before storage; the docs cover tracing, evaluation, prompts, datasets, and self-hosting but never address sensitive-data redaction.

   ⚑ flag

Sdk coverage

1. developerInstrument apps in both Python and JS/TS with officially supported SDKs

   weight 2 · round drawn
   Braintrust~partialclaimed6/10

   Docs confirm instrumentation/tracing is central to Braintrust and JS SDK usage is shown in GitHub examples (Eval function calls), with docs referencing SDK-based tracing and instrumentation broadly, but the evidence pack never explicitly confirms an officially supported Python SDK alongside JS/TS — only JS/TS SDK code is shown. Missing for 10: explicit Python SDK documentation/examples, side-by-side confirmation of parity between Python and JS/TS SDKs, and independent corroboration of both SDKs in production use.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Arize Phoenix~partialclaimed6/10

   Phoenix documents OTLP ingestion and auto-instrumentation for frameworks including JS/TS ones like Vercel AI SDK and Mastra, plus a phoenix-cli npx setup and OpenInference-based instrumentation, indicating both Python and JS/TS support. However, the evidence is mostly high-level marketing docs without explicit JS/TS SDK API reference or code samples analogous to the detailed Python SDK reference (docs-28/29/30/36), and no independent/hands-on confirmation of JS/TS SDK parity. Missing for 10: explicit JS/TS SDK API reference/docs comparable to the Python SDK reference, hands-on/independent confirmation of JS/TS instrumentation working end-to-end, and clarity on feature parity between the two SDKs.

   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
   • [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
   • [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
   • [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”

   proof ↗⚑ flag

Trace capture

1. developerTrace multi-step agent runs as nested spans grouped into sessions or threads

   weight 2 · round to Arize Phoenix
   Braintrust~partialclaimed6/10

   Braintrust documents tracing that captures spans, logs, and full request detail, and lets users browse traces and individual spans via UI or CLI, supporting general span-based tracing. However, the evidence never explicitly describes nested multi-step spans grouped into sessions/threads for agent runs (no mention of parent-child span hierarchies or session/thread grouping semantics). Missing for 10: explicit documentation of nested span hierarchies for multi-step agent runs, explicit session/thread grouping constructs, and independent/hands-on confirmation of this specific structure.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”

   proof ↗⚑ flag

   Arize Phoenix✓fullclaimed8/10

   Docs explicitly describe OTLP-based tracing that captures nested spans for model calls, retrieval, and tool use (docs-38, docs-12/13/14), and Sessions explicitly group related traces into conversations/user sessions while Projects organize traces by application (docs-16/49). Auto-instrumentation for popular agent frameworks (LangChain, LlamaIndex, etc.) supports multi-step agent run tracing out of the box (docs-5, docs-32). Missing for 10: independent/hands-on verification of nested span hierarchy rendering for complex multi-step agents beyond vendor docs.

   • [claimed-docs] “Sessions Group related traces into conversations and user sessions”
   • [claimed-docs] “Group related traces into conversations and user sessions”
   • [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
   • [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
   • [claimed-docs] “Retrieved Documents: Inspect the documents retrieved during a Retriever call, including the score and order in which they were returned”
   • [claimed-docs] “Tool Descriptions: View the descriptions and function signatures of the tools your LLM has been given access to”
   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [claimed-docs] “Projects Organize traces into separate projects for different applications”

   proof ↗⚑ flag

2. developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK

   weight 2 · round to Arize Phoenix
   Braintrust~partialcommunity4/10

   Docs confirm a general instrumentation capability and state you can 'integrate with AI providers and frameworks to send traces to Braintrust,' but no evidence explicitly names LangChain, the OpenAI SDK, or the Vercel AI SDK as supported integrations. A community comment even calls integration options 'limited.' Missing for 10: explicit documentation or hands-on confirmation of dedicated LangChain, OpenAI SDK, and Vercel AI SDK integrations.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [community] “Braintrust: Interesting approach to evaluation but feels like an early-stage product. Documentation is sparse and integration options are li…”

   proof ↗⚑ flag

   Arize Phoenix✓fullclaimed8/10

   Docs explicitly state Phoenix accepts OTLP traces and provides auto-instrumentation for popular frameworks including LangChain and Vercel AI SDK, plus OpenAI is referenced as an adapter/evaluator target and is a widely documented OpenInference integration. Missing for 10: no independent/hands-on verification of these integrations working, and no explicit dedicated docs snippet for OpenAI SDK auto-instrumentation beyond the adapter mention.

   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
   • [claimed-docs] “Model Agnostic via adapters (for OpenAI, LiteLLM, LangChain, AI SDK, and more) — so you can easily switch judge models”
   • [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”

   proof ↗⚑ flag

3. developerCapture multimodal payloads (images, audio, files) inside my traces

   weight 1 · round drawn
   Braintrust—none0/10

   The evidence pack describes tracing capturing inputs, outputs, model parameters, latency, token usage, and metadata, but there is no explicit mention of multimodal payload support (images, audio, files) within traces. Missing for 10: explicit documentation or examples showing image/audio/file capture in traces, SDK support for multimodal attachments, or any UI rendering of multimodal content in trace spans.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”

   proof ↗⚑ flag

   Arize Phoenix—none0/10

   The evidence pack describes tracing of model calls, retrieval, tool use, latency, and token usage, but nowhere mentions capturing images, audio, or file attachments/multimodal payloads within spans or traces. Since this is a plausible capability for a tracing/observability tool, absence of evidence yields 'none'.

   • [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
   • [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
   • [claimed-docs] “Retrieved Documents: Inspect the documents retrieved during a Retriever call, including the score and order in which they were returned”

   proof ↗⚑ flag

4. developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format

   weight 3 · round to Arize Phoenix
   Braintrust—none0/10

   No evidence pack item mentions OpenTelemetry or OTLP support; Braintrust's docs describe its own SDK/instrumentation and integrations but never reference OTLP-based trace ingestion/export. Missing for 10: any mention of OTLP endpoint, OpenTelemetry SDK compatibility, or OTel collector integration.

   ⚑ flag

   Arize Phoenix✓fullclaimed9/10

   Docs explicitly state Phoenix accepts traces over OpenTelemetry (OTLP), provides auto-instrumentation for popular frameworks, and is built on top of OpenTelemetry powered by OpenInference instrumentation — confirming standard OTLP ingestion rather than a proprietary format. Missing for 10: independent/hands-on corroboration of OTLP export/receive behavior beyond vendor docs.

   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
   • [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”

   proof ↗⚑ flag

5. developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage

   weight 3 · round to Arize Phoenix
   Braintrust✓fullclaimed8/10

   Braintrust docs explicitly state instrumentation captures inputs, outputs, model parameters, latency, token usage, and metadata for every LLM call, with SDK examples and CLI/UI browsing of traces and spans. missing for 10: independent hands-on verification of exact captured fields and no third-party benchmark of tracing accuracy/performance overhead.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”

   proof ↗⚑ flag

   Arize Phoenix✓fullclaimed9/10

   Phoenix's docs extensively document capturing OTLP/OpenTelemetry-based traces of LLM calls, including inputs/outputs (span data), application latency, and detailed token usage breakdowns, plus auto-instrumentation for popular frameworks and manual tracing decorators. This is a core, heavily documented first-party capability with a CLI/SDK setup flow. Missing for 10: independent hands-on verification beyond vendor docs/community sentiment.

   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
   • [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
   • [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
   • [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
   • [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
   • [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”

   proof ↗⚑ flag