Arize Phoenix vs W&B Weave

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round drawn
   Arize Phoenix✓fullprobed9/10

   Direct probe evidence confirms llms.txt returns HTTP 200 with agent-oriented content, and docs pages are also available in markdown form (.md) with a documentation index, both explicitly designed for agent/LLM consumption. Missing for 10: no independent/community confirmation of an agent actually consuming these files successfully.

   • [probe] “PROBE llms.txt: HTTP 200 at https://arize.com/llms.txt # Arize — AI & Agent Engineering > Arize is the AI engineering platform for teams bu…”
   • [probe] “PROBE docs-md: HTTP 200 at https://arize.com/docs/phoenix.md > ## Documentation Index > Fetch the complete documentation index at: https://a…”

   proof ↗⚑ flag

   W&B Weave✓fullprobed9/10

   Direct probes confirm llms.txt (HTTP 200 with structured doc index) and a .md variant of docs pages exist and are crawlable, exactly matching the ask for agent-oriented docs, plus an OpenAPI spec and MCP server for further agent integration. Missing for 10: no independent/community confirmation that agents actually consume these docs successfully in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.wandb.ai/llms.txt # Weights & Biases Documentation - [Products (407 pages)](https://docs.wandb.ai/…”
   • [probe] “PROBE docs-md: HTTP 200 at https://docs.wandb.ai/weave.md > ## Documentation Index > Fetch the complete documentation index at: https://docs…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Arize Phoenix
   Arize Phoenix~partialclaimed7/10

   Phoenix ships a CLI (`uvx arize-phoenix serve`), a Docker image pinned for production deployments, and a Python SDK with dataset evaluators explicitly described as forming 'an evaluation harness similar to a unit test suite,' all of which support headless/automated use in CI-like pipelines. However, there is no explicit CI/CD example (e.g., GitHub Actions workflow) or documentation of running evaluations as an automated CI gate. Missing for 10: explicit CI pipeline integration examples/docs, evidence of automated test-gating in CI, independent confirmation of headless CI usage.

   • [claimed-docs] “$ uvx arize-phoenix serve”
   • [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
   • [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
   • [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”

   proof ↗⚑ flag

   W&B Weave~partialclaimed6/10

   Weave's SDK (weave.op, weave.init) and Service API/REST endpoints allow programmatic, non-UI instrumentation and evaluation that can run in scripts or CI pipelines, and the OTel-compatible ingestion endpoint supports headless trace collection. However, there is no explicit documentation of a CI-specific workflow, headless auth/config for pipelines, or a dedicated CLI/automation example confirming CI usage. missing for 10: explicit CI/headless setup guide, documented non-interactive auth flow for automated pipelines, concrete CI example (e.g. GitHub Actions integration).

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
   • [github] “Log and debug language model inputs, outputs, and traces”

   proof ↗⚑ flag

3. ai-native userConnect an agent via an official MCP server

   weight 3 · round drawn
   Arize Phoenix✓fullprobed8/10

   Phoenix documents an official remote MCP server exposing a `/mcp` endpoint that MCP-compatible clients (Claude Code, Cursor, VS Code, etc.) can connect to for searching and operating on projects, traces, datasets, experiments, prompts, and annotations. This is first-party documentation confirming the exact capability. Missing for 10: independent/hands-on community corroboration of the MCP server working in practice, and details on authentication/setup robustness.

   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
   • [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”

   proof ↗⚑ flag

   W&B Weave✓fullprobed8/10

   W&B ships an official MCP server (wandb-mcp-server) enabling coding agents like Claude Code to connect to Weave, read live production data, run evaluations, and execute iteration loops autonomously — this is documented both on the product site and via a dedicated GitHub repo. Missing for 10: deeper documentation of MCP server setup/configuration and independent hands-on corroboration beyond vendor claims.

   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”

   proof ↗⚑ flag

4. ai-native userUse an official CLI

   weight 2 · round to Arize Phoenix
   Arize Phoenix~partialclaimed6/10

   Phoenix ships an official CLI (`@arizeai/phoenix-cli` via `npx -y @arizeai/phoenix-cli setup`) for auto-instrumentation setup, plus a `uvx arize-phoenix serve` command to launch the server, confirming a genuine first-party CLI exists for agentic/automated workflows. However, the CLI's documented scope is narrow (setup + serve) rather than covering the full product surface (datasets, evals, prompts, experiments), and there's no independent/community confirmation of its use. Missing for 10: broader CLI command coverage across Phoenix's feature set, independent hands-on verification of the CLI in practice.

   • [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
   • [claimed-docs] “$ uvx arize-phoenix serve”
   • [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app's root directory.”
   • [claimed-docs] “start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app’s root directory”

   proof ↗⚑ flag

   W&B Weave~partialprobed5/10

   There is evidence of an official W&B CLI (docs.wandb.ai/models/ref/cli), but this CLI is documented under the Models product, not specifically Weave, and no evidence details Weave-specific CLI commands (e.g., managing traces, evaluations, or ops from the terminal) or AI-native/agentic use of it. Missing for 10: Weave-specific CLI command reference, evidence of agentic/programmatic use of the CLI, independent hands-on confirmation.

   • [probe] “official CLI documented at https://docs.wandb.ai/models/ref/cli”

   proof ↗⚑ flag

5. ai-native userDrive the product through a documented public API

   weight 3 · round to W&B Weave
   Arize Phoenix✓fullprobed7/10

   Phoenix documents a full SDK API reference (tracing decorators, dataset/experiment APIs, prompt sync SDKs), OTLP-based trace ingestion, a CLI (phoenix-cli), and an official remote MCP endpoint that lets AI-native clients search/query/operate on projects, traces, datasets, experiments, and prompts—together constituting a documented, programmatically-drivable public API surface. The openapi.json probe returned 404s at standard paths, meaning there's no discoverable formal REST/OpenAPI spec, so the API surface is SDK/CLI/MCP-centric rather than a single canonical REST API. Missing for 10: a published OpenAPI/REST spec, independent third-party confirmation of API robustness.

   • [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
   • [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
   • [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
   • [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
   • [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
   • [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
   • [claimed-docs] “$ uvx arize-phoenix serve”

   proof ↗⚑ flag

   W&B Weave✓fullprobed8/10

   Weave documents a public REST Service API for programmatic access, an openapi.json spec, Python/TypeScript SDKs with @weave.op decorators, and an official MCP server enabling agent-driven interaction with live data and evaluations. missing for 10: independent third-party validation of API stability/versioning and rate-limit documentation beyond first-party docs.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”

   proof ↗⚑ flag

6. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   Arize Phoenix—none0/10

   No evidence in the pack describes scoped or least-privilege API key/credential issuance for agents; Phoenix's docs cover tracing, evaluation, prompt management, and an MCP endpoint, but nothing about credential scoping or access control granularity.

   ⚑ flag

   W&B Weave—none0/10

   W&B Weave is an LLM observability/evaluation tool; the evidence pack covers tracing, evaluations, cost tracking, and an MCP/skills integration, but there is no mention of scoped or least-privilege API credential issuance for agents. Missing for 10: any documentation of API key scoping, permission granularity, or credential management for agent access.

   ⚑ flag

7. ai-native userBuild against official SDKs

   weight 2 · round drawn
   Arize Phoenix✓fullclaimed8/10

   Phoenix ships an official, modular Python SDK plus client SDKs for tracing, prompts, evaluations, and datasets, with OTLP-based tracing decorators, auto-instrumentation, and dedicated SDK/API reference docs. Coverage spans core agentic workflows (tracing, evals, prompt management, datasets/experiments) and is corroborated by GitHub repo and docs pages, though independent hands-on SDK usage reports are thin. Missing for 10: broader independent/third-party validation of SDK usage beyond official docs, and explicit multi-language SDK support details beyond Python/JS CLI.

   • [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
   • [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
   • [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
   • [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
   • [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “Prompts in Code - Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
   • [github] “Datasets - Create versioned datasets of examples for experimen”

   proof ↗⚑ flag

   W&B Weave✓fullprobed8/10

   Weave offers official Python and TypeScript SDKs with decorator-based tracing (@weave.op), a REST Service API, OTel-compatible SDK, and a documented CLI/MCP server, all backed by first-party docs and public GitHub repo. Missing for 10: independent third-party benchmarking or hands-on developer reviews validating SDK stability/completeness beyond vendor docs.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [github] “You can trace any function using `weave.op` - from api calls to OpenAI, Anthropic, Google AI Studio etc to generation calls from Hugging Fac…”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Trace and collect metrics about your agents built with popular SDKs and harnesses using Weave OTel-compatible SDK”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
   • [probe] “official CLI documented at https://docs.wandb.ai/models/ref/cli”

   proof ↗⚑ flag

8. ai-native userSubscribe to events via webhooks

   weight 2 · round to W&B Weave
   Arize Phoenix—none0/10

   No evidence anywhere in the pack of a webhook subscription mechanism; Phoenix's integration surface is OTLP tracing ingestion, an MCP server, and SDKs, but nothing about outbound event webhooks for subscribing to Phoenix events.

   ⚑ flag

   W&B Weave~partialclaimed4/10

   Weave documents alerts that 'trigger webhook automations' from production insights, indicating some outbound webhook mechanism exists, but there is no documentation of a subscription API, event types, payload schema, or configuration steps for webhooks. missing for 10: documented webhook subscription/configuration API, list of subscribable event types, payload format, independent/hands-on confirmation.

   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to W&B Weave
   Arize Phoenix~partialclaimed5/10

   Phoenix offers LLM-as-judge evaluators that return explanations and reasoning alongside scores, and lets users explore traces to spot systematic issues, which counts as AI-generated signal on the data — but this is evaluator-driven scoring, not proactive, unprompted insights/suggestions surfaced automatically inside the product UI. missing for 10: evidence of autonomous/proactive insight generation (e.g., auto-detected anomalies, recommended fixes, natural-language summaries of trends) rather than only judge-model scoring/explanations tied to evals a user configures.

   • [claimed-docs] “Built-in Explanations — all Phoenix LLM evaluations return explanations by default for better results and richer signals”
   • [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model's full reasoning, the final scores, a…”
   • [claimed-docs] “Use Phoenix's trace viewer to explore evaluation traces, identify systematic biases, and continuously improve evaluator performance.”
   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”

   proof ↗⚑ flag

   W&B Weave~partialclaimed6/10

   Weave ships built-in LLM-judge scorers (hallucination detection, summarization quality, embedding similarity) and Guardrails (toxicity, bias, PII detection) that automatically generate AI-based assessments of traced data, plus 'signals' that auto-classify agent interactions — all forms of AI-generated insight surfaced inside the product. However, these are narrow, pre-defined quality/safety classifiers rather than general proactive 'suggestions' or exploratory insight generation across arbitrary data, and most of the deeper analysis (custom scorers, evaluation criteria) requires user-authored code rather than the product generating novel suggestions on its own. Missing for 10: evidence of open-ended AI-generated recommendations/next-step suggestions (not just fixed scorer categories), and independent/hands-on confirmation these signals surface meaningfully useful insights in practice.

   • [claimed-docs] “Weave comes with predefined scorers and local SLM scorers that you can use right away, including: Hallucination detection, Summarization qua…”
   • [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
   • [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to W&B Weave
   Arize Phoenix—none0/10

   Phoenix's evidence covers tracing, evaluation, prompt management, and datasets, but nothing describes scheduled or autonomous background automations (e.g., recurring eval jobs, alerting rules, or triggers) that run without user initiation. The evaluation/experiment features described are invoked by the user (via SDK/UI), not autonomous background processes.

   ⚑ flag

   W&B Weave~partialclaimed5/10

   Weave documents automated background signals and alerting (built-in/custom signals classify agent interactions and trigger Slack/webhook automations) and mentions agents connecting via MCP to 'execute automatic iteration loops on their own,' which suggests some autonomous background automation. However, Weave is primarily a tracing/evaluation/observability tool, not a scheduler or workflow-automation platform, and there's no dedicated docs on setting up persistent background jobs or scheduled autonomous runs beyond alert-triggered webhooks. Missing for 10: dedicated automation/scheduling feature docs, evidence of persistent autonomous background jobs beyond alert webhooks, independent corroboration of the MCP-driven 'automatic iteration loops' claim.

   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”

   proof ↗⚑ flag

3. ai-native userOperate the product with natural-language commands

   weight 2 · round to Arize Phoenix
   Arize Phoenix✓fullprobed7/10

   Phoenix documents a dedicated remote MCP integration letting any MCP-compatible client (Claude Code, Cursor, VS Code, etc.) operate on projects, traces, datasets, experiments, prompts, and annotations via natural-language driven agents, and also supports agent-driven CLI setup. This directly satisfies AI-native natural-language operability. Missing for 10: independent/hands-on validation of the MCP command experience and more detail on the breadth/reliability of natural-language operations beyond the docs claim.

   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
   • [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
   • [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”

   proof ↗⚑ flag

   W&B Weave~partialprobed6/10

   Weave itself is an observability/eval dashboard with no native chat-command interface, but an official MCP server lets AI coding agents like Claude Code read production data, run evaluations, and iterate automatically using natural-language instructions relayed through MCP tools. This gives indirect NL-driven operation rather than a first-party conversational control surface. Missing for 10: a built-in Weave chat/NL console, independent hands-on verification of the MCP-driven workflow, and broader agent support beyond Claude Code.

   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round to W&B Weave
   Arize Phoenix—none0/10

   Docs mention an 'sdk-api-reference' page listing decorators and SDK features, but there is no evidence of an interactive, runnable API reference (e.g., a Swagger/OpenAPI explorer or live code sandbox); a direct probe for OpenAPI/swagger specs returned 404 on all candidate paths, indicating no such interactive reference is discoverable.

   • [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
   • [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
   • [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
   • [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”

   proof ↗⚑ flag

   W&B Weave~partialprobed3/10

   The Weave Service API and an OpenAPI spec (openapi.json) exist, suggesting some form of structured API reference, but there is no evidence of an interactive, in-browser reference with runnable/executable examples (e.g., a Swagger/try-it-out console or live code sandbox). missing for 10: confirmation of an interactive UI with 'try it now' runnable examples, evidence of live execution from the docs, and any independent confirmation of this feature.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round to W&B Weave
   Arize Phoenix—none0/10

   A direct probe for OpenAPI/swagger endpoints at Phoenix's domain returned 404 for all candidate paths, and no docs mention a downloadable machine-readable API spec. Missing for 10: any OpenAPI/Swagger spec, documented REST API reference, or SDK-generated schema.

   • [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”

   proof ↗⚑ flag

   W&B Weave✓fullprobed9/10

   A probe confirms an OpenAPI spec is served at https://docs.wandb.ai/openapi.json (HTTP 200, contains an 'openapi' key), and docs also describe a Service API providing REST endpoints for programmatic access. missing for 10: no independent third-party confirmation that the spec is actively used/maintained beyond the probe check.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Arize Phoenix
   Arize Phoenix~partialclaimed5/10

   Phoenix supports self-hosting with full data isolation (data never leaves your infra) and lets you organize traces into separate 'projects', collect data 'from production, staging, evaluations, and even manually' into datasets, and run experiments/evaluations against those datasets rather than live traffic — all of which let a user test without touching real production data. However, there's no explicit documented 'sandbox environment' feature, no isolated test-instance provisioning, and no guidance on separating staging from prod deployments beyond generic self-hosting flexibility. Missing for 10: explicit sandbox/staging environment tooling, isolation guarantees beyond self-hosting, and independent confirmation that experiments truly never touch production data.

   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
   • [claimed-docs] “Projects Organize traces into separate projects for different applications”
   • [claimed-docs] “Organize traces into separate projects for different applications”

   proof ↗⚑ flag

   W&B Weave~partialclaimed3/10

   Weave's Evaluation framework lets users test against curated Datasets/test examples rather than live production data, and the Playground lets you test prompts/models interactively, which implicitly avoids touching production traffic. However, there is no explicit 'sandbox environment' feature, and other docs (e.g., testing against production traces) actually emphasize using real production data rather than isolating from it. Missing for 10: dedicated sandbox/staging environment concept, explicit data isolation guarantees, and evidence separating test vs production data paths.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
   • [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Arize Phoenix—none0/10

   Evidence shows only generic container/image version pinning (e.g., 'version-8.0.0' Docker tags) but no documented API versioning scheme or deprecation policy for Phoenix's SDK/API; an OpenAPI probe also returned 404s, finding no formal API spec to review versioning against.

   • [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”

   proof ↗⚑ flag

   W&B Weave—none0/10

   No evidence of a versioned API scheme or documented deprecation policy for Weave's SDK/Service API; only an OpenAPI spec presence is shown, not versioning/deprecation commitments. missing for 10: versioning scheme documentation, deprecation policy/notice process, changelog or migration guides for breaking changes.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”

   proof ↗⚑ flag

Monitoring

1. ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics

   weight 2 · round to W&B Weave
   Arize Phoenix—none0/10

   Phoenix's docs describe tracing, span-level latency/cost/token data, and eval scoring, but nowhere is there evidence of a dashboard-building feature (custom charts, widgets, or dashboard layouts) that lets an ML engineer aggregate these metrics into a dashboard view. The evidence only covers per-trace exploration, filtering spans, and projects/sessions organization, not dashboard construction or alerting.

   ⚑ flag

   W&B Weave~partialclaimed6/10

   Weave tracks latency/traces, cost (automatic and custom cost tracking), and eval scores (via Evaluation objects, scorers, leaderboards), and supports alerting via Slack/webhooks on signals — covering most of the metrics named in the story. However, there is no explicit evidence of a customizable dashboard-building UI (e.g., drag-and-drop widgets, custom charts combining these metrics side-by-side) beyond the built-in calls table, trace tree, and leaderboards. missing for 10: explicit custom dashboard/visualization builder evidence, unified view combining latency+error+cost+eval-score in one configurable dashboard, independent/hands-on confirmation of dashboard flexibility.

   • [claimed-docs] “Automatic cost tracking: For supported integrations, Weave captures token usage from the API response and applies built-in pricing for the m…”
   • [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
   • [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”
   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”

   proof ↗⚑ flag

2. developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email

   weight 2 · round to W&B Weave
   Arize Phoenix—none0/10

   The evidence pack covers tracing, evaluation, prompt management, and dataset/experiment features but contains no mention of alerting, notification channels (Slack/PagerDuty/email), or threshold-based triggers for error rates, cost, or eval-score drops.

   ⚑ flag

   W&B Weave~partialclaimed4/10

   Weave marketing docs mention built-in/custom 'signals' that capture and classify agent interactions, with alerts routed via Slack notifications and webhook automations, which could plausibly trigger error-rate or eval-score alerts. However, there is no explicit mention of cost-spike alerts, no native PagerDuty or email notification channels (only Slack + generic webhooks), and no detail on how alert thresholds are configured for error rates or eval-score drops specifically. missing for 10: native PagerDuty integration, native email notification channel, explicit documentation of alert types (error rate, cost spike, eval-score drop) and threshold configuration.

   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”
   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round drawn
   Arize Phoenix~partialclaimed5/10

   Phoenix supports bulk-style operations through datasets (grouping many traces/examples), running experiments and evaluations across thousands of items with built-in concurrency/retry handling, and MCP-based querying/operating on projects, traces, datasets, and prompts. However, there is no explicit documentation of a bulk UI action (e.g., multi-select delete/tag/export) or a dedicated batch API for arbitrary bulk edits across items. missing for 10: explicit bulk UI operations (multi-select edit/delete/export), a documented batch API for arbitrary item updates, and independent/hands-on confirmation of large-scale bulk actions beyond evaluation/experiment runs.

   • [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”

   proof ↗⚑ flag

   W&B Weave~partialclaimed5/10

   Weave's Evaluation object runs scoring across an entire Dataset of many test examples in one call, and the Service API exposes REST endpoints for programmatic access, which together support batch-style automation over many items. However, there is no explicit evidence of bulk trace management operations (bulk delete, bulk tagging, bulk export/update of many logged calls) that a fully bulk-operations story would require. Missing for 10: documented bulk edit/delete/export APIs for traces or datasets, and independent confirmation of large-scale batch throughput.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “The core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples.”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to W&B Weave
   Arize Phoenix—none0/10

   Phoenix's evidence covers tracing, evaluation, datasets, prompt management, and MCP integration, but nothing describes a rules/triggers engine that automatically fires actions on events (e.g., alerting, auto-remediation, webhooks on thresholds). The axis is plausible for an observability platform, but no such capability is evidenced.

   ⚑ flag

   W&B Weave~partialclaimed5/10

   Weave's marketing docs mention custom signals that classify agent interactions and alerts that route to Slack or trigger webhook automations, which matches the idea of rule-based triggers on events. However, this is only briefit is only referenced on the marketing page, with no dedicated guide, configuration reference, or independent confirmation of how rules are authored or scoped. Missing for 10: a dedicated docs page detailing rule/condition syntax, examples of trigger configuration, and independent/hands-on verification that these automations work as described.

   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”

   proof ↗⚑ flag

3. ai-native userVersion, review, and roll back my automations

   weight 1 · round to Arize Phoenix
   Arize Phoenix~partialclaimed5/10

   Phoenix offers prompt versioning ('Version, store, and deploy prompts', 'you can version prompts, test prompt variants... before rolling them out') and dataset/experiment comparisons across app versions, which cover review and some rollback-like workflow for prompts. However, Phoenix is an observability/eval platform, not an automation/workflow builder, so there's no evidence of versioning or rolling back full agentic automations/pipelines themselves — only prompts and datasets. Missing for 10: explicit rollback mechanism for automations/workflows (not just prompts), audit trail of automation changes, and evidence of reverting a live automation to a prior version.

   • [claimed-docs] “Prompt Management — Version, store, and deploy prompts”
   • [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
   • [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
   • [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”

   proof ↗⚑ flag

   W&B Weave~partialclaimed3/10

   Weave documents automatic versioning of traced functions, datasets, and evaluation objects (weave.op(), Evaluation objects) and provides alert/webhook 'automations' for production signals, but there is no evidence of a review or rollback mechanism for these automations/versions. Missing for 10: explicit rollback UI/API for automations, version-history browsing/restore workflow, and evidence tying versioning to the alert/webhook automations themselves.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”

   proof ↗⚑ flag

Cost tracking

1. developerAttribute cost and usage to users, sessions, and features via custom metadata

   weight 2 · round to W&B Weave
   Arize Phoenix~partialclaimed4/10

   Phoenix documents token-usage breakdowns per LLM call and lets you organize traces into projects and group them into sessions/conversations, which provides some basis for usage attribution by session/application. However, there is no evidence of a general custom-metadata/tagging mechanism for arbitrary attribution to users or features, nor any cost aggregation/dashboard that rolls up spend by user, session, or feature tag. missing for 10: explicit custom metadata/tagging API for attributing spans to users or features, cost-specific aggregation/dashboard views by user/session/feature, documentation showing cost roll-ups beyond raw token counts.

   • [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
   • [claimed-docs] “Projects Organize traces into separate projects for different applications”
   • [claimed-docs] “Sessions Group related traces into conversations and user sessions”
   • [claimed-docs] “Organize traces into separate projects for different applications”
   • [claimed-docs] “Group related traces into conversations and user sessions”

   proof ↗⚑ flag

   W&B Weave~partialclaimed5/10

   Weave captures call metadata via weave.op, tracks token usage/cost automatically or via custom cost entries, and groups related calls into Threads (sessions), which together enable some cost/usage attribution. However, there is no explicit documentation of tagging calls with custom user/feature metadata or aggregating/filtering cost by such tags. Missing for 10: explicit custom-attribute tagging API (e.g., user_id/feature tags) and evidence of cost rollups/dashboards filtered by those custom dimensions.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “Threads are collections of Traces related to a single session or conversation. You can use Threads to analyze or score the entire conversati…”
   • [claimed-docs] “Automatic cost tracking: For supported integrations, Weave captures token usage from the API response and applies built-in pricing for the m…”
   • [claimed-docs] “Add a custom cost with the `add_cost` method. The three required fields are `llm_id`, `prompt_token_cost`, and `completion_token_cost`.”
   • [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
   • [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
   • [claimed-docs] “Use custom cost tracking when automatic cost tracking isn’t available, such as with fine-tuned models, self-hosted models, or providers that…”

   proof ↗⚑ flag

2. developerSee cost and token usage per request, model, and time period in dashboards

   weight 3 · round to W&B Weave
   Arize Phoenix~partialclaimed5/10

   Phoenix provides a token usage breakdown per LLM call/span (docs-46) and organizes traces into projects and sessions (docs-15/16/48/49), which supports some per-model and per-time grouping, but there is no evidence of aggregated cost dashboards, dollar-cost calculations, or dedicated per-model/time-period cost views — only per-span token counts. missing for 10: dedicated cost dashboards showing dollar cost (not just tokens), aggregated views by model and by time period, and any UI screenshot/independent confirmation of such dashboards.

   • [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
   • [claimed-docs] “Projects Organize traces into separate projects for different applications”
   • [claimed-docs] “Sessions Group related traces into conversations and user sessions”
   • [claimed-docs] “Organize traces into separate projects for different applications”
   • [claimed-docs] “Group related traces into conversations and user sessions”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed8/10

   Weave automatically tracks token usage and cost per call using built-in pricing for supported integrations, with costs shown in the trace tree and calls table in the Weave UI, plus custom cost support for unsupported models; calls table is filterable/aggregable by model and time via the trace UI. missing for 10: explicit documentation of pre-built cost dashboards aggregating by time period across the whole org, and independent/hands-on confirmation beyond vendor docs.

   • [claimed-docs] “Automatic cost tracking: For supported integrations, Weave captures token usage from the API response and applies built-in pricing for the m…”
   • [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
   • [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
   • [claimed-docs] “Use custom cost tracking when automatic cost tracking isn’t available, such as with fine-tuned models, self-hosted models, or providers that…”
   • [claimed-docs] “Add a custom cost with the `add_cost` method. The three required fields are `llm_id`, `prompt_token_cost`, and `completion_token_cost`.”

   proof ↗⚑ flag

Data export

1. developerBulk-export traces and datasets to blob storage or my data warehouse

   weight 2 · round drawn
   Arize Phoenix—none0/10

   The evidence pack covers tracing, datasets, evaluation, prompt management, and self-hosting, but contains no mention of bulk-exporting traces or datasets to blob storage (S3, GCS) or a data warehouse (Snowflake, BigQuery, etc.). The closest evidence is generic dataset/SDK querying (docs-25, docs-30) which is not the same as a bulk export/warehouse-sync feature.

   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”

   proof ↗⚑ flag

   W&B Weave—none0/10

   Evidence shows Weave has a Service API for programmatic access and OTel import for bringing trace data in, but nothing documents bulk export of traces/datasets to blob storage (S3/GCS) or a data warehouse (Snowflake/BigQuery), which is a reasonable ask for an observability/eval platform.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”

   proof ↗⚑ flag

Ai eval ops

1. ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results

   weight 2 · round to W&B Weave
   Arize Phoenix~partialclaimed6/10

   Phoenix has a Python/SDK-driven dataset creation, experiment/eval-run triggering, and result retrieval (datasets, dataset evaluators, experiments, MCP endpoint letting agents 'operate on datasets, experiments'), which supports an agent programmatically creating datasets, running evals, and reading back scores. However the evidence is all docs-level description without a concrete end-to-end code example or independent confirmation of an agent performing this full loop. Missing for 10: a worked code/API example showing dataset creation -> eval trigger -> result read-back in one flow, and independent/hands-on verification beyond vendor docs.

   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
   • [github] “Datasets - Create versioned datasets of examples for experimen”

   proof ↗⚑ flag

   W&B Weave✓fullprobed8/10

   Weave provides a programmatic Evaluation API with Dataset objects and scoring functions (docs-6, docs-19, docs-25, docs-31), a Service API with REST endpoints for programmatic access to results (docs-15), and an official MCP server explicitly enabling coding agents to 'read live production data, run evaluations, and execute automatic iteration loops on their own' (docs-20, probe-4) — directly matching the agent-driven create-dataset/trigger-eval/read-results workflow. Missing for 10: independent/hands-on confirmation of an agent autonomously completing this full loop end-to-end, and explicit example code showing dataset creation + eval trigger + result read-back in one flow.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Measure every improvement to your harnesses and models with confidence with a flexible imperative evaluation API. Weave provides powerful ev…”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”

   proof ↗⚑ flag

Human review

1. ml engineerRoute outputs to human annotation queues for review and labeling

   weight 2 · round to Arize Phoenix
   Arize Phoenix~partialclaimed6/10

   Phoenix supports attaching human annotations/ground-truth labels directly in the UI on traces/spans, which enables human review and labeling of outputs, and scoring can combine human labels with automated evals. However, there's no evidence of a dedicated 'annotation queue' routing mechanism, task assignment to reviewers, or workflow management for distributing items to human labelers. missing for 10: dedicated annotation queue/task routing to specific reviewers, workflow/assignment management, evidence of multi-user labeling workflows beyond UI annotation.

   • [claimed-docs] “Human annotations — Attach ground truth labels directly in the UI”
   • [claimed-docs] “Attach ground truth labels directly in the UI”
   • [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”

   proof ↗⚑ flag

   W&B Weave—none0/10

   Weave's evidence covers tracing, evaluation, scoring, cost tracking, and feedback collection, but there is no mention of routing outputs to human annotation/labeling queues or any human-in-the-loop review workflow tooling.

   ⚑ flag

Offline evals

1. developerRun evals in CI and gate deployments on their results

   weight 2 · round to Arize Phoenix
   Arize Phoenix~partialclaimed5/10

   Phoenix explicitly frames dataset evaluators as forming 'an evaluation harness similar to a unit test suite' and provides SDK/Python-based experiment running with automatic scoring, which are the building blocks for CI-based evals. However, there is no direct documentation of CI pipeline integration (e.g., GitHub Actions examples), exit-code/pass-fail thresholds, or deployment-gating mechanics tied to eval results. Missing for 10: explicit CI/CD integration docs, pass/fail gating logic, deployment-blocking examples.

   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
   • [claimed-docs] “Dataset Evaluators serve as **test cases** that automatically score outputs when running experiments—forming an evaluation harness similar t…”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”

   proof ↗⚑ flag

   W&B Weave~partialclaimed4/10

   Weave's imperative Evaluation API and Service API (REST) mean evals can be scripted and run programmatically, which a team could wire into a CI job, but the evidence never documents a CI/CD integration, pipeline templates, or a mechanism for gating/blocking deployments based on eval results. Missing for 10: explicit CI/CD integration guides (e.g., GitHub Actions), exit-code/threshold-based gating support, and any documented deployment-blocking workflow.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Measure every improvement to your harnesses and models with confidence with a flexible imperative evaluation API. Weave provides powerful ev…”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”

   proof ↗⚑ flag

2. ml engineerWrite custom code-based scorers and metrics for my evaluations

   weight 2 · round to W&B Weave
   Arize Phoenix✓fullclaimed8/10

   Docs explicitly state Phoenix supports deterministic code-based evaluators (exact match, regex, custom heuristics) alongside LLM-as-a-judge, and Dataset Evaluators act as test cases scoring outputs automatically in experiments similar to a unit test suite, which is exactly code-based custom scoring for evals. Missing for 10: no independent/hands-on corroboration of writing custom Python scorer code, and no detailed API reference/example showing custom scorer implementation.

   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators, where a sec…”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
   • [claimed-docs] “Dataset Evaluators serve as **test cases** that automatically score outputs when running experiments—forming an evaluation harness similar t…”
   • [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed9/10

   Weave's Evaluation object explicitly supports custom scoring functions, and dedicated docs on Custom Scorers describe encoding use-case-specific evaluation criteria beyond built-in scorers, backed by predefined scorers as a baseline. This directly matches writing code-based scorers/metrics for evaluations. Missing for 10: independent/hands-on corroboration beyond vendor docs.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”
   • [claimed-docs] “The core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples.”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [github] “Build rigorous, apples-to-apples evaluations for language model use cases”

   proof ↗⚑ flag

3. ml engineerCompare eval runs side by side to catch regressions between prompt or model versions

   weight 3 · round drawn
   Arize Phoenix✓fullclaimed8/10

   Phoenix's experiments feature explicitly groups traces into datasets, reruns them through different app/prompt/model versions, and compares evaluation results to confirm whether a change improved performance, with automatic scoring via configured evaluators — directly matching the side-by-side regression-catching story. Missing for 10: no independent/hands-on example of an actual side-by-side UI diff view or third-party corroboration beyond vendor docs.

   • [claimed-docs] “optimize your app with experiments that compare changes on the same inputs”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results to confirm whe…”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [github] “Datasets - Create versioned datasets of examples for experimen”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed8/10

   Weave's Evaluation object plus scorers explicitly support comparing runs over time to catch regressions, and docs state comparisons/visualizations exist to 'catch regressions before they reach users,' with leaderboards to aggregate and compare evaluations across versions. missing for 10: no independent/hands-on corroboration of side-by-side UI comparison workflow beyond vendor docs.

   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [claimed-docs] “Measure every improvement to your harnesses and models with confidence with a flexible imperative evaluation API. Weave provides powerful ev…”
   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”
   • [github] “Build rigorous, apples-to-apples evaluations for language model use cases”

   proof ↗⚑ flag

4. ml engineerScore outputs with configurable LLM-as-a-judge evaluators

   weight 3 · round drawn
   Arize Phoenix✓fullclaimed9/10

   Docs explicitly describe LLM-as-a-judge evaluators configurable via model-agnostic adapters (OpenAI, LiteLLM, LangChain, AI SDK), with built-in explanations, automatic scoring of experiment results in the UI or via SDK, and full traceability of judge prompts/reasoning/scores. This is a first-party documented core feature well beyond a single mention. Missing for 10: independent/hands-on corroboration beyond vendor docs.

   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Model Agnostic via adapters (for OpenAI, LiteLLM, LangChain, AI SDK, and more) — so you can easily switch judge models”
   • [claimed-docs] “Built-in Explanations — all Phoenix LLM evaluations return explanations by default for better results and richer signals”
   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators, where a sec…”
   • [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model’s full reasoning, the final scores, a…”
   • [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model's full reasoning, the final scores, a…”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed9/10

   Weave provides a first-class Evaluation object with scoring functions, built-in LLM-judge scorers (hallucination, summarization quality, etc.), and explicit support for custom scorers to encode use-case-specific criteria, plus Guardrails pre-built safety/quality scorers. Missing for 10: independent/hands-on third-party corroboration beyond vendor docs.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”
   • [claimed-docs] “Weave comes with predefined scorers and local SLM scorers that you can use right away, including: Hallucination detection, Summarization qua…”
   • [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [github] “Build rigorous, apples-to-apples evaluations for language model use cases”

   proof ↗⚑ flag

5. ml engineerCurate datasets from production traces and run offline evaluations against them

   weight 3 · round drawn
   Arize Phoenix✓fullclaimed8/10

   Phoenix docs describe grouping production traces into versioned datasets (docs-10, docs-25, gh-1), attaching evaluators (code-based and LLM-as-judge) to datasets and scoring experiment results automatically as an offline eval harness (docs-17, docs-18, docs-26, docs-34), and comparing experiment results across app versions (docs-41). This directly matches curating datasets from traces and running offline evals. missing for 10: independent/hands-on corroboration beyond vendor docs, and concrete example of the full curate-to-evaluate workflow output.

   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [github] “Datasets - Create versioned datasets of examples for experimen”
   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
   • [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
   • [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results to confirm whe…”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed8/10

   Weave supports capturing production traces via @weave.op instrumentation, and explicitly supports building Datasets from these traces for use in its Evaluation object, which runs scoring functions/LLM judges against test examples; docs also mention testing new LLMs/custom models against production traces (offline evaluation). missing for 10: no explicit hands-on/independent example walking through 'export trace → dataset → evaluation' end-to-end, and no third-party corroboration of this specific workflow.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”
   • [claimed-docs] “Weave comes with predefined scorers and local SLM scorers that you can use right away, including: Hallucination detection, Summarization qua…”
   • [github] “Build rigorous, apples-to-apples evaluations for language model use cases”
   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”

   proof ↗⚑ flag

Online evals

1. ml engineerRun evaluators continuously on live production traffic, not just offline datasets

   weight 2 · round to W&B Weave
   Arize Phoenix~partialclaimed5/10

   Docs confirm Phoenix ingests live production traces (OTLP) and can score traces & spans with LLM, code-based, or human evaluators (docs-39, docs-25), suggesting evaluation can be applied beyond static datasets. However, the more detailed evaluation workflow described is dataset/experiment-centric (attach evaluators to datasets, score experiment results) rather than an explicit always-on/scheduled evaluator pipeline running automatically on incoming production traffic. Missing for 10: explicit documentation of a continuous/scheduled evaluation job or streaming eval pipeline that runs on live traces without manual dataset curation, and independent confirmation of this in production use.

   • [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”
   • [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”

   proof ↗⚑ flag

   W&B Weave~partialclaimed6/10

   Weave supports testing against production traces (docs-21), monitoring live agent interactions with signals/alerts (docs-16/17/28/36), and Guardrails scorers can presumably run on live traffic, plus custom/predefined scorers (docs-7,8,22). However, the core Evaluation workflow is explicitly framed around Datasets/lists of test examples run offline (docs-6, docs-31), and there's no explicit documentation of a continuous/streaming online-evaluation pipeline that automatically scores all live production calls in real time as they occur. Missing for 10: explicit documentation of automated/continuous scoring pipelines applied to every live production call (not just ad-hoc production trace sampling), and independent/hands-on confirmation of this online-evaluation mode.

   • [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”
   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to W&B Weave
   Arize Phoenix~partialprobed5/10

   Phoenix's Python/SDK APIs cover most workflows (tracing, datasets, experiments, evals, prompts) matching many UI features, and human annotations/UI-configured evaluators are documented, but there's no comprehensive API/CLI reference confirming full parity with UI-only actions like human annotation labeling or UI-based evaluator configuration, and no public OpenAPI spec was found (404s). missing for 10: a documented OpenAPI/REST spec confirming full UI-to-API parity, explicit API equivalents for UI-only actions like human annotation attachment, and independent verification that every UI action has an API counterpart.

   • [claimed-docs] “Human annotations — Attach ground truth labels directly in the UI”
   • [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
   • [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
   • [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
   • [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”

   proof ↗⚑ flag

   W&B Weave~partialprobed6/10

   Weave exposes a Service API with REST endpoints for programmatic access, plus SDK-level ops for tracing, evaluations, and cost tracking, and an OpenAPI spec is served, indicating broad API coverage. However, some UI-centric features (Playground model comparison/testing, leaderboards, Slack alert configuration) are documented mainly as UI workflows without explicit evidence that every one of these is fully API-exposed. missing for 10: explicit documentation confirming Playground, leaderboards, and alerting/webhook configuration are all fully controllable via the API/SDK rather than just the UI.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
   • [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”
   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to Arize Phoenix
   Arize Phoenix~partialclaimed6/10

   Phoenix is open-source and self-hostable with data staying entirely in the user's infrastructure, and it ingests/exports traces via the open OpenTelemetry (OTLP) standard rather than a proprietary format, which strongly supports data portability and no lock-in. However, there is no explicit documentation of a dedicated 'export all my data' feature (e.g., bulk dataset/trace export to CSV/Parquet or a migration tool) beyond OTLP ingestion and self-hosting. Missing for 10: explicit bulk data export tooling/documentation, independent confirmation of successful full data migration/export by a user.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
   • [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [github] “Datasets - Create versioned datasets of examples for experimen”

   proof ↗⚑ flag

   W&B Weave~partialclaimed3/10

   Weave documents a REST Service API for 'programmatic access to Weave functionality' and OTel-compatible trace import/export interoperability, which could theoretically be used to pull data out in an open format, but there is no explicit documentation of a bulk 'export all my data' feature or guidance for migrating off the platform entirely. missing for 10: explicit bulk export/download feature, documented open-format export (e.g., JSON/OTLP dump of all traces/evals), and any guidance for full data portability or platform exit.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
   • [claimed-docs] “Use this integration when you want to instrument your application with the OpenTelemetry standard and have those traces appear alongside you…”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round to Arize Phoenix
   Arize Phoenix~partialclaimed5/10

   Phoenix's docs state it is 'built by Arize AI and the open-source community' and a public GitHub repo (Arize-ai/phoenix) is referenced, indicating the source is publicly readable. However, no evidence pack item specifies the actual license terms (e.g., MIT, Apache, ELv2) or confirms it is a permissive open-source license. Missing for 10: explicit license identification/text, independent confirmation of license terms, and any discussion of license restrictions.

   • [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
   • [github] “Datasets - Create versioned datasets of examples for experimen”

   proof ↗⚑ flag

   W&B Weave—none0/10

   The evidence pack confirms a public GitHub repository (github.com/wandb/weave) exists with descriptions of its tracing/evaluation code, but none of the citations mention an open-source license (e.g., Apache-2.0/MIT) or any licensing terms at all, so there is no evidence the source is available under an open license.

   • [github] “You can trace any function using `weave.op` - from api calls to OpenAI, Anthropic, Google AI Studio etc to generation calls from Hugging Fac…”
   • [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”
   • [github] “Log and debug language model inputs, outputs, and traces”
   • [github] “Build rigorous, apples-to-apples evaluations for language model use cases”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round to Arize Phoenix
   Arize Phoenix✓fullclaimed9/10

   Docs explicitly state Phoenix is free to self-host with no feature limitations, data stays fully within infrastructure and can be air-gapped, with a simple CLI command (uvx arize-phoenix serve) and Docker image versioning for production deployments. Missing for 10: independent/hands-on confirmation of self-hosting experience beyond vendor docs.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
   • [claimed-docs] “$ uvx arize-phoenix serve”
   • [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”

   proof ↗⚑ flag

   W&B Weave—none0/10

   Weave is documented as a hosted SaaS platform (weave.init() connecting to W&B's cloud) with no evidence pack mentions of a self-hosted or on-prem deployment option for the core Weave product itself; only W&B Models/Platform is known to have enterprise self-hosting but that's not evidenced here for Weave specifically.

   ⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round to Arize Phoenix
   Arize Phoenix~partialclaimed6/10

   Phoenix documents that it is free to self-host with data staying entirely within your own infrastructure and can be fully air-gapped, which lets users choose exactly where their data lives (docs-27/56). However, there is no explicit mention of a managed/cloud offering with selectable regions or residency guarantees, so the 'choose region' capability is only implicit via self-hosting. Missing for 10: explicit region-selection UI/settings, documented data residency options for any hosted/cloud tier, and independent confirmation of deployment flexibility.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
   • [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”

   proof ↗⚑ flag

   W&B Weave—none0/10

   No evidence of region/residency data storage controls for Weave; the pack covers tracing, evaluation, cost tracking, and integrations only, with no mention of self-hosting, EU/US data residency, or region selection options. Missing for 10: any documentation of regional data storage, residency guarantees, or self-hosted/on-prem deployment options.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round to Arize Phoenix
   Arize Phoenix~partialclaimed6/10

   Phoenix's self-hosting docs state that data stays entirely within the user's infrastructure and 'nothing is sent to Arize' and can be fully air-gapped, which strongly implies no data is available to Arize for model training when self-hosted. However, there is no explicit statement of a data-training policy (e.g., for any hosted/cloud offering) or a formal privacy/DPA commitment on this point. Missing for 10: explicit no-training-data policy statement, coverage of hosted/cloud deployment privacy terms, independent verification.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”

   proof ↗⚑ flag

   W&B Weave—none0/10

   No evidence in the pack addresses data usage policies, opt-out of training, or any privacy commitment regarding customer data being used to train models; the evidence pack focuses entirely on tracing, evaluation, and observability features. Missing for 10: any privacy policy statement, training opt-out mechanism, or data usage terms documentation.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round to Arize Phoenix
   Arize Phoenix~partialclaimed5/10

   Phoenix's self-hosting model gives users full control over where data lives ('free to self-host... data stays entirely within your infrastructure... nothing is sent to Arize and can be fully air-gapped'), which implicitly enables retention/deletion control since the operator owns the storage. However, there is no explicit documentation of retention policies, TTL settings, or data-deletion APIs/commands within Phoenix itself. Missing for 10: explicit retention configuration options, documented deletion/purge commands or APIs, and any cloud-hosted retention controls for non-self-hosted deployments.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
   • [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”

   proof ↗⚑ flag

   W&B Weave—none0/10

   The evidence pack covers tracing, evaluation, cost tracking, OTel import, and playground features, but there is no mention of data retention policies, deletion controls, or data lifecycle management for logged traces/data. missing for 10: documentation on data retention periods, user-initiated deletion of traces/projects/data, GDPR/CCPA compliance controls, or any retention configuration options.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round to Arize Phoenix
   Arize Phoenix~partialclaimed6/10

   Phoenix's self-hosting docs state that when self-hosted, 'nothing is sent to Arize and can be fully air-gapped,' implying no forced telemetry to the vendor, which directly supports a privacy-conscious opt-out posture. However, there is no explicit documentation of a dedicated telemetry/usage-tracking toggle (e.g., an env var or CLI flag to disable analytics) for the hosted or default install path. Missing for 10: explicit opt-out mechanism/config flag for anonymous usage analytics, confirmation of what telemetry (if any) is collected by default in non-air-gapped mode.

   • [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
   • [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”

   proof ↗⚑ flag

   W&B Weave—none0/10

   The evidence pack contains no mention of a telemetry opt-out, privacy settings, or usage-tracking controls for Weave itself; all evidence concerns tracing/evaluation features that Weave provides for users' LLM apps, not W&B's own telemetry collection. Since Weave is a SaaS-style observability tool where such an axis plausibly applies, absence of evidence yields 'none'.

   ⚑ flag

Prompt workflow

1. developerIterate on prompts in a playground against real models and variables

   weight 2 · round drawn
   Arize Phoenix✓fullclaimed8/10

   Docs explicitly describe a Prompt Playground where developers can 'play with prompts, models, invocation parameters' and track progress via tracing/experiments, plus use of real production examples/datasets for iteration. This directly matches the story of iterating on prompts against real models and variables. Missing for 10: independent/hands-on user reports specifically about the playground UX (only vendor docs, no third-party corroboration of the playground itself).

   • [claimed-docs] “Play with prompts, models, invocation parameters and track your progress via tracing and experiments”
   • [claimed-docs] “Prompt Playground - Play with prompts, models, invocation parameters and track your progress via tracing and experiments”
   • [claimed-docs] “iterate on your prompts using real production examples”
   • [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
   • [claimed-docs] “Replay the invocation of an LLM. Whether it's an LLM step in an LLM workflow or a router query, you can step into the LLM invocation and see…”
   • [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed8/10

   Weave's Playground explicitly supports prompt editing, message retrying, model comparison, and testing custom/OpenAI-compatible endpoints against real models, plus saved model presets for reusable variable configs, directly matching the story. Missing for 10: independent/hands-on corroboration beyond vendor docs, and explicit detail on templated variable substitution within prompts.

   • [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
   • [claimed-docs] “Custom runtimes: Test OpenAI-compatible API endpoints for custom models.”
   • [claimed-docs] “Saved models: Create and configure a reusable model preset for your workflow.”
   • [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”

   proof ↗⚑ flag

2. developerVersion prompts and deploy changes to production without shipping code

   weight 3 · round to Arize Phoenix
   Arize Phoenix✓fullclaimed8/10

   Phoenix has explicit Prompt Management features supporting versioning, storage, and deployment of prompts, plus 'Prompts in Code' SDKs to sync prompts across environments so changes can be pushed to production without code changes. Docs show a UI-based prompt playground and versioning workflow decoupled from application code deploys. Missing for 10: independent/hands-on verification beyond vendor docs of production rollback/rollout mechanics and no explicit mention of environment-specific staged rollout controls.

   • [claimed-docs] “Prompt Management — Version, store, and deploy prompts”
   • [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
   • [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “Prompts in Code - Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
   • [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”

   proof ↗⚑ flag

   W&B Weave—none0/10

   The evidence pack covers tracing, evaluation, cost tracking, and a Playground for prompt editing/model comparison, but nothing describes a prompt versioning/registry system or a mechanism to push prompt changes to production without redeploying code. Playground's 'prompt editing' (wandb-weave-docs-9) and 'Saved models' preset (wandb-weave-docs-11) are experimentation tools, not a production deployment/versioning workflow for prompts decoupled from code.

   • [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
   • [claimed-docs] “Saved models: Create and configure a reusable model preset for your workflow.”

   proof ↗⚑ flag

Ai observability

1. ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app

   weight 3 · round drawn
   Arize Phoenix✓fullprobed8/10

   Phoenix documents a first-party MCP server (/mcp endpoint) that lets MCP-compatible clients (Claude Code, Cursor, VS Code, etc.) search, query, and operate on projects, traces, datasets, experiments, prompts, and annotations, directly enabling an agent to query traces/metrics/eval results for debugging. This is backed by an official docs page and confirmed by probe evidence. Missing for 10: independent/hands-on third-party validation of the MCP server in practice and a documented general-purpose REST/GraphQL API spec (openapi probes 404) for non-MCP API access.

   • [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
   • [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
   • [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”

   proof ↗⚑ flag

   W&B Weave✓fullprobed8/10

   Weave provides a documented Service API (REST) for programmatic access to traces/evals, plus an official MCP server (wandb-mcp-server) explicitly described as letting coding agents like Claude Code 'read live production data, run evaluations, and execute automatic iteration loops on their own.' This directly matches the story of an agent querying traces/metrics/evals to debug an app. missing for 10: independent/hands-on corroboration of the MCP server in real debugging workflows beyond vendor docs.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”

   proof ↗⚑ flag

Data controls

1. developerMask or redact sensitive data before it is stored in traces

   weight 2 · round drawn
   Arize Phoenix—none0/10

   No evidence pack item mentions masking, redaction, PII scrubbing, or any data-sanitization capability for traces before storage; the docs cover tracing, evaluation, prompts, datasets, and self-hosting but never address sensitive-data redaction.

   ⚑ flag

   W&B Weave—none0/10

   The evidence describes tracing, evaluation, cost tracking, and PII *detection* via Guardrails scorers (wandb-weave-docs-22), but nothing about masking or redacting sensitive data before it is written into stored traces. This is a fair capability to expect from a tracing/instrumentation product, so absence of evidence means 'none' rather than 'na'.

   • [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “When you decorate a function with @weave.op() (Python) or wrap it with weave.op() (TypeScript), Weave automatically captures its code, input…”

   proof ↗⚑ flag

Sdk coverage

1. developerInstrument apps in both Python and JS/TS with officially supported SDKs

   weight 2 · round to W&B Weave
   Arize Phoenix~partialclaimed6/10

   Phoenix documents OTLP ingestion and auto-instrumentation for frameworks including JS/TS ones like Vercel AI SDK and Mastra, plus a phoenix-cli npx setup and OpenInference-based instrumentation, indicating both Python and JS/TS support. However, the evidence is mostly high-level marketing docs without explicit JS/TS SDK API reference or code samples analogous to the detailed Python SDK reference (docs-28/29/30/36), and no independent/hands-on confirmation of JS/TS SDK parity. Missing for 10: explicit JS/TS SDK API reference/docs comparable to the Python SDK reference, hands-on/independent confirmation of JS/TS instrumentation working end-to-end, and clarity on feature parity between the two SDKs.

   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
   • [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
   • [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
   • [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed8/10

   Docs explicitly confirm both Python (@weave.op() decorator) and TypeScript (weave.op() wrap) SDK support for instrumenting functions and LLM calls, with consistent API design across languages. Missing for 10: independent/third-party corroboration of TS SDK parity and maturity, and more detail on JS/TS-specific setup/init beyond the single mention.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “When you decorate a function with @weave.op() (Python) or wrap it with weave.op() (TypeScript), Weave automatically captures its code, input…”
   • [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”
   • [claimed-docs] “Manually instrument your application’s LLM calls and arbitrary functions to trace, version, and collect feedback about your application”

   proof ↗⚑ flag

Trace capture

1. developerTrace multi-step agent runs as nested spans grouped into sessions or threads

   weight 2 · round to W&B Weave
   Arize Phoenix✓fullclaimed8/10

   Docs explicitly describe OTLP-based tracing that captures nested spans for model calls, retrieval, and tool use (docs-38, docs-12/13/14), and Sessions explicitly group related traces into conversations/user sessions while Projects organize traces by application (docs-16/49). Auto-instrumentation for popular agent frameworks (LangChain, LlamaIndex, etc.) supports multi-step agent run tracing out of the box (docs-5, docs-32). Missing for 10: independent/hands-on verification of nested span hierarchy rendering for complex multi-step agents beyond vendor docs.

   • [claimed-docs] “Sessions Group related traces into conversations and user sessions”
   • [claimed-docs] “Group related traces into conversations and user sessions”
   • [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
   • [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
   • [claimed-docs] “Retrieved Documents: Inspect the documents retrieved during a Retriever call, including the score and order in which they were returned”
   • [claimed-docs] “Tool Descriptions: View the descriptions and function signatures of the tools your LLM has been given access to”
   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [claimed-docs] “Projects Organize traces into separate projects for different applications”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed9/10

   Weave supports automatic nested span capture via @weave.op() producing trace trees, plus first-class grouping into Threads/sessions/turns/sub-agents for multi-step agent runs, explicitly designed to navigate agent sessions as executed. Missing for 10: independent hands-on corroboration beyond vendor docs.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “Threads are collections of Traces related to a single session or conversation. You can use Threads to analyze or score the entire conversati…”
   • [claimed-docs] “Weave now brings sessions, turns, steps, tools, and sub-agents as first-class concepts, making it much easier to navigate an agent session t…”
   • [claimed-docs] “Weave organizes traces into sessions and turns from the ground up.”
   • [claimed-docs] “Weave organizes traces into sessions and turns from the ground up. That structure, paired with native analytics tools, makes it easy to trac…”
   • [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”

   proof ↗⚑ flag

2. developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK

   weight 2 · round to Arize Phoenix
   Arize Phoenix✓fullclaimed8/10

   Docs explicitly state Phoenix accepts OTLP traces and provides auto-instrumentation for popular frameworks including LangChain and Vercel AI SDK, plus OpenAI is referenced as an adapter/evaluator target and is a widely documented OpenInference integration. Missing for 10: no independent/hands-on verification of these integrations working, and no explicit dedicated docs snippet for OpenAI SDK auto-instrumentation beyond the adapter mention.

   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
   • [claimed-docs] “Model Agnostic via adapters (for OpenAI, LiteLLM, LangChain, AI SDK, and more) — so you can easily switch judge models”
   • [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”

   proof ↗⚑ flag

   W&B Weave~partialclaimed5/10

   Weave clearly supports auto-instrumentation for OpenAI (and Anthropic, Cohere, Mistral, Hugging Face) via weave.op() decorators/wrappers and has a TypeScript wrap() function, satisfying the OpenAI-SDK part of the story, and it also supports generic OTel-based instrumentation for 'popular SDKs and harnesses.' However, no evidence pack citation explicitly names a LangChain integration or a Vercel AI SDK integration, so those specific framework integrations are unconfirmed. missing for 10: explicit documentation of a first-party LangChain integration, explicit documentation of a Vercel AI SDK integration.

   • [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
   • [github] “You can trace any function using `weave.op` - from api calls to OpenAI, Anthropic, Google AI Studio etc to generation calls from Hugging Fac…”
   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “Trace and collect metrics about your agents built with popular SDKs and harnesses using Weave OTel-compatible SDK”
   • [claimed-docs] “Use this integration when you want to instrument your application with the OpenTelemetry standard and have those traces appear alongside you…”

   proof ↗⚑ flag

3. developerCapture multimodal payloads (images, audio, files) inside my traces

   weight 1 · round drawn
   Arize Phoenix—none0/10

   The evidence pack describes tracing of model calls, retrieval, tool use, latency, and token usage, but nowhere mentions capturing images, audio, or file attachments/multimodal payloads within spans or traces. Since this is a plausible capability for a tracing/observability tool, absence of evidence yields 'none'.

   • [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
   • [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
   • [claimed-docs] “Retrieved Documents: Inspect the documents retrieved during a Retriever call, including the score and order in which they were returned”

   proof ↗⚑ flag

   W&B Weave—none0/10

   The evidence describes Weave's tracing capturing function inputs/outputs, code, and metadata via @weave.op(), but none of the docs or GitHub excerpts mention support for images, audio, or file attachments within traces. Missing for 10: any explicit mention of multimodal payload types (images, audio, files) being captured, rendered, or stored in trace data.

   ⚑ flag

4. developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format

   weight 3 · round to Arize Phoenix
   Arize Phoenix✓fullclaimed9/10

   Docs explicitly state Phoenix accepts traces over OpenTelemetry (OTLP), provides auto-instrumentation for popular frameworks, and is built on top of OpenTelemetry powered by OpenInference instrumentation — confirming standard OTLP ingestion rather than a proprietary format. Missing for 10: independent/hands-on corroboration of OTLP export/receive behavior beyond vendor docs.

   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
   • [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”

   proof ↗⚑ flag

   W&B Weave~partialclaimed7/10

   Weave documents a dedicated OTLP import endpoint and an OTel-compatible SDK so external OpenTelemetry traces can be sent in and appear alongside native Weave traces, rather than requiring the proprietary weave.op format exclusively. However, this is framed as import-only alongside existing pipelines, not a full bidirectional replace-your-format OTLP transport, and there's no evidence of receiving/exporting traces out via OTLP or independent corroboration of interoperability. Missing for 10: evidence of OTLP export/round-trip, independent hands-on confirmation, and clarity that OTel is a full alternative rather than a supplementary ingestion path.

   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
   • [claimed-docs] “Trace and collect metrics about your agents built with popular SDKs and harnesses using Weave OTel-compatible SDK”
   • [claimed-docs] “Use this integration when you want to instrument your application with the OpenTelemetry standard and have those traces appear alongside you…”

   proof ↗⚑ flag

5. developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage

   weight 3 · round drawn
   Arize Phoenix✓fullclaimed9/10

   Phoenix's docs extensively document capturing OTLP/OpenTelemetry-based traces of LLM calls, including inputs/outputs (span data), application latency, and detailed token usage breakdowns, plus auto-instrumentation for popular frameworks and manual tracing decorators. This is a core, heavily documented first-party capability with a CLI/SDK setup flow. Missing for 10: independent hands-on verification beyond vendor docs/community sentiment.

   • [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
   • [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
   • [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
   • [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
   • [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
   • [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
   • [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed9/10

   Weave's @weave.op() decorator automatically captures code, inputs, outputs, and execution metadata for LLM calls, with automatic token usage and cost tracking recorded per call and displayed in the trace tree/UI. Latency is inherently part of the captured trace/execution metadata; OTel-compatible import and GitHub docs corroborate first-party and independent-style evidence. Missing for 10: explicit standalone documentation calling out latency capture by name, and independent (non-vendor) hands-on validation.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
   • [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
   • [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”
   • [github] “Log and debug language model inputs, outputs, and traces”
   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”

   proof ↗⚑ flag