LLM Evals & Observability Arena
Arize Phoenix vs Galileo
Arize Phoenix
Arize AI, Inc.
Arize Phoenix wins · 27–5 (18 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to Arize PhoenixDirect probe evidence confirms llms.txt returns HTTP 200 with agent-oriented content, and docs pages are also available in markdown form (.md) with a documentation index, both explicitly designed for agent/LLM consumption. Missing for 10: no independent/community confirmation of an agent actually consuming these files successfully.
- [probe] “PROBE llms.txt: HTTP 200 at https://arize.com/llms.txt # Arize — AI & Agent Engineering > Arize is the AI engineering platform for teams bu…”
- [probe] “PROBE docs-md: HTTP 200 at https://arize.com/docs/phoenix.md > ## Documentation Index > Fetch the complete documentation index at: https://a…”
Direct probes confirm llms.txt returns HTTP 200 with a documentation index, and individual doc pages provide .md versions with pointers back to llms.txt, making the docs agent-consumable as claimed. missing for 10: no independent/third-party confirmation of an agent actually consuming these files successfully, and no evidence of broader machine-readable spec coverage (e.g., OpenAPI probe returned 404s).
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.galileo.ai/llms.txt # Galileo - [What Is Galileo?](https://docs.galileo.ai/what-is-galileo.md) - […”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.galileo.ai/what-is-galileo.md > ## Documentation Index > Fetch the complete documentation index at: …”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to Arize PhoenixPhoenix ships a CLI (`uvx arize-phoenix serve`), a Docker image pinned for production deployments, and a Python SDK with dataset evaluators explicitly described as forming 'an evaluation harness similar to a unit test suite,' all of which support headless/automated use in CI-like pipelines. However, there is no explicit CI/CD example (e.g., GitHub Actions workflow) or documentation of running evaluations as an automated CI gate. Missing for 10: explicit CI pipeline integration examples/docs, evidence of automated test-gating in CI, independent confirmation of headless CI usage.
- [claimed-docs] “$ uvx arize-phoenix serve”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
Galileo ships a Python SDK (with `@log` decorators, OpenTelemetry distributed tracing, and experiment/dataset APIs) that can be invoked programmatically without the console UI, implying headless/CI usage is possible. However, the evidence never explicitly documents a CI/CD pipeline example, a CLI, or headless-run guidance—experiments are largely framed around the console UI's 'Create Experiment' button. Missing for 10: explicit CI/CD integration docs or examples, a documented CLI/headless entrypoint, and confirmation that experiments can be fully triggered/scored outside the UI.
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
ai-native userConnect an agent via an official MCP server
weight 3 · round drawnPhoenix documents an official remote MCP server exposing a `/mcp` endpoint that MCP-compatible clients (Claude Code, Cursor, VS Code, etc.) can connect to for searching and operating on projects, traces, datasets, experiments, prompts, and annotations. This is first-party documentation confirming the exact capability. Missing for 10: independent/hands-on community corroboration of the MCP server working in practice, and details on authentication/setup robustness.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
Galileo, as an observability/evaluation platform (not itself an agent), documents an official MCP server that lets users access dataset management, experiments, and prompt templates directly from their dev environment, confirmed live via docs page. Missing for 10: independent/hands-on verification beyond first-party docs and details on broader client compatibility.
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
ai-native userUse an official CLI
weight 2 · round to Arize PhoenixPhoenix ships an official CLI (`@arizeai/phoenix-cli` via `npx -y @arizeai/phoenix-cli setup`) for auto-instrumentation setup, plus a `uvx arize-phoenix serve` command to launch the server, confirming a genuine first-party CLI exists for agentic/automated workflows. However, the CLI's documented scope is narrow (setup + serve) rather than covering the full product surface (datasets, evals, prompts, experiments), and there's no independent/community confirmation of its use. Missing for 10: broader CLI command coverage across Phoenix's feature set, independent hands-on verification of the CLI in practice.
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
- [claimed-docs] “$ uvx arize-phoenix serve”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app's root directory.”
- [claimed-docs] “start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app’s root directory”
ai-native userDrive the product through a documented public API
weight 3 · round to Arize PhoenixPhoenix documents a full SDK API reference (tracing decorators, dataset/experiment APIs, prompt sync SDKs), OTLP-based trace ingestion, a CLI (phoenix-cli), and an official remote MCP endpoint that lets AI-native clients search/query/operate on projects, traces, datasets, experiments, and prompts—together constituting a documented, programmatically-drivable public API surface. The openapi.json probe returned 404s at standard paths, meaning there's no discoverable formal REST/OpenAPI spec, so the API surface is SDK/CLI/MCP-centric rather than a single canonical REST API. Missing for 10: a published OpenAPI/REST spec, independent third-party confirmation of API robustness.
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
- [claimed-docs] “$ uvx arize-phoenix serve”
Galileo provides a Python SDK (galileo-python) with decorators for logging traces/spans, an MCP server for programmatic access to datasets/experiments, and OpenTelemetry-based distributed tracing support, indicating a documented API surface for AI-native/agentic use. However, no formal public REST/OpenAPI reference was found (openapi probe returned 404s across all candidate paths), so the API's full documented surface and versioning/auth details are unclear. missing for 10: a discoverable OpenAPI/REST API spec, formal API reference docs beyond SDK/MCP usage, and independent confirmation of API completeness.
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round drawnArize Phoenixnone0/10No evidence in the pack describes scoped or least-privilege API key/credential issuance for agents; Phoenix's docs cover tracing, evaluation, prompt management, and an MCP endpoint, but nothing about credential scoping or access control granularity.
ai-native userBuild against official SDKs
weight 2 · round to Arize PhoenixPhoenix ships an official, modular Python SDK plus client SDKs for tracing, prompts, evaluations, and datasets, with OTLP-based tracing decorators, auto-instrumentation, and dedicated SDK/API reference docs. Coverage spans core agentic workflows (tracing, evals, prompt management, datasets/experiments) and is corroborated by GitHub repo and docs pages, though independent hands-on SDK usage reports are thin. Missing for 10: broader independent/third-party validation of SDK usage beyond official docs, and explicit multi-language SDK support details beyond Python/JS CLI.
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Prompts in Code - Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
- [github] “Datasets - Create versioned datasets of examples for experimen”
Galileo has an official Python SDK (galileo-python) with decorators/logging APIs referenced in GitHub docs, plus MCP server integration for accessing Galileo capabilities from dev environments, supporting AI-native/agentic workflows. However, evidence lacks details on multi-language SDK coverage, versioning/stability, or independent developer corroboration beyond first-party docs, and OpenAPI spec probes all 404'd. Missing for 10: multi-language SDK evidence, independent hands-on validation, public API reference/OpenAPI spec.
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
ai-native userSubscribe to events via webhooks
weight 2 · round drawnArize Phoenixnone0/10No evidence anywhere in the pack of a webhook subscription mechanism; Phoenix's integration surface is OTLP tracing ingestion, an MCP server, and SDKs, but nothing about outbound event webhooks for subscribing to Phoenix events.
Galileonone0/10There is a mention of alerting (galileo-docs-9) but no evidence of webhook subscription support; the OpenAPI/API endpoints probe returned 404s and nothing in the evidence pack describes webhooks or event subscription mechanisms.
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round drawnPhoenix offers LLM-as-judge evaluators that return explanations and reasoning alongside scores, and lets users explore traces to spot systematic issues, which counts as AI-generated signal on the data — but this is evaluator-driven scoring, not proactive, unprompted insights/suggestions surfaced automatically inside the product UI. missing for 10: evidence of autonomous/proactive insight generation (e.g., auto-detected anomalies, recommended fixes, natural-language summaries of trends) rather than only judge-model scoring/explanations tied to evals a user configures.
- [claimed-docs] “Built-in Explanations — all Phoenix LLM evaluations return explanations by default for better results and richer signals”
- [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model's full reasoning, the final scores, a…”
- [claimed-docs] “Use Phoenix's trace viewer to explore evaluation traces, identify systematic biases, and continuously improve evaluator performance.”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
Galileo offers LLM-as-a-judge and custom metrics that can evaluate outputs, natural-language feedback loops that auto-improve metric alignment, and alerts on anomalies, which imply some AI-assisted analysis of data — but there is no explicit documentation of a dedicated 'insights/suggestions' feature that proactively surfaces AI-generated recommendations to users. Missing for 10: a clear insights/suggestions UI or feature description, independent examples of such AI-generated recommendations in use, and confirmation this goes beyond metric scoring to actionable suggestions.
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
- [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”
- [claimed-docs] “Agentic metrics help you measure how well your AI agents perform complex, multi-step tasks—especially when those agents need to use tools, m…”
ai-native userSet up automations that run autonomously in the background
weight 2 · round to GalileoArize Phoenixnone0/10Phoenix's evidence covers tracing, evaluation, prompt management, and datasets, but nothing describes scheduled or autonomous background automations (e.g., recurring eval jobs, alerting rules, or triggers) that run without user initiation. The evaluation/experiment features described are invoked by the user (via SDK/UI), not autonomous background processes.
Galileo documents background alerting ('get alerted whenever unexpected things happen') and continuous automatic capture of traces/spans, which are autonomous background processes, but there is no evidence of a general-purpose automation/scheduling system for agentic workflows that a user configures to run independently. missing for 10: explicit automation/workflow scheduler, triggers/conditions configuration, evidence of autonomous multi-step agent execution beyond monitoring/alerts.
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round drawnArize Phoenixnone0/10Phoenix is an observability/evaluation platform; the evidence describes tracing, evals, prompt management, datasets, and an MCP server that lets *external* agents (Claude Code, Cursor, etc.) operate on Phoenix data — not a built-in AI assistant living inside Phoenix that users delegate tasks to. No docs mention any embedded assistant/chat agent within the Phoenix UI itself.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app's root directory.”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
ai-native userOperate the product with natural-language commands
weight 2 · round to Arize PhoenixPhoenix documents a dedicated remote MCP integration letting any MCP-compatible client (Claude Code, Cursor, VS Code, etc.) operate on projects, traces, datasets, experiments, prompts, and annotations via natural-language driven agents, and also supports agent-driven CLI setup. This directly satisfies AI-native natural-language operability. Missing for 10: independent/hands-on validation of the MCP command experience and more detail on the breadth/reliability of natural-language operations beyond the docs claim.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
Galileo ships an official MCP server that lets AI assistants create/manage datasets, run experiments, and set up prompt templates from a dev environment using natural language, and it supports continuous natural-language feedback to refine metrics — both are concrete NL-driven operation paths. However, there's no evidence of a native chat/NL command interface within the Galileo console itself; missing for 10: first-party in-app NL command console, broader coverage of all product actions via NL, and independent hands-on confirmation of the MCP NL workflow.
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnArize Phoenixnone0/10Docs mention an 'sdk-api-reference' page listing decorators and SDK features, but there is no evidence of an interactive, runnable API reference (e.g., a Swagger/OpenAPI explorer or live code sandbox); a direct probe for OpenAPI/swagger specs returned 404 on all candidate paths, indicating no such interactive reference is discoverable.
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
Galileonone0/10No evidence of an interactive API reference with runnable examples; openapi probes all returned 404, and no Swagger/Redoc-style playground is mentioned anywhere in the docs pack.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round drawnArize Phoenixnone0/10A direct probe for OpenAPI/swagger endpoints at Phoenix's domain returned 404 for all candidate paths, and no docs mention a downloadable machine-readable API spec. Missing for 10: any OpenAPI/Swagger spec, documented REST API reference, or SDK-generated schema.
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
Galileonone0/10A direct probe for OpenAPI/Swagger spec files at all standard locations returned 404, and no documentation references a downloadable machine-readable API spec; only an llms.txt index and MCP server exist, neither of which is an OpenAPI spec.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to Arize PhoenixPhoenix supports self-hosting with full data isolation (data never leaves your infra) and lets you organize traces into separate 'projects', collect data 'from production, staging, evaluations, and even manually' into datasets, and run experiments/evaluations against those datasets rather than live traffic — all of which let a user test without touching real production data. However, there's no explicit documented 'sandbox environment' feature, no isolated test-instance provisioning, and no guidance on separating staging from prod deployments beyond generic self-hosting flexibility. Missing for 10: explicit sandbox/staging environment tooling, isolation guarantees beyond self-hosting, and independent confirmation that experiments truly never touch production data.
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
- [claimed-docs] “Organize traces into separate projects for different applications”
Galileo's Experiments feature lets users evaluate prompts/models against well-defined inputs and datasets can be built from 'synthetic, development, and live production data,' implying some separation between test and production data, but there is no explicit sandbox/staging environment concept described. missing for 10: explicit sandbox/staging environment docs, isolation guarantees from production data, and independent confirmation of non-production testing workflow.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnArize Phoenixnone0/10Evidence shows only generic container/image version pinning (e.g., 'version-8.0.0' Docker tags) but no documented API versioning scheme or deprecation policy for Phoenix's SDK/API; an OpenAPI probe also returned 404s, finding no formal API spec to review versioning against.
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
Galileonone0/10No evidence of API versioning scheme or a documented deprecation policy; OpenAPI spec probes returned 404 and no changelog/versioning docs are present in the evidence pack.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Alerting dashboards — stories about alerting dashboards in this arenaAlerting dashboards
Stories about alerting dashboards in this arena
Monitoring
ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics
weight 2 · round drawnArize Phoenixnone0/10Phoenix's docs describe tracing, span-level latency/cost/token data, and eval scoring, but nowhere is there evidence of a dashboard-building feature (custom charts, widgets, or dashboard layouts) that lets an ML engineer aggregate these metrics into a dashboard view. The evidence only covers per-trace exploration, filtering spans, and projects/sessions organization, not dashboard construction or alerting.
Galileonone0/10Evidence covers logging/tracing (latency, spans), custom metrics, LLM-as-judge evals, and alerting, but no documentation describes a dashboard-building UI or customizable visualization layer combining latency, error, cost, and eval-score metrics. missing for 10: dashboard/widget customization UI, evidence of combining metrics types into a single view, cost-metric tracking, independent/hands-on confirmation of dashboarding.
developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email
weight 2 · round to GalileoArize Phoenixnone0/10The evidence pack covers tracing, evaluation, prompt management, and dataset/experiment features but contains no mention of alerting, notification channels (Slack/PagerDuty/email), or threshold-based triggers for error rates, cost, or eval-score drops.
Galileo docs confirm a general alerting capability ("get alerted whenever unexpected things happen" via a 'set up alerts on logs' guide), but the evidence pack contains no detail on which triggers (error rate, cost spike, eval-score drop) are supported nor which notification channels (Slack, PagerDuty, email) are integrated. Missing for 10: documented list of supported alert conditions, confirmation of Slack/PagerDuty/email integrations, and any hands-on/independent verification of alert delivery.
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round drawnPhoenix supports bulk-style operations through datasets (grouping many traces/examples), running experiments and evaluations across thousands of items with built-in concurrency/retry handling, and MCP-based querying/operating on projects, traces, datasets, and prompts. However, there is no explicit documentation of a bulk UI action (e.g., multi-select delete/tag/export) or a dedicated batch API for arbitrary bulk edits across items. missing for 10: explicit bulk UI operations (multi-select edit/delete/export), a documented batch API for arbitrary item updates, and independent/hands-on confirmation of large-scale bulk actions beyond evaluation/experiment runs.
- [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
Galileo's Experiments feature runs evaluations across datasets of many inputs at once, and MCP/SDK access lets users programmatically create and manage datasets and run experiments in bulk from code rather than one item at a time (galileo-docs-2, galileo-docs-3, galileo-docs-4, galileo-docs-11). However, there is no explicit documentation of bulk edit/delete/tag/annotate operations across arbitrary large sets of existing items in the console or API. Missing for 10: explicit bulk CRUD operations (batch edit/delete/tag) across items, batch API endpoints/rate-limit guidance for large-scale automation, and independent confirmation of bulk-scale reliability.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to GalileoArize Phoenixnone0/10Phoenix's evidence covers tracing, evaluation, datasets, prompt management, and MCP integration, but nothing describes a rules/triggers engine that automatically fires actions on events (e.g., alerting, auto-remediation, webhooks on thresholds). The axis is plausible for an observability platform, but no such capability is evidenced.
Galileo docs mention that users can set up alerts on logs to be notified of unexpected events, which is a basic rule-trigger-on-event capability, but there is no detail on defining custom rule logic, condition types, or automated actions beyond alerting (e.g., webhooks, workflow triggers, remediation actions). missing for 10: rule definition UI/API details, supported trigger conditions, and evidence of automated actions beyond simple alert notifications.
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
ai-native userSchedule recurring jobs or workflows
weight 2 · round drawnArize Phoenixnone0/10Phoenix is an observability/evaluation platform for LLM apps (tracing, evals, prompt management, datasets) but no evidence describes any scheduler, cron-like job runner, or recurring workflow automation feature.
Galileonone0/10Galileo is an LLM observability/evaluation platform with experiments, alerts, and MCP integration, but no evidence describes scheduling recurring jobs or workflows (e.g., cron-like automation, scheduled evaluation runs). Alerts are reactive, not scheduled, and no scheduler feature is documented.
Cost monitoring — stories about cost monitoring in this arenaCost monitoring
Stories about cost monitoring in this arena
Cost tracking
developerAttribute cost and usage to users, sessions, and features via custom metadata
weight 2 · round drawnPhoenix documents token-usage breakdowns per LLM call and lets you organize traces into projects and group them into sessions/conversations, which provides some basis for usage attribution by session/application. However, there is no evidence of a general custom-metadata/tagging mechanism for arbitrary attribution to users or features, nor any cost aggregation/dashboard that rolls up spend by user, session, or feature tag. missing for 10: explicit custom metadata/tagging API for attributing spans to users or features, cost-specific aggregation/dashboard views by user/session/feature, documentation showing cost roll-ups beyond raw token counts.
- [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
- [claimed-docs] “Sessions Group related traces into conversations and user sessions”
- [claimed-docs] “Organize traces into separate projects for different applications”
- [claimed-docs] “Group related traces into conversations and user sessions”
Galileo's logging captures sessions, traces, and spans (galileo-docs-6) and supports custom metrics (galileo-docs-8), and mentions Luna models monitoring traffic at lower cost (galileo-docs-12), implying some usage/cost tracking infrastructure exists. However, there is no explicit documentation of tagging traces/sessions with custom metadata fields (e.g., user ID, feature name) for cost attribution or cost breakdown by dimension. missing for 10: explicit custom metadata tagging API/fields for user/session/feature attribution, cost-per-tag reporting or dashboards, and any hands-on example of cost attribution via metadata.
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
- [claimed-docs] “Distill your optimized evals into Luna models that monitor 100% of your traffic at 96% lower cost.”
developerSee cost and token usage per request, model, and time period in dashboards
weight 3 · round to Arize PhoenixPhoenix provides a token usage breakdown per LLM call/span (docs-46) and organizes traces into projects and sessions (docs-15/16/48/49), which supports some per-model and per-time grouping, but there is no evidence of aggregated cost dashboards, dollar-cost calculations, or dedicated per-model/time-period cost views — only per-span token counts. missing for 10: dedicated cost dashboards showing dollar cost (not just tokens), aggregated views by model and by time period, and any UI screenshot/independent confirmation of such dashboards.
- [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
- [claimed-docs] “Sessions Group related traces into conversations and user sessions”
- [claimed-docs] “Organize traces into separate projects for different applications”
- [claimed-docs] “Group related traces into conversations and user sessions”
Galileonone0/10The evidence pack covers tracing, experiments, metrics, and alerts, but contains no mention of cost or token usage tracking, nor dashboards broken down by request, model, or time period. This is a plausible axis for an LLM observability platform, so absence of evidence yields 'none' rather than 'na'.
Data access export — stories about data access export in this arenaData access export
Stories about data access export in this arena
Data export
developerBulk-export traces and datasets to blob storage or my data warehouse
weight 2 · round drawnArize Phoenixnone0/10The evidence pack covers tracing, datasets, evaluation, prompt management, and self-hosting, but contains no mention of bulk-exporting traces or datasets to blob storage (S3, GCS) or a data warehouse (Snowflake, BigQuery, etc.). The closest evidence is generic dataset/SDK querying (docs-25, docs-30) which is not the same as a bulk export/warehouse-sync feature.
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
Galileonone0/10No evidence of any bulk-export capability to blob storage or a data warehouse; docs cover logging, tracing, experiments, and MCP dataset management but never mention exporting data out to S3/GCS/BigQuery/Snowflake, and the OpenAPI probe returned 404s with no export endpoint mentioned.
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets
Measuring quality — datasets, eval runs, regression tracking
Ai eval ops
ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results
weight 2 · round to GalileoPhoenix has a Python/SDK-driven dataset creation, experiment/eval-run triggering, and result retrieval (datasets, dataset evaluators, experiments, MCP endpoint letting agents 'operate on datasets, experiments'), which supports an agent programmatically creating datasets, running evals, and reading back scores. However the evidence is all docs-level description without a concrete end-to-end code example or independent confirmation of an agent performing this full loop. Missing for 10: a worked code/API example showing dataset creation -> eval trigger -> result read-back in one flow, and independent/hands-on verification beyond vendor docs.
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [github] “Datasets - Create versioned datasets of examples for experimen”
Galileo's official MCP server explicitly exposes dataset creation, experiment (eval) running, and prompt template management directly from an agent's dev environment, and separate SDK/decorator logging plus experiment docs confirm results are captured and queryable. Missing for 10: no hands-on/independent confirmation of an agent actually reading back structured eval results via MCP, and no explicit example showing the full create-dataset→run-eval→read-results loop end-to-end.
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
Human review
ml engineerRoute outputs to human annotation queues for review and labeling
weight 2 · round to Arize PhoenixPhoenix supports attaching human annotations/ground-truth labels directly in the UI on traces/spans, which enables human review and labeling of outputs, and scoring can combine human labels with automated evals. However, there's no evidence of a dedicated 'annotation queue' routing mechanism, task assignment to reviewers, or workflow management for distributing items to human labelers. missing for 10: dedicated annotation queue/task routing to specific reviewers, workflow/assignment management, evidence of multi-user labeling workflows beyond UI annotation.
- [claimed-docs] “Human annotations — Attach ground truth labels directly in the UI”
- [claimed-docs] “Attach ground truth labels directly in the UI”
- [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”
Galileo docs mention capturing 'subject matter expert annotations' to build datasets and using natural-language feedback to align metrics, implying some human-in-the-loop capability, but there is no explicit documentation of a dedicated annotation queue, review workflow, task assignment, or labeling UI for routing outputs to human reviewers. missing for 10: dedicated annotation queue/workflow feature, reviewer assignment mechanism, labeling UI documentation, independent corroboration of human review routing.
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
- [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”
Offline evals
developerRun evals in CI and gate deployments on their results
weight 2 · round to Arize PhoenixPhoenix explicitly frames dataset evaluators as forming 'an evaluation harness similar to a unit test suite' and provides SDK/Python-based experiment running with automatic scoring, which are the building blocks for CI-based evals. However, there is no direct documentation of CI pipeline integration (e.g., GitHub Actions examples), exit-code/pass-fail thresholds, or deployment-gating mechanics tied to eval results. Missing for 10: explicit CI/CD integration docs, pass/fail gating logic, deployment-blocking examples.
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “Dataset Evaluators serve as **test cases** that automatically score outputs when running experiments—forming an evaluation harness similar t…”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
Galileonone0/10Evidence shows experiments/evals can be run via console or SDK, but there is no mention of a CI integration, CLI exit codes, or gating deployments based on eval results. missing for 10: CI/CD integration docs, pass/fail thresholds for gating, pipeline examples (GitHub Actions, Jenkins, etc.), any mention of 'CI' or 'gate' in evidence.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
ml engineerWrite custom code-based scorers and metrics for my evaluations
weight 2 · round to Arize PhoenixDocs explicitly state Phoenix supports deterministic code-based evaluators (exact match, regex, custom heuristics) alongside LLM-as-a-judge, and Dataset Evaluators act as test cases scoring outputs automatically in experiments similar to a unit test suite, which is exactly code-based custom scoring for evals. Missing for 10: no independent/hands-on corroboration of writing custom Python scorer code, and no detailed API reference/example showing custom scorer implementation.
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators, where a sec…”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “Dataset Evaluators serve as **test cases** that automatically score outputs when running experiments—forming an evaluation harness similar t…”
- [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”
Docs explicitly mention custom code-based metrics as an extension to built-in metrics, alongside LLM-as-a-judge metrics, confirming the capability exists. However, evidence lacks concrete SDK examples, API reference, or hands-on walkthroughs showing how to author and register a custom scorer function. Missing for 10: code samples/API reference for writing custom scorers, independent corroboration of usage, details on scorer registration/execution lifecycle.
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
ml engineerCompare eval runs side by side to catch regressions between prompt or model versions
weight 3 · round to Arize PhoenixPhoenix's experiments feature explicitly groups traces into datasets, reruns them through different app/prompt/model versions, and compares evaluation results to confirm whether a change improved performance, with automatic scoring via configured evaluators — directly matching the side-by-side regression-catching story. Missing for 10: no independent/hands-on example of an actual side-by-side UI diff view or third-party corroboration beyond vendor docs.
- [claimed-docs] “optimize your app with experiments that compare changes on the same inputs”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results to confirm whe…”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [github] “Datasets - Create versioned datasets of examples for experimen”
Galileo's Experiments feature lets you evaluate prompts, models, and code against chosen metrics, and the console provides a way to create and add experiments to a project, implying some run-to-run evaluation tracking. However, no evidence explicitly describes a side-by-side comparison view or regression-detection UI/workflow between prompt or model versions. missing for 10: explicit side-by-side comparison UI, diffing/regression alerts between experiment runs, independent user confirmation of comparison workflow.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
ml engineerScore outputs with configurable LLM-as-a-judge evaluators
weight 3 · round to Arize PhoenixDocs explicitly describe LLM-as-a-judge evaluators configurable via model-agnostic adapters (OpenAI, LiteLLM, LangChain, AI SDK), with built-in explanations, automatic scoring of experiment results in the UI or via SDK, and full traceability of judge prompts/reasoning/scores. This is a first-party documented core feature well beyond a single mention. Missing for 10: independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Model Agnostic via adapters (for OpenAI, LiteLLM, LangChain, AI SDK, and more) — so you can easily switch judge models”
- [claimed-docs] “Built-in Explanations — all Phoenix LLM evaluations return explanations by default for better results and richer signals”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators, where a sec…”
- [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model’s full reasoning, the final scores, a…”
- [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model's full reasoning, the final scores, a…”
Docs explicitly describe LLM-as-a-judge as a configurable metric type alongside custom code-based metrics, plus continuous feedback loops to align metrics to domain needs, and experiments to run these metrics against outputs. Missing for 10: independent/hands-on corroboration beyond vendor docs and more detail on configuring specific judge prompts/models.
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
- [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
ml engineerCurate datasets from production traces and run offline evaluations against them
weight 3 · round drawnPhoenix docs describe grouping production traces into versioned datasets (docs-10, docs-25, gh-1), attaching evaluators (code-based and LLM-as-judge) to datasets and scoring experiment results automatically as an offline eval harness (docs-17, docs-18, docs-26, docs-34), and comparing experiment results across app versions (docs-41). This directly matches curating datasets from traces and running offline evals. missing for 10: independent/hands-on corroboration beyond vendor docs, and concrete example of the full curate-to-evaluate workflow output.
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [github] “Datasets - Create versioned datasets of examples for experimen”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results to confirm whe…”
Galileo explicitly supports building datasets from production/live traces (galileo-docs-11), capturing traces/spans in production (galileo-docs-6, galileo-docs-7), and running offline evaluations/experiments against datasets with custom or LLM-as-judge metrics (galileo-docs-2, galileo-docs-8). This directly covers curating datasets from production traces and running offline evals. Missing for 10: independent/hands-on corroboration of the full production-trace-to-dataset-to-experiment workflow beyond vendor docs.
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
Online evals
ml engineerRun evaluators continuously on live production traffic, not just offline datasets
weight 2 · round to GalileoDocs confirm Phoenix ingests live production traces (OTLP) and can score traces & spans with LLM, code-based, or human evaluators (docs-39, docs-25), suggesting evaluation can be applied beyond static datasets. However, the more detailed evaluation workflow described is dataset/experiment-centric (attach evaluators to datasets, score experiment results) rather than an explicit always-on/scheduled evaluator pipeline running automatically on incoming production traffic. Missing for 10: explicit documentation of a continuous/scheduled evaluation job or streaming eval pipeline that runs on live traces without manual dataset curation, and independent confirmation of this in production use.
- [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
Galileo's docs describe real-time capture of every session/trace/span, alerting on live logs, and distilled 'Luna' models that monitor 100% of production traffic at lower cost, which directly supports continuous evaluator execution on live traffic (not just offline datasets), complemented by custom/LLM-as-judge metrics and datasets built from live production data. missing for 10: independent/hands-on verification that evaluators run continuously in production (all evidence is vendor docs) and clearer detail on evaluator scheduling/latency at scale.
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
- [claimed-docs] “Distill your optimized evals into Luna models that monitor 100% of your traffic at 96% lower cost.”
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round drawnPhoenix's Python/SDK APIs cover most workflows (tracing, datasets, experiments, evals, prompts) matching many UI features, and human annotations/UI-configured evaluators are documented, but there's no comprehensive API/CLI reference confirming full parity with UI-only actions like human annotation labeling or UI-based evaluator configuration, and no public OpenAPI spec was found (404s). missing for 10: a documented OpenAPI/REST spec confirming full UI-to-API parity, explicit API equivalents for UI-only actions like human annotation attachment, and independent verification that every UI action has an API counterpart.
- [claimed-docs] “Human annotations — Attach ground truth labels directly in the UI”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
Galileo exposes a Python SDK, decorators, and an MCP server that cover core workflows (tracing, experiments, datasets, prompt templates), suggesting many UI actions have API/SDK equivalents (galileo-docs-2, galileo-docs-4, galileo-gh-1). However, docs explicitly describe some actions (e.g., 'Create Experiment' button) as UI-only, and no public OpenAPI/swagger spec is discoverable (galileo-probe-3), so full API parity is unproven. Missing for 10: an explicit statement or spec confirming 1:1 API/UI feature parity, a discoverable OpenAPI reference, and evidence that console-only features (alerts setup, dashboards) have API equivalents.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [github] “You can also use the `@log` decorator to log spans.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
ai-native userExport all of my data in open formats and leave
weight 3 · round to Arize PhoenixPhoenix is open-source and self-hostable with data staying entirely in the user's infrastructure, and it ingests/exports traces via the open OpenTelemetry (OTLP) standard rather than a proprietary format, which strongly supports data portability and no lock-in. However, there is no explicit documentation of a dedicated 'export all my data' feature (e.g., bulk dataset/trace export to CSV/Parquet or a migration tool) beyond OTLP ingestion and self-hosting. Missing for 10: explicit bulk data export tooling/documentation, independent confirmation of successful full data migration/export by a user.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [github] “Datasets - Create versioned datasets of examples for experimen”
ai-native userRead the product's source under an open license
weight 2 · round to Arize PhoenixPhoenix's docs state it is 'built by Arize AI and the open-source community' and a public GitHub repo (Arize-ai/phoenix) is referenced, indicating the source is publicly readable. However, no evidence pack item specifies the actual license terms (e.g., MIT, Apache, ELv2) or confirms it is a permissive open-source license. Missing for 10: explicit license identification/text, independent confirmation of license terms, and any discussion of license restrictions.
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
- [github] “Datasets - Create versioned datasets of examples for experimen”
Evidence shows only a GitHub repo for the Python client SDK (galileo-python), with no license details or indication that the core Galileo platform/backend is open source. missing for 10: explicit open-source license text, evidence that the full product (not just a client SDK) is source-available, independent confirmation of license terms.
- [github] “You can also use the `@log` decorator to log spans.”
ai-native userSelf-host the core product
weight 3 · round to Arize PhoenixDocs explicitly state Phoenix is free to self-host with no feature limitations, data stays fully within infrastructure and can be air-gapped, with a simple CLI command (uvx arize-phoenix serve) and Docker image versioning for production deployments. Missing for 10: independent/hands-on confirmation of self-hosting experience beyond vendor docs.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “$ uvx arize-phoenix serve”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
Galileonone0/10No evidence of a self-hostable/on-prem version of Galileo; all documentation points to a hosted console/SaaS product with SDKs and MCP integration, not a self-hosted deployment option. missing for 10: any mention of self-hosting, on-prem deployment, Docker/Helm packages, or enterprise private-cloud install instructions.
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round to Arize PhoenixPhoenix documents that it is free to self-host with data staying entirely within your own infrastructure and can be fully air-gapped, which lets users choose exactly where their data lives (docs-27/56). However, there is no explicit mention of a managed/cloud offering with selectable regions or residency guarantees, so the 'choose region' capability is only implicit via self-hosting. Missing for 10: explicit region-selection UI/settings, documented data residency options for any hosted/cloud tier, and independent confirmation of deployment flexibility.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
Galileonone0/10No evidence pack item mentions data residency, regional storage options, or compliance controls for data location; the evidence covers tracing, experiments, metrics, and MCP only. Since Galileo is a SaaS platform where data residency is a plausible and common enterprise concern, absence of evidence means 'none' rather than 'na'.
ai-native userPrevent my data from being used to train AI models
weight 3 · round to Arize PhoenixPhoenix's self-hosting docs state that data stays entirely within the user's infrastructure and 'nothing is sent to Arize' and can be fully air-gapped, which strongly implies no data is available to Arize for model training when self-hosted. However, there is no explicit statement of a data-training policy (e.g., for any hosted/cloud offering) or a formal privacy/DPA commitment on this point. Missing for 10: explicit no-training-data policy statement, coverage of hosted/cloud deployment privacy terms, independent verification.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
ai-native userControl data retention and deletion
weight 2 · round to Arize PhoenixPhoenix's self-hosting model gives users full control over where data lives ('free to self-host... data stays entirely within your infrastructure... nothing is sent to Arize and can be fully air-gapped'), which implicitly enables retention/deletion control since the operator owns the storage. However, there is no explicit documentation of retention policies, TTL settings, or data-deletion APIs/commands within Phoenix itself. Missing for 10: explicit retention configuration options, documented deletion/purge commands or APIs, and any cloud-hosted retention controls for non-self-hosted deployments.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
Galileonone0/10The evidence pack covers tracing, experiments, metrics, and MCP integration but contains no mention of data retention policies, deletion controls, or privacy/compliance configuration options for AI-native users. No documentation cites retention windows, data deletion APIs, or export/purge capabilities.
ai-native userOpt out of telemetry and usage tracking
weight 2 · round to Arize PhoenixPhoenix's self-hosting docs state that when self-hosted, 'nothing is sent to Arize and can be fully air-gapped,' implying no forced telemetry to the vendor, which directly supports a privacy-conscious opt-out posture. However, there is no explicit documentation of a dedicated telemetry/usage-tracking toggle (e.g., an env var or CLI flag to disable analytics) for the hosted or default install path. Missing for 10: explicit opt-out mechanism/config flag for anonymous usage analytics, confirmation of what telemetry (if any) is collected by default in non-air-gapped mode.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
Prompt management — stories about prompt management in this arenaPrompt management
Stories about prompt management in this arena
Prompt workflow
developerIterate on prompts in a playground against real models and variables
weight 2 · round to Arize PhoenixDocs explicitly describe a Prompt Playground where developers can 'play with prompts, models, invocation parameters' and track progress via tracing/experiments, plus use of real production examples/datasets for iteration. This directly matches the story of iterating on prompts against real models and variables. Missing for 10: independent/hands-on user reports specifically about the playground UX (only vendor docs, no third-party corroboration of the playground itself).
- [claimed-docs] “Play with prompts, models, invocation parameters and track your progress via tracing and experiments”
- [claimed-docs] “Prompt Playground - Play with prompts, models, invocation parameters and track your progress via tracing and experiments”
- [claimed-docs] “iterate on your prompts using real production examples”
- [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
- [claimed-docs] “Replay the invocation of an LLM. Whether it's an LLM step in an LLM workflow or a router query, you can step into the LLM invocation and see…”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
Galileo's Experiments feature lets users evaluate prompts and models against defined inputs and metrics via a console UI, and prompt templates can be set up through the MCP integration, which loosely maps to prompt iteration with variables. However, there's no evidence of a dedicated interactive 'playground' for live, real-time prompt testing against models — missing for 10: a documented playground UI, live model response preview, and variable substitution workflow distinct from formal experiment runs.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
developerVersion prompts and deploy changes to production without shipping code
weight 3 · round to Arize PhoenixPhoenix has explicit Prompt Management features supporting versioning, storage, and deployment of prompts, plus 'Prompts in Code' SDKs to sync prompts across environments so changes can be pushed to production without code changes. Docs show a UI-based prompt playground and versioning workflow decoupled from application code deploys. Missing for 10: independent/hands-on verification beyond vendor docs of production rollback/rollout mechanics and no explicit mention of environment-specific staged rollout controls.
- [claimed-docs] “Prompt Management — Version, store, and deploy prompts”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Prompts in Code - Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
Galileonone0/10Evidence shows Galileo supports experiments for evaluating prompts and mentions 'setting up prompt templates' via MCP, but there is no documentation of prompt versioning, a prompt registry, or a mechanism to deploy prompt changes to production independent of code deploys.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation
Instrumenting code and tracing requests end to end
Ai observability
ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app
weight 3 · round to Arize PhoenixPhoenix documents a first-party MCP server (/mcp endpoint) that lets MCP-compatible clients (Claude Code, Cursor, VS Code, etc.) search, query, and operate on projects, traces, datasets, experiments, prompts, and annotations, directly enabling an agent to query traces/metrics/eval results for debugging. This is backed by an official docs page and confirmed by probe evidence. Missing for 10: independent/hands-on third-party validation of the MCP server in practice and a documented general-purpose REST/GraphQL API spec (openapi probes 404) for non-MCP API access.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
Galileo has a documented official MCP server (galileo-docs-4, galileo-probe-4) enabling access to Galileo capabilities from a dev environment, but the explicitly listed MCP capabilities are creating/managing datasets, running experiments, and setting up prompt templates—not querying traces, metrics, or eval results directly. No OpenAPI/API reference was discoverable (galileo-probe-3 returned 404s), so there's no clear evidence an agent can query traces/metrics/eval results programmatically for debugging via API or MCP. missing for 10: explicit MCP/API support for querying traces and metrics, evidence of eval-result retrieval via MCP, and a discoverable REST/OpenAPI spec for programmatic trace queries.
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Data controls
developerMask or redact sensitive data before it is stored in traces
weight 2 · round drawnArize Phoenixnone0/10No evidence pack item mentions masking, redaction, PII scrubbing, or any data-sanitization capability for traces before storage; the docs cover tracing, evaluation, prompts, datasets, and self-hosting but never address sensitive-data redaction.
Sdk coverage
developerInstrument apps in both Python and JS/TS with officially supported SDKs
weight 2 · round to Arize PhoenixPhoenix documents OTLP ingestion and auto-instrumentation for frameworks including JS/TS ones like Vercel AI SDK and Mastra, plus a phoenix-cli npx setup and OpenInference-based instrumentation, indicating both Python and JS/TS support. However, the evidence is mostly high-level marketing docs without explicit JS/TS SDK API reference or code samples analogous to the detailed Python SDK reference (docs-28/29/30/36), and no independent/hands-on confirmation of JS/TS SDK parity. Missing for 10: explicit JS/TS SDK API reference/docs comparable to the Python SDK reference, hands-on/independent confirmation of JS/TS instrumentation working end-to-end, and clarity on feature parity between the two SDKs.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
Evidence confirms a Python SDK (galileo-python) with decorator-based span logging and OTel-based distributed tracing, but no evidence of an official JS/TS SDK or its documentation. missing for 10: JS/TS SDK repo or docs, parity of instrumentation features (decorators, spans) between Python and JS/TS.
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
Trace capture
developerTrace multi-step agent runs as nested spans grouped into sessions or threads
weight 2 · round drawnDocs explicitly describe OTLP-based tracing that captures nested spans for model calls, retrieval, and tool use (docs-38, docs-12/13/14), and Sessions explicitly group related traces into conversations/user sessions while Projects organize traces by application (docs-16/49). Auto-instrumentation for popular agent frameworks (LangChain, LlamaIndex, etc.) supports multi-step agent run tracing out of the box (docs-5, docs-32). Missing for 10: independent/hands-on verification of nested span hierarchy rendering for complex multi-step agents beyond vendor docs.
- [claimed-docs] “Sessions Group related traces into conversations and user sessions”
- [claimed-docs] “Group related traces into conversations and user sessions”
- [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
- [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
- [claimed-docs] “Retrieved Documents: Inspect the documents retrieved during a Retriever call, including the score and order in which they were returned”
- [claimed-docs] “Tool Descriptions: View the descriptions and function signatures of the tools your LLM has been given access to”
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
Docs explicitly describe capturing sessions, traces, and spans with structured logging, and distributed tracing docs show spans joined via shared trace IDs (nested spans under a trace) plus the @log decorator for span-level instrumentation. Missing for 10: explicit worked example showing session/thread grouping across multiple agent runs and independent hands-on corroboration beyond first-party docs.
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “Agentic metrics help you measure how well your AI agents perform complex, multi-step tasks—especially when those agents need to use tools, m…”
developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK
weight 2 · round to Arize PhoenixDocs explicitly state Phoenix accepts OTLP traces and provides auto-instrumentation for popular frameworks including LangChain and Vercel AI SDK, plus OpenAI is referenced as an adapter/evaluator target and is a widely documented OpenInference integration. Missing for 10: no independent/hands-on verification of these integrations working, and no explicit dedicated docs snippet for OpenAI SDK auto-instrumentation beyond the adapter mention.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
- [claimed-docs] “Model Agnostic via adapters (for OpenAI, LiteLLM, LangChain, AI SDK, and more) — so you can easily switch judge models”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
Galileonone0/10The evidence pack shows Galileo's own Python SDK (decorator-based logging) and generic OpenTelemetry distributed tracing support, but contains no mention of pre-built integrations for LangChain, the OpenAI SDK, or the Vercel AI SDK specifically. Since this is a well-known, plausible capability for a tracing/observability product, absence of evidence means 'none' rather than 'na'.
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
developerCapture multimodal payloads (images, audio, files) inside my traces
weight 1 · round drawnArize Phoenixnone0/10The evidence pack describes tracing of model calls, retrieval, tool use, latency, and token usage, but nowhere mentions capturing images, audio, or file attachments/multimodal payloads within spans or traces. Since this is a plausible capability for a tracing/observability tool, absence of evidence yields 'none'.
- [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
- [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
- [claimed-docs] “Retrieved Documents: Inspect the documents retrieved during a Retriever call, including the score and order in which they were returned”
developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format
weight 3 · round to Arize PhoenixDocs explicitly state Phoenix accepts traces over OpenTelemetry (OTLP), provides auto-instrumentation for popular frameworks, and is built on top of OpenTelemetry powered by OpenInference instrumentation — confirming standard OTLP ingestion rather than a proprietary format. Missing for 10: independent/hands-on corroboration of OTLP export/receive behavior beyond vendor docs.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
Docs confirm Galileo's SDK supports distributed tracing using OpenTelemetry's W3C traceparent header to propagate context and stitch spans into a single trace, showing OTel compatibility beyond a fully proprietary format. However, there's no evidence of a dedicated OTLP ingestion/export endpoint or explicit statement that Galileo accepts/emits OTLP-formatted traces from arbitrary OTel collectors. missing for 10: explicit OTLP endpoint/collector documentation, confirmation of both sending and receiving OTLP data, and independent verification of OTLP interoperability outside Galileo's own SDK.
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
- [github] “You can also use the `@log` decorator to log spans.”
developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage
weight 3 · round to Arize PhoenixPhoenix's docs extensively document capturing OTLP/OpenTelemetry-based traces of LLM calls, including inputs/outputs (span data), application latency, and detailed token usage breakdowns, plus auto-instrumentation for popular frameworks and manual tracing decorators. This is a core, heavily documented first-party capability with a CLI/SDK setup flow. Missing for 10: independent hands-on verification beyond vendor docs/community sentiment.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
- [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
- [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
Galileo's docs describe capturing sessions, traces, and spans once instrumented, with a `@log` decorator for logging spans and OTel-based distributed tracing joining spans by trace ID, forming a structured real-time data stream. However, explicit confirmation that captured traces include latency and token usage fields specifically is not directly quoted, only implied via 'structured stream of real-time data.' Missing for 10: explicit documentation/screenshot confirming latency and token-usage fields are captured per trace, and independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “Create and run your first trace in less than 5 minutes.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
- [github] “You can also use the `@log` decorator to log spans.”
Not comparable on these axes
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · not comparableArize Phoenixn/aPhoenix is an observability/evaluation platform, not an agent that performs tasks using tools; its documented MCP integration runs in the opposite direction — Phoenix exposes an /mcp endpoint so external MCP clients (Claude Code, Cursor, VS Code) can query Phoenix's traces/datasets, not the reverse. There is no indication Phoenix itself consumes external MCP servers' tools to extend its own capabilities, which is a different product role (MCP client) than what Phoenix is built for.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
Galileon/aGalileo is an LLM observability/evaluation platform, not an agentic tool-using product; the MCP evidence shows Galileo exposing its own capabilities via an MCP server for other clients (dev environments) to consume, not Galileo itself consuming external MCP servers to gain new tool capabilities. This 'plug servers in so it can use their tools' axis is a category mismatch for this kind of product.
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
ai-native userVersion, review, and roll back my automations
weight 1 · not comparablePhoenix offers prompt versioning ('Version, store, and deploy prompts', 'you can version prompts, test prompt variants... before rolling them out') and dataset/experiment comparisons across app versions, which cover review and some rollback-like workflow for prompts. However, Phoenix is an observability/eval platform, not an automation/workflow builder, so there's no evidence of versioning or rolling back full agentic automations/pipelines themselves — only prompts and datasets. Missing for 10: explicit rollback mechanism for automations/workflows (not just prompts), audit trail of automation changes, and evidence of reverting a live automation to a prior version.
- [claimed-docs] “Prompt Management — Version, store, and deploy prompts”
- [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”