LLM Evals & Observability Arena
LangSmith vs Arize Phoenix
Arize Phoenix wins · 14–22 (15 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to Arize PhoenixDirect probe evidence confirms LangSmith hosts a working llms.txt file (HTTP 200) plus markdown-formatted docs pages that explicitly point agents to the llms.txt index, making the docs agent-consumable by design. missing for 10: no independent/community confirmation that agents actually consume this successfully in practice, and no explicit vendor statement framing this as an intentional agent-facing feature.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.langchain.com/llms.txt # Docs by LangChain > Documentation for LangSmith, Fleet, and our open sour…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.langchain.com/langsmith/home.md > ## Documentation Index > Fetch the complete documentation index at…”
Direct probe evidence confirms llms.txt returns HTTP 200 with agent-oriented content, and docs pages are also available in markdown form (.md) with a documentation index, both explicitly designed for agent/LLM consumption. Missing for 10: no independent/community confirmation of an agent actually consuming these files successfully.
- [probe] “PROBE llms.txt: HTTP 200 at https://arize.com/llms.txt # Arize — AI & Agent Engineering > Arize is the AI engineering platform for teams bu…”
- [probe] “PROBE docs-md: HTTP 200 at https://arize.com/docs/phoenix.md > ## Documentation Index > Fetch the complete documentation index at: https://a…”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to Arize PhoenixLangSmith exposes SDKs (Python/TS/Go/Java) and OpenTelemetry-based tracing that let you programmatically run evaluations and send traces without the UI, and evaluation experiments can be scripted with configurable concurrency/repetitions, which is consistent with running as part of automated pipelines. However there is no explicit CI/CD example, GitHub Actions template, or documented CLI for headless automation. missing for 10: explicit CI/CD integration guide or example, a dedicated headless CLI, and independent confirmation of automated pipeline usage.
- [claimed-docs] “Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.”
- [claimed-docs] “LangSmith supports OpenTelemetry-based tracing, allowing you to send traces from any OpenTelemetry-compatible application.”
- [claimed-docs] “Run evaluations on curated datasets during development to compare versions, benchmark performance, and catch regressions.”
- [claimed-docs] “Trace your preferred framework or integrate LangSmith with any agent stack using our Python, Typescript, Go, or Java SDKs.”
- [github] “client = wrap_openai(openai.Client())”
Phoenix ships a CLI (`uvx arize-phoenix serve`), a Docker image pinned for production deployments, and a Python SDK with dataset evaluators explicitly described as forming 'an evaluation harness similar to a unit test suite,' all of which support headless/automated use in CI-like pipelines. However, there is no explicit CI/CD example (e.g., GitHub Actions workflow) or documentation of running evaluations as an automated CI gate. Missing for 10: explicit CI pipeline integration examples/docs, evidence of automated test-gating in CI, independent confirmation of headless CI usage.
- [claimed-docs] “$ uvx arize-phoenix serve”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
ai-native userConnect an agent via an official MCP server
weight 3 · round to Arize PhoenixThere is direct evidence of an official LangSmith MCP server repository (langsmith-mcp-server), confirming LangSmith ships a first-party MCP server that agents could connect to. However, the evidence pack has no further detail on setup, supported tools/capabilities, or independent confirmation of it working end-to-end with an agent, so the story is only thinly supported. missing for 10: detailed docs on MCP server capabilities/setup, independent/hands-on verification that an agent successfully connects and uses it, and integration guidance beyond the bare repo reference.
- [probe] “official MCP server documented at https://github.com/langchain-ai/langsmith-mcp-server”
Phoenix documents an official remote MCP server exposing a `/mcp` endpoint that MCP-compatible clients (Claude Code, Cursor, VS Code, etc.) can connect to for searching and operating on projects, traces, datasets, experiments, prompts, and annotations. This is first-party documentation confirming the exact capability. Missing for 10: independent/hands-on community corroboration of the MCP server working in practice, and details on authentication/setup robustness.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
ai-native userUse an official CLI
weight 2 · round to Arize PhoenixLangSmithnone0/10No evidence pack item mentions an official LangSmith CLI tool; the SDKs (Python/TS/Go/Java) and APIs are referenced but not a dedicated CLI for AI-native workflows.
Phoenix ships an official CLI (`@arizeai/phoenix-cli` via `npx -y @arizeai/phoenix-cli setup`) for auto-instrumentation setup, plus a `uvx arize-phoenix serve` command to launch the server, confirming a genuine first-party CLI exists for agentic/automated workflows. However, the CLI's documented scope is narrow (setup + serve) rather than covering the full product surface (datasets, evals, prompts, experiments), and there's no independent/community confirmation of its use. Missing for 10: broader CLI command coverage across Phoenix's feature set, independent hands-on verification of the CLI in practice.
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
- [claimed-docs] “$ uvx arize-phoenix serve”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app's root directory.”
- [claimed-docs] “start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app’s root directory”
ai-native userDrive the product through a documented public API
weight 3 · round to Arize PhoenixLangSmith explicitly advertises programmatic access ('Filter, export, share, and compare traces... via the UI or API') and ships official SDKs for Python, TypeScript, Go, and Java plus webhook-based automation, indicating a documented public API surface. However, a probe for a discoverable OpenAPI/swagger spec returned 404s across standard paths, suggesting no easily found canonical API reference, and no independent developer confirmation of full API coverage was found. missing for 10: a public OpenAPI/reference doc, independent/hands-on confirmation of API completeness.
- [claimed-docs] “Filter, export, share, and compare traces via the UI or API.”
- [claimed-docs] “Trace your preferred framework or integrate LangSmith with any agent stack using our Python, Typescript, Go, or Java SDKs.”
- [claimed-docs] “Automate workflows with rules, webhooks, and online evaluations.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langc…”
Phoenix documents a full SDK API reference (tracing decorators, dataset/experiment APIs, prompt sync SDKs), OTLP-based trace ingestion, a CLI (phoenix-cli), and an official remote MCP endpoint that lets AI-native clients search/query/operate on projects, traces, datasets, experiments, and prompts—together constituting a documented, programmatically-drivable public API surface. The openapi.json probe returned 404s at standard paths, meaning there's no discoverable formal REST/OpenAPI spec, so the API surface is SDK/CLI/MCP-centric rather than a single canonical REST API. Missing for 10: a published OpenAPI/REST spec, independent third-party confirmation of API robustness.
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
- [claimed-docs] “$ uvx arize-phoenix serve”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round drawnLangSmithnone0/10The evidence pack covers tracing, evaluation, dashboards, alerts, and self-hosting, but contains no mention of API key scoping, permissions, roles, or least-privilege credential issuance for agents. No documentation of scoped/limited API keys or RBAC-style credential controls is present.
ai-native userBuild against official SDKs
weight 2 · round drawnLangSmith documents official SDKs in Python, TypeScript, Go, and Java, backed by a public GitHub SDK repo with working code examples (e.g., wrap_openai), and also supports OpenTelemetry-based tracing for any compatible stack. Missing for 10: independent hands-on verification of SDK completeness across all four languages, and community notes (langsmith-comm-3) indicate weaker support for non-LangChain frameworks like crewai, which slightly tempers confidence in universal SDK-native coverage.
- [claimed-docs] “Trace your preferred framework or integrate LangSmith with any agent stack using our Python, Typescript, Go, or Java SDKs.”
- [github] “client = wrap_openai(openai.Client())”
- [claimed-docs] “LangSmith supports OpenTelemetry-based tracing, allowing you to send traces from any OpenTelemetry-compatible application.”
- [community] “Langsmith only shows the llm inferences. Also it hardly works with other frameworks like crewai for instance.”
Phoenix ships an official, modular Python SDK plus client SDKs for tracing, prompts, evaluations, and datasets, with OTLP-based tracing decorators, auto-instrumentation, and dedicated SDK/API reference docs. Coverage spans core agentic workflows (tracing, evals, prompt management, datasets/experiments) and is corroborated by GitHub repo and docs pages, though independent hands-on SDK usage reports are thin. Missing for 10: broader independent/third-party validation of SDK usage beyond official docs, and explicit multi-language SDK support details beyond Python/JS CLI.
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Prompts in Code - Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
- [github] “Datasets - Create versioned datasets of examples for experimen”
ai-native userSubscribe to events via webhooks
weight 2 · round to LangSmithLangSmith docs explicitly state that alerts and automation rules can route to webhooks/any HTTP endpoint, letting users subscribe to events like errors, cost, or feedback thresholds via webhook. However, this is scoped to alert/automation rules rather than a general-purpose event subscription API, and there's no independent or hands-on confirmation of webhook reliability or payload structure. missing for 10: broader documentation of webhook event types/payloads beyond alert thresholds, independent/hands-on validation of webhook delivery.
- [claimed-docs] “Automate workflows with rules, webhooks, and online evaluations.”
- [claimed-docs] “Alerts can route to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
- [claimed-docs] “Alerts can [route](#step-4-configure-notification-channel) to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
- [claimed-docs] “LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round drawnLangSmith offers LLM-as-judge evaluators, automated root-cause detection via 'LangSmith Engine' (langsmith-docs-6), and alerting/dashboards that surface quality signals, which constitute AI-generated insight generation embedded in the product. However, evidence doesn't show a general-purpose AI assistant proactively surfacing free-form suggestions across the whole product, and community feedback notes trace visibility is LLM-inference-only and framework-limited (langsmith-comm-3). Missing for 10: broader proactive AI-generated recommendations beyond eval/root-cause detection, first-hand demonstration of LangSmith Engine's insight quality, and independent corroboration of its effectiveness.
- [claimed-docs] “Automatically detect recurring issues in your traces, diagnose their root cause, and resolve them with LangSmith Engine.”
- [claimed-docs] “Create evaluators to score performance: Human review, Code rules, LLM-as-judge, Pairwise comparison”
- [claimed-docs] “Set up evaluators to run automatically on production traces: safety checks, format validation, quality heuristics, and reference-free LLM-as…”
- [claimed-docs] “Human review * Code rules * LLM-as-judge * Pairwise comparison”
- [community] “Langsmith only shows the llm inferences. Also it hardly works with other frameworks like crewai for instance.”
Phoenix offers LLM-as-judge evaluators that return explanations and reasoning alongside scores, and lets users explore traces to spot systematic issues, which counts as AI-generated signal on the data — but this is evaluator-driven scoring, not proactive, unprompted insights/suggestions surfaced automatically inside the product UI. missing for 10: evidence of autonomous/proactive insight generation (e.g., auto-detected anomalies, recommended fixes, natural-language summaries of trends) rather than only judge-model scoring/explanations tied to evals a user configures.
- [claimed-docs] “Built-in Explanations — all Phoenix LLM evaluations return explanations by default for better results and richer signals”
- [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model's full reasoning, the final scores, a…”
- [claimed-docs] “Use Phoenix's trace viewer to explore evaluation traces, identify systematic biases, and continuously improve evaluator performance.”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
ai-native userSet up automations that run autonomously in the background
weight 2 · round to LangSmithLangSmith supports background automations via rules/webhooks, alerts that route to Slack/PagerDuty/HTTP endpoints, and online evaluators that run automatically on live production traces without user intervention (docs-4, docs-10, docs-12/13). It also offers optional 'LangSmith Deployment' to run and manage agents (docs-17), suggesting some autonomous agent execution capability, but this is a secondary/optional feature and detail is thin. Missing for 10: independent/hands-on evidence that these automations actually run unattended over time, more detail on scheduling/triggers beyond alerts, and clearer scope of 'LangSmith Deployment' for autonomous agent runs.
- [claimed-docs] “Automate workflows with rules, webhooks, and online evaluations.”
- [claimed-docs] “Set up evaluators to run automatically on production traces: safety checks, format validation, quality heuristics, and reference-free LLM-as…”
- [claimed-docs] “LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”
- [claimed-docs] “Alerts can route to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable La…”
Arize Phoenixnone0/10Phoenix's evidence covers tracing, evaluation, prompt management, and datasets, but nothing describes scheduled or autonomous background automations (e.g., recurring eval jobs, alerting rules, or triggers) that run without user initiation. The evaluation/experiment features described are invoked by the user (via SDK/UI), not autonomous background processes.
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round to LangSmithLangSmith Engine is described as automatically detecting recurring issues, diagnosing root causes, and resolving them, which functions like a built-in AI assistant for debugging tasks, but this is only a single, thin doc line with no further elaboration, UI walkthrough, or independent corroboration of how tasks are 'delegated' to it. missing for 10: detailed documentation of assistant capabilities/UX, examples of delegated task execution, independent/hands-on confirmation.
- [claimed-docs] “Automatically detect recurring issues in your traces, diagnose their root cause, and resolve them with LangSmith Engine.”
Arize Phoenixnone0/10Phoenix is an observability/evaluation platform; the evidence describes tracing, evals, prompt management, datasets, and an MCP server that lets *external* agents (Claude Code, Cursor, etc.) operate on Phoenix data — not a built-in AI assistant living inside Phoenix that users delegate tasks to. No docs mention any embedded assistant/chat agent within the Phoenix UI itself.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app's root directory.”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
ai-native userOperate the product with natural-language commands
weight 2 · round to Arize PhoenixLangSmithnone0/10LangSmith is an observability/evaluation platform with a UI, API, and SDKs, but there is no evidence of natural-language command interfaces for operating the product itself (e.g., a chat-based control plane). The MCP server exposes tools to agents but that's a different axis from a user issuing NL commands to operate the product. missing for 10: any documented NL command interface, chat-based control of dashboards/alerts/experiments, or evidence of conversational operation.
- [probe] “official MCP server documented at https://github.com/langchain-ai/langsmith-mcp-server”
Phoenix documents a dedicated remote MCP integration letting any MCP-compatible client (Claude Code, Cursor, VS Code, etc.) operate on projects, traces, datasets, experiments, prompts, and annotations via natural-language driven agents, and also supports agent-driven CLI setup. This directly satisfies AI-native natural-language operability. Missing for 10: independent/hands-on validation of the MCP command experience and more detail on the breadth/reliability of natural-language operations beyond the docs claim.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnLangSmithnone0/10No evidence of an interactive API reference with runnable examples; the OpenAPI probe explicitly returned 404s at all candidate paths, and no docs mention a Swagger/Redoc-style interactive reference or embedded runnable code snippets.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langc…”
Arize Phoenixnone0/10Docs mention an 'sdk-api-reference' page listing decorators and SDK features, but there is no evidence of an interactive, runnable API reference (e.g., a Swagger/OpenAPI explorer or live code sandbox); a direct probe for OpenAPI/swagger specs returned 404 on all candidate paths, indicating no such interactive reference is discoverable.
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round drawnLangSmithnone0/10LangSmith exposes a REST API (referenced for filtering/exporting traces) but the evidence pack shows a direct probe for OpenAPI/swagger specs at the docs site returned 404 on all candidate paths, and no other citation points to a downloadable machine-readable API spec.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langc…”
- [claimed-docs] “Filter, export, share, and compare traces via the UI or API.”
Arize Phoenixnone0/10A direct probe for OpenAPI/swagger endpoints at Phoenix's domain returned 404 for all candidate paths, and no docs mention a downloadable machine-readable API spec. Missing for 10: any OpenAPI/Swagger spec, documented REST API reference, or SDK-generated schema.
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round drawnLangSmith supports datasets with curated/synthetic examples and running evaluations/experiments against them separately from production traces, which functions like a sandbox testing workflow rather than testing directly on live production data. However, there's no explicit documentation of an isolated 'sandbox environment' concept, staging/prod environment separation, or data isolation guarantees — evaluation datasets can be built from historical production traces, blurring the separation. missing for 10: explicit sandbox/staging environment documentation, guarantees of production data isolation, and independent confirmation that dataset-based testing avoids touching production systems.
- [claimed-docs] “Create a dataset with examples from manually curated test cases, historical production traces, or synthetic data generation.”
- [claimed-docs] “Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.”
- [claimed-docs] “Run evaluations on curated datasets during development to compare versions, benchmark performance, and catch regressions.”
- [claimed-docs] “Create a [dataset](/langsmith/manage-datasets) with [examples](/langsmith/evaluation-concepts#examples) from manually curated test cases, hi…”
- [claimed-docs] “Execute your application on the dataset to create an [experiment](/langsmith/evaluation-concepts#experiment). Configure [repetitions, concur…”
Phoenix supports self-hosting with full data isolation (data never leaves your infra) and lets you organize traces into separate 'projects', collect data 'from production, staging, evaluations, and even manually' into datasets, and run experiments/evaluations against those datasets rather than live traffic — all of which let a user test without touching real production data. However, there's no explicit documented 'sandbox environment' feature, no isolated test-instance provisioning, and no guidance on separating staging from prod deployments beyond generic self-hosting flexibility. Missing for 10: explicit sandbox/staging environment tooling, isolation guarantees beyond self-hosting, and independent confirmation that experiments truly never touch production data.
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
- [claimed-docs] “Organize traces into separate projects for different applications”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnLangSmithnone0/10No evidence pack item documents API versioning scheme or a deprecation policy; the OpenAPI probe returned 404s and no docs page addresses version support lifecycle or breaking-change policy.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langc…”
Arize Phoenixnone0/10Evidence shows only generic container/image version pinning (e.g., 'version-8.0.0' Docker tags) but no documented API versioning scheme or deprecation policy for Phoenix's SDK/API; an OpenAPI probe also returned 404s, finding no formal API spec to review versioning against.
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
Alerting dashboards — stories about alerting dashboards in this arenaAlerting dashboards
Stories about alerting dashboards in this arena
Monitoring
ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics
weight 2 · round to LangSmithLangSmith docs explicitly describe building custom dashboards ('tailored collections of charts') and prebuilt dashboards covering trace count, error rates, token usage, plus threshold alerts on Run Count, Cost, Errors, Feedback Score, and Latency, covering latency, error, cost, and eval-score metrics with group-by tag/metadata support for slicing. Missing for 10: independent/hands-on confirmation of dashboard-building UX and eval-score charting specifically.
- [claimed-docs] “Prebuilt dashboards are created automatically for each project and cover essential metrics, such as trace count, error rates, token usage, a…”
- [claimed-docs] “Create tailored collections of charts for tracking metrics that matter most for your application.”
- [claimed-docs] “Custom dashboards: Collections of charts you can configure to your needs.”
- [claimed-docs] “You can use group by [run tag or metadata](/langsmith/add-metadata-tags) to split data over attributes that are important to your applicatio…”
- [claimed-docs] “LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”
- [claimed-docs] “You can use group by run tag or metadata to split data over attributes that are important to your application.”
Arize Phoenixnone0/10Phoenix's docs describe tracing, span-level latency/cost/token data, and eval scoring, but nowhere is there evidence of a dashboard-building feature (custom charts, widgets, or dashboard layouts) that lets an ML engineer aggregate these metrics into a dashboard view. The evidence only covers per-trace exploration, filtering spans, and projects/sessions organization, not dashboard construction or alerting.
developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email
weight 2 · round to LangSmithLangSmith docs explicitly describe threshold-based alerting on Errors, Cost, Feedback Score (eval score), Run Count, and Latency, with routing to Slack, PagerDuty, Dynatrace, or any HTTP webhook (which can cover email/other integrations), directly matching the story. Missing for 10: independent/hands-on confirmation of alert delivery reliability and no explicit mention of native email notification channel beyond webhook routing.
- [claimed-docs] “LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”
- [claimed-docs] “Alerts can route to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
- [claimed-docs] “LangSmith provides threshold-based alerting on the following metrics:”
- [claimed-docs] “Alerts can [route](#step-4-configure-notification-channel) to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
- [claimed-docs] “Build dashboards and set alerts to track quality and catch issues early.”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to LangSmithLangSmith documents dataset-wide evaluation runs (execute across many examples with configurable concurrency/repetitions), bulk trace filter/export/share/compare via API, and bulk dataset creation from historical traces or synthetic generation—supporting operations across many items at once. However, there's no explicit documentation of general bulk actions (e.g., bulk delete/tag/annotate across arbitrary trace sets) beyond evaluation and dataset contexts, and no independent/hands-on confirmation of bulk-op reliability at scale. Missing for 10: explicit bulk edit/delete/tag API documentation, independent hands-on verification of large-scale bulk operations.
- [claimed-docs] “Filter, export, share, and compare traces via the UI or API.”
- [claimed-docs] “Create a dataset with examples from manually curated test cases, historical production traces, or synthetic data generation.”
- [claimed-docs] “Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.”
- [claimed-docs] “Create a [dataset](/langsmith/manage-datasets) with [examples](/langsmith/evaluation-concepts#examples) from manually curated test cases, hi…”
- [claimed-docs] “Execute your application on the dataset to create an [experiment](/langsmith/evaluation-concepts#experiment). Configure [repetitions, concur…”
- [github] “client = wrap_openai(openai.Client())”
Phoenix supports bulk-style operations through datasets (grouping many traces/examples), running experiments and evaluations across thousands of items with built-in concurrency/retry handling, and MCP-based querying/operating on projects, traces, datasets, and prompts. However, there is no explicit documentation of a bulk UI action (e.g., multi-select delete/tag/export) or a dedicated batch API for arbitrary bulk edits across items. missing for 10: explicit bulk UI operations (multi-select edit/delete/export), a documented batch API for arbitrary item updates, and independent/hands-on confirmation of large-scale bulk actions beyond evaluation/experiment runs.
- [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to LangSmithLangSmith documents rule-based automation via threshold alerts (run count, cost, errors, feedback score, latency) that route to Slack, PagerDuty, Dynatrace, or webhooks, and separately mentions 'rules, webhooks, and online evaluations' as automation primitives. This covers automatic action-on-event but is scoped mainly to monitoring/alerting metrics rather than a general-purpose rule engine for arbitrary events. Missing for 10: detailed documentation of the 'rules' feature itself (beyond the one-line mention), broader event types beyond the five alert metrics, and independent/hands-on confirmation that rule-triggered actions work reliably in practice.
- [claimed-docs] “Automate workflows with rules, webhooks, and online evaluations.”
- [claimed-docs] “LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”
- [claimed-docs] “Alerts can route to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
- [claimed-docs] “LangSmith provides threshold-based alerting on the following metrics:”
- [claimed-docs] “Alerts can [route](#step-4-configure-notification-channel) to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
Arize Phoenixnone0/10Phoenix's evidence covers tracing, evaluation, datasets, prompt management, and MCP integration, but nothing describes a rules/triggers engine that automatically fires actions on events (e.g., alerting, auto-remediation, webhooks on thresholds). The axis is plausible for an observability platform, but no such capability is evidenced.
ai-native userSchedule recurring jobs or workflows
weight 2 · round drawnLangSmithnone0/10LangSmith documents automation via rules, webhooks, and online evaluations triggered on production traces, but there is no evidence of a scheduler or cron-like capability for recurring jobs/workflows (e.g., scheduled dataset evaluations or periodic report runs). Since automation is otherwise a supported theme for this platform, the absence of scheduling evidence makes this 'none' rather than 'na'.
- [claimed-docs] “Automate workflows with rules, webhooks, and online evaluations.”
- [claimed-docs] “Set up evaluators to run automatically on production traces: safety checks, format validation, quality heuristics, and reference-free LLM-as…”
- [claimed-docs] “Evaluate real user interactions in real-time to detect issues and measure quality on live traffic.”
ai-native userVersion, review, and roll back my automations
weight 1 · round to Arize PhoenixLangSmithnone0/10LangSmith documents automation rules, webhooks, and dashboards but no evidence describes versioning, review workflows, or rollback capabilities for these automations. Missing for 10: version history for rules/automations, review/approval workflow, rollback mechanism, and any documentation or community confirmation of these features.
- [claimed-docs] “Automate workflows with rules, webhooks, and online evaluations.”
- [claimed-docs] “LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”
- [claimed-docs] “Alerts can route to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
Phoenix offers prompt versioning ('Version, store, and deploy prompts', 'you can version prompts, test prompt variants... before rolling them out') and dataset/experiment comparisons across app versions, which cover review and some rollback-like workflow for prompts. However, Phoenix is an observability/eval platform, not an automation/workflow builder, so there's no evidence of versioning or rolling back full agentic automations/pipelines themselves — only prompts and datasets. Missing for 10: explicit rollback mechanism for automations/workflows (not just prompts), audit trail of automation changes, and evidence of reverting a live automation to a prior version.
- [claimed-docs] “Prompt Management — Version, store, and deploy prompts”
- [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
Cost monitoring — stories about cost monitoring in this arenaCost monitoring
Stories about cost monitoring in this arena
Cost tracking
developerAttribute cost and usage to users, sessions, and features via custom metadata
weight 2 · round to LangSmithLangSmith docs show custom metadata/tags can be attached to traces and used to group dashboard metrics (including token usage) by attribute, and alerts can be set on cost, enabling attribution of cost/usage to arbitrary dimensions like user, session, or feature via metadata. Missing for 10: a concrete worked example of per-user/session/feature cost breakdown and independent/hands-on corroboration that metadata-based cost attribution works well in practice.
- [claimed-docs] “You can use group by run tag or metadata to split data over attributes that are important to your application.”
- [claimed-docs] “You can use group by [run tag or metadata](/langsmith/add-metadata-tags) to split data over attributes that are important to your applicatio…”
- [claimed-docs] “LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”
- [claimed-docs] “LangSmith provides threshold-based alerting on the following metrics:”
- [claimed-docs] “Prebuilt dashboards are created automatically for each project and cover essential metrics, such as trace count, error rates, token usage, a…”
- [claimed-docs] “Custom dashboards: Collections of charts you can configure to your needs.”
Phoenix documents token-usage breakdowns per LLM call and lets you organize traces into projects and group them into sessions/conversations, which provides some basis for usage attribution by session/application. However, there is no evidence of a general custom-metadata/tagging mechanism for arbitrary attribution to users or features, nor any cost aggregation/dashboard that rolls up spend by user, session, or feature tag. missing for 10: explicit custom metadata/tagging API for attributing spans to users or features, cost-specific aggregation/dashboard views by user/session/feature, documentation showing cost roll-ups beyond raw token counts.
- [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
- [claimed-docs] “Sessions Group related traces into conversations and user sessions”
- [claimed-docs] “Organize traces into separate projects for different applications”
- [claimed-docs] “Group related traces into conversations and user sessions”
developerSee cost and token usage per request, model, and time period in dashboards
weight 3 · round to LangSmithLangSmith documents prebuilt per-project dashboards covering token usage and cost, threshold-based cost alerting, and custom dashboards with group-by tag/metadata to slice metrics by model or time period, directly matching the story. missing for 10: independent/hands-on confirmation of the cost-per-model breakdown specifically and no third-party validation of dashboard accuracy.
- [claimed-docs] “LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”
- [claimed-docs] “Prebuilt dashboards are created automatically for each project and cover essential metrics, such as trace count, error rates, token usage, a…”
- [claimed-docs] “Create tailored collections of charts for tracking metrics that matter most for your application.”
- [claimed-docs] “You can use group by run tag or metadata to split data over attributes that are important to your application.”
- [claimed-docs] “Custom dashboards: Collections of charts you can configure to your needs.”
- [claimed-docs] “You can use group by [run tag or metadata](/langsmith/add-metadata-tags) to split data over attributes that are important to your applicatio…”
Phoenix provides a token usage breakdown per LLM call/span (docs-46) and organizes traces into projects and sessions (docs-15/16/48/49), which supports some per-model and per-time grouping, but there is no evidence of aggregated cost dashboards, dollar-cost calculations, or dedicated per-model/time-period cost views — only per-span token counts. missing for 10: dedicated cost dashboards showing dollar cost (not just tokens), aggregated views by model and by time period, and any UI screenshot/independent confirmation of such dashboards.
- [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
- [claimed-docs] “Sessions Group related traces into conversations and user sessions”
- [claimed-docs] “Organize traces into separate projects for different applications”
- [claimed-docs] “Group related traces into conversations and user sessions”
Data access export — stories about data access export in this arenaData access export
Stories about data access export in this arena
Data export
developerBulk-export traces and datasets to blob storage or my data warehouse
weight 2 · round to LangSmithDocs mention the ability to 'export' traces via UI or API, but there is no evidence of bulk export tooling, blob storage integration (S3/GCS), or data warehouse connectors (BigQuery/Snowflake) specifically designed for large-scale export. missing for 10: dedicated bulk-export API/CLI docs, blob storage integration, data warehouse connector documentation, and any hands-on confirmation of large-scale export workflows.
- [claimed-docs] “Filter, export, share, and compare traces via the UI or API.”
Arize Phoenixnone0/10The evidence pack covers tracing, datasets, evaluation, prompt management, and self-hosting, but contains no mention of bulk-exporting traces or datasets to blob storage (S3, GCS) or a data warehouse (Snowflake, BigQuery, etc.). The closest evidence is generic dataset/SDK querying (docs-25, docs-30) which is not the same as a bulk export/warehouse-sync feature.
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets
Measuring quality — datasets, eval runs, regression tracking
Ai eval ops
ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results
weight 2 · round to LangSmithLangSmith docs and SDK show programmatic dataset creation (from curated data, traces, or synthetic generation), programmatic evaluator/experiment execution (`Execute your application on the dataset to create an experiment`), and API/SDK access to trace/experiment results, all reachable from Python/TS/Go/Java SDKs, and an official MCP server exists so an agent can invoke these operations as tools. Missing for 10: a concrete end-to-end code sample showing an agent creating a dataset, kicking off an eval, and parsing results back, plus independent hands-on confirmation of the MCP server workflow.
- [claimed-docs] “Create a dataset with examples from manually curated test cases, historical production traces, or synthetic data generation.”
- [claimed-docs] “Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.”
- [claimed-docs] “Create evaluators to score performance: Human review, Code rules, LLM-as-judge, Pairwise comparison”
- [claimed-docs] “Filter, export, share, and compare traces via the UI or API.”
- [github] “client = wrap_openai(openai.Client())”
- [probe] “official MCP server documented at https://github.com/langchain-ai/langsmith-mcp-server”
Phoenix has a Python/SDK-driven dataset creation, experiment/eval-run triggering, and result retrieval (datasets, dataset evaluators, experiments, MCP endpoint letting agents 'operate on datasets, experiments'), which supports an agent programmatically creating datasets, running evals, and reading back scores. However the evidence is all docs-level description without a concrete end-to-end code example or independent confirmation of an agent performing this full loop. Missing for 10: a worked code/API example showing dataset creation -> eval trigger -> result read-back in one flow, and independent/hands-on verification beyond vendor docs.
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [github] “Datasets - Create versioned datasets of examples for experimen”
Human review
ml engineerRoute outputs to human annotation queues for review and labeling
weight 2 · round to LangSmithLangSmith explicitly supports 'Annotate outputs and gather user feedback using queues or inline annotation' and lists 'Human review' as a first-class evaluator type, directly matching the routing-to-annotation-queue story. missing for 10: no independent/hands-on corroboration of the annotation queue workflow beyond first-party docs, and no detail on queue assignment/collaboration features.
- [claimed-docs] “Annotate outputs and gather user feedback using queues or inline annotation.”
- [claimed-docs] “Create evaluators to score performance: Human review, Code rules, LLM-as-judge, Pairwise comparison”
- [claimed-docs] “Human review * Code rules * LLM-as-judge * Pairwise comparison”
Phoenix supports attaching human annotations/ground-truth labels directly in the UI on traces/spans, which enables human review and labeling of outputs, and scoring can combine human labels with automated evals. However, there's no evidence of a dedicated 'annotation queue' routing mechanism, task assignment to reviewers, or workflow management for distributing items to human labelers. missing for 10: dedicated annotation queue/task routing to specific reviewers, workflow/assignment management, evidence of multi-user labeling workflows beyond UI annotation.
- [claimed-docs] “Human annotations — Attach ground truth labels directly in the UI”
- [claimed-docs] “Attach ground truth labels directly in the UI”
- [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”
Offline evals
developerRun evals in CI and gate deployments on their results
weight 2 · round drawnLangSmith supports running evaluations on datasets to compare versions and catch regressions, and can execute experiments programmatically, which could be invoked from a CI pipeline, but there is no documented CI integration, GitHub Actions example, exit-code/pass-fail gating mechanism, or deployment-gating workflow in the evidence pack. missing for 10: explicit CI/CD integration docs, pass/fail threshold gating mechanism, deployment-blocking example, independent confirmation of CI usage.
- [claimed-docs] “Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.”
- [claimed-docs] “Run evaluations on curated datasets during development to compare versions, benchmark performance, and catch regressions.”
- [claimed-docs] “Execute your application on the dataset to create an [experiment](/langsmith/evaluation-concepts#experiment). Configure [repetitions, concur…”
Phoenix explicitly frames dataset evaluators as forming 'an evaluation harness similar to a unit test suite' and provides SDK/Python-based experiment running with automatic scoring, which are the building blocks for CI-based evals. However, there is no direct documentation of CI pipeline integration (e.g., GitHub Actions examples), exit-code/pass-fail thresholds, or deployment-gating mechanics tied to eval results. Missing for 10: explicit CI/CD integration docs, pass/fail gating logic, deployment-blocking examples.
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “Dataset Evaluators serve as **test cases** that automatically score outputs when running experiments—forming an evaluation harness similar t…”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
ml engineerWrite custom code-based scorers and metrics for my evaluations
weight 2 · round drawnLangSmith docs explicitly list 'Code rules' as an evaluator type alongside LLM-as-judge, human review, and pairwise comparison, confirming support for custom code-based scorers/metrics in evaluations. Missing for 10: no concrete code example of writing a custom Python scorer function and no independent/hands-on corroboration of code-based evaluator usage.
- [claimed-docs] “Create evaluators to score performance: Human review, Code rules, LLM-as-judge, Pairwise comparison”
- [claimed-docs] “Human review * Code rules * LLM-as-judge * Pairwise comparison”
- [claimed-docs] “Create [evaluators](/langsmith/evaluation-concepts#evaluators) to score performance:”
- [claimed-docs] “Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.”
Docs explicitly state Phoenix supports deterministic code-based evaluators (exact match, regex, custom heuristics) alongside LLM-as-a-judge, and Dataset Evaluators act as test cases scoring outputs automatically in experiments similar to a unit test suite, which is exactly code-based custom scoring for evals. Missing for 10: no independent/hands-on corroboration of writing custom Python scorer code, and no detailed API reference/example showing custom scorer implementation.
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators, where a sec…”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “Dataset Evaluators serve as **test cases** that automatically score outputs when running experiments—forming an evaluation harness similar t…”
- [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”
ml engineerCompare eval runs side by side to catch regressions between prompt or model versions
weight 3 · round drawnLangSmith's evaluation docs explicitly describe running evaluations on curated datasets to 'compare versions, benchmark performance, and catch regressions,' with experiments generated per run and pairwise comparison evaluators supported, directly matching the story. Missing for 10: independent/hands-on confirmation of the side-by-side comparison UI experience and any community validation specific to regression-catching workflows.
- [claimed-docs] “Run evaluations on curated datasets during development to compare versions, benchmark performance, and catch regressions.”
- [claimed-docs] “Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.”
- [claimed-docs] “Create evaluators to score performance: Human review, Code rules, LLM-as-judge, Pairwise comparison”
- [claimed-docs] “Human review * Code rules * LLM-as-judge * Pairwise comparison”
- [claimed-docs] “Execute your application on the dataset to create an [experiment](/langsmith/evaluation-concepts#experiment). Configure [repetitions, concur…”
Phoenix's experiments feature explicitly groups traces into datasets, reruns them through different app/prompt/model versions, and compares evaluation results to confirm whether a change improved performance, with automatic scoring via configured evaluators — directly matching the side-by-side regression-catching story. Missing for 10: no independent/hands-on example of an actual side-by-side UI diff view or third-party corroboration beyond vendor docs.
- [claimed-docs] “optimize your app with experiments that compare changes on the same inputs”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results to confirm whe…”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [github] “Datasets - Create versioned datasets of examples for experimen”
ml engineerScore outputs with configurable LLM-as-a-judge evaluators
weight 3 · round to Arize PhoenixLangSmith's docs explicitly document creating LLM-as-judge evaluators (alongside human review, code rules, pairwise comparison) for both offline dataset experiments and online production traces, with configurable filters/sampling for cost control. Missing for 10: independent hands-on validation of LLM-as-judge accuracy/configurability and no first-party code snippet showing evaluator configuration in this pack.
- [claimed-docs] “Create evaluators to score performance: Human review, Code rules, LLM-as-judge, Pairwise comparison”
- [claimed-docs] “Set up evaluators to run automatically on production traces: safety checks, format validation, quality heuristics, and reference-free LLM-as…”
- [claimed-docs] “Run evaluations on curated datasets during development to compare versions, benchmark performance, and catch regressions.”
- [claimed-docs] “Evaluate real user interactions in real-time to detect issues and measure quality on live traffic.”
- [claimed-docs] “Human review * Code rules * LLM-as-judge * Pairwise comparison”
- [claimed-docs] “Set up [evaluators](/langsmith/online-evaluations-llm-as-judge) to run automatically on production traces: safety checks, format validation,…”
Docs explicitly describe LLM-as-a-judge evaluators configurable via model-agnostic adapters (OpenAI, LiteLLM, LangChain, AI SDK), with built-in explanations, automatic scoring of experiment results in the UI or via SDK, and full traceability of judge prompts/reasoning/scores. This is a first-party documented core feature well beyond a single mention. Missing for 10: independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Model Agnostic via adapters (for OpenAI, LiteLLM, LangChain, AI SDK, and more) — so you can easily switch judge models”
- [claimed-docs] “Built-in Explanations — all Phoenix LLM evaluations return explanations by default for better results and richer signals”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators, where a sec…”
- [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model’s full reasoning, the final scores, a…”
- [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model's full reasoning, the final scores, a…”
ml engineerCurate datasets from production traces and run offline evaluations against them
weight 3 · round drawnDocs explicitly describe creating datasets from historical production traces and running offline experiments/evaluations with configurable evaluators (human review, code rules, LLM-as-judge, pairwise comparison) to compare versions and catch regressions. Missing for 10: independent/hands-on corroboration of the dataset-curation-from-traces workflow specifically (community comments focus on tracing/observability, not dataset curation).
- [claimed-docs] “Create a dataset with examples from manually curated test cases, historical production traces, or synthetic data generation.”
- [claimed-docs] “Run evaluations on curated datasets during development to compare versions, benchmark performance, and catch regressions.”
- [claimed-docs] “Create evaluators to score performance: Human review, Code rules, LLM-as-judge, Pairwise comparison”
- [claimed-docs] “Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.”
- [claimed-docs] “Traces are the record of what your agents did in production. Use them to debug failures, monitor quality, and build the datasets you evaluat…”
Phoenix docs describe grouping production traces into versioned datasets (docs-10, docs-25, gh-1), attaching evaluators (code-based and LLM-as-judge) to datasets and scoring experiment results automatically as an offline eval harness (docs-17, docs-18, docs-26, docs-34), and comparing experiment results across app versions (docs-41). This directly matches curating datasets from traces and running offline evals. missing for 10: independent/hands-on corroboration beyond vendor docs, and concrete example of the full curate-to-evaluate workflow output.
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [github] “Datasets - Create versioned datasets of examples for experimen”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results to confirm whe…”
Online evals
ml engineerRun evaluators continuously on live production traffic, not just offline datasets
weight 2 · round to LangSmithDocs explicitly describe online evaluators that run automatically on production traces (safety checks, format validation, quality heuristics, reference-free LLM-as-judge) with filters/sampling to control cost, and separately describe real-time evaluation of live traffic distinct from offline dataset evals. This directly matches the story. Missing for 10: independent/hands-on confirmation of continuous online evaluators in practice (community evidence only covers tracing, not online eval specifically).
- [claimed-docs] “Set up evaluators to run automatically on production traces: safety checks, format validation, quality heuristics, and reference-free LLM-as…”
- [claimed-docs] “Evaluate real user interactions in real-time to detect issues and measure quality on live traffic.”
- [claimed-docs] “Set up [evaluators](/langsmith/online-evaluations-llm-as-judge) to run automatically on production traces: safety checks, format validation,…”
- [claimed-docs] “Automate workflows with rules, webhooks, and online evaluations.”
Docs confirm Phoenix ingests live production traces (OTLP) and can score traces & spans with LLM, code-based, or human evaluators (docs-39, docs-25), suggesting evaluation can be applied beyond static datasets. However, the more detailed evaluation workflow described is dataset/experiment-centric (attach evaluators to datasets, score experiment results) rather than an explicit always-on/scheduled evaluator pipeline running automatically on incoming production traffic. Missing for 10: explicit documentation of a continuous/scheduled evaluation job or streaming eval pipeline that runs on live traces without manual dataset curation, and independent confirmation of this in production use.
- [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round drawnDocs explicitly state traces can be filtered, exported, shared, and compared 'via the UI or API', and SDKs (Python/TS/Go/Java) support tracing, dataset creation, and evaluation programmatically, suggesting real API parity for core workflows. However, there's no evidence of a full OpenAPI spec (probe for openapi.json returned 404s) or explicit API support for dashboard/alert creation, and community notes describe friction (e.g., needing a cloud account just to view your own traces) suggesting the API surface isn't fully self-serve/parity for all UI actions. missing for 10: published OpenAPI/API reference showing full UI-equivalent endpoints, explicit API methods for dashboards/alerts management, independent confirmation that all UI actions are scriptable.
- [claimed-docs] “Filter, export, share, and compare traces via the UI or API.”
- [claimed-docs] “Trace your preferred framework or integrate LangSmith with any agent stack using our Python, Typescript, Go, or Java SDKs.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langc…”
- [community] “I built this because LangSmith needs a cloud account to see my own traces.”
Phoenix's Python/SDK APIs cover most workflows (tracing, datasets, experiments, evals, prompts) matching many UI features, and human annotations/UI-configured evaluators are documented, but there's no comprehensive API/CLI reference confirming full parity with UI-only actions like human annotation labeling or UI-based evaluator configuration, and no public OpenAPI spec was found (404s). missing for 10: a documented OpenAPI/REST spec confirming full UI-to-API parity, explicit API equivalents for UI-only actions like human annotation attachment, and independent verification that every UI action has an API counterpart.
- [claimed-docs] “Human annotations — Attach ground truth labels directly in the UI”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
ai-native userExport all of my data in open formats and leave
weight 3 · round to Arize PhoenixLangSmith allows exporting traces via UI/API (langsmith-docs-2) and self-hosting your own instance (langsmith-docs-17), which gives some data portability, but there's no documented bulk-export tool, no explicit open-format guarantee (e.g., standard OTLP export of full trace history), and community comments note traces require a cloud account and lock-in concerns (langsmith-comm-5, langsmith-comm-1). missing for 10: a documented full-account data export/dump feature, explicit open-format (e.g., JSON/OTLP) export guarantees, and independent confirmation that exported data is complete and portable without vendor lock-in.
- [claimed-docs] “Filter, export, share, and compare traces via the UI or API.”
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable La…”
- [community] “I built this because LangSmith needs a cloud account to see my own traces.”
- [community] “LangSmith is excellent, but my usage is quite minimal, and I would prefer a locally hosted version that is easy to customize.”
Phoenix is open-source and self-hostable with data staying entirely in the user's infrastructure, and it ingests/exports traces via the open OpenTelemetry (OTLP) standard rather than a proprietary format, which strongly supports data portability and no lock-in. However, there is no explicit documentation of a dedicated 'export all my data' feature (e.g., bulk dataset/trace export to CSV/Parquet or a migration tool) beyond OTLP ingestion and self-hosting. Missing for 10: explicit bulk data export tooling/documentation, independent confirmation of successful full data migration/export by a user.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [github] “Datasets - Create versioned datasets of examples for experimen”
ai-native userRead the product's source under an open license
weight 2 · round to Arize PhoenixLangSmithnone0/10Evidence shows an open-source client SDK (langsmith-sdk) and MCP server, but no indication that the core LangSmith platform/product itself is released under an open license — self-hosting docs describe running LangSmith in your own infra, not viewing or auditing its source code.
- [github] “client = wrap_openai(openai.Client())”
- [probe] “official MCP server documented at https://github.com/langchain-ai/langsmith-mcp-server”
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable La…”
Phoenix's docs state it is 'built by Arize AI and the open-source community' and a public GitHub repo (Arize-ai/phoenix) is referenced, indicating the source is publicly readable. However, no evidence pack item specifies the actual license terms (e.g., MIT, Apache, ELv2) or confirms it is a permissive open-source license. Missing for 10: explicit license identification/text, independent confirmation of license terms, and any discussion of license restrictions.
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
- [github] “Datasets - Create versioned datasets of examples for experimen”
ai-native userSelf-host the core product
weight 3 · round to Arize PhoenixLangSmith explicitly documents self-hosting the core platform in your own infrastructure for observability, evaluation, and prompt engineering (with optional deployment management), which directly matches the story. However, community evidence shows at least one user built a workaround specifically because default LangSmith usage requires a cloud account to view traces, suggesting self-hosting is not the default/easy path and likely gated behind enterprise tiers rather than freely available like a fully open-source self-host. Missing for 10: independent hands-on confirmation of a successful self-hosted deployment, and clarity on licensing/tier restrictions for self-hosting.
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable La…”
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering.”
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for [observability](/langsmith/observability), [evaluation](/langsmith/evaluation),…”
- [community] “I built this because LangSmith needs a cloud account to see my own traces.”
Docs explicitly state Phoenix is free to self-host with no feature limitations, data stays fully within infrastructure and can be air-gapped, with a simple CLI command (uvx arize-phoenix serve) and Docker image versioning for production deployments. Missing for 10: independent/hands-on confirmation of self-hosting experience beyond vendor docs.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “$ uvx arize-phoenix serve”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round to Arize PhoenixLangSmith supports self-hosting your own instance in your own infrastructure (docs-17/23/32), which lets an organization control where data physically resides, but there is no documented cloud region-selection or data-residency feature (e.g., EU vs US region choice) for the hosted SaaS offering. missing for 10: explicit cloud data-residency/region selection options, documentation of supported regions, and any compliance certifications tied to residency.
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable La…”
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering.”
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for [observability](/langsmith/observability), [evaluation](/langsmith/evaluation),…”
Phoenix documents that it is free to self-host with data staying entirely within your own infrastructure and can be fully air-gapped, which lets users choose exactly where their data lives (docs-27/56). However, there is no explicit mention of a managed/cloud offering with selectable regions or residency guarantees, so the 'choose region' capability is only implicit via self-hosting. Missing for 10: explicit region-selection UI/settings, documented data residency options for any hosted/cloud tier, and independent confirmation of deployment flexibility.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
ai-native userPrevent my data from being used to train AI models
weight 3 · round to Arize PhoenixLangSmithnone0/10The evidence pack contains no mention of a data-training opt-out, privacy policy, or commitment regarding use of customer trace data for model training; all evidence is about tracing, evaluation, dashboards, and self-hosting features, not privacy/training-data posture.
Phoenix's self-hosting docs state that data stays entirely within the user's infrastructure and 'nothing is sent to Arize' and can be fully air-gapped, which strongly implies no data is available to Arize for model training when self-hosted. However, there is no explicit statement of a data-training policy (e.g., for any hosted/cloud offering) or a formal privacy/DPA commitment on this point. Missing for 10: explicit no-training-data policy statement, coverage of hosted/cloud deployment privacy terms, independent verification.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
ai-native userControl data retention and deletion
weight 2 · round to Arize PhoenixLangSmithnone0/10The evidence pack contains no mention of data retention windows, deletion APIs, or user-controlled data lifecycle settings for traces/datasets in LangSmith. Self-hosting (langsmith-docs-17/23/32) offers indirect infrastructure control but is not evidence of an explicit retention/deletion feature.
Phoenix's self-hosting model gives users full control over where data lives ('free to self-host... data stays entirely within your infrastructure... nothing is sent to Arize and can be fully air-gapped'), which implicitly enables retention/deletion control since the operator owns the storage. However, there is no explicit documentation of retention policies, TTL settings, or data-deletion APIs/commands within Phoenix itself. Missing for 10: explicit retention configuration options, documented deletion/purge commands or APIs, and any cloud-hosted retention controls for non-self-hosted deployments.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
ai-native userOpt out of telemetry and usage tracking
weight 2 · round to Arize PhoenixLangSmithnone0/10No evidence in the pack of any telemetry/usage-tracking opt-out setting for LangSmith itself; self-hosting (langsmith-docs-17/23/32) addresses data locality but doesn't address opting out of product telemetry, and community comments (langsmith-comm-1, langsmith-comm-5) actually highlight lack of control, needing a cloud account to view one's own traces.
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable La…”
- [community] “LangSmith is excellent, but my usage is quite minimal, and I would prefer a locally hosted version that is easy to customize.”
- [community] “I built this because LangSmith needs a cloud account to see my own traces.”
Phoenix's self-hosting docs state that when self-hosted, 'nothing is sent to Arize and can be fully air-gapped,' implying no forced telemetry to the vendor, which directly supports a privacy-conscious opt-out posture. However, there is no explicit documentation of a dedicated telemetry/usage-tracking toggle (e.g., an env var or CLI flag to disable analytics) for the hosted or default install path. Missing for 10: explicit opt-out mechanism/config flag for anonymous usage analytics, confirmation of what telemetry (if any) is collected by default in non-air-gapped mode.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
Prompt management — stories about prompt management in this arenaPrompt management
Stories about prompt management in this arena
Prompt workflow
developerIterate on prompts in a playground against real models and variables
weight 2 · round to Arize PhoenixLangSmithnone0/10The evidence pack mentions 'prompt engineering' and a 'prompt-context-hub#prompts' link in passing (langsmith-docs-17/23/32) but contains no description of a playground UI, testing against real models, or variable substitution for prompt iteration. Without concrete documentation of this specific capability, it cannot be credited as delivered.
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable La…”
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering.”
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for [observability](/langsmith/observability), [evaluation](/langsmith/evaluation),…”
Docs explicitly describe a Prompt Playground where developers can 'play with prompts, models, invocation parameters' and track progress via tracing/experiments, plus use of real production examples/datasets for iteration. This directly matches the story of iterating on prompts against real models and variables. Missing for 10: independent/hands-on user reports specifically about the playground UX (only vendor docs, no third-party corroboration of the playground itself).
- [claimed-docs] “Play with prompts, models, invocation parameters and track your progress via tracing and experiments”
- [claimed-docs] “Prompt Playground - Play with prompts, models, invocation parameters and track your progress via tracing and experiments”
- [claimed-docs] “iterate on your prompts using real production examples”
- [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
- [claimed-docs] “Replay the invocation of an LLM. Whether it's an LLM step in an LLM workflow or a router query, you can step into the LLM invocation and see…”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
developerVersion prompts and deploy changes to production without shipping code
weight 3 · round to Arize PhoenixThe evidence only references 'prompt engineering' and a 'prompt-context-hub#prompts' link within self-hosted/observability docs, implying a prompt management feature exists, but there is no detail on versioning, promotion workflows, or deploying prompt changes to production without code changes. Missing for 10: explicit docs on prompt version history, tagging/promotion to production, and evidence that apps pull prompts at runtime without redeploy.
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable La…”
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering.”
- [claimed-docs] “Host an instance of LangSmith in your own infrastructure for [observability](/langsmith/observability), [evaluation](/langsmith/evaluation),…”
Phoenix has explicit Prompt Management features supporting versioning, storage, and deployment of prompts, plus 'Prompts in Code' SDKs to sync prompts across environments so changes can be pushed to production without code changes. Docs show a UI-based prompt playground and versioning workflow decoupled from application code deploys. Missing for 10: independent/hands-on verification beyond vendor docs of production rollback/rollout mechanics and no explicit mention of environment-specific staged rollout controls.
- [claimed-docs] “Prompt Management — Version, store, and deploy prompts”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Prompts in Code - Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation
Instrumenting code and tracing requests end to end
Ai observability
ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app
weight 3 · round to Arize PhoenixLangSmith exposes traces via a documented API (filter/export via API) and has an official MCP server repo (langsmith-mcp-server) confirming agent-queryable access, but the evidence pack lacks detail on whether the MCP server or API surfaces metrics and eval results specifically, and no OpenAPI spec was found. Missing for 10: documented API/MCP coverage of metrics dashboards and eval-result querying, and independent confirmation the MCP server works well in practice.
- [probe] “official MCP server documented at https://github.com/langchain-ai/langsmith-mcp-server”
- [claimed-docs] “Filter, export, share, and compare traces via the UI or API.”
- [claimed-docs] “Trace your preferred framework or integrate LangSmith with any agent stack using our Python, Typescript, Go, or Java SDKs.”
Phoenix documents a first-party MCP server (/mcp endpoint) that lets MCP-compatible clients (Claude Code, Cursor, VS Code, etc.) search, query, and operate on projects, traces, datasets, experiments, prompts, and annotations, directly enabling an agent to query traces/metrics/eval results for debugging. This is backed by an official docs page and confirmed by probe evidence. Missing for 10: independent/hands-on third-party validation of the MCP server in practice and a documented general-purpose REST/GraphQL API spec (openapi probes 404) for non-MCP API access.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
Data controls
developerMask or redact sensitive data before it is stored in traces
weight 2 · round drawnLangSmithnone0/10No evidence in the pack mentions PII masking, redaction, or data filtering before trace storage; the docs cover tracing, evaluation, dashboards, alerts, and self-hosting but nothing about redacting sensitive data in traces.
Sdk coverage
developerInstrument apps in both Python and JS/TS with officially supported SDKs
weight 2 · round to LangSmithDocs and GitHub SDK repo confirm official Python and TypeScript SDKs (plus Go/Java) for instrumenting apps, with wrap_openai example and OpenTelemetry-based tracing support as alternatives. missing for 10: independent hands-on corroboration of JS/TS SDK parity and depth beyond docs, and community feedback is mixed on framework compatibility outside core LangChain use cases.
- [claimed-docs] “Trace your preferred framework or integrate LangSmith with any agent stack using our Python, Typescript, Go, or Java SDKs.”
- [github] “client = wrap_openai(openai.Client())”
- [claimed-docs] “LangSmith supports OpenTelemetry-based tracing, allowing you to send traces from any OpenTelemetry-compatible application.”
- [community] “Langsmith only shows the llm inferences. Also it hardly works with other frameworks like crewai for instance.”
- [community] “This is primarily based on the LangChain framework so it's fairly turnkey, but has no integration with the rest of your application... It's …”
Phoenix documents OTLP ingestion and auto-instrumentation for frameworks including JS/TS ones like Vercel AI SDK and Mastra, plus a phoenix-cli npx setup and OpenInference-based instrumentation, indicating both Python and JS/TS support. However, the evidence is mostly high-level marketing docs without explicit JS/TS SDK API reference or code samples analogous to the detailed Python SDK reference (docs-28/29/30/36), and no independent/hands-on confirmation of JS/TS SDK parity. Missing for 10: explicit JS/TS SDK API reference/docs comparable to the Python SDK reference, hands-on/independent confirmation of JS/TS instrumentation working end-to-end, and clarity on feature parity between the two SDKs.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
Trace capture
developerTrace multi-step agent runs as nested spans grouped into sessions or threads
weight 2 · round to Arize PhoenixDocs and community comments confirm LangSmith traces multi-step agent runs and lets you inspect each step in a graph/execution ('full traces of moving through the graph ... inputs and outputs for each step'), and OpenTelemetry-based tracing implies nested span semantics. However, the evidence pack never explicitly documents grouping into 'sessions' or 'threads' as a first-class concept, and one hands-on report claims tracing is limited to LLM inferences and works poorly with non-LangChain frameworks like CrewAI, undercutting universal multi-step nested tracing. missing for 10: explicit docs on session/thread grouping of traces, and stronger cross-framework corroboration of full nested-span coverage beyond LLM calls.
- [claimed-docs] “Traces are the record of what your agents did in production. Use them to debug failures, monitor quality, and build the datasets you evaluat…”
- [claimed-docs] “LangSmith supports OpenTelemetry-based tracing, allowing you to send traces from any OpenTelemetry-compatible application.”
- [github] “client = wrap_openai(openai.Client())”
- [community] “The power of langsmith is seeing full traces of moving through the graph and being able to inspect the inputs and outputs for each step... l…”
- [community] “Langsmith only shows the llm inferences. Also it hardly works with other frameworks like crewai for instance.”
Docs explicitly describe OTLP-based tracing that captures nested spans for model calls, retrieval, and tool use (docs-38, docs-12/13/14), and Sessions explicitly group related traces into conversations/user sessions while Projects organize traces by application (docs-16/49). Auto-instrumentation for popular agent frameworks (LangChain, LlamaIndex, etc.) supports multi-step agent run tracing out of the box (docs-5, docs-32). Missing for 10: independent/hands-on verification of nested span hierarchy rendering for complex multi-step agents beyond vendor docs.
- [claimed-docs] “Sessions Group related traces into conversations and user sessions”
- [claimed-docs] “Group related traces into conversations and user sessions”
- [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
- [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
- [claimed-docs] “Retrieved Documents: Inspect the documents retrieved during a Retriever call, including the score and order in which they were returned”
- [claimed-docs] “Tool Descriptions: View the descriptions and function signatures of the tools your LLM has been given access to”
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK
weight 2 · round to Arize PhoenixLangSmith is built natively for LangChain and has an explicit OpenAI SDK wrapper (wrap_openai) plus OTel-based tracing and generic SDKs (Python/TS/Go/Java) for other stacks, but no evidence names a Vercel AI SDK integration specifically, and community feedback notes friction integrating non-LangChain frameworks (e.g. 'hardly works with other frameworks like crewai', 'no integration with the rest of your application'). missing for 10: explicit Vercel AI SDK integration documentation, broader evidence that non-LangChain framework instrumentation works smoothly.
- [github] “client = wrap_openai(openai.Client())”
- [claimed-docs] “LangSmith supports OpenTelemetry-based tracing, allowing you to send traces from any OpenTelemetry-compatible application.”
- [claimed-docs] “Trace your preferred framework or integrate LangSmith with any agent stack using our Python, Typescript, Go, or Java SDKs.”
- [community] “Langsmith only shows the llm inferences. Also it hardly works with other frameworks like crewai for instance.”
- [community] “This is primarily based on the LangChain framework so it's fairly turnkey, but has no integration with the rest of your application... It's …”
Docs explicitly state Phoenix accepts OTLP traces and provides auto-instrumentation for popular frameworks including LangChain and Vercel AI SDK, plus OpenAI is referenced as an adapter/evaluator target and is a widely documented OpenInference integration. Missing for 10: no independent/hands-on verification of these integrations working, and no explicit dedicated docs snippet for OpenAI SDK auto-instrumentation beyond the adapter mention.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
- [claimed-docs] “Model Agnostic via adapters (for OpenAI, LiteLLM, LangChain, AI SDK, and more) — so you can easily switch judge models”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
developerCapture multimodal payloads (images, audio, files) inside my traces
weight 1 · round drawnLangSmithnone0/10No evidence in the pack mentions multimodal payload capture (images, audio, files) within traces; documentation covers general tracing, evaluation, dashboards, alerts, and SDK integrations but never addresses multimodal content in traces. Missing for 10: any mention of image/audio/file attachment support in trace payloads, UI rendering of multimodal content, or SDK helpers for multimodal logging.
Arize Phoenixnone0/10The evidence pack describes tracing of model calls, retrieval, tool use, latency, and token usage, but nowhere mentions capturing images, audio, or file attachments/multimodal payloads within spans or traces. Since this is a plausible capability for a tracing/observability tool, absence of evidence yields 'none'.
- [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
- [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
- [claimed-docs] “Retrieved Documents: Inspect the documents retrieved during a Retriever call, including the score and order in which they were returned”
developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format
weight 3 · round to Arize PhoenixLangSmith's docs explicitly state OpenTelemetry-based tracing support, letting any OTel-compatible app send traces to LangSmith instead of using a proprietary SDK format. However, evidence is limited to a single doc page with no independent/hands-on corroboration or detail on receiving/exporting via OTLP endpoints. Missing for 10: independent verification of OTLP ingestion working in practice, and documentation on OTLP export/interop beyond a single sentence.
- [claimed-docs] “LangSmith supports OpenTelemetry-based tracing, allowing you to send traces from any OpenTelemetry-compatible application.”
Docs explicitly state Phoenix accepts traces over OpenTelemetry (OTLP), provides auto-instrumentation for popular frameworks, and is built on top of OpenTelemetry powered by OpenInference instrumentation — confirming standard OTLP ingestion rather than a proprietary format. Missing for 10: independent/hands-on corroboration of OTLP export/receive behavior beyond vendor docs.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage
weight 3 · round to Arize PhoenixDocs and SDK examples (wrap_openai) confirm capturing full traces with inputs/outputs, and dashboards explicitly track token usage, latency, and error rates; OpenTelemetry support extends this beyond native SDKs. Community feedback corroborates seeing full input/output traces per step, though one user notes it mainly captures LLM inferences and integration with non-LangChain frameworks (e.g., CrewAI) is weaker. missing for 10: independent benchmark of latency/token accuracy, broader non-LangChain framework validation.
- [claimed-docs] “Traces are the record of what your agents did in production. Use them to debug failures, monitor quality, and build the datasets you evaluat…”
- [claimed-docs] “Filter, export, share, and compare traces via the UI or API.”
- [claimed-docs] “LangSmith supports OpenTelemetry-based tracing, allowing you to send traces from any OpenTelemetry-compatible application.”
- [claimed-docs] “Prebuilt dashboards are created automatically for each project and cover essential metrics, such as trace count, error rates, token usage, a…”
- [github] “client = wrap_openai(openai.Client())”
- [community] “The power of langsmith is seeing full traces of moving through the graph and being able to inspect the inputs and outputs for each step... l…”
- [community] “Langsmith only shows the llm inferences. Also it hardly works with other frameworks like crewai for instance.”
Phoenix's docs extensively document capturing OTLP/OpenTelemetry-based traces of LLM calls, including inputs/outputs (span data), application latency, and detailed token usage breakdowns, plus auto-instrumentation for popular frameworks and manual tracing decorators. This is a core, heavily documented first-party capability with a CLI/SDK setup flow. Missing for 10: independent hands-on verification beyond vendor docs/community sentiment.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
- [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
- [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
Not comparable on these axes
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · not comparableLangSmithn/aLangSmith is an observability/evaluation platform for LLM applications, not an agent that executes tasks using tools; the only MCP evidence found (langsmith-probe-4) shows LangSmith exposes itself AS an MCP server for others to query traces/evals, which is the opposite direction from the story's ask (LangSmith consuming external MCP servers' tools). There is no indication LangSmith has an agent runtime that plugs into MCP servers as a client, so this axis does not apply to this product category.
- [probe] “official MCP server documented at https://github.com/langchain-ai/langsmith-mcp-server”
Arize Phoenixn/aPhoenix is an observability/evaluation platform, not an agent that performs tasks using tools; its documented MCP integration runs in the opposite direction — Phoenix exposes an /mcp endpoint so external MCP clients (Claude Code, Cursor, VS Code) can query Phoenix's traces/datasets, not the reverse. There is no indication Phoenix itself consumes external MCP servers' tools to extend its own capabilities, which is a different product role (MCP client) than what Phoenix is built for.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”