LLM Evals & Observability Arena
Arize Phoenix vs Helicone
Arize Phoenix
Arize AI, Inc.
Arize Phoenix wins · 23–10 (13 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to Arize PhoenixDirect probe evidence confirms llms.txt returns HTTP 200 with agent-oriented content, and docs pages are also available in markdown form (.md) with a documentation index, both explicitly designed for agent/LLM consumption. Missing for 10: no independent/community confirmation of an agent actually consuming these files successfully.
- [probe] “PROBE llms.txt: HTTP 200 at https://arize.com/llms.txt # Arize — AI & Agent Engineering > Arize is the AI engineering platform for teams bu…”
- [probe] “PROBE docs-md: HTTP 200 at https://arize.com/docs/phoenix.md > ## Documentation Index > Fetch the complete documentation index at: https://a…”
Helicone has a confirmed llms.txt at docs.helicone.ai/llms.txt (HTTP 200) with an index of docs, plus per-page .md versions and an OpenAPI/swagger.json spec, all directly probeable and agent-consumable. missing for 10: no independent/community confirmation that agents actually consume this llms.txt successfully in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.helicone.ai/llms.txt # Helicone OSS LLM Observability - [Quickstart](https://docs.helicone.ai/gett…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.helicone.ai/getting-started/quick-start.md > ## Documentation Index > Fetch the complete documentati…”
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to Arize PhoenixPhoenix ships a CLI (`uvx arize-phoenix serve`), a Docker image pinned for production deployments, and a Python SDK with dataset evaluators explicitly described as forming 'an evaluation harness similar to a unit test suite,' all of which support headless/automated use in CI-like pipelines. However, there is no explicit CI/CD example (e.g., GitHub Actions workflow) or documentation of running evaluations as an automated CI gate. Missing for 10: explicit CI pipeline integration examples/docs, evidence of automated test-gating in CI, independent confirmation of headless CI usage.
- [claimed-docs] “$ uvx arize-phoenix serve”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
Helicone is fundamentally an API/SDK-based proxy and observability layer — one-line SDK integration, REST API for queries, webhooks, and self-hosted Docker deployment all imply it can run without any UI, i.e., headlessly in scripts or pipelines. However, there is no explicit documentation or example of CI/CD usage, automated test pipelines, or headless agent orchestration specifically. Missing for 10: explicit CI/CD integration guide or example, documented non-interactive/automation-focused workflow, and independent confirmation of headless operation in production pipelines.
- [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
ai-native userConnect an agent via an official MCP server
weight 3 · round to Arize PhoenixPhoenix documents an official remote MCP server exposing a `/mcp` endpoint that MCP-compatible clients (Claude Code, Cursor, VS Code, etc.) can connect to for searching and operating on projects, traces, datasets, experiments, prompts, and annotations. This is first-party documentation confirming the exact capability. Missing for 10: independent/hands-on community corroboration of the MCP server working in practice, and details on authentication/setup robustness.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
Heliconenone0/10Helicone is an LLM observability/gateway platform, and this axis (offering an official MCP server so agents can connect) plausibly applies since it has an ecosystem of integrations, docs, and APIs. However, no evidence in the pack mentions an MCP server, MCP integration, or any agent-connection protocol offered by Helicone.
ai-native userUse an official CLI
weight 2 · round to Arize PhoenixPhoenix ships an official CLI (`@arizeai/phoenix-cli` via `npx -y @arizeai/phoenix-cli setup`) for auto-instrumentation setup, plus a `uvx arize-phoenix serve` command to launch the server, confirming a genuine first-party CLI exists for agentic/automated workflows. However, the CLI's documented scope is narrow (setup + serve) rather than covering the full product surface (datasets, evals, prompts, experiments), and there's no independent/community confirmation of its use. Missing for 10: broader CLI command coverage across Phoenix's feature set, independent hands-on verification of the CLI in practice.
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
- [claimed-docs] “$ uvx arize-phoenix serve”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app's root directory.”
- [claimed-docs] “start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app’s root directory”
ai-native userDrive the product through a documented public API
weight 3 · round to HeliconePhoenix documents a full SDK API reference (tracing decorators, dataset/experiment APIs, prompt sync SDKs), OTLP-based trace ingestion, a CLI (phoenix-cli), and an official remote MCP endpoint that lets AI-native clients search/query/operate on projects, traces, datasets, experiments, and prompts—together constituting a documented, programmatically-drivable public API surface. The openapi.json probe returned 404s at standard paths, meaning there's no discoverable formal REST/OpenAPI spec, so the API surface is SDK/CLI/MCP-centric rather than a single canonical REST API. Missing for 10: a published OpenAPI/REST spec, independent third-party confirmation of API robustness.
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
- [claimed-docs] “$ uvx arize-phoenix serve”
Helicone publishes a documented REST API (swagger/OpenAPI spec, point-query endpoints, HQL) plus SDK/gateway APIs, all accessible via public docs and confirmed by a live OpenAPI probe. This gives AI-native users a documented programmatic surface beyond the UI. Missing for 10: deeper docs/examples of full CRUD API coverage and independent third-party corroboration of API usage beyond ingestion/logging.
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.helicone.ai/llms.txt # Helicone OSS LLM Observability - [Quickstart](https://docs.helicone.ai/gett…”
- [github] “AI Gateway: Access 100+ AI models with 1 API key through the OpenAI API with intelligent routing and automatic fallbacks.”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round drawnArize Phoenixnone0/10No evidence in the pack describes scoped or least-privilege API key/credential issuance for agents; Phoenix's docs cover tracing, evaluation, prompt management, and an MCP endpoint, but nothing about credential scoping or access control granularity.
Heliconenone0/10No evidence Helicone supports issuing scoped or least-privilege API credentials/keys for agents; docs mention bringing your own provider keys or using Helicone's own key, but nothing about granular permission scoping. A community comment even flags concern about sending a full API key on every request, with no rebuttal describing scoped-key mitigation.
- [claimed-docs] “Want more control? You can bring your own provider keys instead.”
- [community] “Hmm, so to integrate I have to basically send my api key to you on every request? Not great”
ai-native userBuild against official SDKs
weight 2 · round drawnPhoenix ships an official, modular Python SDK plus client SDKs for tracing, prompts, evaluations, and datasets, with OTLP-based tracing decorators, auto-instrumentation, and dedicated SDK/API reference docs. Coverage spans core agentic workflows (tracing, evals, prompt management, datasets/experiments) and is corroborated by GitHub repo and docs pages, though independent hands-on SDK usage reports are thin. Missing for 10: broader independent/third-party validation of SDK usage beyond official docs, and explicit multi-language SDK support details beyond Python/JS CLI.
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Prompts in Code - Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
- [github] “Datasets - Create versioned datasets of examples for experimen”
Helicone works by wrapping official/familiar SDKs (OpenAI SDK, Anthropic, LangChain, Gemini, Vercel AI SDK) with one-line integration, backed by docs, GitHub README, and independent hands-on community confirmation of easy SDK-based integration. missing for 10: no dedicated first-party Helicone SDK docs page enumerated (language coverage) and no independent benchmark of SDK completeness beyond community anecdotes.
- [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”
- [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
- [community] “Happy Helicone customer here. It's a dead simple setup. It's great to have the extra charts and logging to debug issues and make sure all is…”
- [community] “Your onboarding is impressive, one of the few products where 'get set up in one line of code' is true.”
ai-native userSubscribe to events via webhooks
weight 2 · round to HeliconeArize Phoenixnone0/10No evidence anywhere in the pack of a webhook subscription mechanism; Phoenix's integration surface is OTLP tracing ingestion, an MCP server, and SDKs, but nothing about outbound event webhooks for subscribing to Phoenix events.
Helicone documents a dedicated webhooks feature that sends instant notifications on LLM request completion, supports property-based filtering, and enables automation/scoring/integration with external systems, directly matching the story's subscribe-to-events use case. missing for 10: independent hands-on confirmation of webhook reliability/usage and richer docs on payload schema or delivery guarantees.
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Only requests matching ALL specified properties will trigger webhooks.”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round to Arize PhoenixPhoenix offers LLM-as-judge evaluators that return explanations and reasoning alongside scores, and lets users explore traces to spot systematic issues, which counts as AI-generated signal on the data — but this is evaluator-driven scoring, not proactive, unprompted insights/suggestions surfaced automatically inside the product UI. missing for 10: evidence of autonomous/proactive insight generation (e.g., auto-detected anomalies, recommended fixes, natural-language summaries of trends) rather than only judge-model scoring/explanations tied to evals a user configures.
- [claimed-docs] “Built-in Explanations — all Phoenix LLM evaluations return explanations by default for better results and richer signals”
- [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model's full reasoning, the final scores, a…”
- [claimed-docs] “Use Phoenix's trace viewer to explore evaluation traces, identify systematic biases, and continuously improve evaluator performance.”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
Helicone advertises 'real-time evaluation' that automatically scores and evaluates LLM responses for quality, safety, and relevance, which is a form of AI-generated insight on the user's own request data, but the evidence pack shows no broader AI-generated dashboard summaries, anomaly explanations, or proactive suggestions beyond this single scoring feature. missing for 10: evidence of AI-generated narrative insights/recommendations across usage/cost data, natural-language querying of analytics, or proactive suggestion features beyond automated response scoring.
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round to HeliconeArize Phoenixnone0/10Docs mention an 'sdk-api-reference' page listing decorators and SDK features, but there is no evidence of an interactive, runnable API reference (e.g., a Swagger/OpenAPI explorer or live code sandbox); a direct probe for OpenAPI/swagger specs returned 404 on all candidate paths, indicating no such interactive reference is discoverable.
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
Helicone publishes a formal OpenAPI/swagger spec and REST API reference pages, and separately offers a 'Playground' for testing/iterating on prompts in its UI, suggesting some interactive tooling exists. However, there is no direct evidence of a Swagger-UI-style interactive reference page where a user can execute runnable request examples against the API from the docs themselves. Missing for 10: confirmation of an actual 'try it out' runnable console in the API reference docs, and evidence of code-sample generation across languages.
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [claimed-docs] “Get Requests (Point Queries)”
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round to HeliconeArize Phoenixnone0/10A direct probe for OpenAPI/swagger endpoints at Phoenix's domain returned 404 for all candidate paths, and no docs mention a downloadable machine-readable API spec. Missing for 10: any OpenAPI/Swagger spec, documented REST API reference, or SDK-generated schema.
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
Helicone exposes a machine-readable OpenAPI spec at /swagger.json (HTTP 200, valid 'openapi' key), and its REST endpoints are documented (e.g., request query endpoints). missing for 10: no independent/community confirmation of the OpenAPI spec's completeness or usage.
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [claimed-docs] “Get Requests (Point Queries)”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to Arize PhoenixPhoenix supports self-hosting with full data isolation (data never leaves your infra) and lets you organize traces into separate 'projects', collect data 'from production, staging, evaluations, and even manually' into datasets, and run experiments/evaluations against those datasets rather than live traffic — all of which let a user test without touching real production data. However, there's no explicit documented 'sandbox environment' feature, no isolated test-instance provisioning, and no guidance on separating staging from prod deployments beyond generic self-hosting flexibility. Missing for 10: explicit sandbox/staging environment tooling, isolation guarantees beyond self-hosting, and independent confirmation that experiments truly never touch production data.
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
- [claimed-docs] “Organize traces into separate projects for different applications”
Helicone offers a Playground for testing/iterating on prompts (not necessarily isolated from production logging) and self-host Docker Compose options for local development, which could function as an ad-hoc sandbox, but there is no explicit documented 'sandbox environment' or test-mode that guarantees isolation from production data. Missing for 10: a dedicated sandbox/staging mode, explicit statement about not touching production data, and independent verification of data isolation.
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnArize Phoenixnone0/10Evidence shows only generic container/image version pinning (e.g., 'version-8.0.0' Docker tags) but no documented API versioning scheme or deprecation policy for Phoenix's SDK/API; an OpenAPI probe also returned 404s, finding no formal API spec to review versioning against.
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
Heliconenone0/10An OpenAPI spec exists (helicone-probe-3) confirming a REST API, but there is no evidence of API versioning scheme (e.g., v1/v2 paths with migration guides) or a documented deprecation policy for endpoints/models; nothing in the docs pack addresses lifecycle or backward-compatibility commitments.
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [claimed-docs] “Get Requests (Point Queries)”
Alerting dashboards — stories about alerting dashboards in this arenaAlerting dashboards
Stories about alerting dashboards in this arena
Monitoring
ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics
weight 2 · round to HeliconeArize Phoenixnone0/10Phoenix's docs describe tracing, span-level latency/cost/token data, and eval scoring, but nowhere is there evidence of a dashboard-building feature (custom charts, widgets, or dashboard layouts) that lets an ML engineer aggregate these metrics into a dashboard view. The evidence only covers per-trace exploration, filtering spans, and projects/sessions organization, not dashboard construction or alerting.
Helicone provides building blocks for custom analytics — HQL query language for ad-hoc queries, REST API point queries, one-line export to PostHog for custom dashboards, real-time eval scoring via webhooks, and alerts on error rates/cost — but evidence does not show a native in-product dashboard builder combining latency, error, cost, and eval-score metrics together; users must rely on external tools (PostHog) or raw queries (HQL) to assemble true custom dashboards. Missing for 10: evidence of a first-party customizable dashboard UI, direct proof eval scores can be plotted alongside latency/cost/error in one dashboard, and independent hands-on confirmation of dashboard customization.
- [claimed-docs] “HQL (Query Language)”
- [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
- [github] “Export to PostHog in one-line for custom dashboards”
- [claimed-docs] “Get Requests (Point Queries)”
- [community] “Discussion of Helicone's architecture allowing users to write SQL directly to a shared ClickHouse instance, with commenters noting noisy-nei…”
developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email
weight 2 · round to HeliconeArize Phoenixnone0/10The evidence pack covers tracing, evaluation, prompt management, and dataset/experiment features but contains no mention of alerting, notification channels (Slack/PagerDuty/email), or threshold-based triggers for error rates, cost, or eval-score drops.
Helicone documents a dedicated Alerts feature for monitoring error rates and cost spikes (helicone-docs-11), and webhooks for real-time notifications tied to request events (helicone-docs-9, helicone-docs-23), but the evidence pack never mentions eval-score-drop alerts or specific delivery channels like Slack, PagerDuty, or email. Missing for 10: eval-score-based alert triggers, and documented integrations/notification channels (Slack, PagerDuty, email).
- [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to Arize PhoenixPhoenix supports bulk-style operations through datasets (grouping many traces/examples), running experiments and evaluations across thousands of items with built-in concurrency/retry handling, and MCP-based querying/operating on projects, traces, datasets, and prompts. However, there is no explicit documentation of a bulk UI action (e.g., multi-select delete/tag/export) or a dedicated batch API for arbitrary bulk edits across items. missing for 10: explicit bulk UI operations (multi-select edit/delete/export), a documented batch API for arbitrary item updates, and independent/hands-on confirmation of large-scale bulk actions beyond evaluation/experiment runs.
- [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
Helicone offers query/export mechanisms (HQL query language, REST 'point queries' API, one-line export to PostHog) that could be used to operate on many logged requests at once, but there is no explicit documentation of bulk actions like batch tagging, batch deletion, or batch updates across many items. missing for 10: dedicated bulk-edit/bulk-delete/batch-processing API, documentation of batch size limits, and independent confirmation of bulk operation reliability.
- [claimed-docs] “HQL (Query Language)”
- [claimed-docs] “Get Requests (Point Queries)”
- [github] “Export to PostHog in one-line for custom dashboards”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to HeliconeArize Phoenixnone0/10Phoenix's evidence covers tracing, evaluation, datasets, prompt management, and MCP integration, but nothing describes a rules/triggers engine that automatically fires actions on events (e.g., alerting, auto-remediation, webhooks on thresholds). The axis is plausible for an observability platform, but no such capability is evidenced.
Helicone supports webhooks that fire on request-completion events and can be filtered so only requests matching ALL specified properties trigger them, plus alerts that monitor error rates/costs — both are event-triggered automations. However, this is a narrow, fixed set of trigger types (webhooks, alerts) rather than a general user-defined rules engine with configurable conditions and arbitrary actions. Missing for 10: evidence of a flexible rule-builder UI/API for custom conditions and actions beyond webhooks/alerts, and independent confirmation these automations work reliably in production.
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Only requests matching ALL specified properties will trigger webhooks.”
- [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
ai-native userVersion, review, and roll back my automations
weight 1 · round to HeliconePhoenix offers prompt versioning ('Version, store, and deploy prompts', 'you can version prompts, test prompt variants... before rolling them out') and dataset/experiment comparisons across app versions, which cover review and some rollback-like workflow for prompts. However, Phoenix is an observability/eval platform, not an automation/workflow builder, so there's no evidence of versioning or rolling back full agentic automations/pipelines themselves — only prompts and datasets. Missing for 10: explicit rollback mechanism for automations/workflows (not just prompts), audit trail of automation changes, and evidence of reverting a live automation to a prior version.
- [claimed-docs] “Prompt Management — Version, store, and deploy prompts”
- [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
Helicone's prompts feature explicitly supports versioning and rollback ('Track every change, compare versions, and rollback instantly if something goes wrong') and instant deployment of prompt changes without redeploying, which covers version/rollback for prompt-based automations. However, this is scoped to prompts specifically, not broader automation workflows (e.g., webhooks, alerts pipelines), and there's no 'review' workflow (approvals, diffs, comments) documented, nor independent/community corroboration of the rollback feature working in practice. Missing for 10: review/approval workflow evidence, broader automation (not just prompts) versioning, independent hands-on confirmation of rollback reliability.
- [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”
- [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
- [claimed-docs] “Use your prompt instantly by referencing its ID in your AI Gateway. No code changes, no rebuilds.”
Cost monitoring — stories about cost monitoring in this arenaCost monitoring
Stories about cost monitoring in this arena
Cost tracking
developerAttribute cost and usage to users, sessions, and features via custom metadata
weight 2 · round drawnPhoenix documents token-usage breakdowns per LLM call and lets you organize traces into projects and group them into sessions/conversations, which provides some basis for usage attribution by session/application. However, there is no evidence of a general custom-metadata/tagging mechanism for arbitrary attribution to users or features, nor any cost aggregation/dashboard that rolls up spend by user, session, or feature tag. missing for 10: explicit custom metadata/tagging API for attributing spans to users or features, cost-specific aggregation/dashboard views by user/session/feature, documentation showing cost roll-ups beyond raw token counts.
- [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
- [claimed-docs] “Sessions Group related traces into conversations and user sessions”
- [claimed-docs] “Organize traces into separate projects for different applications”
- [claimed-docs] “Group related traces into conversations and user sessions”
Helicone's docs show a Sessions feature that groups related requests to trace agent flows (helicone-docs-5, helicone-docs-6) and per-user cache namespaces (helicone-docs-13), which support some usage segmentation, but there is no explicit evidence of a general custom-metadata/custom-properties system for tagging requests by user, feature, or session to attribute cost and usage in dashboards or queries. Missing for 10: explicit custom-properties/tagging API, cost breakdown by user/feature dashboards, and documentation showing metadata-based cost attribution reports.
- [claimed-docs] “Sessions group these related requests together, letting you trace the entire agent flow from initial user input to final response in one uni…”
- [claimed-docs] “Requests with the same path represent the same "type" of work, even if they happen at different times.”
- [claimed-docs] “Create separate cache namespaces for different users or contexts.”
developerSee cost and token usage per request, model, and time period in dashboards
weight 3 · round drawnPhoenix provides a token usage breakdown per LLM call/span (docs-46) and organizes traces into projects and sessions (docs-15/16/48/49), which supports some per-model and per-time grouping, but there is no evidence of aggregated cost dashboards, dollar-cost calculations, or dedicated per-model/time-period cost views — only per-span token counts. missing for 10: dedicated cost dashboards showing dollar cost (not just tokens), aggregated views by model and by time period, and any UI screenshot/independent confirmation of such dashboards.
- [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
- [claimed-docs] “Sessions Group related traces into conversations and user sessions”
- [claimed-docs] “Organize traces into separate projects for different applications”
- [claimed-docs] “Group related traces into conversations and user sessions”
Community feedback confirms cost/logging visibility ('extra charts and logging to debug issues', 'how much it costs', 'analytics/observability') and docs reference request query APIs and an HQL query language for analysis, but no first-party documentation explicitly describes dashboards broken down by request, model, and time period. Missing for 10: explicit docs/screenshots of dashboard UI showing cost/token breakdowns by model and time period, and independent confirmation of that specific breakdown granularity.
- [community] “Happy Helicone customer here. It's a dead simple setup. It's great to have the extra charts and logging to debug issues and make sure all is…”
- [community] “When your chains get long/complex enough in LangChain, it's really hard to understand from debug output what's final prompt that actually be…”
- [community] “justin - helicone's great. I login in ~10 times a day. Great for debugging - I'm mostly using for analytics / observability.”
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
Data access export — stories about data access export in this arenaData access export
Stories about data access export in this arena
Data export
developerBulk-export traces and datasets to blob storage or my data warehouse
weight 2 · round to HeliconeArize Phoenixnone0/10The evidence pack covers tracing, datasets, evaluation, prompt management, and self-hosting, but contains no mention of bulk-exporting traces or datasets to blob storage (S3, GCS) or a data warehouse (Snowflake, BigQuery, etc.). The closest evidence is generic dataset/SDK querying (docs-25, docs-30) which is not the same as a bulk export/warehouse-sync feature.
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
Helicone offers a REST API for point queries, an HQL query language for direct SQL access to logged data, and a one-line PostHog export integration, which give some programmatic data-access/export capability. However, no evidence documents a dedicated bulk-export feature to blob storage (S3/GCS) or to a data warehouse (Snowflake/BigQuery) specifically for traces/datasets. Missing for 10: explicit blob-storage export, data-warehouse connector/sync, bulk/batch export tooling or docs beyond point queries and third-party PostHog export.
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
- [github] “Export to PostHog in one-line for custom dashboards”
- [community] “Discussion of Helicone's architecture allowing users to write SQL directly to a shared ClickHouse instance, with commenters noting noisy-nei…”
Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets
Measuring quality — datasets, eval runs, regression tracking
Ai eval ops
ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results
weight 2 · round to Arize PhoenixPhoenix has a Python/SDK-driven dataset creation, experiment/eval-run triggering, and result retrieval (datasets, dataset evaluators, experiments, MCP endpoint letting agents 'operate on datasets, experiments'), which supports an agent programmatically creating datasets, running evals, and reading back scores. However the evidence is all docs-level description without a concrete end-to-end code example or independent confirmation of an agent performing this full loop. Missing for 10: a worked code/API example showing dataset creation -> eval trigger -> result read-back in one flow, and independent/hands-on verification beyond vendor docs.
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [github] “Datasets - Create versioned datasets of examples for experimen”
Heliconenone0/10Evidence shows observability features (sessions, webhooks, HQL query, REST API for point queries) and a vague mention of 'real-time evaluation' scoring, but there is no documentation of a dataset-creation API, a way to programmatically trigger an eval run, or an API to read back eval results — the core building blocks of this story are absent from the evidence pack.
Human review
ml engineerRoute outputs to human annotation queues for review and labeling
weight 2 · round to Arize PhoenixPhoenix supports attaching human annotations/ground-truth labels directly in the UI on traces/spans, which enables human review and labeling of outputs, and scoring can combine human labels with automated evals. However, there's no evidence of a dedicated 'annotation queue' routing mechanism, task assignment to reviewers, or workflow management for distributing items to human labelers. missing for 10: dedicated annotation queue/task routing to specific reviewers, workflow/assignment management, evidence of multi-user labeling workflows beyond UI annotation.
- [claimed-docs] “Human annotations — Attach ground truth labels directly in the UI”
- [claimed-docs] “Attach ground truth labels directly in the UI”
- [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”
Offline evals
developerRun evals in CI and gate deployments on their results
weight 2 · round to Arize PhoenixPhoenix explicitly frames dataset evaluators as forming 'an evaluation harness similar to a unit test suite' and provides SDK/Python-based experiment running with automatic scoring, which are the building blocks for CI-based evals. However, there is no direct documentation of CI pipeline integration (e.g., GitHub Actions examples), exit-code/pass-fail thresholds, or deployment-gating mechanics tied to eval results. Missing for 10: explicit CI/CD integration docs, pass/fail gating logic, deployment-blocking examples.
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “Dataset Evaluators serve as **test cases** that automatically score outputs when running experiments—forming an evaluation harness similar t…”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “This means you can run thousands of evaluations without writing any retry or concurrency logic yourself.”
Heliconenone0/10Helicone offers observability, webhooks, real-time scoring, caching, and prompt versioning, but there is no evidence of a CI-integrated eval runner, test suite, or deployment gating mechanism tied to eval results. Nothing in the evidence pack mentions CI pipelines, exit codes, or blocking deploys based on evaluation scores.
ml engineerWrite custom code-based scorers and metrics for my evaluations
weight 2 · round to Arize PhoenixDocs explicitly state Phoenix supports deterministic code-based evaluators (exact match, regex, custom heuristics) alongside LLM-as-a-judge, and Dataset Evaluators act as test cases scoring outputs automatically in experiments similar to a unit test suite, which is exactly code-based custom scoring for evals. Missing for 10: no independent/hands-on corroboration of writing custom Python scorer code, and no detailed API reference/example showing custom scorer implementation.
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators, where a sec…”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “Dataset Evaluators serve as **test cases** that automatically score outputs when running experiments—forming an evaluation harness similar t…”
- [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”
Helicone's webhooks feature mentions enabling users to 'score responses' and lists 'Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance', implying some custom scoring hook exists, but there is no documented SDK, API, or framework for writing custom code-based scorers/metrics as part of an evals workflow. missing for 10: dedicated evals/scorer API or SDK docs, examples of custom metric code, integration with a datasets/evals product surface.
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
ml engineerCompare eval runs side by side to catch regressions between prompt or model versions
weight 3 · round to Arize PhoenixPhoenix's experiments feature explicitly groups traces into datasets, reruns them through different app/prompt/model versions, and compares evaluation results to confirm whether a change improved performance, with automatic scoring via configured evaluators — directly matching the side-by-side regression-catching story. Missing for 10: no independent/hands-on example of an actual side-by-side UI diff view or third-party corroboration beyond vendor docs.
- [claimed-docs] “optimize your app with experiments that compare changes on the same inputs”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results to confirm whe…”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [github] “Datasets - Create versioned datasets of examples for experimen”
Helicone documents prompt version tracking/comparison and rollback (docs-18) plus real-time automated scoring via webhooks (docs-23) and a playground for iterating on prompts (gh-3/gh-6), which are adjacent building blocks for regression detection, but there is no explicit evidence of a dedicated 'eval run' comparison view or side-by-side regression report across prompt/model versions. Missing for 10: a documented eval-run comparison UI, dataset-based batch evaluation runs, and any hands-on/community confirmation of side-by-side regression catching.
- [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
ml engineerScore outputs with configurable LLM-as-a-judge evaluators
weight 3 · round to Arize PhoenixDocs explicitly describe LLM-as-a-judge evaluators configurable via model-agnostic adapters (OpenAI, LiteLLM, LangChain, AI SDK), with built-in explanations, automatic scoring of experiment results in the UI or via SDK, and full traceability of judge prompts/reasoning/scores. This is a first-party documented core feature well beyond a single mention. Missing for 10: independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Model Agnostic via adapters (for OpenAI, LiteLLM, LangChain, AI SDK, and more) — so you can easily switch judge models”
- [claimed-docs] “Built-in Explanations — all Phoenix LLM evaluations return explanations by default for better results and richer signals”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators, where a sec…”
- [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model’s full reasoning, the final scores, a…”
- [claimed-docs] “Every evaluation execution captures the input data, the exact prompts sent to the judge LLM, the model's full reasoning, the final scores, a…”
Helicone's docs mention webhooks enabling 'real-time evaluation' to automatically score responses for quality/safety/relevance, which points toward LLM-as-judge scoring, but there's no evidence of a configurable evaluator framework, built-in judge models, or dataset-based eval workflows that an ML engineer could set up and customize. missing for 10: dedicated evals/scoring product docs, configurable judge model selection, dataset-linked evaluation runs, independent validation of eval accuracy.
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
ml engineerCurate datasets from production traces and run offline evaluations against them
weight 3 · round to Arize PhoenixPhoenix docs describe grouping production traces into versioned datasets (docs-10, docs-25, gh-1), attaching evaluators (code-based and LLM-as-judge) to datasets and scoring experiment results automatically as an offline eval harness (docs-17, docs-18, docs-26, docs-34), and comparing experiment results across app versions (docs-41). This directly matches curating datasets from traces and running offline evals. missing for 10: independent/hands-on corroboration beyond vendor docs, and concrete example of the full curate-to-evaluate workflow output.
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [github] “Datasets - Create versioned datasets of examples for experimen”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments—forming an evaluation harness similar to a …”
- [claimed-docs] “Dataset Evaluators serve as test cases that automatically score outputs when running experiments”
- [claimed-docs] “You can group traces into datasets, rerun them through different versions of your application, and compare evaluation results to confirm whe…”
Helicone provides adjacent capabilities—HQL query language and point-query APIs to pull production request data (helicone-docs-15, helicone-docs-16), a Playground to test/iterate on prompts, sessions and traces (helicone-gh-3, helicone-gh-6), and webhook-driven 'real-time evaluation' scoring (helicone-docs-23)—but there is no explicit mention of a dataset-curation workflow or a dedicated offline/batch evaluation feature run against saved datasets. missing for 10: explicit dataset creation/curation from traces, offline evaluation harness or benchmark-running against curated sets, and independent confirmation these workflows work end-to-end.
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
Online evals
ml engineerRun evaluators continuously on live production traffic, not just offline datasets
weight 2 · round drawnDocs confirm Phoenix ingests live production traces (OTLP) and can score traces & spans with LLM, code-based, or human evaluators (docs-39, docs-25), suggesting evaluation can be applied beyond static datasets. However, the more detailed evaluation workflow described is dataset/experiment-centric (attach evaluators to datasets, score experiment results) rather than an explicit always-on/scheduled evaluator pipeline running automatically on incoming production traffic. Missing for 10: explicit documentation of a continuous/scheduled evaluation job or streaming eval pipeline that runs on live traces without manual dataset curation, and independent confirmation of this in production use.
- [claimed-docs] “You can score traces & spans with LLM-based evaluators, code-based checks, or human labels so you can track performance and identify failure…”
- [claimed-docs] “Datasets allow you to collect data from production, staging, evaluations, and even manually.”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Phoenix supports both deterministic code-based evaluators (exact match, regex, custom heuristics) and LLM-as-a-judge evaluators”
Helicone offers webhooks for real-time scoring/evaluation of production requests as they complete ('Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance') plus alerts on error rates/costs, which supports continuous evaluation on live traffic. However, this is implemented via webhook-triggered external scoring rather than a documented native 'evaluator' framework applied continuously to production traffic, and there's no evidence of built-in evaluator templates, dashboards specifically for eval results over time, or comparison against offline dataset evals. Missing for 10: dedicated evaluator/eval-suite feature applied to live traffic, evidence of eval results surfaced in-product over time, and independent/hands-on confirmation that this works as described.
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Only requests matching ALL specified properties will trigger webhooks.”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
- [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to HeliconePhoenix's Python/SDK APIs cover most workflows (tracing, datasets, experiments, evals, prompts) matching many UI features, and human annotations/UI-configured evaluators are documented, but there's no comprehensive API/CLI reference confirming full parity with UI-only actions like human annotation labeling or UI-based evaluator configuration, and no public OpenAPI spec was found (404s). missing for 10: a documented OpenAPI/REST spec confirming full UI-to-API parity, explicit API equivalents for UI-only actions like human annotation attachment, and independent verification that every UI action has an API counterpart.
- [claimed-docs] “Human annotations — Attach ground truth labels directly in the UI”
- [claimed-docs] “Configure evaluators in the Phoenix UI and attach them to your datasets. Phoenix scores experiment results automatically — no code required.”
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [probe] “PROBE openapi: all candidate paths 404 (https://arize.com/openapi.json, https://arize.com/swagger.json, https://arize.com/api/openapi.json, …”
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
Helicone exposes a REST API and OpenAPI spec (swagger.json) with query endpoints, and HQL for querying logs, plus API-driven prompt versioning/webhooks/caching that mirror UI features. However, some UI-centric features like the interactive Playground for testing prompts/sessions are explicitly UI-only, and there's no evidence of full API parity for all dashboard/analytics/alert configuration workflows. missing for 10: explicit documentation of API endpoints for alerts configuration, full analytics dashboard equivalents, and Playground-equivalent API/CLI tooling.
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
ai-native userExport all of my data in open formats and leave
weight 3 · round to Arize PhoenixPhoenix is open-source and self-hostable with data staying entirely in the user's infrastructure, and it ingests/exports traces via the open OpenTelemetry (OTLP) standard rather than a proprietary format, which strongly supports data portability and no lock-in. However, there is no explicit documentation of a dedicated 'export all my data' feature (e.g., bulk dataset/trace export to CSV/Parquet or a migration tool) beyond OTLP ingestion and self-hosting. Missing for 10: explicit bulk data export tooling/documentation, independent confirmation of successful full data migration/export by a user.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [github] “Datasets - Create versioned datasets of examples for experimen”
Helicone offers a REST API for querying/retrieving request data (JSON) and one-line export to PostHog, plus self-hosting options that let users keep their own data in open infra (Postgres/ClickHouse). However there's no explicit documented bulk 'export all your data' or account-portability feature, and the 'open source' framing itself was disputed by the community over licensing (Commons Clause), which weakens the openness narrative. Missing for 10: an explicit full-data export/download tool, documented data-portability guarantees, and independent confirmation that self-hosted/exported data is complete and usable outside Helicone.
- [claimed-docs] “Get Requests (Point Queries)”
- [github] “Export to PostHog in one-line for custom dashboards”
- [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
- [community] “Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typi…”
ai-native userRead the product's source under an open license
weight 2 · round to Arize PhoenixPhoenix's docs state it is 'built by Arize AI and the open-source community' and a public GitHub repo (Arize-ai/phoenix) is referenced, indicating the source is publicly readable. However, no evidence pack item specifies the actual license terms (e.g., MIT, Apache, ELv2) or confirms it is a permissive open-source license. Missing for 10: explicit license identification/text, independent confirmation of license terms, and any discussion of license restrictions.
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
- [github] “Datasets - Create versioned datasets of examples for experimen”
Heliconedisputedcontradicted5/10Helicone's source is publicly viewable on GitHub and marketed as open-source/self-hostable (helicone-docs-14, helicone-docs-20, helicone-docs-21), but a community commenter concretely disputes the 'open source' framing, noting the project uses the Commons Clause, which is not an OSI-recognized open license (helicone-comm-5). This is a direct contradiction of the openness claim rather than mere skepticism. Missing for 10: clear documentation of the actual license terms, confirmation of unrestricted OSI-approved licensing, and resolution of the Commons Clause concern.
- [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
- [claimed-docs] “Helicone gives you more provider flexibility, is open-source, and scales more cost-effectively.”
- [community] “Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typi…”
ai-native userSelf-host the core product
weight 3 · round to Arize PhoenixDocs explicitly state Phoenix is free to self-host with no feature limitations, data stays fully within infrastructure and can be air-gapped, with a simple CLI command (uvx arize-phoenix serve) and Docker image versioning for production deployments. Missing for 10: independent/hands-on confirmation of self-hosting experience beyond vendor docs.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “$ uvx arize-phoenix serve”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
Helicone documents multiple self-host deployment methods (e.g., Docker Compose) via a dedicated self-host overview page, and claims to be open-source with provider flexibility, supporting AI-native self-hosting. However, community evidence disputes the 'open source' label since it uses the Commons Clause license, and there's noted risk around shared ClickHouse infra suggesting the hosted version may differ architecturally from a fully self-hostable stack. Missing for 10: independent verification of a full self-hosted deployment working end-to-end, clarity on license restrictions for self-hosting, and confirmation that all core features (AI Gateway, sessions, caching) work identically when self-hosted.
- [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
- [claimed-docs] “Helicone gives you more provider flexibility, is open-source, and scales more cost-effectively.”
- [community] “Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typi…”
- [community] “Discussion of Helicone's architecture allowing users to write SQL directly to a shared ClickHouse instance, with commenters noting noisy-nei…”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round to Arize PhoenixPhoenix documents that it is free to self-host with data staying entirely within your own infrastructure and can be fully air-gapped, which lets users choose exactly where their data lives (docs-27/56). However, there is no explicit mention of a managed/cloud offering with selectable regions or residency guarantees, so the 'choose region' capability is only implicit via self-hosting. Missing for 10: explicit region-selection UI/settings, documented data residency options for any hosted/cloud tier, and independent confirmation of deployment flexibility.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
Helicone's self-hosting options (Docker Compose, multiple deployment methods) let a user control where their own infrastructure runs, which indirectly enables data residency, but there is no explicit region-selection feature or documented data-residency/compliance options for the hosted SaaS offering. missing for 10: explicit region/residency selection UI or docs, compliance certifications (SOC2/GDPR data-location guarantees), and any mention of choosing storage location within the hosted product.
- [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
ai-native userControl data retention and deletion
weight 2 · round to Arize PhoenixPhoenix's self-hosting model gives users full control over where data lives ('free to self-host... data stays entirely within your infrastructure... nothing is sent to Arize and can be fully air-gapped'), which implicitly enables retention/deletion control since the operator owns the storage. However, there is no explicit documentation of retention policies, TTL settings, or data-deletion APIs/commands within Phoenix itself. Missing for 10: explicit retention configuration options, documented deletion/purge commands or APIs, and any cloud-hosted retention controls for non-self-hosted deployments.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
- [claimed-docs] “Pin to a specific version (e.g., `arizephoenix/phoenix:version-8.0.0`) for production deployments.”
Heliconenone0/10The evidence pack has no mention of data retention settings, deletion controls, TTL policies, or privacy/compliance controls for stored logs and cached data; it only covers logging, caching, prompts, and observability features. Self-hosting is mentioned but not tied to retention/deletion controls specifically.
ai-native userOpt out of telemetry and usage tracking
weight 2 · round to Arize PhoenixPhoenix's self-hosting docs state that when self-hosted, 'nothing is sent to Arize and can be fully air-gapped,' implying no forced telemetry to the vendor, which directly supports a privacy-conscious opt-out posture. However, there is no explicit documentation of a dedicated telemetry/usage-tracking toggle (e.g., an env var or CLI flag to disable analytics) for the hosted or default install path. Missing for 10: explicit opt-out mechanism/config flag for anonymous usage analytics, confirmation of what telemetry (if any) is collected by default in non-air-gapped mode.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
Helicone's docs describe self-hosting options (Docker Compose, multiple deployment methods) which would let an AI-native user avoid sending usage data to Helicone's cloud entirely, but there is no explicit documented telemetry/analytics opt-out toggle within the product itself for cloud users. missing for 10: explicit opt-out setting for product usage telemetry, documentation of what data Helicone collects about its own users, and confirmation that self-hosting fully disables any phone-home telemetry.
- [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
Prompt management — stories about prompt management in this arenaPrompt management
Stories about prompt management in this arena
Prompt workflow
developerIterate on prompts in a playground against real models and variables
weight 2 · round to Arize PhoenixDocs explicitly describe a Prompt Playground where developers can 'play with prompts, models, invocation parameters' and track progress via tracing/experiments, plus use of real production examples/datasets for iteration. This directly matches the story of iterating on prompts against real models and variables. Missing for 10: independent/hands-on user reports specifically about the playground UX (only vendor docs, no third-party corroboration of the playground itself).
- [claimed-docs] “Play with prompts, models, invocation parameters and track your progress via tracing and experiments”
- [claimed-docs] “Prompt Playground - Play with prompts, models, invocation parameters and track your progress via tracing and experiments”
- [claimed-docs] “iterate on your prompts using real production examples”
- [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
- [claimed-docs] “Replay the invocation of an LLM. Whether it's an LLM step in an LLM workflow or a router query, you can step into the LLM invocation and see…”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
GitHub docs explicitly advertise a Playground for rapidly testing and iterating on prompts, sessions and traces in the UI, and prompt docs mention instant testing/deploying of prompt changes and version tracking. However, there's no explicit evidence of variable/templating support within the playground UI, no hands-on/independent confirmation of the playground experience, and no detail on live model comparison or variable injection workflow. Missing for 10: independent/hands-on validation of the playground, explicit documentation of variable substitution in playground testing, and detail on comparing multiple models side-by-side in that UI.
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
- [claimed-docs] “Use your prompt instantly by referencing its ID in your AI Gateway. No code changes, no rebuilds.”
- [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”
developerVersion prompts and deploy changes to production without shipping code
weight 3 · round drawnPhoenix has explicit Prompt Management features supporting versioning, storage, and deployment of prompts, plus 'Prompts in Code' SDKs to sync prompts across environments so changes can be pushed to production without code changes. Docs show a UI-based prompt playground and versioning workflow decoupled from application code deploys. Missing for 10: independent/hands-on verification beyond vendor docs of production rollback/rollout mechanics and no explicit mention of environment-specific staged rollout controls.
- [claimed-docs] “Prompt Management — Version, store, and deploy prompts”
- [claimed-docs] “Prompts in Code — Sync prompts across environments via SDK”
- [claimed-docs] “Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Prompts in Code - Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.”
- [claimed-docs] “Phoenix helps you iterate on prompts using real examples from your application. You can version prompts, test prompt variants across dataset…”
Helicone's Prompts feature explicitly supports versioning ('Track every change, compare versions, and rollback instantly') and production deployment without code changes ('Test and deploy prompt changes instantly without rebuilding or redeploying your application' by referencing a prompt ID in the AI Gateway). This directly matches the story of versioning and deploying prompt changes without shipping code. Missing for 10: independent/hands-on community corroboration specifically of the prompt-versioning workflow (community evidence covers observability/logging, not prompt deployment) and detail on rollback UX.
- [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
- [claimed-docs] “Use your prompt instantly by referencing its ID in your AI Gateway. No code changes, no rebuilds.”
- [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”
Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation
Instrumenting code and tracing requests end to end
Ai observability
ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app
weight 3 · round to Arize PhoenixPhoenix documents a first-party MCP server (/mcp endpoint) that lets MCP-compatible clients (Claude Code, Cursor, VS Code, etc.) search, query, and operate on projects, traces, datasets, experiments, prompts, and annotations, directly enabling an agent to query traces/metrics/eval results for debugging. This is backed by an official docs page and confirmed by probe evidence. Missing for 10: independent/hands-on third-party validation of the MCP server in practice and a documented general-purpose REST/GraphQL API spec (openapi probes 404) for non-MCP API access.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
- [claimed-docs] “Spans — Query and analyze traces with powerful filtering capabilities”
Helicone documents a REST API for querying requests (helicone-docs-15) and an HQL query language (helicone-docs-16), plus a discoverable OpenAPI spec (helicone-probe-3) and llms.txt for AI-agent-friendly docs (helicone-probe-1), enabling programmatic access to traces/sessions/metrics for debugging. However, there is no evidence of a dedicated MCP server offering, nor explicit documentation of querying eval results via API, so an agent's ability to pull all three (traces, metrics, evals) through a single API/MCP interface is only partially substantiated. missing for 10: explicit MCP server documentation, API endpoints/examples for querying eval results specifically, and independent confirmation of agent-based API usage for debugging.
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.helicone.ai/llms.txt # Helicone OSS LLM Observability - [Quickstart](https://docs.helicone.ai/gett…”
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
Data controls
developerMask or redact sensitive data before it is stored in traces
weight 2 · round drawnArize Phoenixnone0/10No evidence pack item mentions masking, redaction, PII scrubbing, or any data-sanitization capability for traces before storage; the docs cover tracing, evaluation, prompts, datasets, and self-hosting but never address sensitive-data redaction.
Heliconenone0/10No evidence in the pack mentions data masking, redaction, PII scrubbing, or any privacy-preserving controls applied before storing traces; the documentation excerpts cover gateway, sessions, caching, prompts, webhooks, and pricing but none address sensitive data redaction. missing for 10: any mention of masking/redaction feature, PII detection, or configuration options to omit/obfuscate sensitive fields before storage.
Sdk coverage
developerInstrument apps in both Python and JS/TS with officially supported SDKs
weight 2 · round drawnPhoenix documents OTLP ingestion and auto-instrumentation for frameworks including JS/TS ones like Vercel AI SDK and Mastra, plus a phoenix-cli npx setup and OpenInference-based instrumentation, indicating both Python and JS/TS support. However, the evidence is mostly high-level marketing docs without explicit JS/TS SDK API reference or code samples analogous to the detailed Python SDK reference (docs-28/29/30/36), and no independent/hands-on confirmation of JS/TS SDK parity. Missing for 10: explicit JS/TS SDK API reference/docs comparable to the Python SDK reference, hands-on/independent confirmation of JS/TS instrumentation working end-to-end, and clarity on feature parity between the two SDKs.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
- [claimed-docs] “Phoenix’s Python SDK is modular by design, allowing you to install only what you need.”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
Evidence shows one-line integration via the OpenAI SDK and support for multiple frameworks (OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK) which span both Python and JS/TS ecosystems, and community reports confirm the same style easy setup in real projects. However, the pack never explicitly names distinct 'Helicone Python SDK' and 'Helicone JS/TS SDK' packages or shows side-by-side code samples for each language. Missing for 10: explicit dedicated Python SDK docs/package reference, explicit dedicated JS/TS SDK docs/package reference, and independent hands-on confirmation for each language separately.
- [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”
- [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
- [community] “Your onboarding is impressive, one of the few products where 'get set up in one line of code' is true.”
Trace capture
developerTrace multi-step agent runs as nested spans grouped into sessions or threads
weight 2 · round drawnDocs explicitly describe OTLP-based tracing that captures nested spans for model calls, retrieval, and tool use (docs-38, docs-12/13/14), and Sessions explicitly group related traces into conversations/user sessions while Projects organize traces by application (docs-16/49). Auto-instrumentation for popular agent frameworks (LangChain, LlamaIndex, etc.) supports multi-step agent run tracing out of the box (docs-5, docs-32). Missing for 10: independent/hands-on verification of nested span hierarchy rendering for complex multi-step agents beyond vendor docs.
- [claimed-docs] “Sessions Group related traces into conversations and user sessions”
- [claimed-docs] “Group related traces into conversations and user sessions”
- [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
- [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
- [claimed-docs] “Retrieved Documents: Inspect the documents retrieved during a Retriever call, including the score and order in which they were returned”
- [claimed-docs] “Tool Descriptions: View the descriptions and function signatures of the tools your LLM has been given access to”
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Projects Organize traces into separate projects for different applications”
Helicone's Sessions feature explicitly groups related requests together to trace an entire agent flow in one unified view, with path-based grouping to identify recurring 'types' of work, and the Playground supports testing/iterating on prompts, sessions and traces. This directly matches nested/grouped multi-step agent tracing, though evidence lacks detail on explicit parent/child span nesting depth or independent hands-on verification of the sessions UI. missing for 10: independent/hands-on corroboration of nested span hierarchy depth, more detail on thread-level grouping beyond path-based sessions.
- [claimed-docs] “Sessions group these related requests together, letting you trace the entire agent flow from initial user input to final response in one uni…”
- [claimed-docs] “Requests with the same path represent the same "type" of work, even if they happen at different times.”
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK
weight 2 · round drawnDocs explicitly state Phoenix accepts OTLP traces and provides auto-instrumentation for popular frameworks including LangChain and Vercel AI SDK, plus OpenAI is referenced as an adapter/evaluator target and is a widely documented OpenInference integration. Missing for 10: no independent/hands-on verification of these integrations working, and no explicit dedicated docs snippet for OpenAI SDK auto-instrumentation beyond the adapter mention.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
- [claimed-docs] “Model Agnostic via adapters (for OpenAI, LiteLLM, LangChain, AI SDK, and more) — so you can easily switch judge models”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
GitHub docs explicitly advertise one-line integration for OpenAI, Anthropic, LangChain, Gemini, and Vercel AI SDK, and community testimonials corroborate easy OpenAI SDK/LangChain integration and quick onboarding. Missing for 10: dedicated docs pages per-framework (e.g., a LangChain-specific integration guide) and independent hands-on confirmation for Vercel AI SDK specifically.
- [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
- [community] “When your chains get long/complex enough in LangChain, it's really hard to understand from debug output what's final prompt that actually be…”
- [community] “Your onboarding is impressive, one of the few products where 'get set up in one line of code' is true.”
- [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”
developerCapture multimodal payloads (images, audio, files) inside my traces
weight 1 · round drawnArize Phoenixnone0/10The evidence pack describes tracing of model calls, retrieval, tool use, latency, and token usage, but nowhere mentions capturing images, audio, or file attachments/multimodal payloads within spans or traces. Since this is a plausible capability for a tracing/observability tool, absence of evidence yields 'none'.
- [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
- [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
- [claimed-docs] “Retrieved Documents: Inspect the documents retrieved during a Retriever call, including the score and order in which they were returned”
developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format
weight 3 · round to Arize PhoenixDocs explicitly state Phoenix accepts traces over OpenTelemetry (OTLP), provides auto-instrumentation for popular frameworks, and is built on top of OpenTelemetry powered by OpenInference instrumentation — confirming standard OTLP ingestion rather than a proprietary format. Missing for 10: independent/hands-on corroboration of OTLP export/receive behavior beyond vendor docs.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
- [claimed-docs] “Phoenix is built by Arize AI and the open-source community. It is built on top of OpenTelemetry and is powered by OpenInference instrumentat…”
developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage
weight 3 · round drawnPhoenix's docs extensively document capturing OTLP/OpenTelemetry-based traces of LLM calls, including inputs/outputs (span data), application latency, and detailed token usage breakdowns, plus auto-instrumentation for popular frameworks and manual tracing decorators. This is a core, heavily documented first-party capability with a CLI/SDK setup flow. Missing for 10: independent hands-on verification beyond vendor docs/community sentiment.
- [claimed-docs] “Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for popular frameworks (LlamaIndex, LangChain, DSPy, Mast…”
- [claimed-docs] “Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application”
- [claimed-docs] “Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.”
- [claimed-docs] “A trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent.”
- [claimed-docs] “Zero-config tracing — Enable auto_instrument=True to automatically trace AI libraries”
- [claimed-docs] “Tracing decorators — @tracer.chain, @tracer.tool, and more for manual instrumentation”
- [claimed-docs] “Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks”
Helicone's core product is LLM request logging with one-line integration capturing inputs/outputs, latency, tokens, and cost, corroborated by both docs and independent HN community reports of debugging and analytics use. Missing for 10: no explicit first-party doc snippet enumerating latency/token-usage fields captured per trace, relying on inference from 'automatic logging, observability' and community mentions of cost/debugging.
- [claimed-docs] “Get your first LLM request logged with Helicone in under 2 minutes using the AI Gateway.”
- [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”
- [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
- [community] “Happy Helicone customer here. It's a dead simple setup. It's great to have the extra charts and logging to debug issues and make sure all is…”
- [community] “When your chains get long/complex enough in LangChain, it's really hard to understand from debug output what's final prompt that actually be…”
- [community] “justin - helicone's great. I login in ~10 times a day. Great for debugging - I'm mostly using for analytics / observability.”
Not comparable on these axes
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · not comparableArize Phoenixn/aPhoenix is an observability/evaluation platform, not an agent that performs tasks using tools; its documented MCP integration runs in the opposite direction — Phoenix exposes an /mcp endpoint so external MCP clients (Claude Code, Cursor, VS Code) can query Phoenix's traces/datasets, not the reverse. There is no indication Phoenix itself consumes external MCP servers' tools to extend its own capabilities, which is a different product role (MCP client) than what Phoenix is built for.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
ai-native userSet up automations that run autonomously in the background
weight 2 · not comparableArize Phoenixnone0/10Phoenix's evidence covers tracing, evaluation, prompt management, and datasets, but nothing describes scheduled or autonomous background automations (e.g., recurring eval jobs, alerting rules, or triggers) that run without user initiation. The evaluation/experiment features described are invoked by the user (via SDK/UI), not autonomous background processes.
Heliconen/aHelicone is an LLM observability/gateway platform for logging, monitoring, and debugging LLM requests, not an automation/agent-runner product; it has no evidence of scheduled or autonomous background task execution, and this axis is a category error for its product type (webhooks/alerts are reactive, not autonomous automations).
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · not comparableArize Phoenixnone0/10Phoenix is an observability/evaluation platform; the evidence describes tracing, evals, prompt management, datasets, and an MCP server that lets *external* agents (Claude Code, Cursor, etc.) operate on Phoenix data — not a built-in AI assistant living inside Phoenix that users delegate tasks to. No docs mention any embedded assistant/chat agent within the Phoenix UI itself.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run `npx -y @arizeai/phoenix-cli setup` from your app's root directory.”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
ai-native userOperate the product with natural-language commands
weight 2 · not comparablePhoenix documents a dedicated remote MCP integration letting any MCP-compatible client (Claude Code, Cursor, VS Code, etc.) operate on projects, traces, datasets, experiments, prompts, and annotations via natural-language driven agents, and also supports agent-driven CLI setup. This directly satisfies AI-native natural-language operability. Missing for 10: independent/hands-on validation of the MCP command experience and more detail on the breadth/reliability of natural-language operations beyond the docs claim.
- [claimed-docs] “Point any MCP-compatible client (Claude Code, Cursor, VS Code, and others) at your Phoenix instance’s `/mcp` endpoint and it can search, que…”
- [probe] “official MCP server documented at https://arize.com/docs/phoenix/integrations/remote-mcp”
- [claimed-docs] “Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app's root directory.”
ai-native userSchedule recurring jobs or workflows
weight 2 · not comparableArize Phoenixnone0/10Phoenix is an observability/evaluation platform for LLM apps (tracing, evals, prompt management, datasets) but no evidence describes any scheduler, cron-like job runner, or recurring workflow automation feature.
ai-native userPrevent my data from being used to train AI models
weight 3 · not comparablePhoenix's self-hosting docs state that data stays entirely within the user's infrastructure and 'nothing is sent to Arize' and can be fully air-gapped, which strongly implies no data is available to Arize for model training when self-hosted. However, there is no explicit statement of a data-training policy (e.g., for any hosted/cloud offering) or a formal privacy/DPA commitment on this point. Missing for 10: explicit no-training-data policy statement, coverage of hosted/cloud deployment privacy terms, independent verification.
- [claimed-docs] “Phoenix is free to self-host with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize and c…”
- [claimed-docs] “Phoenix is **free to self-host** with no feature limitations. Your data stays entirely within your infrastructure—nothing is sent to Arize a…”
Heliconen/aHelicone is an LLM observability/gateway tool for developers logging their own API requests; it has no relationship to controlling whether end-user data is used to train third-party AI models. This story concerns AI-native consumer privacy posture, which is a wrong axis for this developer infrastructure product.