Braintrust vs Helicone

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Helicone
   Braintrust~partialprobed6/10

   A live probe confirms llms.txt exists at braintrust.dev/llms.txt returning 200 with a concise agent-oriented description, directly satisfying the story's core requirement. However, docs.md and OpenAPI/machine-readable doc endpoints return 404, indicating incomplete agent-oriented documentation coverage beyond the single llms.txt file. missing for 10: full docs-as-markdown or OpenAPI endpoints for deeper agent consumption, independent confirmation that agents successfully use llms.txt in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://www.braintrust.dev/llms.txt # Braintrust > Braintrust is the active observability platform for instrume…”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.braintrust.dev/docs.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Helicone✓fullprobed8/10

   Helicone has a confirmed llms.txt at docs.helicone.ai/llms.txt (HTTP 200) with an index of docs, plus per-page .md versions and an OpenAPI/swagger.json spec, all directly probeable and agent-consumable. missing for 10: no independent/community confirmation that agents actually consume this llms.txt successfully in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.helicone.ai/llms.txt # Helicone OSS LLM Observability - [Quickstart](https://docs.helicone.ai/gett…”
   • [probe] “PROBE docs-md: HTTP 200 at https://docs.helicone.ai/getting-started/quick-start.md > ## Documentation Index > Fetch the complete documentati…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust supports headless/CI usage via code-based Eval() runs, an API for programmatic control, a CLI (`bt`) for auth/eval/log operations from the terminal, and explicit documentation of CI/CD integration to run evals on every pull request. This combination clearly enables automated, non-interactive operation in pipelines. Missing for 10: independent hands-on confirmation of CI pipeline usage beyond vendor docs.

   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Helicone~partialprobed6/10

   Helicone is fundamentally an API/SDK-based proxy and observability layer — one-line SDK integration, REST API for queries, webhooks, and self-hosted Docker deployment all imply it can run without any UI, i.e., headlessly in scripts or pipelines. However, there is no explicit documentation or example of CI/CD usage, automated test pipelines, or headless agent orchestration specifically. Missing for 10: explicit CI/CD integration guide or example, documented non-interactive/automation-focused workflow, and independent confirmation of headless operation in production pipelines.

   • [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
   • [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
   • [claimed-docs] “Get Requests (Point Queries)”
   • [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”

   proof ↗⚑ flag

3. ai-native userConnect an agent via an official MCP server

   weight 3 · round to Braintrust
   Braintrust✓fullprobed8/10

   Braintrust documents an official MCP server that lets Claude Code, Cursor, Codex, VS Code, and other MCP-compatible clients query logs, author prompts/scorers, configure monitoring, and run evals — a clear server-side MCP offering that agents can connect to. Missing for 10: independent/hands-on third-party confirmation that the MCP server works as documented (only first-party docs and a probe confirm its existence).

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   Helicone—none0/10

   Helicone is an LLM observability/gateway platform, and this axis (offering an official MCP server so agents can connect) plausibly applies since it has an ecosystem of integrations, docs, and APIs. However, no evidence in the pack mentions an MCP server, MCP integration, or any agent-connection protocol offered by Helicone.

   ⚑ flag

4. ai-native userUse an official CLI

   weight 2 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust ships an official `bt` CLI, documented with a dedicated quickstart, allowing authentication, tracing coding-agent sessions, running evals, browsing/querying logs, syncing data, and managing functions from the terminal — directly matching AI-native/agentic CLI usage. Missing for 10: independent/hands-on community corroboration of the CLI itself (community evidence is generic, not CLI-specific).

   • [claimed-docs] “Install the `bt` CLI to set up, instrument, and run Braintrust with your coding agent”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Helicone—none0/10

   No evidence of an official Helicone CLI tool; integration is via SDKs, API keys, gateway, and REST/OpenAPI, but no CLI is mentioned anywhere in docs, GitHub, or community sources.

   ⚑ flag

5. ai-native userDrive the product through a documented public API

   weight 3 · round drawn
   Braintrust✓fullprobed8/10

   Braintrust documents a full public API reference for managing projects, experiments, datasets, prompts, scorers, and logging programmatically (braintrust-docs-11, -34), backed by SDKs (braintrust-gh-1/2), a CLI (braintrust-docs-33), and MCP server for agent-driven access (braintrust-docs-9, -25). Missing for 10: no discoverable OpenAPI/swagger spec (braintrust-probe-3 shows 404s) and no independent third-party corroboration of API robustness beyond docs.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Helicone✓fullprobed8/10

   Helicone publishes a documented REST API (swagger/OpenAPI spec, point-query endpoints, HQL) plus SDK/gateway APIs, all accessible via public docs and confirmed by a live OpenAPI probe. This gives AI-native users a documented programmatic surface beyond the UI. Missing for 10: deeper docs/examples of full CRUD API coverage and independent third-party corroboration of API usage beyond ingestion/logging.

   • [claimed-docs] “Get Requests (Point Queries)”
   • [claimed-docs] “HQL (Query Language)”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.helicone.ai/llms.txt # Helicone OSS LLM Observability - [Quickstart](https://docs.helicone.ai/gett…”
   • [github] “AI Gateway: Access 100+ AI models with 1 API key through the OpenAI API with intelligent routing and automatic fallbacks.”

   proof ↗⚑ flag

6. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   Braintrust—none0/10

   The evidence describes Braintrust's general API, CLI, and MCP integrations but contains no mention of scoped, role-based, or least-privilege API key/credential issuance for agents; the only security-related item is a breach report telling customers to rotate keys, which does not demonstrate a scoping/least-privilege capability.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [community] “AI evaluation startup Braintrust confirms breach, tells every customer to rotate sensitive keys.”

   proof ↗⚑ flag

   Helicone—none0/10

   No evidence Helicone supports issuing scoped or least-privilege API credentials/keys for agents; docs mention bringing your own provider keys or using Helicone's own key, but nothing about granular permission scoping. A community comment even flags concern about sending a full API key on every request, with no rebuttal describing scoped-key mitigation.

   • [claimed-docs] “Want more control? You can bring your own provider keys instead.”
   • [community] “Hmm, so to integrate I have to basically send my api key to you on every request? Not great”

   proof ↗⚑ flag

7. ai-native userBuild against official SDKs

   weight 2 · round drawn
   Braintrust✓fullprobed8/10

   Braintrust ships official JS SDK (github examples using Eval()), an API reference for programmatic access, a CLI (bt), and an MCP server, giving AI-native developers multiple official, documented ways to build against the platform. One community comment calls docs 'sparse' but this is general skepticism, not a concrete contradiction of SDK functionality. missing for 10: independent hands-on validation of the SDK itself (beyond vendor docs/examples) and a public OpenAPI spec (probe found 404s).

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”
   • [community] “Braintrust: Interesting approach to evaluation but feels like an early-stage product. Documentation is sparse and integration options are li…”

   proof ↗⚑ flag

   Helicone✓fullcommunity8/10

   Helicone works by wrapping official/familiar SDKs (OpenAI SDK, Anthropic, LangChain, Gemini, Vercel AI SDK) with one-line integration, backed by docs, GitHub README, and independent hands-on community confirmation of easy SDK-based integration. missing for 10: no dedicated first-party Helicone SDK docs page enumerated (language coverage) and no independent benchmark of SDK completeness beyond community anecdotes.

   • [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”
   • [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
   • [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
   • [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
   • [community] “Happy Helicone customer here. It's a dead simple setup. It's great to have the extra charts and logging to debug issues and make sure all is…”
   • [community] “Your onboarding is impressive, one of the few products where 'get set up in one line of code' is true.”

   proof ↗⚑ flag

8. ai-native userSubscribe to events via webhooks

   weight 2 · round to Helicone
   Braintrust—none0/10

   No evidence in the pack mentions webhooks or any event-subscription mechanism; Braintrust's documented interfaces are API, CLI, MCP server, and UI, none of which are shown to support webhook subscriptions.

   ⚑ flag

   Helicone✓fullclaimed7/10

   Helicone documents a dedicated webhooks feature that sends instant notifications on LLM request completion, supports property-based filtering, and enables automation/scoring/integration with external systems, directly matching the story's subscribe-to-events use case. missing for 10: independent hands-on confirmation of webhook reliability/usage and richer docs on payload schema or delivery guarantees.

   • [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
   • [claimed-docs] “Only requests matching ALL specified properties will trigger webhooks.”
   • [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust ships Loop, an AI agent that investigates logs via natural language and can build scorers/datasets/dashboards, plus Patterns which proactively scans trace backlogs for recurring issues and suggests fixes, and Topics which auto-clusters/classifies traces against business dimensions — all directly generating AI insights/suggestions inside the product. Missing for 10: independent/hands-on validation of Loop/Patterns quality (evidence is vendor docs only) and no third-party review confirming suggestion usefulness.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Define the dimensions that matter to your business, like use case, customer segment, compliance, or tone. Topics continuously clusters every…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review. Each trace is analyzed by facets that extract sho”

   proof ↗⚑ flag

   Helicone~partialclaimed4/10

   Helicone advertises 'real-time evaluation' that automatically scores and evaluates LLM responses for quality, safety, and relevance, which is a form of AI-generated insight on the user's own request data, but the evidence pack shows no broader AI-generated dashboard summaries, anomaly explanations, or proactive suggestions beyond this single scoring feature. missing for 10: evidence of AI-generated narrative insights/recommendations across usage/cost data, natural-language querying of analytics, or proactive suggestion features beyond automated response scoring.

   • [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round to Helicone
   Braintrust—none0/10

   Braintrust has an API reference page (braintrust-docs-11/34) but the evidence pack shows no interactive or runnable-example capability — probes for an OpenAPI/swagger spec (needed for typical interactive API explorers) all returned 404, and there's no mention of a try-it-now console or embedded runnable snippets in the API reference itself. missing for 10: evidence of an interactive API console, runnable code snippets embedded in the reference, or a machine-readable OpenAPI spec powering such interactivity.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.braintrust.dev/docs.md”

   proof ↗⚑ flag

   Helicone~partialprobed5/10

   Helicone publishes a formal OpenAPI/swagger spec and REST API reference pages, and separately offers a 'Playground' for testing/iterating on prompts in its UI, suggesting some interactive tooling exists. However, there is no direct evidence of a Swagger-UI-style interactive reference page where a user can execute runnable request examples against the API from the docs themselves. Missing for 10: confirmation of an actual 'try it out' runnable console in the API reference docs, and evidence of code-sample generation across languages.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
   • [claimed-docs] “Get Requests (Point Queries)”
   • [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
   • [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round to Helicone
   Braintrust—none0/10

   Braintrust documents a REST API (api-reference) but explicit probes for OpenAPI/swagger specs at all standard paths returned 404, and no docs mention a downloadable machine-readable spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Helicone✓fullprobed9/10

   Helicone exposes a machine-readable OpenAPI spec at /swagger.json (HTTP 200, valid 'openapi' key), and its REST endpoints are documented (e.g., request query endpoints). missing for 10: no independent/community confirmation of the OpenAPI spec's completeness or usage.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
   • [claimed-docs] “Get Requests (Point Queries)”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Braintrust
   Braintrust~partialclaimed4/10

   Braintrust supports evals/experiments/datasets and separate projects that could function as a sandbox for testing without hitting production logs, and self-hosting offers infrastructure isolation, but there is no explicit documentation of a dedicated 'sandbox environment' or isolation guarantee preventing test runs from touching production data. missing for 10: explicit sandbox/staging environment concept, documentation on isolating test evals from production datasets/logs, and independent confirmation that eval runs never write to production stores.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”

   proof ↗⚑ flag

   Helicone~partialclaimed3/10

   Helicone offers a Playground for testing/iterating on prompts (not necessarily isolated from production logging) and self-host Docker Compose options for local development, which could function as an ad-hoc sandbox, but there is no explicit documented 'sandbox environment' or test-mode that guarantees isolation from production data. Missing for 10: a dedicated sandbox/staging mode, explicit statement about not touching production data, and independent verification of data isolation.

   • [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
   • [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
   • [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Braintrust—none0/10

   There is an API reference documenting functionality, but no evidence of API versioning scheme or any documented deprecation policy; probes even show no OpenAPI spec discoverable at standard locations. Missing for 10: versioning scheme documentation, explicit deprecation policy, changelog/migration guides.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Helicone—none0/10

   An OpenAPI spec exists (helicone-probe-3) confirming a REST API, but there is no evidence of API versioning scheme (e.g., v1/v2 paths with migration guides) or a documented deprecation policy for endpoints/models; nothing in the docs pack addresses lifecycle or backward-compatibility commitments.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
   • [claimed-docs] “Get Requests (Point Queries)”

   proof ↗⚑ flag

Monitoring

1. ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics

   weight 2 · round to Helicone
   Braintrust~partialclaimed5/10

   Braintrust captures latency, token/cost, and eval-score metrics via tracing and experiments (braintrust-docs-1, braintrust-docs-27), and one doc line explicitly says you can 'ask Braintrust's AI agent to investigate your data and build scorers, datasets, and dashboards' (braintrust-docs-18), implying custom dashboard creation. However there is no dedicated dashboard-building documentation, no detail on customizing charts/widgets, and no explicit mention of error-rate metrics or alerting tied to dashboards. Missing for 10: dedicated dashboard-configuration docs, explicit error-metric support, and independent/hands-on evidence of building a custom dashboard.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”

   proof ↗⚑ flag

   Helicone~partialcommunity6/10

   Helicone provides building blocks for custom analytics — HQL query language for ad-hoc queries, REST API point queries, one-line export to PostHog for custom dashboards, real-time eval scoring via webhooks, and alerts on error rates/cost — but evidence does not show a native in-product dashboard builder combining latency, error, cost, and eval-score metrics together; users must rely on external tools (PostHog) or raw queries (HQL) to assemble true custom dashboards. Missing for 10: evidence of a first-party customizable dashboard UI, direct proof eval scores can be plotted alongside latency/cost/error in one dashboard, and independent hands-on confirmation of dashboard customization.

   • [claimed-docs] “HQL (Query Language)”
   • [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
   • [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
   • [github] “Export to PostHog in one-line for custom dashboards”
   • [claimed-docs] “Get Requests (Point Queries)”
   • [community] “Discussion of Helicone's architecture allowing users to write SQL directly to a shared ClickHouse instance, with commenters noting noisy-nei…”

   proof ↗⚑ flag

2. developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email

   weight 2 · round to Helicone
   Braintrust—none0/10

   The evidence pack covers tracing, evals, online scoring, Loop/Patterns/Topics analysis, and self-hosting, but contains no mention of configurable alerts, thresholds, or notification integrations (Slack, PagerDuty, email) for error rates, cost spikes, or eval-score drops.

   ⚑ flag

   Helicone~partialclaimed4/10

   Helicone documents a dedicated Alerts feature for monitoring error rates and cost spikes (helicone-docs-11), and webhooks for real-time notifications tied to request events (helicone-docs-9, helicone-docs-23), but the evidence pack never mentions eval-score-drop alerts or specific delivery channels like Slack, PagerDuty, or email. Missing for 10: eval-score-based alert triggers, and documented integrations/notification channels (Slack, PagerDuty, email).

   • [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
   • [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
   • [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed7/10

   Braintrust evals run against entire datasets in one operation, Patterns scans the full trace backlog on a schedule, Topics classifies every trace automatically, and the CLI/API support bulk sync ('bt sync pull'), bulk log queries, and programmatic management of experiments/datasets/prompts at scale. Missing for 10: explicit documentation of bulk edit/delete/update operations on datasets or logs, and independent (non-vendor) confirmation of bulk-operation performance at scale.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Helicone~partialclaimed4/10

   Helicone offers query/export mechanisms (HQL query language, REST 'point queries' API, one-line export to PostHog) that could be used to operate on many logged requests at once, but there is no explicit documentation of bulk actions like batch tagging, batch deletion, or batch updates across many items. missing for 10: dedicated bulk-edit/bulk-delete/batch-processing API, documentation of batch size limits, and independent confirmation of bulk operation reliability.

   • [claimed-docs] “HQL (Query Language)”
   • [claimed-docs] “Get Requests (Point Queries)”
   • [github] “Export to PostHog in one-line for custom dashboards”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round drawn
   Braintrust~partialclaimed5/10

   Braintrust supports several automatic, event-driven behaviors — online scoring runs automatically on every new production trace, evals run automatically in CI/CD on pull requests, and Patterns runs Loop against traces on a schedule — but these are fixed built-in automations rather than a general user-defined 'if event then action' rules engine (e.g., no evidence of custom triggers/webhooks/alerts tied to arbitrary conditions). Missing for 10: a documented rules/alerting framework letting users define arbitrary trigger conditions and custom actions, and independent confirmation of this automation depth.

   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”

   proof ↗⚑ flag

   Helicone~partialclaimed5/10

   Helicone supports webhooks that fire on request-completion events and can be filtered so only requests matching ALL specified properties trigger them, plus alerts that monitor error rates/costs — both are event-triggered automations. However, this is a narrow, fixed set of trigger types (webhooks, alerts) rather than a general user-defined rules engine with configurable conditions and arbitrary actions. Missing for 10: evidence of a flexible rule-builder UI/API for custom conditions and actions beyond webhooks/alerts, and independent confirmation these automations work reliably in production.

   • [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
   • [claimed-docs] “Only requests matching ALL specified properties will trigger webhooks.”
   • [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
   • [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”

   proof ↗⚑ flag

3. ai-native userVersion, review, and roll back my automations

   weight 1 · round to Helicone
   Braintrust~partialclaimed5/10

   Braintrust documents versioned datasets and immutable, comparable experiment records, plus CLI/API management of prompts, scorers, and functions, which gives some version/review capability for its automation artifacts (evals, scorers, prompts). However, there is no explicit documentation of a rollback mechanism for automations (e.g., reverting a prompt, scorer, or online-scoring rule to a prior version) or a review/approval workflow before rollback. Missing for 10: explicit rollback functionality, version history UI/CLI commands, and independent confirmation that rollback works in practice.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Helicone~partialclaimed6/10

   Helicone's prompts feature explicitly supports versioning and rollback ('Track every change, compare versions, and rollback instantly if something goes wrong') and instant deployment of prompt changes without redeploying, which covers version/rollback for prompt-based automations. However, this is scoped to prompts specifically, not broader automation workflows (e.g., webhooks, alerts pipelines), and there's no 'review' workflow (approvals, diffs, comments) documented, nor independent/community corroboration of the rollback feature working in practice. Missing for 10: review/approval workflow evidence, broader automation (not just prompts) versioning, independent hands-on confirmation of rollback reliability.

   • [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”
   • [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
   • [claimed-docs] “Use your prompt instantly by referencing its ID in your AI Gateway. No code changes, no rebuilds.”

   proof ↗⚑ flag

Cost tracking

1. developerAttribute cost and usage to users, sessions, and features via custom metadata

   weight 2 · round drawn
   Braintrust~partialclaimed4/10

   Braintrust's tracing captures token usage and custom metadata for every request, which is the raw ingredient for cost attribution, but no evidence describes dashboards, filters, or reports that actually roll up cost/usage by user, session, or feature tags. missing for 10: explicit cost-attribution UI/reporting by custom metadata dimensions, user/session/feature breakdown examples, and independent confirmation this works in practice.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”

   proof ↗⚑ flag

   Helicone~partialclaimed4/10

   Helicone's docs show a Sessions feature that groups related requests to trace agent flows (helicone-docs-5, helicone-docs-6) and per-user cache namespaces (helicone-docs-13), which support some usage segmentation, but there is no explicit evidence of a general custom-metadata/custom-properties system for tagging requests by user, feature, or session to attribute cost and usage in dashboards or queries. Missing for 10: explicit custom-properties/tagging API, cost breakdown by user/feature dashboards, and documentation showing metadata-based cost attribution reports.

   • [claimed-docs] “Sessions group these related requests together, letting you trace the entire agent flow from initial user input to final response in one uni…”
   • [claimed-docs] “Requests with the same path represent the same "type" of work, even if they happen at different times.”
   • [claimed-docs] “Create separate cache namespaces for different users or contexts.”

   proof ↗⚑ flag

2. developerSee cost and token usage per request, model, and time period in dashboards

   weight 3 · round drawn
   Braintrust~partialclaimed5/10

   Braintrust's tracing explicitly captures token usage per request (braintrust-docs-1) and traces/spans can be browsed in the UI or terminal (braintrust-docs-14/19/35), which supports basic usage visibility, but the evidence never mentions a cost metric, cost calculation, or a dedicated dashboard aggregating cost/tokens by model or time period. Missing for 10: explicit cost tracking/computation, dashboard views aggregating usage by model, dashboard views aggregating usage by time period, and any independent confirmation of these dashboard features.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`]”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”

   proof ↗⚑ flag

   Helicone~partialcommunity5/10

   Community feedback confirms cost/logging visibility ('extra charts and logging to debug issues', 'how much it costs', 'analytics/observability') and docs reference request query APIs and an HQL query language for analysis, but no first-party documentation explicitly describes dashboards broken down by request, model, and time period. Missing for 10: explicit docs/screenshots of dashboard UI showing cost/token breakdowns by model and time period, and independent confirmation of that specific breakdown granularity.

   • [community] “Happy Helicone customer here. It's a dead simple setup. It's great to have the extra charts and logging to debug issues and make sure all is…”
   • [community] “When your chains get long/complex enough in LangChain, it's really hard to understand from debug output what's final prompt that actually be…”
   • [community] “justin - helicone's great. I login in ~10 times a day. Great for debugging - I'm mostly using for analytics / observability.”
   • [claimed-docs] “Get Requests (Point Queries)”
   • [claimed-docs] “HQL (Query Language)”

   proof ↗⚑ flag

Data export

1. developerBulk-export traces and datasets to blob storage or my data warehouse

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust supports exporting logs/traces via CLI (`bt sync pull`, download as CSV/JSON) and a full API for programmatic access to traces, datasets, and experiments, which developers could pipe into blob storage or a warehouse themselves. However, there is no documented native connector or bulk-export feature specifically targeting blob storage (S3/GCS) or data warehouses (Snowflake/BigQuery) — missing for 10: dedicated warehouse/blob-storage export integrations, scheduled/automated bulk export pipelines, and independent confirmation of large-scale export performance.

   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”

   proof ↗⚑ flag

   Helicone~partialcommunity4/10

   Helicone offers a REST API for point queries, an HQL query language for direct SQL access to logged data, and a one-line PostHog export integration, which give some programmatic data-access/export capability. However, no evidence documents a dedicated bulk-export feature to blob storage (S3/GCS) or to a data warehouse (Snowflake/BigQuery) specifically for traces/datasets. Missing for 10: explicit blob-storage export, data-warehouse connector/sync, bulk/batch export tooling or docs beyond point queries and third-party PostHog export.

   • [claimed-docs] “Get Requests (Point Queries)”
   • [claimed-docs] “HQL (Query Language)”
   • [github] “Export to PostHog in one-line for custom dashboards”
   • [community] “Discussion of Helicone's architecture allowing users to write SQL directly to a shared ClickHouse instance, with commenters noting noisy-nei…”

   proof ↗⚑ flag

Ai eval ops

1. ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust provides programmatic APIs and MCP/CLI tooling that let an agent create datasets, trigger evals, and read back results: the API supports creating/managing datasets and experiments, the SDK's Eval() function runs evals programmatically, and the MCP server/CLI explicitly let agents like Claude Code/Cursor 'run evals' and 'query logs' from within their workflow. missing for 10: a concrete end-to-end example showing an agent specifically creating a dataset via API/MCP and then reading back eval results in one flow, and independent (non-vendor) confirmation of this specific workflow.

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Helicone—none0/10

   Evidence shows observability features (sessions, webhooks, HQL query, REST API for point queries) and a vague mention of 'real-time evaluation' scoring, but there is no documentation of a dataset-creation API, a way to programmatically trigger an eval run, or an API to read back eval results — the core building blocks of this story are absent from the evidence pack.

   ⚑ flag

Human review

1. ml engineerRoute outputs to human annotation queues for review and labeling

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust supports building custom annotation interfaces for human review and labeling (docs-12) and datasets can be built/curated from production logs and manual curation (docs-5, docs-31), implying human review workflows, but there's no explicit evidence of a dedicated 'routing' mechanism, queue assignment, reviewer task distribution, or workflow status tracking for human annotators. missing for 10: explicit queue/routing mechanism to assign outputs to specific human reviewers, task assignment/status tracking, and independent corroboration of the annotation workflow in practice.

   • [claimed-docs] “Build annotation interfaces that match your team's workflow. Review support conversations differently than co”
   • [claimed-docs] “Build datasets from production logs, user feedback, manual curation, or generate them with Loop.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”

   proof ↗⚑ flag

   Helicone—none0/10

   Helicone's evidence covers observability, logging, prompts, caching, webhooks, and real-time automated scoring, but there is no mention of routing outputs to human annotation queues or workflows for manual review/labeling by human annotators.

   ⚑ flag

Offline evals

1. developerRun evals in CI and gate deployments on their results

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Docs explicitly describe running evals via code/CLI, integrating into CI/CD to catch regressions, and gating on experiment results ('Automate in CI/CD — Run evals on every pull request to catch regressions'; 'integrate into CI/CD to catch regressions before they reach production'), backed by SDK eval code examples. Missing for 10: independent/hands-on confirmation of actual CI gating in production and details on failure-threshold/gate configuration.

   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”

   proof ↗⚑ flag

   Helicone—none0/10

   Helicone offers observability, webhooks, real-time scoring, caching, and prompt versioning, but there is no evidence of a CI-integrated eval runner, test suite, or deployment gating mechanism tied to eval results. Nothing in the evidence pack mentions CI pipelines, exit codes, or blocking deploys based on evaluation scores.

   ⚑ flag

2. ml engineerWrite custom code-based scorers and metrics for my evaluations

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust's SDK examples show Eval() calls with custom `scores` arrays (e.g. LevenshteinScorer), demonstrating code-based custom scorers, and docs describe managing/authoring scorers via API, CLI, and playgrounds. Missing for 10: independent hands-on validation of writing a fully custom scorer function beyond the built-in Levenshtein example, and more detail on scorer authoring API/signature.

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”

   proof ↗⚑ flag

   Helicone~partialclaimed3/10

   Helicone's webhooks feature mentions enabling users to 'score responses' and lists 'Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance', implying some custom scoring hook exists, but there is no documented SDK, API, or framework for writing custom code-based scorers/metrics as part of an evals workflow. missing for 10: dedicated evals/scorer API or SDK docs, examples of custom metric code, integration with a datasets/evals product surface.

   • [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
   • [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”

   proof ↗⚑ flag

3. ml engineerCompare eval runs side by side to catch regressions between prompt or model versions

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed9/10

   Braintrust's Experiments feature is explicitly described as an immutable, comparable record of eval runs that tracks progress over time and integrates into CI/CD to catch regressions, and Playgrounds let users run full evaluations and compare results side-by-side across prompt/model configurations. This directly matches the story of comparing eval runs to catch regressions between versions. Missing for 10: independent/hands-on corroboration of the side-by-side comparison UI beyond first-party docs.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”

   proof ↗⚑ flag

   Helicone~partialclaimed3/10

   Helicone documents prompt version tracking/comparison and rollback (docs-18) plus real-time automated scoring via webhooks (docs-23) and a playground for iterating on prompts (gh-3/gh-6), which are adjacent building blocks for regression detection, but there is no explicit evidence of a dedicated 'eval run' comparison view or side-by-side regression report across prompt/model versions. Missing for 10: a documented eval-run comparison UI, dataset-based batch evaluation runs, and any hands-on/community confirmation of side-by-side regression catching.

   • [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”
   • [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
   • [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
   • [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
   • [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”

   proof ↗⚑ flag

4. ml engineerScore outputs with configurable LLM-as-a-judge evaluators

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust's docs describe configurable scorers used within evals and playgrounds (LLM-as-a-judge style scorers), online scoring of production traces, and code examples show custom scorer configuration (e.g., LevenshteinScorer swapped in evals). Docs explicitly mention 'author scorers' via CLI/MCP and building scorers with Loop, supporting configurability. Missing for 10: explicit documentation naming 'LLM-as-a-judge' scorer templates/library and independent hands-on validation of judge configurability beyond vendor docs.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Helicone~partialclaimed4/10

   Helicone's docs mention webhooks enabling 'real-time evaluation' to automatically score responses for quality/safety/relevance, which points toward LLM-as-judge scoring, but there's no evidence of a configurable evaluator framework, built-in judge models, or dataset-based eval workflows that an ML engineer could set up and customize. missing for 10: dedicated evals/scoring product docs, configurable judge model selection, dataset-linked evaluation runs, independent validation of eval accuracy.

   • [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
   • [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”

   proof ↗⚑ flag

5. ml engineerCurate datasets from production traces and run offline evaluations against them

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed9/10

   Docs explicitly describe building versioned datasets from production logs/traces (docs-5, docs-31) and running offline evaluations against them via Experiments, code, or Playgrounds (docs-27, docs-23, gh-1/gh-2 SDK examples), with CI/CD integration for regression testing (docs-2). Missing for 10: independent/hands-on third-party validation of the end-to-end curate-from-production-to-offline-eval workflow beyond first-party docs.

   • [claimed-docs] “Build datasets from production logs, user feedback, manual curation, or generate them with Loop.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Helicone~partialclaimed4/10

   Helicone provides adjacent capabilities—HQL query language and point-query APIs to pull production request data (helicone-docs-15, helicone-docs-16), a Playground to test/iterate on prompts, sessions and traces (helicone-gh-3, helicone-gh-6), and webhook-driven 'real-time evaluation' scoring (helicone-docs-23)—but there is no explicit mention of a dataset-curation workflow or a dedicated offline/batch evaluation feature run against saved datasets. missing for 10: explicit dataset creation/curation from traces, offline evaluation harness or benchmark-running against curated sets, and independent confirmation these workflows work end-to-end.

   • [claimed-docs] “Get Requests (Point Queries)”
   • [claimed-docs] “HQL (Query Language)”
   • [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
   • [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
   • [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”

   proof ↗⚑ flag

Online evals

1. ml engineerRun evaluators continuously on live production traffic, not just offline datasets

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust explicitly supports 'online scoring' that evaluates production traces automatically as they're logged asynchronously with no latency impact, plus continuous background analysis via Patterns/Topics scanning trace backlogs on a schedule for issues no scorer already catches. This directly matches continuous evaluation on live traffic beyond offline datasets. Missing for 10: independent/hands-on verification of online scoring behavior in production and more detail on configuring which evaluators run continuously.

   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Define the dimensions that matter to your business, like use case, customer segment, compliance, or tone. Topics continuously clusters every…”

   proof ↗⚑ flag

   Helicone~partialclaimed5/10

   Helicone offers webhooks for real-time scoring/evaluation of production requests as they complete ('Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance') plus alerts on error rates/costs, which supports continuous evaluation on live traffic. However, this is implemented via webhook-triggered external scoring rather than a documented native 'evaluator' framework applied continuously to production traffic, and there's no evidence of built-in evaluator templates, dashboards specifically for eval results over time, or comparison against offline dataset evals. Missing for 10: dedicated evaluator/eval-suite feature applied to live traffic, evidence of eval results surfaced in-product over time, and independent/hands-on confirmation that this works as described.

   • [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
   • [claimed-docs] “Only requests matching ALL specified properties will trigger webhooks.”
   • [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
   • [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Braintrust
   Braintrust~partialprobed7/10

   Braintrust offers a comprehensive API/SDK ('interact with all aspects of the Braintrust platform programmatically'), a full-featured CLI (bt) for auth, tracing, evals, log queries, and function management, and an MCP server letting agents query logs, author scorers, configure monitoring, and run evals — covering most UI functionality outside the terminal. However, some UI-only features like Playgrounds (explicitly 'no-code workspace') and annotation-interface building appear UI-centric without clear API/CLI parity, and no public OpenAPI spec was found to confirm full API coverage. missing for 10: confirmed OpenAPI/spec proving 1:1 API-UI parity, explicit API/CLI equivalent for Playgrounds and custom annotation UI workflows, independent verification of completeness.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Helicone~partialprobed6/10

   Helicone exposes a REST API and OpenAPI spec (swagger.json) with query endpoints, and HQL for querying logs, plus API-driven prompt versioning/webhooks/caching that mirror UI features. However, some UI-centric features like the interactive Playground for testing prompts/sessions are explicitly UI-only, and there's no evidence of full API parity for all dashboard/analytics/alert configuration workflows. missing for 10: explicit documentation of API endpoints for alerts configuration, full analytics dashboard equivalents, and Playground-equivalent API/CLI tooling.

   • [claimed-docs] “Get Requests (Point Queries)”
   • [claimed-docs] “HQL (Query Language)”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
   • [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
   • [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
   • [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   Braintrust~partialclaimed5/10

   Braintrust provides data export via `bt sync pull`, `bt view logs` downloads as CSV/JSON, and a full API to programmatically extract projects, experiments, datasets, traces, and prompts, plus a self-hosting option for data control. However, there's no explicit documentation of full-account export/portability guarantees, no mention of open/standard interchange formats beyond CSV/JSON logs, and no independent confirmation of a clean 'leave' workflow. Missing for 10: comprehensive account-level export documentation, open-format guarantees for all entity types (prompts, scorers, configs), and third-party validation of export completeness.

   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Helicone~partialcommunity5/10

   Helicone offers a REST API for querying/retrieving request data (JSON) and one-line export to PostHog, plus self-hosting options that let users keep their own data in open infra (Postgres/ClickHouse). However there's no explicit documented bulk 'export all your data' or account-portability feature, and the 'open source' framing itself was disputed by the community over licensing (Commons Clause), which weakens the openness narrative. Missing for 10: an explicit full-data export/download tool, documented data-portability guarantees, and independent confirmation that self-hosted/exported data is complete and usable outside Helicone.

   • [claimed-docs] “Get Requests (Point Queries)”
   • [github] “Export to PostHog in one-line for custom dashboards”
   • [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
   • [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
   • [community] “Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typi…”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round to Helicone
   Braintrust—none0/10

   Evidence shows only client SDK repos on GitHub (braintrust-sdk-javascript) and self-hosted deployment of infrastructure, but the core platform/UI is explicitly described as Braintrust-managed and not released under an open license; no evidence of the product's full source being open-licensed.

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Helicone!disputedcontradicted5/10

   Helicone's source is publicly viewable on GitHub and marketed as open-source/self-hostable (helicone-docs-14, helicone-docs-20, helicone-docs-21), but a community commenter concretely disputes the 'open source' framing, noting the project uses the Commons Clause, which is not an OSI-recognized open license (helicone-comm-5). This is a direct contradiction of the openness claim rather than mere skepticism. Missing for 10: clear documentation of the actual license terms, confirmation of unrestricted OSI-approved licensing, and resolution of the Commons Clause concern.

   • [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
   • [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
   • [claimed-docs] “Helicone gives you more provider flexibility, is open-source, and scales more cost-effectively.”
   • [community] “Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typi…”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   Braintrust~partialclaimed6/10

   Braintrust documents an official self-hosted deployment option, but it explicitly separates data storage (self-hosted) from platform management (Braintrust-managed UI, auth, updates), meaning it's not a fully self-hostable core product. Missing for 10: evidence of fully self-hosting the entire platform (UI, control plane, auth) independent of Braintrust's managed services, and independent/community confirmation of self-hosting experience.

   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”

   proof ↗⚑ flag

   Helicone~partialcommunity6/10

   Helicone documents multiple self-host deployment methods (e.g., Docker Compose) via a dedicated self-host overview page, and claims to be open-source with provider flexibility, supporting AI-native self-hosting. However, community evidence disputes the 'open source' label since it uses the Commons Clause license, and there's noted risk around shared ClickHouse infra suggesting the hosted version may differ architecturally from a fully self-hostable stack. Missing for 10: independent verification of a full self-hosted deployment working end-to-end, clarity on license restrictions for self-hosting, and confirmation that all core features (AI Gateway, sessions, caching) work identically when self-hosted.

   • [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
   • [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
   • [claimed-docs] “Helicone gives you more provider flexibility, is open-source, and scales more cost-effectively.”
   • [community] “Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typi…”
   • [community] “Discussion of Helicone's architecture allowing users to write SQL directly to a shared ClickHouse instance, with commenters noting noisy-nei…”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust's self-hosting docs explicitly state it's designed to meet data residency requirements by keeping customer data within your own cloud account and region, giving users region/residency control — but only via full self-hosted deployment, not a simple region-selection toggle in the managed cloud offering. Missing for 10: evidence of region choice within the standard hosted/SaaS plan, list of specific supported regions, and compliance certifications tied to residency.

   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Helicone~partialclaimed4/10

   Helicone's self-hosting options (Docker Compose, multiple deployment methods) let a user control where their own infrastructure runs, which indirectly enables data residency, but there is no explicit region-selection feature or documented data-residency/compliance options for the hosted SaaS offering. missing for 10: explicit region/residency selection UI or docs, compliance certifications (SOC2/GDPR data-location guarantees), and any mention of choosing storage location within the hosted product.

   • [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
   • [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”

   proof ↗⚑ flag

2. ai-native userControl data retention and deletion

   weight 2 · round to Braintrust
   Braintrust~partialclaimed4/10

   Braintrust's self-hosted deployment option lets customers keep all data within their own cloud account/region for compliance and residency, giving some control over where data lives, and the API/CLI allow programmatic management of logs and datasets. However, there is no explicit documentation of a retention policy, TTL settings, or a delete/purge API for logs, traces, or datasets on the managed (non-self-hosted) offering. Missing for 10: explicit retention window configuration, an in-platform delete/purge mechanism for traces or datasets, and confirmation that these controls exist outside of self-hosting.

   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”

   proof ↗⚑ flag

   Helicone—none0/10

   The evidence pack has no mention of data retention settings, deletion controls, TTL policies, or privacy/compliance controls for stored logs and cached data; it only covers logging, caching, prompts, and observability features. Self-hosting is mentioned but not tied to retention/deletion controls specifically.

   ⚑ flag

3. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round to Helicone
   Braintrust—none0/10

   No evidence of any telemetry opt-out or usage-tracking control setting for Braintrust itself as a tool; the evidence covers Braintrust's own tracing/observability product features (for user's AI apps), not the vendor's own telemetry collection practices or an opt-out mechanism. Self-hosting controls where customer AI data resides but says nothing about product telemetry/usage tracking opt-out.

   ⚑ flag

   Helicone~partialclaimed3/10

   Helicone's docs describe self-hosting options (Docker Compose, multiple deployment methods) which would let an AI-native user avoid sending usage data to Helicone's cloud entirely, but there is no explicit documented telemetry/analytics opt-out toggle within the product itself for cloud users. missing for 10: explicit opt-out setting for product usage telemetry, documentation of what data Helicone collects about its own users, and confirmation that self-hosting fully disables any phone-home telemetry.

   • [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
   • [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”

   proof ↗⚑ flag

Prompt workflow

1. developerIterate on prompts in a playground against real models and variables

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Docs explicitly describe Playgrounds as a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets, running full evaluations in real-time against real models with variable inputs, and comparing results side-by-side (braintrust-docs-3, braintrust-docs-23, braintrust-docs-17). This directly matches the story of iterating on prompts in a playground against real models and variables. Missing for 10: independent/hands-on user corroboration of the playground experience specifically (community evidence is generic/mixed) and no detail on variable-substitution mechanics beyond the general description.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Measure and improve AI application quality with playgrounds and evals”

   proof ↗⚑ flag

   Helicone~partialclaimed6/10

   GitHub docs explicitly advertise a Playground for rapidly testing and iterating on prompts, sessions and traces in the UI, and prompt docs mention instant testing/deploying of prompt changes and version tracking. However, there's no explicit evidence of variable/templating support within the playground UI, no hands-on/independent confirmation of the playground experience, and no detail on live model comparison or variable injection workflow. Missing for 10: independent/hands-on validation of the playground, explicit documentation of variable substitution in playground testing, and detail on comparing multiple models side-by-side in that UI.

   • [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
   • [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
   • [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
   • [claimed-docs] “Use your prompt instantly by referencing its ID in your AI Gateway. No code changes, no rebuilds.”
   • [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”

   proof ↗⚑ flag

2. developerVersion prompts and deploy changes to production without shipping code

   weight 3 · round to Helicone
   Braintrust~partialclaimed6/10

   Braintrust supports managing prompts via playgrounds, API, and CLI ("Manage prompts, tools, and scorers"), and datasets/experiments are versioned, suggesting prompt versioning is possible; however, there's no explicit evidence of a distinct 'deploy to production without code' workflow (e.g., prompt promotion/publish/rollback mechanism) separate from logging or evals. missing for 10: explicit prompt versioning/history UI, a documented deploy/publish/rollback mechanism for prompts, and independent confirmation that production apps can swap prompts without redeploying code.

   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”

   proof ↗⚑ flag

   Helicone✓fullclaimed8/10

   Helicone's Prompts feature explicitly supports versioning ('Track every change, compare versions, and rollback instantly') and production deployment without code changes ('Test and deploy prompt changes instantly without rebuilding or redeploying your application' by referencing a prompt ID in the AI Gateway). This directly matches the story of versioning and deploying prompt changes without shipping code. Missing for 10: independent/hands-on community corroboration specifically of the prompt-versioning workflow (community evidence covers observability/logging, not prompt deployment) and detail on rollback UX.

   • [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
   • [claimed-docs] “Use your prompt instantly by referencing its ID in your AI Gateway. No code changes, no rebuilds.”
   • [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”

   proof ↗⚑ flag

Ai observability

1. ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app

   weight 3 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust documents a first-party MCP server enabling MCP-compatible agents (Claude Code, Cursor, Codex, etc.) to query logs, run evals, author scorers, and configure monitoring, plus a full programmatic API for traces/experiments/datasets, and Loop, Braintrust's own AI agent, for natural-language investigation of logs and eval data. This directly matches the story of an agent querying traces, metrics, and eval results via API or MCP to debug an app. Missing for 10: independent hands-on corroboration of the MCP server working end-to-end (only vendor docs/probe confirm existence, no third-party review of actual debugging workflow).

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   Helicone~partialprobed5/10

   Helicone documents a REST API for querying requests (helicone-docs-15) and an HQL query language (helicone-docs-16), plus a discoverable OpenAPI spec (helicone-probe-3) and llms.txt for AI-agent-friendly docs (helicone-probe-1), enabling programmatic access to traces/sessions/metrics for debugging. However, there is no evidence of a dedicated MCP server offering, nor explicit documentation of querying eval results via API, so an agent's ability to pull all three (traces, metrics, evals) through a single API/MCP interface is only partially substantiated. missing for 10: explicit MCP server documentation, API endpoints/examples for querying eval results specifically, and independent confirmation of agent-based API usage for debugging.

   • [claimed-docs] “Get Requests (Point Queries)”
   • [claimed-docs] “HQL (Query Language)”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.helicone.ai/llms.txt # Helicone OSS LLM Observability - [Quickstart](https://docs.helicone.ai/gett…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
   • [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”

   proof ↗⚑ flag

Data controls

1. developerMask or redact sensitive data before it is stored in traces

   weight 2 · round drawn
   Braintrust—none0/10

   No evidence in the pack describes any PII masking, redaction, or scrubbing feature for traces before storage; documentation covers instrumentation, tracing, evals, and observability but never mentions data masking or redaction capabilities. This is a plausible and common axis for a tracing/observability platform, so absence of evidence yields 'none' rather than 'na'.

   ⚑ flag

   Helicone—none0/10

   No evidence in the pack mentions data masking, redaction, PII scrubbing, or any privacy-preserving controls applied before storing traces; the documentation excerpts cover gateway, sessions, caching, prompts, webhooks, and pricing but none address sensitive data redaction. missing for 10: any mention of masking/redaction feature, PII detection, or configuration options to omit/obfuscate sensitive fields before storage.

   ⚑ flag

Sdk coverage

1. developerInstrument apps in both Python and JS/TS with officially supported SDKs

   weight 2 · round drawn
   Braintrust~partialclaimed6/10

   Docs confirm instrumentation/tracing is central to Braintrust and JS SDK usage is shown in GitHub examples (Eval function calls), with docs referencing SDK-based tracing and instrumentation broadly, but the evidence pack never explicitly confirms an officially supported Python SDK alongside JS/TS — only JS/TS SDK code is shown. Missing for 10: explicit Python SDK documentation/examples, side-by-side confirmation of parity between Python and JS/TS SDKs, and independent corroboration of both SDKs in production use.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Helicone~partialcommunity6/10

   Evidence shows one-line integration via the OpenAI SDK and support for multiple frameworks (OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK) which span both Python and JS/TS ecosystems, and community reports confirm the same style easy setup in real projects. However, the pack never explicitly names distinct 'Helicone Python SDK' and 'Helicone JS/TS SDK' packages or shows side-by-side code samples for each language. Missing for 10: explicit dedicated Python SDK docs/package reference, explicit dedicated JS/TS SDK docs/package reference, and independent hands-on confirmation for each language separately.

   • [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”
   • [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
   • [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
   • [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
   • [community] “Your onboarding is impressive, one of the few products where 'get set up in one line of code' is true.”

   proof ↗⚑ flag

Trace capture

1. developerTrace multi-step agent runs as nested spans grouped into sessions or threads

   weight 2 · round to Helicone
   Braintrust~partialclaimed6/10

   Braintrust documents tracing that captures spans, logs, and full request detail, and lets users browse traces and individual spans via UI or CLI, supporting general span-based tracing. However, the evidence never explicitly describes nested multi-step spans grouped into sessions/threads for agent runs (no mention of parent-child span hierarchies or session/thread grouping semantics). Missing for 10: explicit documentation of nested span hierarchies for multi-step agent runs, explicit session/thread grouping constructs, and independent/hands-on confirmation of this specific structure.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”

   proof ↗⚑ flag

   Helicone✓fullclaimed8/10

   Helicone's Sessions feature explicitly groups related requests together to trace an entire agent flow in one unified view, with path-based grouping to identify recurring 'types' of work, and the Playground supports testing/iterating on prompts, sessions and traces. This directly matches nested/grouped multi-step agent tracing, though evidence lacks detail on explicit parent/child span nesting depth or independent hands-on verification of the sessions UI. missing for 10: independent/hands-on corroboration of nested span hierarchy depth, more detail on thread-level grouping beyond path-based sessions.

   • [claimed-docs] “Sessions group these related requests together, letting you trace the entire agent flow from initial user input to final response in one uni…”
   • [claimed-docs] “Requests with the same path represent the same "type" of work, even if they happen at different times.”
   • [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
   • [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”

   proof ↗⚑ flag

2. developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK

   weight 2 · round to Helicone
   Braintrust~partialcommunity4/10

   Docs confirm a general instrumentation capability and state you can 'integrate with AI providers and frameworks to send traces to Braintrust,' but no evidence explicitly names LangChain, the OpenAI SDK, or the Vercel AI SDK as supported integrations. A community comment even calls integration options 'limited.' Missing for 10: explicit documentation or hands-on confirmation of dedicated LangChain, OpenAI SDK, and Vercel AI SDK integrations.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [community] “Braintrust: Interesting approach to evaluation but feels like an early-stage product. Documentation is sparse and integration options are li…”

   proof ↗⚑ flag

   Helicone✓fullcommunity8/10

   GitHub docs explicitly advertise one-line integration for OpenAI, Anthropic, LangChain, Gemini, and Vercel AI SDK, and community testimonials corroborate easy OpenAI SDK/LangChain integration and quick onboarding. Missing for 10: dedicated docs pages per-framework (e.g., a LangChain-specific integration guide) and independent hands-on confirmation for Vercel AI SDK specifically.

   • [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
   • [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
   • [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
   • [community] “When your chains get long/complex enough in LangChain, it's really hard to understand from debug output what's final prompt that actually be…”
   • [community] “Your onboarding is impressive, one of the few products where 'get set up in one line of code' is true.”
   • [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”

   proof ↗⚑ flag

3. developerCapture multimodal payloads (images, audio, files) inside my traces

   weight 1 · round drawn
   Braintrust—none0/10

   The evidence pack describes tracing capturing inputs, outputs, model parameters, latency, token usage, and metadata, but there is no explicit mention of multimodal payload support (images, audio, files) within traces. Missing for 10: explicit documentation or examples showing image/audio/file capture in traces, SDK support for multimodal attachments, or any UI rendering of multimodal content in trace spans.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”

   proof ↗⚑ flag

   Helicone—none0/10

   The evidence pack covers logging, sessions, prompts, webhooks, caching, and deployment options, but contains no mention of capturing images, audio, or file payloads within traces—no documentation, GitHub feature, or community report addresses multimodal payload capture.

   ⚑ flag

4. developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format

   weight 3 · round drawn
   Braintrust—none0/10

   No evidence pack item mentions OpenTelemetry or OTLP support; Braintrust's docs describe its own SDK/instrumentation and integrations but never reference OTLP-based trace ingestion/export. Missing for 10: any mention of OTLP endpoint, OpenTelemetry SDK compatibility, or OTel collector integration.

   ⚑ flag

   Helicone—none0/10

   No evidence pack items mention OpenTelemetry, OTLP, or any standard tracing protocol support; Helicone's docs describe proprietary logging via SDK integration, sessions, and REST API, not OTLP ingestion/export.

   ⚑ flag

5. developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage

   weight 3 · round to Helicone
   Braintrust✓fullclaimed8/10

   Braintrust docs explicitly state instrumentation captures inputs, outputs, model parameters, latency, token usage, and metadata for every LLM call, with SDK examples and CLI/UI browsing of traces and spans. missing for 10: independent hands-on verification of exact captured fields and no third-party benchmark of tracing accuracy/performance overhead.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”

   proof ↗⚑ flag

   Helicone✓fullcommunity9/10

   Helicone's core product is LLM request logging with one-line integration capturing inputs/outputs, latency, tokens, and cost, corroborated by both docs and independent HN community reports of debugging and analytics use. Missing for 10: no explicit first-party doc snippet enumerating latency/token-usage fields captured per trace, relying on inference from 'automatic logging, observability' and community mentions of cost/debugging.

   • [claimed-docs] “Get your first LLM request logged with Helicone in under 2 minutes using the AI Gateway.”
   • [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”
   • [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
   • [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
   • [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
   • [community] “Happy Helicone customer here. It's a dead simple setup. It's great to have the extra charts and logging to debug issues and make sure all is…”
   • [community] “When your chains get long/complex enough in LangChain, it's really hard to understand from debug output what's final prompt that actually be…”
   • [community] “justin - helicone's great. I login in ~10 times a day. Great for debugging - I'm mostly using for analytics / observability.”

   proof ↗⚑ flag