Braintrust vs Galileo

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Galileo
   Braintrust~partialprobed6/10

   A live probe confirms llms.txt exists at braintrust.dev/llms.txt returning 200 with a concise agent-oriented description, directly satisfying the story's core requirement. However, docs.md and OpenAPI/machine-readable doc endpoints return 404, indicating incomplete agent-oriented documentation coverage beyond the single llms.txt file. missing for 10: full docs-as-markdown or OpenAPI endpoints for deeper agent consumption, independent confirmation that agents successfully use llms.txt in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://www.braintrust.dev/llms.txt # Braintrust > Braintrust is the active observability platform for instrume…”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.braintrust.dev/docs.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Galileo✓fullprobed8/10

   Direct probes confirm llms.txt returns HTTP 200 with a documentation index, and individual doc pages provide .md versions with pointers back to llms.txt, making the docs agent-consumable as claimed. missing for 10: no independent/third-party confirmation of an agent actually consuming these files successfully, and no evidence of broader machine-readable spec coverage (e.g., OpenAPI probe returned 404s).

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.galileo.ai/llms.txt # Galileo - [What Is Galileo?](https://docs.galileo.ai/what-is-galileo.md) - […”
   • [probe] “PROBE docs-md: HTTP 200 at https://docs.galileo.ai/what-is-galileo.md > ## Documentation Index > Fetch the complete documentation index at: …”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust supports headless/CI usage via code-based Eval() runs, an API for programmatic control, a CLI (`bt`) for auth/eval/log operations from the terminal, and explicit documentation of CI/CD integration to run evals on every pull request. This combination clearly enables automated, non-interactive operation in pipelines. Missing for 10: independent hands-on confirmation of CI pipeline usage beyond vendor docs.

   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Galileo~partialclaimed6/10

   Galileo ships a Python SDK (with `@log` decorators, OpenTelemetry distributed tracing, and experiment/dataset APIs) that can be invoked programmatically without the console UI, implying headless/CI usage is possible. However, the evidence never explicitly documents a CI/CD pipeline example, a CLI, or headless-run guidance—experiments are largely framed around the console UI's 'Create Experiment' button. Missing for 10: explicit CI/CD integration docs or examples, a documented CLI/headless entrypoint, and confirmation that experiments can be fully triggered/scored outside the UI.

   • [github] “You can also use the `@log` decorator to log spans.”
   • [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
   • [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”

   proof ↗⚑ flag

3. ai-native userConnect an agent via an official MCP server

   weight 3 · round drawn
   Braintrust✓fullprobed8/10

   Braintrust documents an official MCP server that lets Claude Code, Cursor, Codex, VS Code, and other MCP-compatible clients query logs, author prompts/scorers, configure monitoring, and run evals — a clear server-side MCP offering that agents can connect to. Missing for 10: independent/hands-on third-party confirmation that the MCP server works as documented (only first-party docs and a probe confirm its existence).

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   Galileo✓fullprobed8/10

   Galileo, as an observability/evaluation platform (not itself an agent), documents an official MCP server that lets users access dataset management, experiments, and prompt templates directly from their dev environment, confirmed live via docs page. Missing for 10: independent/hands-on verification beyond first-party docs and details on broader client compatibility.

   • [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
   • [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”

   proof ↗⚑ flag

4. ai-native userUse an official CLI

   weight 2 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust ships an official `bt` CLI, documented with a dedicated quickstart, allowing authentication, tracing coding-agent sessions, running evals, browsing/querying logs, syncing data, and managing functions from the terminal — directly matching AI-native/agentic CLI usage. Missing for 10: independent/hands-on community corroboration of the CLI itself (community evidence is generic, not CLI-specific).

   • [claimed-docs] “Install the `bt` CLI to set up, instrument, and run Braintrust with your coding agent”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Galileo—none0/10

   Evidence shows Python SDK, MCP server integration, and REST-like console features, but no mention of an official CLI tool for Galileo. missing for 10: any documentation of a dedicated CLI binary/command, install instructions, or command reference.

   ⚑ flag

5. ai-native userDrive the product through a documented public API

   weight 3 · round to Braintrust
   Braintrust✓fullprobed8/10

   Braintrust documents a full public API reference for managing projects, experiments, datasets, prompts, scorers, and logging programmatically (braintrust-docs-11, -34), backed by SDKs (braintrust-gh-1/2), a CLI (braintrust-docs-33), and MCP server for agent-driven access (braintrust-docs-9, -25). Missing for 10: no discoverable OpenAPI/swagger spec (braintrust-probe-3 shows 404s) and no independent third-party corroboration of API robustness beyond docs.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Galileo~partialprobed6/10

   Galileo provides a Python SDK (galileo-python) with decorators for logging traces/spans, an MCP server for programmatic access to datasets/experiments, and OpenTelemetry-based distributed tracing support, indicating a documented API surface for AI-native/agentic use. However, no formal public REST/OpenAPI reference was found (openapi probe returned 404s across all candidate paths), so the API's full documented surface and versioning/auth details are unclear. missing for 10: a discoverable OpenAPI/REST API spec, formal API reference docs beyond SDK/MCP usage, and independent confirmation of API completeness.

   • [github] “You can also use the `@log` decorator to log spans.”
   • [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
   • [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
   • [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”

   proof ↗⚑ flag

6. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   Braintrust—none0/10

   The evidence describes Braintrust's general API, CLI, and MCP integrations but contains no mention of scoped, role-based, or least-privilege API key/credential issuance for agents; the only security-related item is a breach report telling customers to rotate keys, which does not demonstrate a scoping/least-privilege capability.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [community] “AI evaluation startup Braintrust confirms breach, tells every customer to rotate sensitive keys.”

   proof ↗⚑ flag

   Galileo—none0/10

   Galileo is an AI observability/evaluation platform; evidence covers tracing, metrics, experiments, and MCP integration, but there is no mention of scoped or least-privilege API credential/key management for agents.

   ⚑ flag

7. ai-native userBuild against official SDKs

   weight 2 · round to Braintrust
   Braintrust✓fullprobed8/10

   Braintrust ships official JS SDK (github examples using Eval()), an API reference for programmatic access, a CLI (bt), and an MCP server, giving AI-native developers multiple official, documented ways to build against the platform. One community comment calls docs 'sparse' but this is general skepticism, not a concrete contradiction of SDK functionality. missing for 10: independent hands-on validation of the SDK itself (beyond vendor docs/examples) and a public OpenAPI spec (probe found 404s).

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”
   • [community] “Braintrust: Interesting approach to evaluation but feels like an early-stage product. Documentation is sparse and integration options are li…”

   proof ↗⚑ flag

   Galileo~partialprobed6/10

   Galileo has an official Python SDK (galileo-python) with decorators/logging APIs referenced in GitHub docs, plus MCP server integration for accessing Galileo capabilities from dev environments, supporting AI-native/agentic workflows. However, evidence lacks details on multi-language SDK coverage, versioning/stability, or independent developer corroboration beyond first-party docs, and OpenAPI spec probes all 404'd. Missing for 10: multi-language SDK evidence, independent hands-on validation, public API reference/OpenAPI spec.

   • [github] “You can also use the `@log` decorator to log spans.”
   • [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
   • [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”

   proof ↗⚑ flag

8. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Braintrust—none0/10

   No evidence in the pack mentions webhooks or any event-subscription mechanism; Braintrust's documented interfaces are API, CLI, MCP server, and UI, none of which are shown to support webhook subscriptions.

   ⚑ flag

   Galileo—none0/10

   There is a mention of alerting (galileo-docs-9) but no evidence of webhook subscription support; the OpenAPI/API endpoints probe returned 404s and nothing in the evidence pack describes webhooks or event subscription mechanisms.

   • [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust ships Loop, an AI agent that investigates logs via natural language and can build scorers/datasets/dashboards, plus Patterns which proactively scans trace backlogs for recurring issues and suggests fixes, and Topics which auto-clusters/classifies traces against business dimensions — all directly generating AI insights/suggestions inside the product. Missing for 10: independent/hands-on validation of Loop/Patterns quality (evidence is vendor docs only) and no third-party review confirming suggestion usefulness.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Define the dimensions that matter to your business, like use case, customer segment, compliance, or tone. Topics continuously clusters every…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review. Each trace is analyzed by facets that extract sho”

   proof ↗⚑ flag

   Galileo~partialclaimed5/10

   Galileo offers LLM-as-a-judge and custom metrics that can evaluate outputs, natural-language feedback loops that auto-improve metric alignment, and alerts on anomalies, which imply some AI-assisted analysis of data — but there is no explicit documentation of a dedicated 'insights/suggestions' feature that proactively surfaces AI-generated recommendations to users. Missing for 10: a clear insights/suggestions UI or feature description, independent examples of such AI-generated recommendations in use, and confirmation this goes beyond metric scoring to actionable suggestions.

   • [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
   • [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
   • [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”
   • [claimed-docs] “Agentic metrics help you measure how well your AI agents perform complex, multi-step tasks—especially when those agents need to use tools, m…”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust supports autonomous background automation via 'Patterns', which runs Loop (its AI agent) on a schedule against the trace backlog to surface recurring issues and suggested fixes, plus online scoring that runs asynchronously on production traces and CI/CD-triggered evals. However, this is narrower than general-purpose autonomous automations — it's scoped to trace analysis/scoring rather than arbitrary user-defined background tasks, and there's no independent/hands-on corroboration of Patterns actually running unattended. Missing for 10: broader/customizable scheduled automation beyond Patterns and online scoring, independent evidence of reliability, and detail on failure handling or configurability of these background jobs.

   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”

   proof ↗⚑ flag

   Galileo~partialclaimed3/10

   Galileo documents background alerting ('get alerted whenever unexpected things happen') and continuous automatic capture of traces/spans, which are autonomous background processes, but there is no evidence of a general-purpose automation/scheduling system for agentic workflows that a user configures to run independently. missing for 10: explicit automation/workflow scheduler, triggers/conditions configuration, evidence of autonomous multi-step agent execution beyond monitoring/alerts.

   • [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”

   proof ↗⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed7/10

   Braintrust ships "Loop," a built-in AI agent that can investigate logs via natural language, build scorers/datasets/dashboards, and "Patterns" which runs Loop automatically on trace backlogs to surface issues and fixes — a clear built-in AI assistant delegated to perform tasks within the product. missing for 10: independent/hands-on user validation of Loop's task delegation (community evidence is thin/mixed and doesn't specifically address Loop), and more detail on the scope/limits of tasks it can autonomously complete.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Loop is Braintrust's AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”

   proof ↗⚑ flag

   Galileo—none0/10

   Galileo's evidence covers evaluating and monitoring external AI agents (agentic metrics, tracing, MCP access to its own capabilities from a dev environment) but nothing about a built-in assistant inside Galileo's own product that a user can delegate tasks to.

   ⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round to Braintrust
   Braintrust✓fullprobed8/10

   Braintrust's Loop agent lets users investigate logs and build scorers/datasets/dashboards via natural language instead of writing queries, and this is exposed through an official MCP server so natural-language commands work from Claude Code, Cursor, Codex, etc. Missing for 10: independent hands-on validation of natural-language command accuracy/reliability and broader coverage beyond Loop's stated scope.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Loop is Braintrust's AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   Galileo~partialprobed5/10

   Galileo ships an official MCP server that lets AI assistants create/manage datasets, run experiments, and set up prompt templates from a dev environment using natural language, and it supports continuous natural-language feedback to refine metrics — both are concrete NL-driven operation paths. However, there's no evidence of a native chat/NL command interface within the Galileo console itself; missing for 10: first-party in-app NL command console, broader coverage of all product actions via NL, and independent hands-on confirmation of the MCP NL workflow.

   • [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
   • [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”
   • [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Braintrust—none0/10

   Braintrust has an API reference page (braintrust-docs-11/34) but the evidence pack shows no interactive or runnable-example capability — probes for an OpenAPI/swagger spec (needed for typical interactive API explorers) all returned 404, and there's no mention of a try-it-now console or embedded runnable snippets in the API reference itself. missing for 10: evidence of an interactive API console, runnable code snippets embedded in the reference, or a machine-readable OpenAPI spec powering such interactivity.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.braintrust.dev/docs.md”

   proof ↗⚑ flag

   Galileo—none0/10

   No evidence of an interactive API reference with runnable examples; openapi probes all returned 404, and no Swagger/Redoc-style playground is mentioned anywhere in the docs pack.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   Braintrust—none0/10

   Braintrust documents a REST API (api-reference) but explicit probes for OpenAPI/swagger specs at all standard paths returned 404, and no docs mention a downloadable machine-readable spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Galileo—none0/10

   A direct probe for OpenAPI/Swagger spec files at all standard locations returned 404, and no documentation references a downloadable machine-readable API spec; only an llms.txt index and MCP server exist, neither of which is an OpenAPI spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round drawn
   Braintrust~partialclaimed4/10

   Braintrust supports evals/experiments/datasets and separate projects that could function as a sandbox for testing without hitting production logs, and self-hosting offers infrastructure isolation, but there is no explicit documentation of a dedicated 'sandbox environment' or isolation guarantee preventing test runs from touching production data. missing for 10: explicit sandbox/staging environment concept, documentation on isolating test evals from production datasets/logs, and independent confirmation that eval runs never write to production stores.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”

   proof ↗⚑ flag

   Galileo~partialclaimed4/10

   Galileo's Experiments feature lets users evaluate prompts/models against well-defined inputs and datasets can be built from 'synthetic, development, and live production data,' implying some separation between test and production data, but there is no explicit sandbox/staging environment concept described. missing for 10: explicit sandbox/staging environment docs, isolation guarantees from production data, and independent confirmation of non-production testing workflow.

   • [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
   • [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Braintrust—none0/10

   There is an API reference documenting functionality, but no evidence of API versioning scheme or any documented deprecation policy; probes even show no OpenAPI spec discoverable at standard locations. Missing for 10: versioning scheme documentation, explicit deprecation policy, changelog/migration guides.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Galileo—none0/10

   No evidence of API versioning scheme or a documented deprecation policy; OpenAPI spec probes returned 404 and no changelog/versioning docs are present in the evidence pack.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”

   proof ↗⚑ flag

Monitoring

1. ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust captures latency, token/cost, and eval-score metrics via tracing and experiments (braintrust-docs-1, braintrust-docs-27), and one doc line explicitly says you can 'ask Braintrust's AI agent to investigate your data and build scorers, datasets, and dashboards' (braintrust-docs-18), implying custom dashboard creation. However there is no dedicated dashboard-building documentation, no detail on customizing charts/widgets, and no explicit mention of error-rate metrics or alerting tied to dashboards. Missing for 10: dedicated dashboard-configuration docs, explicit error-metric support, and independent/hands-on evidence of building a custom dashboard.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”

   proof ↗⚑ flag

   Galileo—none0/10

   Evidence covers logging/tracing (latency, spans), custom metrics, LLM-as-judge evals, and alerting, but no documentation describes a dashboard-building UI or customizable visualization layer combining latency, error, cost, and eval-score metrics. missing for 10: dashboard/widget customization UI, evidence of combining metrics types into a single view, cost-metric tracking, independent/hands-on confirmation of dashboarding.

   ⚑ flag

2. developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email

   weight 2 · round to Galileo
   Braintrust—none0/10

   The evidence pack covers tracing, evals, online scoring, Loop/Patterns/Topics analysis, and self-hosting, but contains no mention of configurable alerts, thresholds, or notification integrations (Slack, PagerDuty, email) for error rates, cost spikes, or eval-score drops.

   ⚑ flag

   Galileo~partialclaimed5/10

   Galileo docs confirm a general alerting capability ("get alerted whenever unexpected things happen" via a 'set up alerts on logs' guide), but the evidence pack contains no detail on which triggers (error rate, cost spike, eval-score drop) are supported nor which notification channels (Slack, PagerDuty, email) are integrated. Missing for 10: documented list of supported alert conditions, confirmation of Slack/PagerDuty/email integrations, and any hands-on/independent verification of alert delivery.

   • [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed7/10

   Braintrust evals run against entire datasets in one operation, Patterns scans the full trace backlog on a schedule, Topics classifies every trace automatically, and the CLI/API support bulk sync ('bt sync pull'), bulk log queries, and programmatic management of experiments/datasets/prompts at scale. Missing for 10: explicit documentation of bulk edit/delete/update operations on datasets or logs, and independent (non-vendor) confirmation of bulk-operation performance at scale.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Galileo~partialclaimed5/10

   Galileo's Experiments feature runs evaluations across datasets of many inputs at once, and MCP/SDK access lets users programmatically create and manage datasets and run experiments in bulk from code rather than one item at a time (galileo-docs-2, galileo-docs-3, galileo-docs-4, galileo-docs-11). However, there is no explicit documentation of bulk edit/delete/tag/annotate operations across arbitrary large sets of existing items in the console or API. Missing for 10: explicit bulk CRUD operations (batch edit/delete/tag) across items, batch API endpoints/rate-limit guidance for large-scale automation, and independent confirmation of bulk-scale reliability.

   • [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
   • [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
   • [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
   • [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust supports several automatic, event-driven behaviors — online scoring runs automatically on every new production trace, evals run automatically in CI/CD on pull requests, and Patterns runs Loop against traces on a schedule — but these are fixed built-in automations rather than a general user-defined 'if event then action' rules engine (e.g., no evidence of custom triggers/webhooks/alerts tied to arbitrary conditions). Missing for 10: a documented rules/alerting framework letting users define arbitrary trigger conditions and custom actions, and independent confirmation of this automation depth.

   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”

   proof ↗⚑ flag

   Galileo~partialclaimed3/10

   Galileo docs mention that users can set up alerts on logs to be notified of unexpected events, which is a basic rule-trigger-on-event capability, but there is no detail on defining custom rule logic, condition types, or automated actions beyond alerting (e.g., webhooks, workflow triggers, remediation actions). missing for 10: rule definition UI/API details, supported trigger conditions, and evidence of automated actions beyond simple alert notifications.

   • [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust documents 'Patterns,' which runs Loop on the trace backlog 'on a schedule' to find recurring issues, and separately supports triggering evals via CI/CD — both are forms of recurring automation. However, there's no evidence of general-purpose, user-configurable recurring job/workflow scheduling (e.g., cron-like scheduling for arbitrary evals, datasets, or pipelines) beyond this specific built-in feature. Missing for 10: user-defined scheduling API/UI for arbitrary evals or workflows, documentation of schedule configuration options, and independent confirmation of reliability.

   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”

   proof ↗⚑ flag

   Galileo—none0/10

   Galileo is an LLM observability/evaluation platform with experiments, alerts, and MCP integration, but no evidence describes scheduling recurring jobs or workflows (e.g., cron-like automation, scheduled evaluation runs). Alerts are reactive, not scheduled, and no scheduler feature is documented.

   ⚑ flag

Cost tracking

1. developerAttribute cost and usage to users, sessions, and features via custom metadata

   weight 2 · round drawn
   Braintrust~partialclaimed4/10

   Braintrust's tracing captures token usage and custom metadata for every request, which is the raw ingredient for cost attribution, but no evidence describes dashboards, filters, or reports that actually roll up cost/usage by user, session, or feature tags. missing for 10: explicit cost-attribution UI/reporting by custom metadata dimensions, user/session/feature breakdown examples, and independent confirmation this works in practice.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”

   proof ↗⚑ flag

   Galileo~partialclaimed4/10

   Galileo's logging captures sessions, traces, and spans (galileo-docs-6) and supports custom metrics (galileo-docs-8), and mentions Luna models monitoring traffic at lower cost (galileo-docs-12), implying some usage/cost tracking infrastructure exists. However, there is no explicit documentation of tagging traces/sessions with custom metadata fields (e.g., user ID, feature name) for cost attribution or cost breakdown by dimension. missing for 10: explicit custom metadata tagging API/fields for user/session/feature attribution, cost-per-tag reporting or dashboards, and any hands-on example of cost attribution via metadata.

   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
   • [claimed-docs] “Distill your optimized evals into Luna models that monitor 100% of your traffic at 96% lower cost.”

   proof ↗⚑ flag

2. developerSee cost and token usage per request, model, and time period in dashboards

   weight 3 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust's tracing explicitly captures token usage per request (braintrust-docs-1) and traces/spans can be browsed in the UI or terminal (braintrust-docs-14/19/35), which supports basic usage visibility, but the evidence never mentions a cost metric, cost calculation, or a dedicated dashboard aggregating cost/tokens by model or time period. Missing for 10: explicit cost tracking/computation, dashboard views aggregating usage by model, dashboard views aggregating usage by time period, and any independent confirmation of these dashboard features.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`]”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”

   proof ↗⚑ flag

   Galileo—none0/10

   The evidence pack covers tracing, experiments, metrics, and alerts, but contains no mention of cost or token usage tracking, nor dashboards broken down by request, model, or time period. This is a plausible axis for an LLM observability platform, so absence of evidence yields 'none' rather than 'na'.

   ⚑ flag

Data export

1. developerBulk-export traces and datasets to blob storage or my data warehouse

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust supports exporting logs/traces via CLI (`bt sync pull`, download as CSV/JSON) and a full API for programmatic access to traces, datasets, and experiments, which developers could pipe into blob storage or a warehouse themselves. However, there is no documented native connector or bulk-export feature specifically targeting blob storage (S3/GCS) or data warehouses (Snowflake/BigQuery) — missing for 10: dedicated warehouse/blob-storage export integrations, scheduled/automated bulk export pipelines, and independent confirmation of large-scale export performance.

   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”

   proof ↗⚑ flag

   Galileo—none0/10

   No evidence of any bulk-export capability to blob storage or a data warehouse; docs cover logging, tracing, experiments, and MCP dataset management but never mention exporting data out to S3/GCS/BigQuery/Snowflake, and the OpenAPI probe returned 404s with no export endpoint mentioned.

   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”

   proof ↗⚑ flag

Ai eval ops

1. ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust provides programmatic APIs and MCP/CLI tooling that let an agent create datasets, trigger evals, and read back results: the API supports creating/managing datasets and experiments, the SDK's Eval() function runs evals programmatically, and the MCP server/CLI explicitly let agents like Claude Code/Cursor 'run evals' and 'query logs' from within their workflow. missing for 10: a concrete end-to-end example showing an agent specifically creating a dataset via API/MCP and then reading back eval results in one flow, and independent (non-vendor) confirmation of this specific workflow.

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Galileo✓fullprobed7/10

   Galileo's official MCP server explicitly exposes dataset creation, experiment (eval) running, and prompt template management directly from an agent's dev environment, and separate SDK/decorator logging plus experiment docs confirm results are captured and queryable. Missing for 10: no hands-on/independent confirmation of an agent actually reading back structured eval results via MCP, and no explicit example showing the full create-dataset→run-eval→read-results loop end-to-end.

   • [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
   • [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
   • [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
   • [github] “You can also use the `@log` decorator to log spans.”
   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”

   proof ↗⚑ flag

Human review

1. ml engineerRoute outputs to human annotation queues for review and labeling

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust supports building custom annotation interfaces for human review and labeling (docs-12) and datasets can be built/curated from production logs and manual curation (docs-5, docs-31), implying human review workflows, but there's no explicit evidence of a dedicated 'routing' mechanism, queue assignment, reviewer task distribution, or workflow status tracking for human annotators. missing for 10: explicit queue/routing mechanism to assign outputs to specific human reviewers, task assignment/status tracking, and independent corroboration of the annotation workflow in practice.

   • [claimed-docs] “Build annotation interfaces that match your team's workflow. Review support conversations differently than co”
   • [claimed-docs] “Build datasets from production logs, user feedback, manual curation, or generate them with Loop.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”

   proof ↗⚑ flag

   Galileo~partialclaimed4/10

   Galileo docs mention capturing 'subject matter expert annotations' to build datasets and using natural-language feedback to align metrics, implying some human-in-the-loop capability, but there is no explicit documentation of a dedicated annotation queue, review workflow, task assignment, or labeling UI for routing outputs to human reviewers. missing for 10: dedicated annotation queue/workflow feature, reviewer assignment mechanism, labeling UI documentation, independent corroboration of human review routing.

   • [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
   • [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”

   proof ↗⚑ flag

Offline evals

1. developerRun evals in CI and gate deployments on their results

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Docs explicitly describe running evals via code/CLI, integrating into CI/CD to catch regressions, and gating on experiment results ('Automate in CI/CD — Run evals on every pull request to catch regressions'; 'integrate into CI/CD to catch regressions before they reach production'), backed by SDK eval code examples. Missing for 10: independent/hands-on confirmation of actual CI gating in production and details on failure-threshold/gate configuration.

   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”

   proof ↗⚑ flag

   Galileo—none0/10

   Evidence shows experiments/evals can be run via console or SDK, but there is no mention of a CI integration, CLI exit codes, or gating deployments based on eval results. missing for 10: CI/CD integration docs, pass/fail thresholds for gating, pipeline examples (GitHub Actions, Jenkins, etc.), any mention of 'CI' or 'gate' in evidence.

   • [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
   • [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”

   proof ↗⚑ flag

2. ml engineerWrite custom code-based scorers and metrics for my evaluations

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust's SDK examples show Eval() calls with custom `scores` arrays (e.g. LevenshteinScorer), demonstrating code-based custom scorers, and docs describe managing/authoring scorers via API, CLI, and playgrounds. Missing for 10: independent hands-on validation of writing a fully custom scorer function beyond the built-in Levenshtein example, and more detail on scorer authoring API/signature.

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”

   proof ↗⚑ flag

   Galileo~partialclaimed6/10

   Docs explicitly mention custom code-based metrics as an extension to built-in metrics, alongside LLM-as-a-judge metrics, confirming the capability exists. However, evidence lacks concrete SDK examples, API reference, or hands-on walkthroughs showing how to author and register a custom scorer function. Missing for 10: code samples/API reference for writing custom scorers, independent corroboration of usage, details on scorer registration/execution lifecycle.

   • [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”

   proof ↗⚑ flag

3. ml engineerCompare eval runs side by side to catch regressions between prompt or model versions

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed9/10

   Braintrust's Experiments feature is explicitly described as an immutable, comparable record of eval runs that tracks progress over time and integrates into CI/CD to catch regressions, and Playgrounds let users run full evaluations and compare results side-by-side across prompt/model configurations. This directly matches the story of comparing eval runs to catch regressions between versions. Missing for 10: independent/hands-on corroboration of the side-by-side comparison UI beyond first-party docs.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”

   proof ↗⚑ flag

   Galileo~partialclaimed5/10

   Galileo's Experiments feature lets you evaluate prompts, models, and code against chosen metrics, and the console provides a way to create and add experiments to a project, implying some run-to-run evaluation tracking. However, no evidence explicitly describes a side-by-side comparison view or regression-detection UI/workflow between prompt or model versions. missing for 10: explicit side-by-side comparison UI, diffing/regression alerts between experiment runs, independent user confirmation of comparison workflow.

   • [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
   • [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”

   proof ↗⚑ flag

4. ml engineerScore outputs with configurable LLM-as-a-judge evaluators

   weight 3 · round drawn
   Braintrust✓fullclaimed8/10

   Braintrust's docs describe configurable scorers used within evals and playgrounds (LLM-as-a-judge style scorers), online scoring of production traces, and code examples show custom scorer configuration (e.g., LevenshteinScorer swapped in evals). Docs explicitly mention 'author scorers' via CLI/MCP and building scorers with Loop, supporting configurability. Missing for 10: explicit documentation naming 'LLM-as-a-judge' scorer templates/library and independent hands-on validation of judge configurability beyond vendor docs.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Galileo✓fullclaimed8/10

   Docs explicitly describe LLM-as-a-judge as a configurable metric type alongside custom code-based metrics, plus continuous feedback loops to align metrics to domain needs, and experiments to run these metrics against outputs. Missing for 10: independent/hands-on corroboration beyond vendor docs and more detail on configuring specific judge prompts/models.

   • [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
   • [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”
   • [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”

   proof ↗⚑ flag

5. ml engineerCurate datasets from production traces and run offline evaluations against them

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed9/10

   Docs explicitly describe building versioned datasets from production logs/traces (docs-5, docs-31) and running offline evaluations against them via Experiments, code, or Playgrounds (docs-27, docs-23, gh-1/gh-2 SDK examples), with CI/CD integration for regression testing (docs-2). Missing for 10: independent/hands-on third-party validation of the end-to-end curate-from-production-to-offline-eval workflow beyond first-party docs.

   • [claimed-docs] “Build datasets from production logs, user feedback, manual curation, or generate them with Loop.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Galileo✓fullclaimed8/10

   Galileo explicitly supports building datasets from production/live traces (galileo-docs-11), capturing traces/spans in production (galileo-docs-6, galileo-docs-7), and running offline evaluations/experiments against datasets with custom or LLM-as-judge metrics (galileo-docs-2, galileo-docs-8). This directly covers curating datasets from production traces and running offline evals. Missing for 10: independent/hands-on corroboration of the full production-trace-to-dataset-to-experiment workflow beyond vendor docs.

   • [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
   • [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
   • [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”

   proof ↗⚑ flag

Online evals

1. ml engineerRun evaluators continuously on live production traffic, not just offline datasets

   weight 2 · round drawn
   Braintrust✓fullclaimed8/10

   Braintrust explicitly supports 'online scoring' that evaluates production traces automatically as they're logged asynchronously with no latency impact, plus continuous background analysis via Patterns/Topics scanning trace backlogs on a schedule for issues no scorer already catches. This directly matches continuous evaluation on live traffic beyond offline datasets. Missing for 10: independent/hands-on verification of online scoring behavior in production and more detail on configuring which evaluators run continuously.

   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Define the dimensions that matter to your business, like use case, customer segment, compliance, or tone. Topics continuously clusters every…”

   proof ↗⚑ flag

   Galileo✓fullclaimed8/10

   Galileo's docs describe real-time capture of every session/trace/span, alerting on live logs, and distilled 'Luna' models that monitor 100% of production traffic at lower cost, which directly supports continuous evaluator execution on live traffic (not just offline datasets), complemented by custom/LLM-as-judge metrics and datasets built from live production data. missing for 10: independent/hands-on verification that evaluators run continuously in production (all evidence is vendor docs) and clearer detail on evaluator scheduling/latency at scale.

   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
   • [claimed-docs] “Distill your optimized evals into Luna models that monitor 100% of your traffic at 96% lower cost.”
   • [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
   • [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Braintrust
   Braintrust~partialprobed7/10

   Braintrust offers a comprehensive API/SDK ('interact with all aspects of the Braintrust platform programmatically'), a full-featured CLI (bt) for auth, tracing, evals, log queries, and function management, and an MCP server letting agents query logs, author scorers, configure monitoring, and run evals — covering most UI functionality outside the terminal. However, some UI-only features like Playgrounds (explicitly 'no-code workspace') and annotation-interface building appear UI-centric without clear API/CLI parity, and no public OpenAPI spec was found to confirm full API coverage. missing for 10: confirmed OpenAPI/spec proving 1:1 API-UI parity, explicit API/CLI equivalent for Playgrounds and custom annotation UI workflows, independent verification of completeness.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Galileo~partialprobed5/10

   Galileo exposes a Python SDK, decorators, and an MCP server that cover core workflows (tracing, experiments, datasets, prompt templates), suggesting many UI actions have API/SDK equivalents (galileo-docs-2, galileo-docs-4, galileo-gh-1). However, docs explicitly describe some actions (e.g., 'Create Experiment' button) as UI-only, and no public OpenAPI/swagger spec is discoverable (galileo-probe-3), so full API parity is unproven. Missing for 10: an explicit statement or spec confirming 1:1 API/UI feature parity, a discoverable OpenAPI reference, and evidence that console-only features (alerts setup, dashboards) have API equivalents.

   • [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
   • [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
   • [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
   • [github] “You can also use the `@log` decorator to log spans.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust provides data export via `bt sync pull`, `bt view logs` downloads as CSV/JSON, and a full API to programmatically extract projects, experiments, datasets, traces, and prompts, plus a self-hosting option for data control. However, there's no explicit documentation of full-account export/portability guarantees, no mention of open/standard interchange formats beyond CSV/JSON logs, and no independent confirmation of a clean 'leave' workflow. Missing for 10: comprehensive account-level export documentation, open-format guarantees for all entity types (prompts, scorers, configs), and third-party validation of export completeness.

   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Galileo—none0/10

   No evidence in the pack describes any data export feature, open-format export, or data portability mechanism for traces, datasets, or experiments — only ingestion, logging, and metric features are documented.

   ⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round to Galileo
   Braintrust—none0/10

   Evidence shows only client SDK repos on GitHub (braintrust-sdk-javascript) and self-hosted deployment of infrastructure, but the core platform/UI is explicitly described as Braintrust-managed and not released under an open license; no evidence of the product's full source being open-licensed.

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Galileo~partialclaimed3/10

   Evidence shows only a GitHub repo for the Python client SDK (galileo-python), with no license details or indication that the core Galileo platform/backend is open source. missing for 10: explicit open-source license text, evidence that the full product (not just a client SDK) is source-available, independent confirmation of license terms.

   • [github] “You can also use the `@log` decorator to log spans.”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust documents an official self-hosted deployment option, but it explicitly separates data storage (self-hosted) from platform management (Braintrust-managed UI, auth, updates), meaning it's not a fully self-hostable core product. Missing for 10: evidence of fully self-hosting the entire platform (UI, control plane, auth) independent of Braintrust's managed services, and independent/community confirmation of self-hosting experience.

   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”

   proof ↗⚑ flag

   Galileo—none0/10

   No evidence of a self-hostable/on-prem version of Galileo; all documentation points to a hosted console/SaaS product with SDKs and MCP integration, not a self-hosted deployment option. missing for 10: any mention of self-hosting, on-prem deployment, Docker/Helm packages, or enterprise private-cloud install instructions.

   ⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust's self-hosting docs explicitly state it's designed to meet data residency requirements by keeping customer data within your own cloud account and region, giving users region/residency control — but only via full self-hosted deployment, not a simple region-selection toggle in the managed cloud offering. Missing for 10: evidence of region choice within the standard hosted/SaaS plan, list of specific supported regions, and compliance certifications tied to residency.

   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Galileo—none0/10

   No evidence pack item mentions data residency, regional storage options, or compliance controls for data location; the evidence covers tracing, experiments, metrics, and MCP only. Since Galileo is a SaaS platform where data residency is a plausible and common enterprise concern, absence of evidence means 'none' rather than 'na'.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Braintrust—none0/10

   No evidence pack item addresses data-training opt-out, a no-training policy, or any statement about whether customer data is used to train AI models. Self-hosting docs address data residency/compliance but do not mention model training use. missing for 10: any explicit no-training-on-customer-data policy, opt-out controls, or terms-of-service statement about AI training use.

   ⚑ flag

   Galileo—none0/10

   The evidence pack covers Galileo's tracing, experiments, metrics, and MCP features but contains no mention of data usage policies, opt-out of model training, or privacy controls regarding customer data being used to train AI models.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round to Braintrust
   Braintrust~partialclaimed4/10

   Braintrust's self-hosted deployment option lets customers keep all data within their own cloud account/region for compliance and residency, giving some control over where data lives, and the API/CLI allow programmatic management of logs and datasets. However, there is no explicit documentation of a retention policy, TTL settings, or a delete/purge API for logs, traces, or datasets on the managed (non-self-hosted) offering. Missing for 10: explicit retention window configuration, an in-platform delete/purge mechanism for traces or datasets, and confirmation that these controls exist outside of self-hosting.

   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”

   proof ↗⚑ flag

   Galileo—none0/10

   The evidence pack covers tracing, experiments, metrics, and MCP integration but contains no mention of data retention policies, deletion controls, or privacy/compliance configuration options for AI-native users. No documentation cites retention windows, data deletion APIs, or export/purge capabilities.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Braintrust—none0/10

   No evidence of any telemetry opt-out or usage-tracking control setting for Braintrust itself as a tool; the evidence covers Braintrust's own tracing/observability product features (for user's AI apps), not the vendor's own telemetry collection practices or an opt-out mechanism. Self-hosting controls where customer AI data resides but says nothing about product telemetry/usage tracking opt-out.

   ⚑ flag

   Galileo—none0/10

   No evidence pack items mention telemetry opt-out, usage tracking controls, or privacy settings for Galileo itself; the evidence only covers product features like tracing, experiments, and metrics.

   ⚑ flag

Prompt workflow

1. developerIterate on prompts in a playground against real models and variables

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Docs explicitly describe Playgrounds as a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets, running full evaluations in real-time against real models with variable inputs, and comparing results side-by-side (braintrust-docs-3, braintrust-docs-23, braintrust-docs-17). This directly matches the story of iterating on prompts in a playground against real models and variables. Missing for 10: independent/hands-on user corroboration of the playground experience specifically (community evidence is generic/mixed) and no detail on variable-substitution mechanics beyond the general description.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Measure and improve AI application quality with playgrounds and evals”

   proof ↗⚑ flag

   Galileo~partialclaimed5/10

   Galileo's Experiments feature lets users evaluate prompts and models against defined inputs and metrics via a console UI, and prompt templates can be set up through the MCP integration, which loosely maps to prompt iteration with variables. However, there's no evidence of a dedicated interactive 'playground' for live, real-time prompt testing against models — missing for 10: a documented playground UI, live model response preview, and variable substitution workflow distinct from formal experiment runs.

   • [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
   • [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
   • [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”

   proof ↗⚑ flag

2. developerVersion prompts and deploy changes to production without shipping code

   weight 3 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust supports managing prompts via playgrounds, API, and CLI ("Manage prompts, tools, and scorers"), and datasets/experiments are versioned, suggesting prompt versioning is possible; however, there's no explicit evidence of a distinct 'deploy to production without code' workflow (e.g., prompt promotion/publish/rollback mechanism) separate from logging or evals. missing for 10: explicit prompt versioning/history UI, a documented deploy/publish/rollback mechanism for prompts, and independent confirmation that production apps can swap prompts without redeploying code.

   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”

   proof ↗⚑ flag

   Galileo—none0/10

   Evidence shows Galileo supports experiments for evaluating prompts and mentions 'setting up prompt templates' via MCP, but there is no documentation of prompt versioning, a prompt registry, or a mechanism to deploy prompt changes to production independent of code deploys.

   • [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
   • [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”

   proof ↗⚑ flag

Ai observability

1. ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app

   weight 3 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust documents a first-party MCP server enabling MCP-compatible agents (Claude Code, Cursor, Codex, etc.) to query logs, run evals, author scorers, and configure monitoring, plus a full programmatic API for traces/experiments/datasets, and Loop, Braintrust's own AI agent, for natural-language investigation of logs and eval data. This directly matches the story of an agent querying traces, metrics, and eval results via API or MCP to debug an app. Missing for 10: independent hands-on corroboration of the MCP server working end-to-end (only vendor docs/probe confirm existence, no third-party review of actual debugging workflow).

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   Galileo~partialprobed5/10

   Galileo has a documented official MCP server (galileo-docs-4, galileo-probe-4) enabling access to Galileo capabilities from a dev environment, but the explicitly listed MCP capabilities are creating/managing datasets, running experiments, and setting up prompt templates—not querying traces, metrics, or eval results directly. No OpenAPI/API reference was discoverable (galileo-probe-3 returned 404s), so there's no clear evidence an agent can query traces/metrics/eval results programmatically for debugging via API or MCP. missing for 10: explicit MCP/API support for querying traces and metrics, evidence of eval-result retrieval via MCP, and a discoverable REST/OpenAPI spec for programmatic trace queries.

   • [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
   • [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”

   proof ↗⚑ flag

Data controls

1. developerMask or redact sensitive data before it is stored in traces

   weight 2 · round drawn
   Braintrust—none0/10

   No evidence in the pack describes any PII masking, redaction, or scrubbing feature for traces before storage; documentation covers instrumentation, tracing, evals, and observability but never mentions data masking or redaction capabilities. This is a plausible and common axis for a tracing/observability platform, so absence of evidence yields 'none' rather than 'na'.

   ⚑ flag

   Galileo—none0/10

   No evidence in the pack mentions masking, redaction, PII scrubbing, or any data sanitization mechanism applied before storing traces; documentation focuses on logging, tracing, metrics, and experiments without addressing sensitive data handling.

   ⚑ flag

Sdk coverage

1. developerInstrument apps in both Python and JS/TS with officially supported SDKs

   weight 2 · round to Braintrust
   Braintrust~partialclaimed6/10

   Docs confirm instrumentation/tracing is central to Braintrust and JS SDK usage is shown in GitHub examples (Eval function calls), with docs referencing SDK-based tracing and instrumentation broadly, but the evidence pack never explicitly confirms an officially supported Python SDK alongside JS/TS — only JS/TS SDK code is shown. Missing for 10: explicit Python SDK documentation/examples, side-by-side confirmation of parity between Python and JS/TS SDKs, and independent corroboration of both SDKs in production use.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Galileo~partialclaimed4/10

   Evidence confirms a Python SDK (galileo-python) with decorator-based span logging and OTel-based distributed tracing, but no evidence of an official JS/TS SDK or its documentation. missing for 10: JS/TS SDK repo or docs, parity of instrumentation features (decorators, spans) between Python and JS/TS.

   • [github] “You can also use the `@log` decorator to log spans.”
   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”

   proof ↗⚑ flag

Trace capture

1. developerTrace multi-step agent runs as nested spans grouped into sessions or threads

   weight 2 · round to Galileo
   Braintrust~partialclaimed6/10

   Braintrust documents tracing that captures spans, logs, and full request detail, and lets users browse traces and individual spans via UI or CLI, supporting general span-based tracing. However, the evidence never explicitly describes nested multi-step spans grouped into sessions/threads for agent runs (no mention of parent-child span hierarchies or session/thread grouping semantics). Missing for 10: explicit documentation of nested span hierarchies for multi-step agent runs, explicit session/thread grouping constructs, and independent/hands-on confirmation of this specific structure.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”

   proof ↗⚑ flag

   Galileo✓fullclaimed8/10

   Docs explicitly describe capturing sessions, traces, and spans with structured logging, and distributed tracing docs show spans joined via shared trace IDs (nested spans under a trace) plus the @log decorator for span-level instrumentation. Missing for 10: explicit worked example showing session/thread grouping across multiple agent runs and independent hands-on corroboration beyond first-party docs.

   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
   • [github] “You can also use the `@log` decorator to log spans.”
   • [claimed-docs] “Agentic metrics help you measure how well your AI agents perform complex, multi-step tasks—especially when those agents need to use tools, m…”

   proof ↗⚑ flag

2. developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK

   weight 2 · round to Braintrust
   Braintrust~partialcommunity4/10

   Docs confirm a general instrumentation capability and state you can 'integrate with AI providers and frameworks to send traces to Braintrust,' but no evidence explicitly names LangChain, the OpenAI SDK, or the Vercel AI SDK as supported integrations. A community comment even calls integration options 'limited.' Missing for 10: explicit documentation or hands-on confirmation of dedicated LangChain, OpenAI SDK, and Vercel AI SDK integrations.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [community] “Braintrust: Interesting approach to evaluation but feels like an early-stage product. Documentation is sparse and integration options are li…”

   proof ↗⚑ flag

   Galileo—none0/10

   The evidence pack shows Galileo's own Python SDK (decorator-based logging) and generic OpenTelemetry distributed tracing support, but contains no mention of pre-built integrations for LangChain, the OpenAI SDK, or the Vercel AI SDK specifically. Since this is a well-known, plausible capability for a tracing/observability product, absence of evidence means 'none' rather than 'na'.

   • [github] “You can also use the `@log` decorator to log spans.”
   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”

   proof ↗⚑ flag

3. developerCapture multimodal payloads (images, audio, files) inside my traces

   weight 1 · round drawn
   Braintrust—none0/10

   The evidence pack describes tracing capturing inputs, outputs, model parameters, latency, token usage, and metadata, but there is no explicit mention of multimodal payload support (images, audio, files) within traces. Missing for 10: explicit documentation or examples showing image/audio/file capture in traces, SDK support for multimodal attachments, or any UI rendering of multimodal content in trace spans.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”

   proof ↗⚑ flag

   Galileo—none0/10

   The evidence pack covers tracing, spans, metrics, alerts, and MCP integration, but contains no mention of multimodal payload capture (images, audio, files) within traces or spans.

   ⚑ flag

4. developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format

   weight 3 · round to Galileo
   Braintrust—none0/10

   No evidence pack item mentions OpenTelemetry or OTLP support; Braintrust's docs describe its own SDK/instrumentation and integrations but never reference OTLP-based trace ingestion/export. Missing for 10: any mention of OTLP endpoint, OpenTelemetry SDK compatibility, or OTel collector integration.

   ⚑ flag

   Galileo~partialclaimed5/10

   Docs confirm Galileo's SDK supports distributed tracing using OpenTelemetry's W3C traceparent header to propagate context and stitch spans into a single trace, showing OTel compatibility beyond a fully proprietary format. However, there's no evidence of a dedicated OTLP ingestion/export endpoint or explicit statement that Galileo accepts/emits OTLP-formatted traces from arbitrary OTel collectors. missing for 10: explicit OTLP endpoint/collector documentation, confirmation of both sending and receiving OTLP data, and independent verification of OTLP interoperability outside Galileo's own SDK.

   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
   • [github] “You can also use the `@log` decorator to log spans.”

   proof ↗⚑ flag

5. developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust docs explicitly state instrumentation captures inputs, outputs, model parameters, latency, token usage, and metadata for every LLM call, with SDK examples and CLI/UI browsing of traces and spans. missing for 10: independent hands-on verification of exact captured fields and no third-party benchmark of tracing accuracy/performance overhead.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”

   proof ↗⚑ flag

   Galileo✓fullclaimed7/10

   Galileo's docs describe capturing sessions, traces, and spans once instrumented, with a `@log` decorator for logging spans and OTel-based distributed tracing joining spans by trace ID, forming a structured real-time data stream. However, explicit confirmation that captured traces include latency and token usage fields specifically is not directly quoted, only implied via 'structured stream of real-time data.' Missing for 10: explicit documentation/screenshot confirming latency and token-usage fields are captured per trace, and independent/hands-on corroboration beyond vendor docs.

   • [claimed-docs] “Create and run your first trace in less than 5 minutes.”
   • [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
   • [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
   • [github] “You can also use the `@log` decorator to log spans.”

   proof ↗⚑ flag