Braintrust vs W&B Weave

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to W&B Weave
   Braintrust~partialprobed6/10

   A live probe confirms llms.txt exists at braintrust.dev/llms.txt returning 200 with a concise agent-oriented description, directly satisfying the story's core requirement. However, docs.md and OpenAPI/machine-readable doc endpoints return 404, indicating incomplete agent-oriented documentation coverage beyond the single llms.txt file. missing for 10: full docs-as-markdown or OpenAPI endpoints for deeper agent consumption, independent confirmation that agents successfully use llms.txt in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://www.braintrust.dev/llms.txt # Braintrust > Braintrust is the active observability platform for instrume…”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.braintrust.dev/docs.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   W&B Weave✓fullprobed9/10

   Direct probes confirm llms.txt (HTTP 200 with structured doc index) and a .md variant of docs pages exist and are crawlable, exactly matching the ask for agent-oriented docs, plus an OpenAPI spec and MCP server for further agent integration. Missing for 10: no independent/community confirmation that agents actually consume these docs successfully in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.wandb.ai/llms.txt # Weights & Biases Documentation - [Products (407 pages)](https://docs.wandb.ai/…”
   • [probe] “PROBE docs-md: HTTP 200 at https://docs.wandb.ai/weave.md > ## Documentation Index > Fetch the complete documentation index at: https://docs…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust supports headless/CI usage via code-based Eval() runs, an API for programmatic control, a CLI (`bt`) for auth/eval/log operations from the terminal, and explicit documentation of CI/CD integration to run evals on every pull request. This combination clearly enables automated, non-interactive operation in pipelines. Missing for 10: independent hands-on confirmation of CI pipeline usage beyond vendor docs.

   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   W&B Weave~partialclaimed6/10

   Weave's SDK (weave.op, weave.init) and Service API/REST endpoints allow programmatic, non-UI instrumentation and evaluation that can run in scripts or CI pipelines, and the OTel-compatible ingestion endpoint supports headless trace collection. However, there is no explicit documentation of a CI-specific workflow, headless auth/config for pipelines, or a dedicated CLI/automation example confirming CI usage. missing for 10: explicit CI/headless setup guide, documented non-interactive auth flow for automated pipelines, concrete CI example (e.g. GitHub Actions integration).

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
   • [github] “Log and debug language model inputs, outputs, and traces”

   proof ↗⚑ flag

3. ai-native userConnect an agent via an official MCP server

   weight 3 · round drawn
   Braintrust✓fullprobed8/10

   Braintrust documents an official MCP server that lets Claude Code, Cursor, Codex, VS Code, and other MCP-compatible clients query logs, author prompts/scorers, configure monitoring, and run evals — a clear server-side MCP offering that agents can connect to. Missing for 10: independent/hands-on third-party confirmation that the MCP server works as documented (only first-party docs and a probe confirm its existence).

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   W&B Weave✓fullprobed8/10

   W&B ships an official MCP server (wandb-mcp-server) enabling coding agents like Claude Code to connect to Weave, read live production data, run evaluations, and execute iteration loops autonomously — this is documented both on the product site and via a dedicated GitHub repo. Missing for 10: deeper documentation of MCP server setup/configuration and independent hands-on corroboration beyond vendor claims.

   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”

   proof ↗⚑ flag

4. ai-native userUse an official CLI

   weight 2 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust ships an official `bt` CLI, documented with a dedicated quickstart, allowing authentication, tracing coding-agent sessions, running evals, browsing/querying logs, syncing data, and managing functions from the terminal — directly matching AI-native/agentic CLI usage. Missing for 10: independent/hands-on community corroboration of the CLI itself (community evidence is generic, not CLI-specific).

   • [claimed-docs] “Install the `bt` CLI to set up, instrument, and run Braintrust with your coding agent”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   W&B Weave~partialprobed5/10

   There is evidence of an official W&B CLI (docs.wandb.ai/models/ref/cli), but this CLI is documented under the Models product, not specifically Weave, and no evidence details Weave-specific CLI commands (e.g., managing traces, evaluations, or ops from the terminal) or AI-native/agentic use of it. Missing for 10: Weave-specific CLI command reference, evidence of agentic/programmatic use of the CLI, independent hands-on confirmation.

   • [probe] “official CLI documented at https://docs.wandb.ai/models/ref/cli”

   proof ↗⚑ flag

5. ai-native userDrive the product through a documented public API

   weight 3 · round drawn
   Braintrust✓fullprobed8/10

   Braintrust documents a full public API reference for managing projects, experiments, datasets, prompts, scorers, and logging programmatically (braintrust-docs-11, -34), backed by SDKs (braintrust-gh-1/2), a CLI (braintrust-docs-33), and MCP server for agent-driven access (braintrust-docs-9, -25). Missing for 10: no discoverable OpenAPI/swagger spec (braintrust-probe-3 shows 404s) and no independent third-party corroboration of API robustness beyond docs.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   W&B Weave✓fullprobed8/10

   Weave documents a public REST Service API for programmatic access, an openapi.json spec, Python/TypeScript SDKs with @weave.op decorators, and an official MCP server enabling agent-driven interaction with live data and evaluations. missing for 10: independent third-party validation of API stability/versioning and rate-limit documentation beyond first-party docs.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”

   proof ↗⚑ flag

6. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   Braintrust—none0/10

   The evidence describes Braintrust's general API, CLI, and MCP integrations but contains no mention of scoped, role-based, or least-privilege API key/credential issuance for agents; the only security-related item is a breach report telling customers to rotate keys, which does not demonstrate a scoping/least-privilege capability.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [community] “AI evaluation startup Braintrust confirms breach, tells every customer to rotate sensitive keys.”

   proof ↗⚑ flag

   W&B Weave—none0/10

   W&B Weave is an LLM observability/evaluation tool; the evidence pack covers tracing, evaluations, cost tracking, and an MCP/skills integration, but there is no mention of scoped or least-privilege API credential issuance for agents. Missing for 10: any documentation of API key scoping, permission granularity, or credential management for agent access.

   ⚑ flag

7. ai-native userBuild against official SDKs

   weight 2 · round drawn
   Braintrust✓fullprobed8/10

   Braintrust ships official JS SDK (github examples using Eval()), an API reference for programmatic access, a CLI (bt), and an MCP server, giving AI-native developers multiple official, documented ways to build against the platform. One community comment calls docs 'sparse' but this is general skepticism, not a concrete contradiction of SDK functionality. missing for 10: independent hands-on validation of the SDK itself (beyond vendor docs/examples) and a public OpenAPI spec (probe found 404s).

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”
   • [community] “Braintrust: Interesting approach to evaluation but feels like an early-stage product. Documentation is sparse and integration options are li…”

   proof ↗⚑ flag

   W&B Weave✓fullprobed8/10

   Weave offers official Python and TypeScript SDKs with decorator-based tracing (@weave.op), a REST Service API, OTel-compatible SDK, and a documented CLI/MCP server, all backed by first-party docs and public GitHub repo. Missing for 10: independent third-party benchmarking or hands-on developer reviews validating SDK stability/completeness beyond vendor docs.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [github] “You can trace any function using `weave.op` - from api calls to OpenAI, Anthropic, Google AI Studio etc to generation calls from Hugging Fac…”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Trace and collect metrics about your agents built with popular SDKs and harnesses using Weave OTel-compatible SDK”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
   • [probe] “official CLI documented at https://docs.wandb.ai/models/ref/cli”

   proof ↗⚑ flag

8. ai-native userSubscribe to events via webhooks

   weight 2 · round to W&B Weave
   Braintrust—none0/10

   No evidence in the pack mentions webhooks or any event-subscription mechanism; Braintrust's documented interfaces are API, CLI, MCP server, and UI, none of which are shown to support webhook subscriptions.

   ⚑ flag

   W&B Weave~partialclaimed4/10

   Weave documents alerts that 'trigger webhook automations' from production insights, indicating some outbound webhook mechanism exists, but there is no documentation of a subscription API, event types, payload schema, or configuration steps for webhooks. missing for 10: documented webhook subscription/configuration API, list of subscribable event types, payload format, independent/hands-on confirmation.

   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust ships Loop, an AI agent that investigates logs via natural language and can build scorers/datasets/dashboards, plus Patterns which proactively scans trace backlogs for recurring issues and suggests fixes, and Topics which auto-clusters/classifies traces against business dimensions — all directly generating AI insights/suggestions inside the product. Missing for 10: independent/hands-on validation of Loop/Patterns quality (evidence is vendor docs only) and no third-party review confirming suggestion usefulness.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Define the dimensions that matter to your business, like use case, customer segment, compliance, or tone. Topics continuously clusters every…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review. Each trace is analyzed by facets that extract sho”

   proof ↗⚑ flag

   W&B Weave~partialclaimed6/10

   Weave ships built-in LLM-judge scorers (hallucination detection, summarization quality, embedding similarity) and Guardrails (toxicity, bias, PII detection) that automatically generate AI-based assessments of traced data, plus 'signals' that auto-classify agent interactions — all forms of AI-generated insight surfaced inside the product. However, these are narrow, pre-defined quality/safety classifiers rather than general proactive 'suggestions' or exploratory insight generation across arbitrary data, and most of the deeper analysis (custom scorers, evaluation criteria) requires user-authored code rather than the product generating novel suggestions on its own. Missing for 10: evidence of open-ended AI-generated recommendations/next-step suggestions (not just fixed scorer categories), and independent/hands-on confirmation these signals surface meaningfully useful insights in practice.

   • [claimed-docs] “Weave comes with predefined scorers and local SLM scorers that you can use right away, including: Hallucination detection, Summarization qua…”
   • [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
   • [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust supports autonomous background automation via 'Patterns', which runs Loop (its AI agent) on a schedule against the trace backlog to surface recurring issues and suggested fixes, plus online scoring that runs asynchronously on production traces and CI/CD-triggered evals. However, this is narrower than general-purpose autonomous automations — it's scoped to trace analysis/scoring rather than arbitrary user-defined background tasks, and there's no independent/hands-on corroboration of Patterns actually running unattended. Missing for 10: broader/customizable scheduled automation beyond Patterns and online scoring, independent evidence of reliability, and detail on failure handling or configurability of these background jobs.

   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”

   proof ↗⚑ flag

   W&B Weave~partialclaimed5/10

   Weave documents automated background signals and alerting (built-in/custom signals classify agent interactions and trigger Slack/webhook automations) and mentions agents connecting via MCP to 'execute automatic iteration loops on their own,' which suggests some autonomous background automation. However, Weave is primarily a tracing/evaluation/observability tool, not a scheduler or workflow-automation platform, and there's no dedicated docs on setting up persistent background jobs or scheduled autonomous runs beyond alert-triggered webhooks. Missing for 10: dedicated automation/scheduling feature docs, evidence of persistent autonomous background jobs beyond alert webhooks, independent corroboration of the MCP-driven 'automatic iteration loops' claim.

   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”

   proof ↗⚑ flag

3. ai-native userOperate the product with natural-language commands

   weight 2 · round to Braintrust
   Braintrust✓fullprobed8/10

   Braintrust's Loop agent lets users investigate logs and build scorers/datasets/dashboards via natural language instead of writing queries, and this is exposed through an official MCP server so natural-language commands work from Claude Code, Cursor, Codex, etc. Missing for 10: independent hands-on validation of natural-language command accuracy/reliability and broader coverage beyond Loop's stated scope.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Loop is Braintrust's AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   W&B Weave~partialprobed6/10

   Weave itself is an observability/eval dashboard with no native chat-command interface, but an official MCP server lets AI coding agents like Claude Code read production data, run evaluations, and iterate automatically using natural-language instructions relayed through MCP tools. This gives indirect NL-driven operation rather than a first-party conversational control surface. Missing for 10: a built-in Weave chat/NL console, independent hands-on verification of the MCP-driven workflow, and broader agent support beyond Claude Code.

   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round to W&B Weave
   Braintrust—none0/10

   Braintrust has an API reference page (braintrust-docs-11/34) but the evidence pack shows no interactive or runnable-example capability — probes for an OpenAPI/swagger spec (needed for typical interactive API explorers) all returned 404, and there's no mention of a try-it-now console or embedded runnable snippets in the API reference itself. missing for 10: evidence of an interactive API console, runnable code snippets embedded in the reference, or a machine-readable OpenAPI spec powering such interactivity.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.braintrust.dev/docs.md”

   proof ↗⚑ flag

   W&B Weave~partialprobed3/10

   The Weave Service API and an OpenAPI spec (openapi.json) exist, suggesting some form of structured API reference, but there is no evidence of an interactive, in-browser reference with runnable/executable examples (e.g., a Swagger/try-it-out console or live code sandbox). missing for 10: confirmation of an interactive UI with 'try it now' runnable examples, evidence of live execution from the docs, and any independent confirmation of this feature.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round to W&B Weave
   Braintrust—none0/10

   Braintrust documents a REST API (api-reference) but explicit probes for OpenAPI/swagger specs at all standard paths returned 404, and no docs mention a downloadable machine-readable spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   W&B Weave✓fullprobed9/10

   A probe confirms an OpenAPI spec is served at https://docs.wandb.ai/openapi.json (HTTP 200, contains an 'openapi' key), and docs also describe a Service API providing REST endpoints for programmatic access. missing for 10: no independent third-party confirmation that the spec is actively used/maintained beyond the probe check.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Braintrust
   Braintrust~partialclaimed4/10

   Braintrust supports evals/experiments/datasets and separate projects that could function as a sandbox for testing without hitting production logs, and self-hosting offers infrastructure isolation, but there is no explicit documentation of a dedicated 'sandbox environment' or isolation guarantee preventing test runs from touching production data. missing for 10: explicit sandbox/staging environment concept, documentation on isolating test evals from production datasets/logs, and independent confirmation that eval runs never write to production stores.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”

   proof ↗⚑ flag

   W&B Weave~partialclaimed3/10

   Weave's Evaluation framework lets users test against curated Datasets/test examples rather than live production data, and the Playground lets you test prompts/models interactively, which implicitly avoids touching production traffic. However, there is no explicit 'sandbox environment' feature, and other docs (e.g., testing against production traces) actually emphasize using real production data rather than isolating from it. Missing for 10: dedicated sandbox/staging environment concept, explicit data isolation guarantees, and evidence separating test vs production data paths.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
   • [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Braintrust—none0/10

   There is an API reference documenting functionality, but no evidence of API versioning scheme or any documented deprecation policy; probes even show no OpenAPI spec discoverable at standard locations. Missing for 10: versioning scheme documentation, explicit deprecation policy, changelog/migration guides.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   W&B Weave—none0/10

   No evidence of a versioned API scheme or documented deprecation policy for Weave's SDK/Service API; only an OpenAPI spec presence is shown, not versioning/deprecation commitments. missing for 10: versioning scheme documentation, deprecation policy/notice process, changelog or migration guides for breaking changes.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”

   proof ↗⚑ flag

Monitoring

1. ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics

   weight 2 · round to W&B Weave
   Braintrust~partialclaimed5/10

   Braintrust captures latency, token/cost, and eval-score metrics via tracing and experiments (braintrust-docs-1, braintrust-docs-27), and one doc line explicitly says you can 'ask Braintrust's AI agent to investigate your data and build scorers, datasets, and dashboards' (braintrust-docs-18), implying custom dashboard creation. However there is no dedicated dashboard-building documentation, no detail on customizing charts/widgets, and no explicit mention of error-rate metrics or alerting tied to dashboards. Missing for 10: dedicated dashboard-configuration docs, explicit error-metric support, and independent/hands-on evidence of building a custom dashboard.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”

   proof ↗⚑ flag

   W&B Weave~partialclaimed6/10

   Weave tracks latency/traces, cost (automatic and custom cost tracking), and eval scores (via Evaluation objects, scorers, leaderboards), and supports alerting via Slack/webhooks on signals — covering most of the metrics named in the story. However, there is no explicit evidence of a customizable dashboard-building UI (e.g., drag-and-drop widgets, custom charts combining these metrics side-by-side) beyond the built-in calls table, trace tree, and leaderboards. missing for 10: explicit custom dashboard/visualization builder evidence, unified view combining latency+error+cost+eval-score in one configurable dashboard, independent/hands-on confirmation of dashboard flexibility.

   • [claimed-docs] “Automatic cost tracking: For supported integrations, Weave captures token usage from the API response and applies built-in pricing for the m…”
   • [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
   • [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”
   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”

   proof ↗⚑ flag

2. developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email

   weight 2 · round to W&B Weave
   Braintrust—none0/10

   The evidence pack covers tracing, evals, online scoring, Loop/Patterns/Topics analysis, and self-hosting, but contains no mention of configurable alerts, thresholds, or notification integrations (Slack, PagerDuty, email) for error rates, cost spikes, or eval-score drops.

   ⚑ flag

   W&B Weave~partialclaimed4/10

   Weave marketing docs mention built-in/custom 'signals' that capture and classify agent interactions, with alerts routed via Slack notifications and webhook automations, which could plausibly trigger error-rate or eval-score alerts. However, there is no explicit mention of cost-spike alerts, no native PagerDuty or email notification channels (only Slack + generic webhooks), and no detail on how alert thresholds are configured for error rates or eval-score drops specifically. missing for 10: native PagerDuty integration, native email notification channel, explicit documentation of alert types (error rate, cost spike, eval-score drop) and threshold configuration.

   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”
   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed7/10

   Braintrust evals run against entire datasets in one operation, Patterns scans the full trace backlog on a schedule, Topics classifies every trace automatically, and the CLI/API support bulk sync ('bt sync pull'), bulk log queries, and programmatic management of experiments/datasets/prompts at scale. Missing for 10: explicit documentation of bulk edit/delete/update operations on datasets or logs, and independent (non-vendor) confirmation of bulk-operation performance at scale.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   W&B Weave~partialclaimed5/10

   Weave's Evaluation object runs scoring across an entire Dataset of many test examples in one call, and the Service API exposes REST endpoints for programmatic access, which together support batch-style automation over many items. However, there is no explicit evidence of bulk trace management operations (bulk delete, bulk tagging, bulk export/update of many logged calls) that a fully bulk-operations story would require. Missing for 10: documented bulk edit/delete/export APIs for traces or datasets, and independent confirmation of large-scale batch throughput.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “The core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples.”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round drawn
   Braintrust~partialclaimed5/10

   Braintrust supports several automatic, event-driven behaviors — online scoring runs automatically on every new production trace, evals run automatically in CI/CD on pull requests, and Patterns runs Loop against traces on a schedule — but these are fixed built-in automations rather than a general user-defined 'if event then action' rules engine (e.g., no evidence of custom triggers/webhooks/alerts tied to arbitrary conditions). Missing for 10: a documented rules/alerting framework letting users define arbitrary trigger conditions and custom actions, and independent confirmation of this automation depth.

   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”

   proof ↗⚑ flag

   W&B Weave~partialclaimed5/10

   Weave's marketing docs mention custom signals that classify agent interactions and alerts that route to Slack or trigger webhook automations, which matches the idea of rule-based triggers on events. However, this is only briefit is only referenced on the marketing page, with no dedicated guide, configuration reference, or independent confirmation of how rules are authored or scoped. Missing for 10: a dedicated docs page detailing rule/condition syntax, examples of trigger configuration, and independent/hands-on verification that these automations work as described.

   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”

   proof ↗⚑ flag

3. ai-native userVersion, review, and roll back my automations

   weight 1 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust documents versioned datasets and immutable, comparable experiment records, plus CLI/API management of prompts, scorers, and functions, which gives some version/review capability for its automation artifacts (evals, scorers, prompts). However, there is no explicit documentation of a rollback mechanism for automations (e.g., reverting a prompt, scorer, or online-scoring rule to a prior version) or a review/approval workflow before rollback. Missing for 10: explicit rollback functionality, version history UI/CLI commands, and independent confirmation that rollback works in practice.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   W&B Weave~partialclaimed3/10

   Weave documents automatic versioning of traced functions, datasets, and evaluation objects (weave.op(), Evaluation objects) and provides alert/webhook 'automations' for production signals, but there is no evidence of a review or rollback mechanism for these automations/versions. Missing for 10: explicit rollback UI/API for automations, version-history browsing/restore workflow, and evidence tying versioning to the alert/webhook automations themselves.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”

   proof ↗⚑ flag

Cost tracking

1. developerAttribute cost and usage to users, sessions, and features via custom metadata

   weight 2 · round to W&B Weave
   Braintrust~partialclaimed4/10

   Braintrust's tracing captures token usage and custom metadata for every request, which is the raw ingredient for cost attribution, but no evidence describes dashboards, filters, or reports that actually roll up cost/usage by user, session, or feature tags. missing for 10: explicit cost-attribution UI/reporting by custom metadata dimensions, user/session/feature breakdown examples, and independent confirmation this works in practice.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”

   proof ↗⚑ flag

   W&B Weave~partialclaimed5/10

   Weave captures call metadata via weave.op, tracks token usage/cost automatically or via custom cost entries, and groups related calls into Threads (sessions), which together enable some cost/usage attribution. However, there is no explicit documentation of tagging calls with custom user/feature metadata or aggregating/filtering cost by such tags. Missing for 10: explicit custom-attribute tagging API (e.g., user_id/feature tags) and evidence of cost rollups/dashboards filtered by those custom dimensions.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “Threads are collections of Traces related to a single session or conversation. You can use Threads to analyze or score the entire conversati…”
   • [claimed-docs] “Automatic cost tracking: For supported integrations, Weave captures token usage from the API response and applies built-in pricing for the m…”
   • [claimed-docs] “Add a custom cost with the `add_cost` method. The three required fields are `llm_id`, `prompt_token_cost`, and `completion_token_cost`.”
   • [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
   • [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
   • [claimed-docs] “Use custom cost tracking when automatic cost tracking isn’t available, such as with fine-tuned models, self-hosted models, or providers that…”

   proof ↗⚑ flag

2. developerSee cost and token usage per request, model, and time period in dashboards

   weight 3 · round to W&B Weave
   Braintrust~partialclaimed5/10

   Braintrust's tracing explicitly captures token usage per request (braintrust-docs-1) and traces/spans can be browsed in the UI or terminal (braintrust-docs-14/19/35), which supports basic usage visibility, but the evidence never mentions a cost metric, cost calculation, or a dedicated dashboard aggregating cost/tokens by model or time period. Missing for 10: explicit cost tracking/computation, dashboard views aggregating usage by model, dashboard views aggregating usage by time period, and any independent confirmation of these dashboard features.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`]”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed8/10

   Weave automatically tracks token usage and cost per call using built-in pricing for supported integrations, with costs shown in the trace tree and calls table in the Weave UI, plus custom cost support for unsupported models; calls table is filterable/aggregable by model and time via the trace UI. missing for 10: explicit documentation of pre-built cost dashboards aggregating by time period across the whole org, and independent/hands-on confirmation beyond vendor docs.

   • [claimed-docs] “Automatic cost tracking: For supported integrations, Weave captures token usage from the API response and applies built-in pricing for the m…”
   • [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
   • [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
   • [claimed-docs] “Use custom cost tracking when automatic cost tracking isn’t available, such as with fine-tuned models, self-hosted models, or providers that…”
   • [claimed-docs] “Add a custom cost with the `add_cost` method. The three required fields are `llm_id`, `prompt_token_cost`, and `completion_token_cost`.”

   proof ↗⚑ flag

Data export

1. developerBulk-export traces and datasets to blob storage or my data warehouse

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust supports exporting logs/traces via CLI (`bt sync pull`, download as CSV/JSON) and a full API for programmatic access to traces, datasets, and experiments, which developers could pipe into blob storage or a warehouse themselves. However, there is no documented native connector or bulk-export feature specifically targeting blob storage (S3/GCS) or data warehouses (Snowflake/BigQuery) — missing for 10: dedicated warehouse/blob-storage export integrations, scheduled/automated bulk export pipelines, and independent confirmation of large-scale export performance.

   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”

   proof ↗⚑ flag

   W&B Weave—none0/10

   Evidence shows Weave has a Service API for programmatic access and OTel import for bringing trace data in, but nothing documents bulk export of traces/datasets to blob storage (S3/GCS) or a data warehouse (Snowflake/BigQuery), which is a reasonable ask for an observability/eval platform.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”

   proof ↗⚑ flag

Ai eval ops

1. ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results

   weight 2 · round drawn
   Braintrust✓fullclaimed8/10

   Braintrust provides programmatic APIs and MCP/CLI tooling that let an agent create datasets, trigger evals, and read back results: the API supports creating/managing datasets and experiments, the SDK's Eval() function runs evals programmatically, and the MCP server/CLI explicitly let agents like Claude Code/Cursor 'run evals' and 'query logs' from within their workflow. missing for 10: a concrete end-to-end example showing an agent specifically creating a dataset via API/MCP and then reading back eval results in one flow, and independent (non-vendor) confirmation of this specific workflow.

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   W&B Weave✓fullprobed8/10

   Weave provides a programmatic Evaluation API with Dataset objects and scoring functions (docs-6, docs-19, docs-25, docs-31), a Service API with REST endpoints for programmatic access to results (docs-15), and an official MCP server explicitly enabling coding agents to 'read live production data, run evaluations, and execute automatic iteration loops on their own' (docs-20, probe-4) — directly matching the agent-driven create-dataset/trigger-eval/read-results workflow. Missing for 10: independent/hands-on confirmation of an agent autonomously completing this full loop end-to-end, and explicit example code showing dataset creation + eval trigger + result read-back in one flow.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Measure every improvement to your harnesses and models with confidence with a flexible imperative evaluation API. Weave provides powerful ev…”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”

   proof ↗⚑ flag

Human review

1. ml engineerRoute outputs to human annotation queues for review and labeling

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust supports building custom annotation interfaces for human review and labeling (docs-12) and datasets can be built/curated from production logs and manual curation (docs-5, docs-31), implying human review workflows, but there's no explicit evidence of a dedicated 'routing' mechanism, queue assignment, reviewer task distribution, or workflow status tracking for human annotators. missing for 10: explicit queue/routing mechanism to assign outputs to specific human reviewers, task assignment/status tracking, and independent corroboration of the annotation workflow in practice.

   • [claimed-docs] “Build annotation interfaces that match your team's workflow. Review support conversations differently than co”
   • [claimed-docs] “Build datasets from production logs, user feedback, manual curation, or generate them with Loop.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”

   proof ↗⚑ flag

   W&B Weave—none0/10

   Weave's evidence covers tracing, evaluation, scoring, cost tracking, and feedback collection, but there is no mention of routing outputs to human annotation/labeling queues or any human-in-the-loop review workflow tooling.

   ⚑ flag

Offline evals

1. developerRun evals in CI and gate deployments on their results

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Docs explicitly describe running evals via code/CLI, integrating into CI/CD to catch regressions, and gating on experiment results ('Automate in CI/CD — Run evals on every pull request to catch regressions'; 'integrate into CI/CD to catch regressions before they reach production'), backed by SDK eval code examples. Missing for 10: independent/hands-on confirmation of actual CI gating in production and details on failure-threshold/gate configuration.

   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”

   proof ↗⚑ flag

   W&B Weave~partialclaimed4/10

   Weave's imperative Evaluation API and Service API (REST) mean evals can be scripted and run programmatically, which a team could wire into a CI job, but the evidence never documents a CI/CD integration, pipeline templates, or a mechanism for gating/blocking deployments based on eval results. Missing for 10: explicit CI/CD integration guides (e.g., GitHub Actions), exit-code/threshold-based gating support, and any documented deployment-blocking workflow.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Measure every improvement to your harnesses and models with confidence with a flexible imperative evaluation API. Weave provides powerful ev…”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”

   proof ↗⚑ flag

2. ml engineerWrite custom code-based scorers and metrics for my evaluations

   weight 2 · round to W&B Weave
   Braintrust✓fullclaimed8/10

   Braintrust's SDK examples show Eval() calls with custom `scores` arrays (e.g. LevenshteinScorer), demonstrating code-based custom scorers, and docs describe managing/authoring scorers via API, CLI, and playgrounds. Missing for 10: independent hands-on validation of writing a fully custom scorer function beyond the built-in Levenshtein example, and more detail on scorer authoring API/signature.

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed9/10

   Weave's Evaluation object explicitly supports custom scoring functions, and dedicated docs on Custom Scorers describe encoding use-case-specific evaluation criteria beyond built-in scorers, backed by predefined scorers as a baseline. This directly matches writing code-based scorers/metrics for evaluations. Missing for 10: independent/hands-on corroboration beyond vendor docs.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”
   • [claimed-docs] “The core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples.”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [github] “Build rigorous, apples-to-apples evaluations for language model use cases”

   proof ↗⚑ flag

3. ml engineerCompare eval runs side by side to catch regressions between prompt or model versions

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed9/10

   Braintrust's Experiments feature is explicitly described as an immutable, comparable record of eval runs that tracks progress over time and integrates into CI/CD to catch regressions, and Playgrounds let users run full evaluations and compare results side-by-side across prompt/model configurations. This directly matches the story of comparing eval runs to catch regressions between versions. Missing for 10: independent/hands-on corroboration of the side-by-side comparison UI beyond first-party docs.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed8/10

   Weave's Evaluation object plus scorers explicitly support comparing runs over time to catch regressions, and docs state comparisons/visualizations exist to 'catch regressions before they reach users,' with leaderboards to aggregate and compare evaluations across versions. missing for 10: no independent/hands-on corroboration of side-by-side UI comparison workflow beyond vendor docs.

   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [claimed-docs] “Measure every improvement to your harnesses and models with confidence with a flexible imperative evaluation API. Weave provides powerful ev…”
   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”
   • [github] “Build rigorous, apples-to-apples evaluations for language model use cases”

   proof ↗⚑ flag

4. ml engineerScore outputs with configurable LLM-as-a-judge evaluators

   weight 3 · round to W&B Weave
   Braintrust✓fullclaimed8/10

   Braintrust's docs describe configurable scorers used within evals and playgrounds (LLM-as-a-judge style scorers), online scoring of production traces, and code examples show custom scorer configuration (e.g., LevenshteinScorer swapped in evals). Docs explicitly mention 'author scorers' via CLI/MCP and building scorers with Loop, supporting configurability. Missing for 10: explicit documentation naming 'LLM-as-a-judge' scorer templates/library and independent hands-on validation of judge configurability beyond vendor docs.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed9/10

   Weave provides a first-class Evaluation object with scoring functions, built-in LLM-judge scorers (hallucination, summarization quality, etc.), and explicit support for custom scorers to encode use-case-specific criteria, plus Guardrails pre-built safety/quality scorers. Missing for 10: independent/hands-on third-party corroboration beyond vendor docs.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”
   • [claimed-docs] “Weave comes with predefined scorers and local SLM scorers that you can use right away, including: Hallucination detection, Summarization qua…”
   • [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [github] “Build rigorous, apples-to-apples evaluations for language model use cases”

   proof ↗⚑ flag

5. ml engineerCurate datasets from production traces and run offline evaluations against them

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed9/10

   Docs explicitly describe building versioned datasets from production logs/traces (docs-5, docs-31) and running offline evaluations against them via Experiments, code, or Playgrounds (docs-27, docs-23, gh-1/gh-2 SDK examples), with CI/CD integration for regression testing (docs-2). Missing for 10: independent/hands-on third-party validation of the end-to-end curate-from-production-to-offline-eval workflow beyond first-party docs.

   • [claimed-docs] “Build datasets from production logs, user feedback, manual curation, or generate them with Loop.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed8/10

   Weave supports capturing production traces via @weave.op instrumentation, and explicitly supports building Datasets from these traces for use in its Evaluation object, which runs scoring functions/LLM judges against test examples; docs also mention testing new LLMs/custom models against production traces (offline evaluation). missing for 10: no explicit hands-on/independent example walking through 'export trace → dataset → evaluation' end-to-end, and no third-party corroboration of this specific workflow.

   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”
   • [claimed-docs] “Weave comes with predefined scorers and local SLM scorers that you can use right away, including: Hallucination detection, Summarization qua…”
   • [github] “Build rigorous, apples-to-apples evaluations for language model use cases”
   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”

   proof ↗⚑ flag

Online evals

1. ml engineerRun evaluators continuously on live production traffic, not just offline datasets

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust explicitly supports 'online scoring' that evaluates production traces automatically as they're logged asynchronously with no latency impact, plus continuous background analysis via Patterns/Topics scanning trace backlogs on a schedule for issues no scorer already catches. This directly matches continuous evaluation on live traffic beyond offline datasets. Missing for 10: independent/hands-on verification of online scoring behavior in production and more detail on configuring which evaluators run continuously.

   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Define the dimensions that matter to your business, like use case, customer segment, compliance, or tone. Topics continuously clusters every…”

   proof ↗⚑ flag

   W&B Weave~partialclaimed6/10

   Weave supports testing against production traces (docs-21), monitoring live agent interactions with signals/alerts (docs-16/17/28/36), and Guardrails scorers can presumably run on live traffic, plus custom/predefined scorers (docs-7,8,22). However, the core Evaluation workflow is explicitly framed around Datasets/lists of test examples run offline (docs-6, docs-31), and there's no explicit documentation of a continuous/streaming online-evaluation pipeline that automatically scores all live production calls in real time as they occur. Missing for 10: explicit documentation of automated/continuous scoring pipelines applied to every live production call (not just ad-hoc production trace sampling), and independent/hands-on confirmation of this online-evaluation mode.

   • [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”
   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
   • [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
   • [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
   • [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
   • [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Braintrust
   Braintrust~partialprobed7/10

   Braintrust offers a comprehensive API/SDK ('interact with all aspects of the Braintrust platform programmatically'), a full-featured CLI (bt) for auth, tracing, evals, log queries, and function management, and an MCP server letting agents query logs, author scorers, configure monitoring, and run evals — covering most UI functionality outside the terminal. However, some UI-only features like Playgrounds (explicitly 'no-code workspace') and annotation-interface building appear UI-centric without clear API/CLI parity, and no public OpenAPI spec was found to confirm full API coverage. missing for 10: confirmed OpenAPI/spec proving 1:1 API-UI parity, explicit API/CLI equivalent for Playgrounds and custom annotation UI workflows, independent verification of completeness.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   W&B Weave~partialprobed6/10

   Weave exposes a Service API with REST endpoints for programmatic access, plus SDK-level ops for tracing, evaluations, and cost tracking, and an OpenAPI spec is served, indicating broad API coverage. However, some UI-centric features (Playground model comparison/testing, leaderboards, Slack alert configuration) are documented mainly as UI workflows without explicit evidence that every one of these is fully API-exposed. missing for 10: explicit documentation confirming Playground, leaderboards, and alerting/webhook configuration are all fully controllable via the API/SDK rather than just the UI.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
   • [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
   • [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”
   • [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust provides data export via `bt sync pull`, `bt view logs` downloads as CSV/JSON, and a full API to programmatically extract projects, experiments, datasets, traces, and prompts, plus a self-hosting option for data control. However, there's no explicit documentation of full-account export/portability guarantees, no mention of open/standard interchange formats beyond CSV/JSON logs, and no independent confirmation of a clean 'leave' workflow. Missing for 10: comprehensive account-level export documentation, open-format guarantees for all entity types (prompts, scorers, configs), and third-party validation of export completeness.

   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   W&B Weave~partialclaimed3/10

   Weave documents a REST Service API for 'programmatic access to Weave functionality' and OTel-compatible trace import/export interoperability, which could theoretically be used to pull data out in an open format, but there is no explicit documentation of a bulk 'export all my data' feature or guidance for migrating off the platform entirely. missing for 10: explicit bulk export/download feature, documented open-format export (e.g., JSON/OTLP dump of all traces/evals), and any guidance for full data portability or platform exit.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
   • [claimed-docs] “Use this integration when you want to instrument your application with the OpenTelemetry standard and have those traces appear alongside you…”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Braintrust—none0/10

   Evidence shows only client SDK repos on GitHub (braintrust-sdk-javascript) and self-hosted deployment of infrastructure, but the core platform/UI is explicitly described as Braintrust-managed and not released under an open license; no evidence of the product's full source being open-licensed.

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   W&B Weave—none0/10

   The evidence pack confirms a public GitHub repository (github.com/wandb/weave) exists with descriptions of its tracing/evaluation code, but none of the citations mention an open-source license (e.g., Apache-2.0/MIT) or any licensing terms at all, so there is no evidence the source is available under an open license.

   • [github] “You can trace any function using `weave.op` - from api calls to OpenAI, Anthropic, Google AI Studio etc to generation calls from Hugging Fac…”
   • [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”
   • [github] “Log and debug language model inputs, outputs, and traces”
   • [github] “Build rigorous, apples-to-apples evaluations for language model use cases”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust documents an official self-hosted deployment option, but it explicitly separates data storage (self-hosted) from platform management (Braintrust-managed UI, auth, updates), meaning it's not a fully self-hostable core product. Missing for 10: evidence of fully self-hosting the entire platform (UI, control plane, auth) independent of Braintrust's managed services, and independent/community confirmation of self-hosting experience.

   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”

   proof ↗⚑ flag

   W&B Weave—none0/10

   Weave is documented as a hosted SaaS platform (weave.init() connecting to W&B's cloud) with no evidence pack mentions of a self-hosted or on-prem deployment option for the core Weave product itself; only W&B Models/Platform is known to have enterprise self-hosting but that's not evidenced here for Weave specifically.

   ⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust's self-hosting docs explicitly state it's designed to meet data residency requirements by keeping customer data within your own cloud account and region, giving users region/residency control — but only via full self-hosted deployment, not a simple region-selection toggle in the managed cloud offering. Missing for 10: evidence of region choice within the standard hosted/SaaS plan, list of specific supported regions, and compliance certifications tied to residency.

   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   W&B Weave—none0/10

   No evidence of region/residency data storage controls for Weave; the pack covers tracing, evaluation, cost tracking, and integrations only, with no mention of self-hosting, EU/US data residency, or region selection options. Missing for 10: any documentation of regional data storage, residency guarantees, or self-hosted/on-prem deployment options.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Braintrust—none0/10

   No evidence pack item addresses data-training opt-out, a no-training policy, or any statement about whether customer data is used to train AI models. Self-hosting docs address data residency/compliance but do not mention model training use. missing for 10: any explicit no-training-on-customer-data policy, opt-out controls, or terms-of-service statement about AI training use.

   ⚑ flag

   W&B Weave—none0/10

   No evidence in the pack addresses data usage policies, opt-out of training, or any privacy commitment regarding customer data being used to train models; the evidence pack focuses entirely on tracing, evaluation, and observability features. Missing for 10: any privacy policy statement, training opt-out mechanism, or data usage terms documentation.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round to Braintrust
   Braintrust~partialclaimed4/10

   Braintrust's self-hosted deployment option lets customers keep all data within their own cloud account/region for compliance and residency, giving some control over where data lives, and the API/CLI allow programmatic management of logs and datasets. However, there is no explicit documentation of a retention policy, TTL settings, or a delete/purge API for logs, traces, or datasets on the managed (non-self-hosted) offering. Missing for 10: explicit retention window configuration, an in-platform delete/purge mechanism for traces or datasets, and confirmation that these controls exist outside of self-hosting.

   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”

   proof ↗⚑ flag

   W&B Weave—none0/10

   The evidence pack covers tracing, evaluation, cost tracking, OTel import, and playground features, but there is no mention of data retention policies, deletion controls, or data lifecycle management for logged traces/data. missing for 10: documentation on data retention periods, user-initiated deletion of traces/projects/data, GDPR/CCPA compliance controls, or any retention configuration options.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Braintrust—none0/10

   No evidence of any telemetry opt-out or usage-tracking control setting for Braintrust itself as a tool; the evidence covers Braintrust's own tracing/observability product features (for user's AI apps), not the vendor's own telemetry collection practices or an opt-out mechanism. Self-hosting controls where customer AI data resides but says nothing about product telemetry/usage tracking opt-out.

   ⚑ flag

   W&B Weave—none0/10

   The evidence pack contains no mention of a telemetry opt-out, privacy settings, or usage-tracking controls for Weave itself; all evidence concerns tracing/evaluation features that Weave provides for users' LLM apps, not W&B's own telemetry collection. Since Weave is a SaaS-style observability tool where such an axis plausibly applies, absence of evidence yields 'none'.

   ⚑ flag

Prompt workflow

1. developerIterate on prompts in a playground against real models and variables

   weight 2 · round drawn
   Braintrust✓fullclaimed8/10

   Docs explicitly describe Playgrounds as a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets, running full evaluations in real-time against real models with variable inputs, and comparing results side-by-side (braintrust-docs-3, braintrust-docs-23, braintrust-docs-17). This directly matches the story of iterating on prompts in a playground against real models and variables. Missing for 10: independent/hands-on user corroboration of the playground experience specifically (community evidence is generic/mixed) and no detail on variable-substitution mechanics beyond the general description.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Measure and improve AI application quality with playgrounds and evals”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed8/10

   Weave's Playground explicitly supports prompt editing, message retrying, model comparison, and testing custom/OpenAI-compatible endpoints against real models, plus saved model presets for reusable variable configs, directly matching the story. Missing for 10: independent/hands-on corroboration beyond vendor docs, and explicit detail on templated variable substitution within prompts.

   • [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
   • [claimed-docs] “Custom runtimes: Test OpenAI-compatible API endpoints for custom models.”
   • [claimed-docs] “Saved models: Create and configure a reusable model preset for your workflow.”
   • [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”

   proof ↗⚑ flag

2. developerVersion prompts and deploy changes to production without shipping code

   weight 3 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust supports managing prompts via playgrounds, API, and CLI ("Manage prompts, tools, and scorers"), and datasets/experiments are versioned, suggesting prompt versioning is possible; however, there's no explicit evidence of a distinct 'deploy to production without code' workflow (e.g., prompt promotion/publish/rollback mechanism) separate from logging or evals. missing for 10: explicit prompt versioning/history UI, a documented deploy/publish/rollback mechanism for prompts, and independent confirmation that production apps can swap prompts without redeploying code.

   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”

   proof ↗⚑ flag

   W&B Weave—none0/10

   The evidence pack covers tracing, evaluation, cost tracking, and a Playground for prompt editing/model comparison, but nothing describes a prompt versioning/registry system or a mechanism to push prompt changes to production without redeploying code. Playground's 'prompt editing' (wandb-weave-docs-9) and 'Saved models' preset (wandb-weave-docs-11) are experimentation tools, not a production deployment/versioning workflow for prompts decoupled from code.

   • [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
   • [claimed-docs] “Saved models: Create and configure a reusable model preset for your workflow.”

   proof ↗⚑ flag

Ai observability

1. ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app

   weight 3 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust documents a first-party MCP server enabling MCP-compatible agents (Claude Code, Cursor, Codex, etc.) to query logs, run evals, author scorers, and configure monitoring, plus a full programmatic API for traces/experiments/datasets, and Loop, Braintrust's own AI agent, for natural-language investigation of logs and eval data. This directly matches the story of an agent querying traces, metrics, and eval results via API or MCP to debug an app. Missing for 10: independent hands-on corroboration of the MCP server working end-to-end (only vendor docs/probe confirm existence, no third-party review of actual debugging workflow).

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   W&B Weave✓fullprobed8/10

   Weave provides a documented Service API (REST) for programmatic access to traces/evals, plus an official MCP server (wandb-mcp-server) explicitly described as letting coding agents like Claude Code 'read live production data, run evaluations, and execute automatic iteration loops on their own.' This directly matches the story of an agent querying traces/metrics/evals to debug an app. missing for 10: independent/hands-on corroboration of the MCP server in real debugging workflows beyond vendor docs.

   • [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
   • [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
   • [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”

   proof ↗⚑ flag

Data controls

1. developerMask or redact sensitive data before it is stored in traces

   weight 2 · round drawn
   Braintrust—none0/10

   No evidence in the pack describes any PII masking, redaction, or scrubbing feature for traces before storage; documentation covers instrumentation, tracing, evals, and observability but never mentions data masking or redaction capabilities. This is a plausible and common axis for a tracing/observability platform, so absence of evidence yields 'none' rather than 'na'.

   ⚑ flag

   W&B Weave—none0/10

   The evidence describes tracing, evaluation, cost tracking, and PII *detection* via Guardrails scorers (wandb-weave-docs-22), but nothing about masking or redacting sensitive data before it is written into stored traces. This is a fair capability to expect from a tracing/instrumentation product, so absence of evidence means 'none' rather than 'na'.

   • [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “When you decorate a function with @weave.op() (Python) or wrap it with weave.op() (TypeScript), Weave automatically captures its code, input…”

   proof ↗⚑ flag

Sdk coverage

1. developerInstrument apps in both Python and JS/TS with officially supported SDKs

   weight 2 · round to W&B Weave
   Braintrust~partialclaimed6/10

   Docs confirm instrumentation/tracing is central to Braintrust and JS SDK usage is shown in GitHub examples (Eval function calls), with docs referencing SDK-based tracing and instrumentation broadly, but the evidence pack never explicitly confirms an officially supported Python SDK alongside JS/TS — only JS/TS SDK code is shown. Missing for 10: explicit Python SDK documentation/examples, side-by-side confirmation of parity between Python and JS/TS SDKs, and independent corroboration of both SDKs in production use.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed8/10

   Docs explicitly confirm both Python (@weave.op() decorator) and TypeScript (weave.op() wrap) SDK support for instrumenting functions and LLM calls, with consistent API design across languages. Missing for 10: independent/third-party corroboration of TS SDK parity and maturity, and more detail on JS/TS-specific setup/init beyond the single mention.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “When you decorate a function with @weave.op() (Python) or wrap it with weave.op() (TypeScript), Weave automatically captures its code, input…”
   • [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”
   • [claimed-docs] “Manually instrument your application’s LLM calls and arbitrary functions to trace, version, and collect feedback about your application”

   proof ↗⚑ flag

Trace capture

1. developerTrace multi-step agent runs as nested spans grouped into sessions or threads

   weight 2 · round to W&B Weave
   Braintrust~partialclaimed6/10

   Braintrust documents tracing that captures spans, logs, and full request detail, and lets users browse traces and individual spans via UI or CLI, supporting general span-based tracing. However, the evidence never explicitly describes nested multi-step spans grouped into sessions/threads for agent runs (no mention of parent-child span hierarchies or session/thread grouping semantics). Missing for 10: explicit documentation of nested span hierarchies for multi-step agent runs, explicit session/thread grouping constructs, and independent/hands-on confirmation of this specific structure.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed9/10

   Weave supports automatic nested span capture via @weave.op() producing trace trees, plus first-class grouping into Threads/sessions/turns/sub-agents for multi-step agent runs, explicitly designed to navigate agent sessions as executed. Missing for 10: independent hands-on corroboration beyond vendor docs.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “Threads are collections of Traces related to a single session or conversation. You can use Threads to analyze or score the entire conversati…”
   • [claimed-docs] “Weave now brings sessions, turns, steps, tools, and sub-agents as first-class concepts, making it much easier to navigate an agent session t…”
   • [claimed-docs] “Weave organizes traces into sessions and turns from the ground up.”
   • [claimed-docs] “Weave organizes traces into sessions and turns from the ground up. That structure, paired with native analytics tools, makes it easy to trac…”
   • [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”

   proof ↗⚑ flag

2. developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK

   weight 2 · round to W&B Weave
   Braintrust~partialcommunity4/10

   Docs confirm a general instrumentation capability and state you can 'integrate with AI providers and frameworks to send traces to Braintrust,' but no evidence explicitly names LangChain, the OpenAI SDK, or the Vercel AI SDK as supported integrations. A community comment even calls integration options 'limited.' Missing for 10: explicit documentation or hands-on confirmation of dedicated LangChain, OpenAI SDK, and Vercel AI SDK integrations.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [community] “Braintrust: Interesting approach to evaluation but feels like an early-stage product. Documentation is sparse and integration options are li…”

   proof ↗⚑ flag

   W&B Weave~partialclaimed5/10

   Weave clearly supports auto-instrumentation for OpenAI (and Anthropic, Cohere, Mistral, Hugging Face) via weave.op() decorators/wrappers and has a TypeScript wrap() function, satisfying the OpenAI-SDK part of the story, and it also supports generic OTel-based instrumentation for 'popular SDKs and harnesses.' However, no evidence pack citation explicitly names a LangChain integration or a Vercel AI SDK integration, so those specific framework integrations are unconfirmed. missing for 10: explicit documentation of a first-party LangChain integration, explicit documentation of a Vercel AI SDK integration.

   • [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
   • [github] “You can trace any function using `weave.op` - from api calls to OpenAI, Anthropic, Google AI Studio etc to generation calls from Hugging Fac…”
   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “Trace and collect metrics about your agents built with popular SDKs and harnesses using Weave OTel-compatible SDK”
   • [claimed-docs] “Use this integration when you want to instrument your application with the OpenTelemetry standard and have those traces appear alongside you…”

   proof ↗⚑ flag

3. developerCapture multimodal payloads (images, audio, files) inside my traces

   weight 1 · round drawn
   Braintrust—none0/10

   The evidence pack describes tracing capturing inputs, outputs, model parameters, latency, token usage, and metadata, but there is no explicit mention of multimodal payload support (images, audio, files) within traces. Missing for 10: explicit documentation or examples showing image/audio/file capture in traces, SDK support for multimodal attachments, or any UI rendering of multimodal content in trace spans.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”

   proof ↗⚑ flag

   W&B Weave—none0/10

   The evidence describes Weave's tracing capturing function inputs/outputs, code, and metadata via @weave.op(), but none of the docs or GitHub excerpts mention support for images, audio, or file attachments within traces. Missing for 10: any explicit mention of multimodal payload types (images, audio, files) being captured, rendered, or stored in trace data.

   ⚑ flag

4. developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format

   weight 3 · round to W&B Weave
   Braintrust—none0/10

   No evidence pack item mentions OpenTelemetry or OTLP support; Braintrust's docs describe its own SDK/instrumentation and integrations but never reference OTLP-based trace ingestion/export. Missing for 10: any mention of OTLP endpoint, OpenTelemetry SDK compatibility, or OTel collector integration.

   ⚑ flag

   W&B Weave~partialclaimed7/10

   Weave documents a dedicated OTLP import endpoint and an OTel-compatible SDK so external OpenTelemetry traces can be sent in and appear alongside native Weave traces, rather than requiring the proprietary weave.op format exclusively. However, this is framed as import-only alongside existing pipelines, not a full bidirectional replace-your-format OTLP transport, and there's no evidence of receiving/exporting traces out via OTLP or independent corroboration of interoperability. Missing for 10: evidence of OTLP export/round-trip, independent hands-on confirmation, and clarity that OTel is a full alternative rather than a supplementary ingestion path.

   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
   • [claimed-docs] “Trace and collect metrics about your agents built with popular SDKs and harnesses using Weave OTel-compatible SDK”
   • [claimed-docs] “Use this integration when you want to instrument your application with the OpenTelemetry standard and have those traces appear alongside you…”

   proof ↗⚑ flag

5. developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage

   weight 3 · round to W&B Weave
   Braintrust✓fullclaimed8/10

   Braintrust docs explicitly state instrumentation captures inputs, outputs, model parameters, latency, token usage, and metadata for every LLM call, with SDK examples and CLI/UI browsing of traces and spans. missing for 10: independent hands-on verification of exact captured fields and no third-party benchmark of tracing accuracy/performance overhead.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”

   proof ↗⚑ flag

   W&B Weave✓fullclaimed9/10

   Weave's @weave.op() decorator automatically captures code, inputs, outputs, and execution metadata for LLM calls, with automatic token usage and cost tracking recorded per call and displayed in the trace tree/UI. Latency is inherently part of the captured trace/execution metadata; OTel-compatible import and GitHub docs corroborate first-party and independent-style evidence. Missing for 10: explicit standalone documentation calling out latency capture by name, and independent (non-vendor) hands-on validation.

   • [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
   • [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
   • [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
   • [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”
   • [github] “Log and debug language model inputs, outputs, and traces”
   • [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”

   proof ↗⚑ flag