Braintrust vs Cekura

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Cekura
   Braintrust~partialprobed6/10

   A live probe confirms llms.txt exists at braintrust.dev/llms.txt returning 200 with a concise agent-oriented description, directly satisfying the story's core requirement. However, docs.md and OpenAPI/machine-readable doc endpoints return 404, indicating incomplete agent-oriented documentation coverage beyond the single llms.txt file. missing for 10: full docs-as-markdown or OpenAPI endpoints for deeper agent consumption, independent confirmation that agents successfully use llms.txt in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://www.braintrust.dev/llms.txt # Braintrust > Braintrust is the active observability platform for instrume…”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.braintrust.dev/docs.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Cekura✓fullprobed9/10

   Cekura has a confirmed live llms.txt endpoint (HTTP 200) plus per-page .md documentation format and an index pointing agents to llms.txt, directly enabling agent-oriented doc consumption; this is corroborated by both docs and live probes. Missing for 10: independent third-party confirmation of an agent actually consuming llms.txt successfully in the wild.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”
   • [probe] “PROBE docs-md: HTTP 200 at https://docs.cekura.ai/documentation/introduction.md > ## Documentation Index > Fetch the complete documentation …”
   • [claimed-docs] “create a comprehensive plan to integrate cekura observability”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust supports headless/CI usage via code-based Eval() runs, an API for programmatic control, a CLI (`bt`) for auth/eval/log operations from the terminal, and explicit documentation of CI/CD integration to run evals on every pull request. This combination clearly enables automated, non-interactive operation in pipelines. Missing for 10: independent hands-on confirmation of CI pipeline usage beyond vendor docs.

   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Cekura✓fullprobed8/10

   Cekura documents a CLI/SDK for programmatic and terminal-based agent/scenario/run management, a REST API with OpenAPI spec and API keys, cron job scheduling for automated evaluation workflows, and explicit GitHub Actions CI/CD integration for testing agents on every change—together these clearly support headless/CI automation. Missing for 10: independent hands-on confirmation of the GitHub Actions or cron setup actually working in a real pipeline beyond docs.

   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
   • [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
   • [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
   • [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round drawn
   Braintrust—none0/10

   All MCP evidence describes Braintrust exposing an MCP server that other clients (Claude Code, Cursor, Codex) connect to in order to use Braintrust's tools — the reverse of this story, which asks whether Braintrust can consume external MCP servers' tools. No evidence shows Braintrust or its Loop agent acting as an MCP client that plugs into third-party MCP servers.

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”
   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Loop is Braintrust's AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”

   proof ↗⚑ flag

   Cekura—none0/10

   All Cekura MCP evidence describes Cekura exposing its own MCP server so external AI assistants can call Cekura's tools (docs-3, probe-4), not Cekura itself acting as an MCP client that consumes third-party MCP servers' tools. No evidence shows Cekura can plug in and use external MCP servers.

   • [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
   • [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round drawn
   Braintrust✓fullprobed8/10

   Braintrust documents an official MCP server that lets Claude Code, Cursor, Codex, VS Code, and other MCP-compatible clients query logs, author prompts/scorers, configure monitoring, and run evals — a clear server-side MCP offering that agents can connect to. Missing for 10: independent/hands-on third-party confirmation that the MCP server works as documented (only first-party docs and a probe confirm its existence).

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   Cekura✓fullprobed8/10

   Cekura is not itself an agent but a testing/observability platform, so the axis applies; it documents an official MCP server ("Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations") confirmed live via probe. Missing for 10: independent hands-on verification of the MCP server working end-to-end and more detail on supported tool/resource surface.

   • [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
   • [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust ships an official `bt` CLI, documented with a dedicated quickstart, allowing authentication, tracing coding-agent sessions, running evals, browsing/querying logs, syncing data, and managing functions from the terminal — directly matching AI-native/agentic CLI usage. Missing for 10: independent/hands-on community corroboration of the CLI itself (community evidence is generic, not CLI-specific).

   • [claimed-docs] “Install the `bt` CLI to set up, instrument, and run Braintrust with your coding agent”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Cekura✓fullprobed8/10

   Cekura documents an official CLI ('Manage agents, scenarios, runs, and call data from your terminal') alongside an SDK, confirmed by both docs and probe evidence. Missing for 10: independent hands-on review of the CLI's usability/completeness beyond first-party docs.

   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
   • [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to Cekura
   Braintrust✓fullprobed8/10

   Braintrust documents a full public API reference for managing projects, experiments, datasets, prompts, scorers, and logging programmatically (braintrust-docs-11, -34), backed by SDKs (braintrust-gh-1/2), a CLI (braintrust-docs-33), and MCP server for agent-driven access (braintrust-docs-9, -25). Missing for 10: no discoverable OpenAPI/swagger spec (braintrust-probe-3 shows 404s) and no independent third-party corroboration of API robustness beyond docs.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”

   proof ↗⚑ flag

   Cekura✓fullprobed9/10

   Cekura publishes a documented public API with OpenAPI spec (probe confirms live openapi.json), API key auth, dedicated API reference endpoints (e.g. send-calls), plus SDKs (sync/async) and a CLI for programmatic/terminal control—clearly enabling an AI-native user to drive the product via documented API access. Missing for 10: independent third-party review/hands-on validation of the API beyond vendor docs and probes.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
   • [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to Cekura
   Braintrust—none0/10

   The evidence describes Braintrust's general API, CLI, and MCP integrations but contains no mention of scoped, role-based, or least-privilege API key/credential issuance for agents; the only security-related item is a breach report telling customers to rotate keys, which does not demonstrate a scoping/least-privilege capability.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [community] “AI evaluation startup Braintrust confirms breach, tells every customer to rotate sensitive keys.”

   proof ↗⚑ flag

   Cekura~partialclaimed4/10

   Cekura documents API key creation restricted to project-level scope, showing some credential scoping capability, but there's no evidence of finer-grained least-privilege controls like read/write permission tiers, resource-specific scoping, or per-agent credential issuance. missing for 10: granular permission scopes (read-only vs write), per-agent or per-resource credential issuance, role-based access control documentation.

   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round drawn
   Braintrust✓fullprobed8/10

   Braintrust ships official JS SDK (github examples using Eval()), an API reference for programmatic access, a CLI (bt), and an MCP server, giving AI-native developers multiple official, documented ways to build against the platform. One community comment calls docs 'sparse' but this is general skepticism, not a concrete contradiction of SDK functionality. missing for 10: independent hands-on validation of the SDK itself (beyond vendor docs/examples) and a public OpenAPI spec (probe found 404s).

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”
   • [probe] “official CLI documented at https://www.braintrust.dev/docs/reference/cli/quickstart”
   • [community] “Braintrust: Interesting approach to evaluation but feels like an early-stage product. Documentation is sparse and integration options are li…”

   proof ↗⚑ flag

   Cekura✓fullprobed8/10

   Cekura documents official sync/async SDKs and a CLI for programmatic access ('Manage agents, scenarios, runs, and call data from your terminal', 'sync and async clients for application code'), plus an OpenAPI spec, backed by probe confirmation of live docs and endpoints. missing for 10: independent hands-on developer reports specifically about SDK usage (only community praise for the platform generally, not SDK-specific).

   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
   • [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Braintrust—none0/10

   No evidence in the pack mentions webhooks or any event-subscription mechanism; Braintrust's documented interfaces are API, CLI, MCP server, and UI, none of which are shown to support webhook subscriptions.

   ⚑ flag

   Cekura—none0/10

   Evidence shows Cekura receiving inbound webhooks (agent's provider posts call data to Cekura) but nothing about Cekura emitting outbound webhooks that a user could subscribe to for events like evaluation completion or drift detection. Missing for 10: documented outbound webhook/event subscription mechanism, webhook configuration UI/API, event types list.

   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust ships Loop, an AI agent that investigates logs via natural language and can build scorers/datasets/dashboards, plus Patterns which proactively scans trace backlogs for recurring issues and suggests fixes, and Topics which auto-clusters/classifies traces against business dimensions — all directly generating AI insights/suggestions inside the product. Missing for 10: independent/hands-on validation of Loop/Patterns quality (evidence is vendor docs only) and no third-party review confirming suggestion usefulness.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Define the dimensions that matter to your business, like use case, customer segment, compliance, or tone. Topics continuously clusters every…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review. Each trace is analyzed by facets that extract sho”

   proof ↗⚑ flag

   Cekura~partialclaimed6/10

   Cekura provides LLM-judge evaluation metrics, live drift detection, and auto-generated fix suggestions ('flags issues → reproduces in simulation → suggests fixes automatically'), showing AI-generated insights derived from call/agent data. However, this is narrowly scoped to voice-agent testing/observability rather than broad, ad-hoc AI-generated insights across arbitrary product data, and there's no independent hands-on evidence of the 'suggests fixes automatically' claim beyond marketing copy. Missing for 10: independent/hands-on verification of the auto-fix-suggestion feature, broader insight/analytics generation beyond voice-agent metrics, and detail on how suggestions are surfaced to users.

   • [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
   • [claimed-docs] “Live drift detection across every call.”
   • [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
   • [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round drawn
   Braintrust~partialclaimed6/10

   Braintrust supports autonomous background automation via 'Patterns', which runs Loop (its AI agent) on a schedule against the trace backlog to surface recurring issues and suggested fixes, plus online scoring that runs asynchronously on production traces and CI/CD-triggered evals. However, this is narrower than general-purpose autonomous automations — it's scoped to trace analysis/scoring rather than arbitrary user-defined background tasks, and there's no independent/hands-on corroboration of Patterns actually running unattended. Missing for 10: broader/customizable scheduled automation beyond Patterns and online scoring, independent evidence of reliability, and detail on failure handling or configurability of these background jobs.

   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”

   proof ↗⚑ flag

   Cekura~partialclaimed6/10

   Cekura supports autonomous background automation via cron jobs for scheduled testing/evaluation runs, GitHub Actions CI/CD integration, and live drift detection across calls, plus automatic flagging/reproduction/fix-suggestion pipelines. However, evidence is entirely first-party docs describing scheduling/triggering mechanisms rather than a broad general-purpose 'automation builder' for arbitrary agentic workflows. missing for 10: independent/hands-on confirmation that cron/CI automations run reliably unattended, evidence of more flexible trigger types beyond cron/CI/webhooks, and detail on monitoring/error-handling for long-running autonomous automations.

   • [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
   • [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
   • [claimed-docs] “Live drift detection across every call.”
   • [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”

   proof ↗⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed7/10

   Braintrust ships "Loop," a built-in AI agent that can investigate logs via natural language, build scorers/datasets/dashboards, and "Patterns" which runs Loop automatically on trace backlogs to surface issues and fixes — a clear built-in AI assistant delegated to perform tasks within the product. missing for 10: independent/hands-on user validation of Loop's task delegation (community evidence is thin/mixed and doesn't specifically address Loop), and more detail on the scope/limits of tasks it can autonomously complete.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Loop is Braintrust's AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”

   proof ↗⚑ flag

   Cekura—none0/10

   Cekura's AI-assistant integrations (Skills, MCP, CLI) are designed so external AI assistants like Claude or Cursor can drive Cekura's testing/evaluation features — this is the reverse relationship of an AI-native user delegating tasks to a built-in assistant inside Cekura itself. The 'flags issues → reproduces → suggests fixes automatically' pipeline is an automated feature, not a delegatable in-product assistant, so there is no evidence of a built-in AI assistant you can converse with or delegate open-ended tasks to.

   • [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
   • [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
   • [claimed-docs] “npx skills add cekura-ai/cekura-skills --all”

   proof ↗⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round to Braintrust
   Braintrust✓fullprobed8/10

   Braintrust's Loop agent lets users investigate logs and build scorers/datasets/dashboards via natural language instead of writing queries, and this is exposed through an official MCP server so natural-language commands work from Claude Code, Cursor, Codex, etc. Missing for 10: independent hands-on validation of natural-language command accuracy/reliability and broader coverage beyond Loop's stated scope.

   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Loop is Braintrust's AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   Cekura~partialprobed6/10

   Cekura ships an MCP server and Skills package so an AI assistant can design, run, and improve voice-agent evaluations via natural-language commands, plus an LLM-judge metric for natural-language evaluation criteria; this supports agentic/NL operation for a subset of workflows. missing for 10: independent hands-on evidence of natural-language command execution beyond vendor docs, and confirmation that the full breadth of platform actions (not just evaluation/testing tasks) can be driven purely by NL commands.

   • [claimed-docs] “npx skills add cekura-ai/cekura-skills --all”
   • [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
   • [github] “13 Skills, 14 Commands in one plugin”
   • [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
   • [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round to Cekura
   Braintrust—none0/10

   Braintrust has an API reference page (braintrust-docs-11/34) but the evidence pack shows no interactive or runnable-example capability — probes for an OpenAPI/swagger spec (needed for typical interactive API explorers) all returned 404, and there's no mention of a try-it-now console or embedded runnable snippets in the API reference itself. missing for 10: evidence of an interactive API console, runnable code snippets embedded in the reference, or a machine-readable OpenAPI spec powering such interactivity.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.braintrust.dev/docs.md”

   proof ↗⚑ flag

   Cekura~partialprobed4/10

   Cekura publishes an OpenAPI spec and structured API-reference docs (e.g., send-calls, create-api-key endpoints) confirming a machine-readable interactive reference exists, but no evidence shows runnable 'try it' examples or a sandboxed console in the docs. Missing for 10: explicit runnable/try-it example UI, evidence of live request execution, independent confirmation of interactivity.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round to Cekura
   Braintrust—none0/10

   Braintrust documents a REST API (api-reference) but explicit probes for OpenAPI/swagger specs at all standard paths returned 404, and no docs mention a downloadable machine-readable spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Cekura✓fullprobed9/10

   A direct probe confirms an OpenAPI spec is publicly downloadable at https://docs.cekura.ai/openapi.json, returning HTTP 200 with a valid "openapi" key, corroborated by API reference docs built from it. Missing for 10: no independent third-party confirmation of spec completeness/versioning beyond the probe.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Cekura
   Braintrust~partialclaimed4/10

   Braintrust supports evals/experiments/datasets and separate projects that could function as a sandbox for testing without hitting production logs, and self-hosting offers infrastructure isolation, but there is no explicit documentation of a dedicated 'sandbox environment' or isolation guarantee preventing test runs from touching production data. missing for 10: explicit sandbox/staging environment concept, documentation on isolating test evals from production datasets/logs, and independent confirmation that eval runs never write to production stores.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”

   proof ↗⚑ flag

   Cekura~partialclaimed5/10

   Cekura explicitly supports testing voice agents via synthetic/simulated conversations 'before go-live' and running scenarios without external API keys, which implies a sandbox-like testing flow separate from live production calls. However, there is no explicit documentation of a dedicated sandbox environment, data isolation guarantees, or explicit statements that test runs never touch production data/logs. missing for 10: explicit sandbox/staging environment documentation, data isolation guarantees between test and production calls, independent confirmation that simulated runs are fully separated from production data.

   • [claimed-docs] “Learn how to quickly create and test voice agents on Cekura without requiring any external API keys or integrations.”
   • [claimed-docs] “Run thousands of synthetic conversations before go-live.”
   • [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Braintrust—none0/10

   There is an API reference documenting functionality, but no evidence of API versioning scheme or any documented deprecation policy; probes even show no OpenAPI spec discoverable at standard locations. Missing for 10: versioning scheme documentation, explicit deprecation policy, changelog/migration guides.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Cekura—none0/10

   There is an OpenAPI spec and API-key auth docs, but no evidence of API versioning scheme or a documented deprecation policy anywhere in the pack. missing for 10: versioning scheme documentation, deprecation policy, changelog/migration guides.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”

   proof ↗⚑ flag

Monitoring

1. ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics

   weight 2 · round to Cekura
   Braintrust~partialclaimed5/10

   Braintrust captures latency, token/cost, and eval-score metrics via tracing and experiments (braintrust-docs-1, braintrust-docs-27), and one doc line explicitly says you can 'ask Braintrust's AI agent to investigate your data and build scorers, datasets, and dashboards' (braintrust-docs-18), implying custom dashboard creation. However there is no dedicated dashboard-building documentation, no detail on customizing charts/widgets, and no explicit mention of error-rate metrics or alerting tied to dashboards. Missing for 10: dedicated dashboard-configuration docs, explicit error-metric support, and independent/hands-on evidence of building a custom dashboard.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”

   proof ↗⚑ flag

   Cekura~partialclaimed6/10

   Cekura docs explicitly describe building custom dashboards with widgets to visualize call data, metrics, and metadata, which covers eval-score and general metric visualization for an ML engineer's needs. However, the docs don't explicitly enumerate latency, error rate, or cost as first-class dashboard metrics, and there's no independent/hands-on corroboration of dashboard flexibility. Missing for 10: explicit mention of latency/error/cost metric widgets, and third-party evidence of dashboard customization in practice.

   • [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”

   proof ↗⚑ flag

2. developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email

   weight 2 · round drawn
   Braintrust—none0/10

   The evidence pack covers tracing, evals, online scoring, Loop/Patterns/Topics analysis, and self-hosting, but contains no mention of configurable alerts, thresholds, or notification integrations (Slack, PagerDuty, email) for error rates, cost spikes, or eval-score drops.

   ⚑ flag

   Cekura—none0/10

   Evidence covers dashboards, metrics, PII redaction, CI/CD, and cron jobs but no mention of alerting/notification integrations for error rates, cost spikes, or eval-score drops, nor any Slack/PagerDuty/email notification channels.

   ⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed7/10

   Braintrust evals run against entire datasets in one operation, Patterns scans the full trace backlog on a schedule, Topics classifies every trace automatically, and the CLI/API support bulk sync ('bt sync pull'), bulk log queries, and programmatic management of experiments/datasets/prompts at scale. Missing for 10: explicit documentation of bulk edit/delete/update operations on datasets or logs, and independent (non-vendor) confirmation of bulk-operation performance at scale.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Cekura~partialclaimed6/10

   Cekura's docs advertise running thousands of synthetic conversations/scenarios at once and provide a CLI/SDK to manage agents, scenarios, runs, and call data programmatically, which supports bulk test execution and automation via cron/CI. However, there's no explicit documentation of bulk edit/delete/update operations across many existing items (e.g., batch API endpoints for updating many scenarios or calls at once). Missing for 10: explicit bulk CRUD API/CLI commands operating on many items simultaneously, and any hands-on evidence of large-scale batch throughput.

   • [claimed-docs] “Run thousands of synthetic conversations before go-live.”
   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
   • [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
   • [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Cekura
   Braintrust~partialclaimed5/10

   Braintrust supports several automatic, event-driven behaviors — online scoring runs automatically on every new production trace, evals run automatically in CI/CD on pull requests, and Patterns runs Loop against traces on a schedule — but these are fixed built-in automations rather than a general user-defined 'if event then action' rules engine (e.g., no evidence of custom triggers/webhooks/alerts tied to arbitrary conditions). Missing for 10: a documented rules/alerting framework letting users define arbitrary trigger conditions and custom actions, and independent confirmation of this automation depth.

   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”

   proof ↗⚑ flag

   Cekura~partialclaimed6/10

   Cekura supports automation triggers like scheduled cron jobs for test workflows, GitHub Actions CI/CD hooks that fire tests on code changes, and an automatic flag→reproduce→suggest-fix pipeline triggered by detected issues, which are event-driven automations. However, there's no evidence of a general-purpose user-defined rules engine (e.g., 'if X metric drops, then do Y') beyond these fixed automation patterns. Missing for 10: a documented custom rule/condition builder for arbitrary events, and independent confirmation these automations behave as advertised.

   • [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
   • [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
   • [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
   • [claimed-docs] “Live drift detection across every call.”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round to Cekura
   Braintrust~partialclaimed5/10

   Braintrust documents 'Patterns,' which runs Loop on the trace backlog 'on a schedule' to find recurring issues, and separately supports triggering evals via CI/CD — both are forms of recurring automation. However, there's no evidence of general-purpose, user-configurable recurring job/workflow scheduling (e.g., cron-like scheduling for arbitrary evals, datasets, or pipelines) beyond this specific built-in feature. Missing for 10: user-defined scheduling API/UI for arbitrary evals or workflows, documentation of schedule configuration options, and independent confirmation of reliability.

   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”

   proof ↗⚑ flag

   Cekura✓fullclaimed7/10

   Cekura documents dedicated cron job support for scheduling automated testing/evaluation workflows (cekura-docs-13), plus GitHub Actions integration for triggering workflows on changes (cekura-docs-14), giving AI-native users concrete recurring automation options via CLI/SDK/API. Missing for 10: independent/hands-on confirmation that cron scheduling works reliably in production and more detail on scheduling granularity/limits.

   • [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
   • [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”

   proof ↗⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust documents versioned datasets and immutable, comparable experiment records, plus CLI/API management of prompts, scorers, and functions, which gives some version/review capability for its automation artifacts (evals, scorers, prompts). However, there is no explicit documentation of a rollback mechanism for automations (e.g., reverting a prompt, scorer, or online-scoring rule to a prior version) or a review/approval workflow before rollback. Missing for 10: explicit rollback functionality, version history UI/CLI commands, and independent confirmation that rollback works in practice.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Cekura~partialclaimed3/10

   Cekura supports comparing two versions of an agent side-by-side (A/B testing) and provides dashboards/metrics for reviewing agent performance, which touches 'version' and 'review' aspects, but there is no evidence of a rollback mechanism to revert an automation/agent to a prior version. Missing for 10: explicit version history/versioning system, rollback/undo capability, and any documented workflow for reverting agent configurations.

   • [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
   • [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”

   proof ↗⚑ flag

Cost tracking

1. developerAttribute cost and usage to users, sessions, and features via custom metadata

   weight 2 · round to Braintrust
   Braintrust~partialclaimed4/10

   Braintrust's tracing captures token usage and custom metadata for every request, which is the raw ingredient for cost attribution, but no evidence describes dashboards, filters, or reports that actually roll up cost/usage by user, session, or feature tags. missing for 10: explicit cost-attribution UI/reporting by custom metadata dimensions, user/session/feature breakdown examples, and independent confirmation this works in practice.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”

   proof ↗⚑ flag

   Cekura—none0/10

   Cekura's docs show metadata being sent with calls and custom dashboards for visualizing metrics/metadata, but there is no evidence of cost tracking, cost attribution, or usage-cost breakdown by user/session/feature — the platform is framed purely around voice-agent evaluation and observability, not cost accounting.

   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”

   proof ↗⚑ flag

2. developerSee cost and token usage per request, model, and time period in dashboards

   weight 3 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust's tracing explicitly captures token usage per request (braintrust-docs-1) and traces/spans can be browsed in the UI or terminal (braintrust-docs-14/19/35), which supports basic usage visibility, but the evidence never mentions a cost metric, cost calculation, or a dedicated dashboard aggregating cost/tokens by model or time period. Missing for 10: explicit cost tracking/computation, dashboard views aggregating usage by model, dashboard views aggregating usage by time period, and any independent confirmation of these dashboard features.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`]”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”

   proof ↗⚑ flag

   Cekura—none0/10

   Cekura documents customizable dashboards for call data, metrics, and metadata (cekura-docs-11), but nothing in the evidence pack mentions cost or token usage tracking per request, model, or time period. As an observability platform for voice agents, this axis plausibly applies, but no evidence supports the specific cost/token dashboard capability.

   ⚑ flag

Data export

1. developerBulk-export traces and datasets to blob storage or my data warehouse

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust supports exporting logs/traces via CLI (`bt sync pull`, download as CSV/JSON) and a full API for programmatic access to traces, datasets, and experiments, which developers could pipe into blob storage or a warehouse themselves. However, there is no documented native connector or bulk-export feature specifically targeting blob storage (S3/GCS) or data warehouses (Snowflake/BigQuery) — missing for 10: dedicated warehouse/blob-storage export integrations, scheduled/automated bulk export pipelines, and independent confirmation of large-scale export performance.

   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”

   proof ↗⚑ flag

   Cekura—none0/10

   Evidence shows Cekura has an API, CLI, SDK, and dashboards for accessing call data, but there is no mention of bulk-export functionality to blob storage (S3/GCS) or data warehouse integrations (Snowflake, BigQuery, etc). Missing for 10: bulk export feature, blob storage integration, data warehouse connectors/sinks, any documentation or API endpoint for exporting datasets in bulk.

   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
   • [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”

   proof ↗⚑ flag

Ai eval ops

1. ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results

   weight 2 · round drawn
   Braintrust✓fullclaimed8/10

   Braintrust provides programmatic APIs and MCP/CLI tooling that let an agent create datasets, trigger evals, and read back results: the API supports creating/managing datasets and experiments, the SDK's Eval() function runs evals programmatically, and the MCP server/CLI explicitly let agents like Claude Code/Cursor 'run evals' and 'query logs' from within their workflow. missing for 10: a concrete end-to-end example showing an agent specifically creating a dataset via API/MCP and then reading back eval results in one flow, and independent (non-vendor) confirmation of this specific workflow.

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Cekura✓fullprobed8/10

   Cekura ships a CLI/SDK explicitly for managing agents, scenarios (datasets), runs, and call data from code, plus an official MCP server letting an AI assistant 'design, run, and improve' evaluations, backed by a documented OpenAPI spec and API-key auth for programmatic access — together covering dataset creation, triggering eval runs, and reading back results. missing for 10: independent/hands-on confirmation that an agent using the MCP/CLI actually completes the full create-dataset-to-trigger-run-to-read-results loop end-to-end.

   • [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
   • [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”

   proof ↗⚑ flag

Human review

1. ml engineerRoute outputs to human annotation queues for review and labeling

   weight 2 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust supports building custom annotation interfaces for human review and labeling (docs-12) and datasets can be built/curated from production logs and manual curation (docs-5, docs-31), implying human review workflows, but there's no explicit evidence of a dedicated 'routing' mechanism, queue assignment, reviewer task distribution, or workflow status tracking for human annotators. missing for 10: explicit queue/routing mechanism to assign outputs to specific human reviewers, task assignment/status tracking, and independent corroboration of the annotation workflow in practice.

   • [claimed-docs] “Build annotation interfaces that match your team's workflow. Review support conversations differently than co”
   • [claimed-docs] “Build datasets from production logs, user feedback, manual curation, or generate them with Loop.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”

   proof ↗⚑ flag

   Cekura—none0/10

   Cekura's evidence pack covers voice-agent testing, evaluation metrics (LLM-judge, Python), dashboards, PII redaction, and observability, but there is no mention of routing outputs to human annotation queues, labeling workflows, or human-in-the-loop review pipelines. Absence of evidence for this applicable ML-eval capability yields none. Missing for 10: any mention of human annotation queues, labeling UI/workflow, or reviewer assignment features.

   ⚑ flag

Offline evals

1. developerRun evals in CI and gate deployments on their results

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Docs explicitly describe running evals via code/CLI, integrating into CI/CD to catch regressions, and gating on experiment results ('Automate in CI/CD — Run evals on every pull request to catch regressions'; 'integrate into CI/CD to catch regressions before they reach production'), backed by SDK eval code examples. Missing for 10: independent/hands-on confirmation of actual CI gating in production and details on failure-threshold/gate configuration.

   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”

   proof ↗⚑ flag

   Cekura~partialclaimed6/10

   Cekura provides a dedicated GitHub Actions CI/CD guide plus CLI/SDK and cron job automation for testing agents on each change, indicating CI integration is a first-class capability. However, the evidence pack only shows the guide's title/description, not concrete details on exit-code/gating behavior for deployments, and there is no independent corroboration of this specific workflow. Missing for 10: explicit documentation of deployment-gating logic (pass/fail thresholds blocking merges/deploys), and independent/hands-on confirmation of the GitHub Actions integration.

   • [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
   • [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”

   proof ↗⚑ flag

2. ml engineerWrite custom code-based scorers and metrics for my evaluations

   weight 2 · round drawn
   Braintrust✓fullclaimed8/10

   Braintrust's SDK examples show Eval() calls with custom `scores` arrays (e.g. LevenshteinScorer), demonstrating code-based custom scorers, and docs describe managing/authoring scorers via API, CLI, and playgrounds. Missing for 10: independent hands-on validation of writing a fully custom scorer function beyond the built-in Levenshtein example, and more detail on scorer authoring API/signature.

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”

   proof ↗⚑ flag

   Cekura✓fullclaimed8/10

   Cekura explicitly documents Python metrics that let engineers 'write custom evaluation logic in Python to evaluate your AI agent's performance with complete control', alongside LLM-judge metrics as an alternative, giving ML engineers code-based scorer capability. missing for 10: no independent hands-on example of a custom Python metric in use, and no detail on SDK/CLI invocation of custom scorers beyond the docs page.

   • [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
   • [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”

   proof ↗⚑ flag

3. ml engineerCompare eval runs side by side to catch regressions between prompt or model versions

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed9/10

   Braintrust's Experiments feature is explicitly described as an immutable, comparable record of eval runs that tracks progress over time and integrates into CI/CD to catch regressions, and Playgrounds let users run full evaluations and compare results side-by-side across prompt/model configurations. This directly matches the story of comparing eval runs to catch regressions between versions. Missing for 10: independent/hands-on corroboration of the side-by-side comparison UI beyond first-party docs.

   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”

   proof ↗⚑ flag

   Cekura~partialclaimed6/10

   Cekura has an explicit A/B testing feature to compare two agent versions side-by-side to measure prompt, model, or configuration changes, plus dashboards for visualizing metrics/call data, which supports regression detection between versions. However, the story specifically frames this as an ML-engineer 'eval run' comparison workflow, and evidence is limited to a single docs page description without detail on run history diffing, metric-level regression flags, or independent hands-on confirmation. Missing for 10: detailed UI/workflow evidence of comparing full eval-run history (not just two agent versions), regression-flagging mechanics, and independent/community corroboration of the comparison feature itself.

   • [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
   • [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
   • [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”

   proof ↗⚑ flag

4. ml engineerScore outputs with configurable LLM-as-a-judge evaluators

   weight 3 · round drawn
   Braintrust✓fullclaimed8/10

   Braintrust's docs describe configurable scorers used within evals and playgrounds (LLM-as-a-judge style scorers), online scoring of production traces, and code examples show custom scorer configuration (e.g., LevenshteinScorer swapped in evals). Docs explicitly mention 'author scorers' via CLI/MCP and building scorers with Loop, supporting configurability. Missing for 10: explicit documentation naming 'LLM-as-a-judge' scorer templates/library and independent hands-on validation of judge configurability beyond vendor docs.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”

   proof ↗⚑ flag

   Cekura✓fullclaimed8/10

   Cekura documents an LLM-judge metric type that evaluates calls using natural language criteria, alongside custom Python metrics for full control, giving configurable LLM-as-a-judge evaluators for scoring agent outputs. This is corroborated by dashboards/A-B testing docs showing these metrics feed into broader evaluation workflows. Missing for 10: no independent hands-on benchmark of judge accuracy/configurability, and no detail on rubric customization depth or multi-model judge support.

   • [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
   • [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
   • [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”

   proof ↗⚑ flag

5. ml engineerCurate datasets from production traces and run offline evaluations against them

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed9/10

   Docs explicitly describe building versioned datasets from production logs/traces (docs-5, docs-31) and running offline evaluations against them via Experiments, code, or Playgrounds (docs-27, docs-23, gh-1/gh-2 SDK examples), with CI/CD integration for regression testing (docs-2). Missing for 10: independent/hands-on third-party validation of the end-to-end curate-from-production-to-offline-eval workflow beyond first-party docs.

   • [claimed-docs] “Build datasets from production logs, user feedback, manual curation, or generate them with Loop.”
   • [claimed-docs] “Datasets are versioned collections of test cases that you use to run evaluations and track improvements over time. Build datasets from produ…”
   • [claimed-docs] “Experiments are the immutable, comparable record of your eval runs. Run them from code or in the UI, track progress over time, and integrate…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Automate in CI/CD — Run evals on every pull request to catch regressions”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Cekura~partialclaimed5/10

   Cekura stores production call data as CallLog objects via its observability API and lets teams run metric evaluations (Python/LLM-judge) and scenario-based tests, which supports building and evaluating datasets from real traces, but there is no explicit documentation of a 'dataset curation' workflow (filtering/tagging/versioning production traces into an eval set) or an offline batch-eval mode distinct from live/simulated runs. missing for 10: dedicated dataset-curation/versioning feature for production traces, explicit offline-evaluation-on-curated-dataset workflow, independent evidence of this specific use case.

   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
   • [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
   • [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
   • [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”

   proof ↗⚑ flag

Online evals

1. ml engineerRun evaluators continuously on live production traffic, not just offline datasets

   weight 2 · round drawn
   Braintrust✓fullclaimed8/10

   Braintrust explicitly supports 'online scoring' that evaluates production traces automatically as they're logged asynchronously with no latency impact, plus continuous background analysis via Patterns/Topics scanning trace backlogs on a schedule for issues no scorer already catches. This directly matches continuous evaluation on live traffic beyond offline datasets. Missing for 10: independent/hands-on verification of online scoring behavior in production and more detail on configuring which evaluators run continuously.

   • [claimed-docs] “Online scoring evaluates production traces automatically as they’re logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Online scoring evaluates production traces automatically as they're logged, running asynchronously with no impact on latency.”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Patterns puts Loop to work on your trace backlog on a schedule. It searches for recurring problems and trends that no scorer already measure…”
   • [claimed-docs] “Topics automatically analyze and classify your logs without manual review.”
   • [claimed-docs] “Define the dimensions that matter to your business, like use case, customer segment, compliance, or tone. Topics continuously clusters every…”

   proof ↗⚑ flag

   Cekura✓fullprobed8/10

   Cekura's observability pipeline explicitly ingests live production calls via webhook/API, stores them as CallLogs, and 'schedules metric evaluation' automatically, with marketing copy specifically citing 'Live drift detection across every call' as a continuous production-monitoring feature distinct from offline simulation runs. missing for 10: independent/hands-on confirmation of evaluator behavior on live traffic at scale (only vendor docs/marketing cited).

   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Live drift detection across every call.”
   • [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round drawn
   Braintrust~partialprobed7/10

   Braintrust offers a comprehensive API/SDK ('interact with all aspects of the Braintrust platform programmatically'), a full-featured CLI (bt) for auth, tracing, evals, log queries, and function management, and an MCP server letting agents query logs, author scorers, configure monitoring, and run evals — covering most UI functionality outside the terminal. However, some UI-only features like Playgrounds (explicitly 'no-code workspace') and annotation-interface building appear UI-centric without clear API/CLI parity, and no public OpenAPI spec was found to confirm full API coverage. missing for 10: confirmed OpenAPI/spec proving 1:1 API-UI parity, explicit API/CLI equivalent for Playgrounds and custom annotation UI workflows, independent verification of completeness.

   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, sync data, and manage functions without leaving you…”
   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.braintrust.dev/openapi.json, https://www.braintrust.dev/swagger.json, https://www.braint…”

   proof ↗⚑ flag

   Cekura~partialprobed7/10

   Cekura ships a full OpenAPI spec, CLI, and SDK that cover core workflows—managing agents, scenarios, runs, call data, sending calls, and metrics—giving strong programmatic parity with the UI (cekura-docs-4, cekura-docs-5, cekura-docs-6, cekura-probe-3, cekura-probe-5). However, some UI-specific features like custom dashboard building (cekura-docs-11) and cron/CI setup aren't clearly confirmed as fully API-driven rather than UI-configured. Missing for 10: explicit confirmation that dashboard creation and all UI-configurable settings are exposed via API/CLI, and independent verification of complete feature parity.

   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
   • [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
   • [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to Braintrust
   Braintrust~partialclaimed5/10

   Braintrust provides data export via `bt sync pull`, `bt view logs` downloads as CSV/JSON, and a full API to programmatically extract projects, experiments, datasets, traces, and prompts, plus a self-hosting option for data control. However, there's no explicit documentation of full-account export/portability guarantees, no mention of open/standard interchange formats beyond CSV/JSON logs, and no independent confirmation of a clean 'leave' workflow. Missing for 10: comprehensive account-level export documentation, open-format guarantees for all entity types (prompts, scorers, configs), and third-party validation of export completeness.

   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “`bt` is the command-line interface for Braintrust. It lets you authenticate, trace coding-agent sessions, run evals, browse and query logs, …”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Cekura—none0/10

   Cekura offers CLI/SDK/API access to call data and metrics, but there is no evidence of a bulk data export feature in open/portable formats or any account-closure/data-portability guarantee for users wishing to leave the platform.

   ⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Braintrust—none0/10

   Evidence shows only client SDK repos on GitHub (braintrust-sdk-javascript) and self-hosted deployment of infrastructure, but the core platform/UI is explicitly described as Braintrust-managed and not released under an open license; no evidence of the product's full source being open-licensed.

   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Cekura—none0/10

   Cekura is a closed SaaS platform with a GitHub repo only for a skills/plugin package (cekura-skills), not the product's core source code, and no evidence of an open-source license for the platform itself. No license file or open-source repo for the core product is mentioned anywhere in the evidence.

   ⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust documents an official self-hosted deployment option, but it explicitly separates data storage (self-hosted) from platform management (Braintrust-managed UI, auth, updates), meaning it's not a fully self-hostable core product. Missing for 10: evidence of fully self-hosting the entire platform (UI, control plane, auth) independent of Braintrust's managed services, and independent/community confirmation of self-hosting experience.

   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”

   proof ↗⚑ flag

   Cekura—none0/10

   Cekura is presented as a hosted SaaS platform (API keys, cloud dashboards, webhooks) with no mention of a self-hosted or on-premises deployment option anywhere in the docs, CLI, SDK, or website copy. No evidence of Docker images, self-hosting instructions, or open-source core.

   • [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
   • [claimed-docs] “create a comprehensive plan to integrate cekura observability”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust's self-hosting docs explicitly state it's designed to meet data residency requirements by keeping customer data within your own cloud account and region, giving users region/residency control — but only via full self-hosted deployment, not a simple region-selection toggle in the managed cloud offering. Missing for 10: evidence of region choice within the standard hosted/SaaS plan, list of specific supported regions, and compliance certifications tied to residency.

   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”

   proof ↗⚑ flag

   Cekura—none0/10

   No evidence anywhere in the pack mentions data residency, region selection, or geographic storage options for Cekura; PII redaction is mentioned but that's a different concern than data location choice.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Braintrust—none0/10

   No evidence pack item addresses data-training opt-out, a no-training policy, or any statement about whether customer data is used to train AI models. Self-hosting docs address data residency/compliance but do not mention model training use. missing for 10: any explicit no-training-on-customer-data policy, opt-out controls, or terms-of-service statement about AI training use.

   ⚑ flag

   Cekura—none0/10

   Evidence shows PII redaction features for transcripts but nothing about opting out of AI model training on customer data, data-use policies, or training-data controls.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round to Braintrust
   Braintrust~partialclaimed4/10

   Braintrust's self-hosted deployment option lets customers keep all data within their own cloud account/region for compliance and residency, giving some control over where data lives, and the API/CLI allow programmatic management of logs and datasets. However, there is no explicit documentation of a retention policy, TTL settings, or a delete/purge API for logs, traces, or datasets on the managed (non-self-hosted) offering. Missing for 10: explicit retention window configuration, an in-platform delete/purge mechanism for traces or datasets, and confirmation that these controls exist outside of self-hosting.

   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management. You deploy and control the infrastru…”
   • [claimed-docs] “Braintrust offers a self-hosted deployment option that separates data storage from platform management.”
   • [claimed-docs] “Self-hosting is designed for organizations with specific requirements: Data residency and compliance: Meet regulatory or contractual obligat…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”

   proof ↗⚑ flag

   Cekura—none0/10

   Cekura documents PII redaction to remove sensitive info from transcripts, but there is no evidence of user-facing data retention controls or deletion policies/APIs (e.g., data export, retention windows, right-to-delete). missing for 10: documented retention policy, deletion/export mechanism, user control over stored call data lifecycle.

   • [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”

   proof ↗⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Braintrust—none0/10

   No evidence of any telemetry opt-out or usage-tracking control setting for Braintrust itself as a tool; the evidence covers Braintrust's own tracing/observability product features (for user's AI apps), not the vendor's own telemetry collection practices or an opt-out mechanism. Self-hosting controls where customer AI data resides but says nothing about product telemetry/usage tracking opt-out.

   ⚑ flag

   Cekura—none0/10

   No evidence of any telemetry opt-out or usage tracking controls; Cekura's docs focus on PII redaction for customer call data, not on the vendor's own product telemetry. Missing for 10: any mention of telemetry/analytics collection, opt-out mechanism, or privacy settings for usage data.

   ⚑ flag

Prompt workflow

1. developerIterate on prompts in a playground against real models and variables

   weight 2 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Docs explicitly describe Playgrounds as a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets, running full evaluations in real-time against real models with variable inputs, and comparing results side-by-side (braintrust-docs-3, braintrust-docs-23, braintrust-docs-17). This directly matches the story of iterating on prompts in a playground against real models and variables. Missing for 10: independent/hands-on user corroboration of the playground experience specifically (community evidence is generic/mixed) and no detail on variable-substitution mechanics beyond the general description.

   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Playgrounds provide a no-code workspace for rapidly iterating on prompts, models, scorers, and datasets. Run full evaluations in real-time, …”
   • [claimed-docs] “Measure and improve AI application quality with playgrounds and evals”

   proof ↗⚑ flag

   Cekura—none0/10

   Cekura's docs describe testing/evaluation workflows (scenario runs, A/B testing agent versions, synthetic conversations) but there is no evidence of an interactive playground where a developer iterates on prompt text against real models with variables in real time — the closest items (ab-testing, testing-agents/overview) describe comparing pre-built agent versions, not a live prompt-editing sandbox.

   • [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
   • [claimed-docs] “Learn how to quickly create and test voice agents on Cekura without requiring any external API keys or integrations.”
   • [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”

   proof ↗⚑ flag

Ai observability

1. ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app

   weight 3 · round to Braintrust
   Braintrust✓fullprobed9/10

   Braintrust documents a first-party MCP server enabling MCP-compatible agents (Claude Code, Cursor, Codex, etc.) to query logs, run evals, author scorers, and configure monitoring, plus a full programmatic API for traces/experiments/datasets, and Loop, Braintrust's own AI agent, for natural-language investigation of logs and eval data. This directly matches the story of an agent querying traces, metrics, and eval results via API or MCP to debug an app. Missing for 10: independent hands-on corroboration of the MCP server working end-to-end (only vendor docs/probe confirm existence, no third-party review of actual debugging workflow).

   • [claimed-docs] “Query production logs, author prompts and scorers, configure monitoring, and run evals from Claude Code, Cursor, Codex, VS Code, and any oth…”
   • [claimed-docs] “Connect Claude Code, Cursor, Codex, and other MCP clients to Braintrust. Query logs, author scorers, configure Topics, and run evals from yo…”
   • [claimed-docs] “Loop is Braintrust’s AI agent. It understands your data structure, so you can investigate your logs through natural language instead of writ…”
   • [claimed-docs] “Ask Braintrust’s AI agent to investigate your data and build scorers, datasets, and dashboards”
   • [claimed-docs] “The Braintrust API allows you to interact with all aspects of the Braintrust platform programmatically.”
   • [claimed-docs] “You can use it to: Create and manage projects, experiments, and datasets; Log traces and metrics; Manage prompts, tools, and scorers”
   • [probe] “official MCP server documented at https://www.braintrust.dev/docs/integrations/developer-tools/mcp”

   proof ↗⚑ flag

   Cekura~partialprobed6/10

   Cekura has documented API access (OpenAPI spec, send-calls, SDK/CLI for agents/scenarios/runs/call data) and an official MCP server, but the MCP server's documented purpose is to 'design, run, and improve voice-agent evaluations' rather than explicitly querying traces/metrics/eval results for debugging — no direct evidence the MCP tools expose trace/metric/eval query endpoints for agent-driven debugging. missing for 10: explicit MCP tool list showing trace/metric/eval query capabilities, independent hands-on confirmation of an agent using MCP or API to debug via traces/metrics/evals.

   • [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
   • [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”

   proof ↗⚑ flag

Data controls

1. developerMask or redact sensitive data before it is stored in traces

   weight 2 · round to Cekura
   Braintrust—none0/10

   No evidence in the pack describes any PII masking, redaction, or scrubbing feature for traces before storage; documentation covers instrumentation, tracing, evals, and observability but never mentions data masking or redaction capabilities. This is a plausible and common axis for a tracing/observability platform, so absence of evidence yields 'none' rather than 'na'.

   ⚑ flag

   Cekura✓fullclaimed8/10

   Cekura has a dedicated PII redaction feature that automatically detects and removes sensitive information from transcripts and audio recordings before/at storage, directly addressing masking/redaction in traces. missing for 10: no independent/hands-on corroboration of redaction accuracy or configurability details beyond the doc summary.

   • [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”

   proof ↗⚑ flag

Sdk coverage

1. developerInstrument apps in both Python and JS/TS with officially supported SDKs

   weight 2 · round to Braintrust
   Braintrust~partialclaimed6/10

   Docs confirm instrumentation/tracing is central to Braintrust and JS SDK usage is shown in GitHub examples (Eval function calls), with docs referencing SDK-based tracing and instrumentation broadly, but the evidence pack never explicitly confirms an officially supported Python SDK alongside JS/TS — only JS/TS SDK code is shown. Missing for 10: explicit Python SDK documentation/examples, side-by-side confirmation of parity between Python and JS/TS SDKs, and independent corroboration of both SDKs in production use.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, { input: "Bar", expected: "Hello Bar" }, ], task: (in…”

   proof ↗⚑ flag

   Cekura~partialprobed3/10

   Cekura documents a general SDK ('sync and async clients for application code') and a Python-specific custom metric guide, implying Python SDK support, but there is no explicit mention of an officially supported JS/TS SDK anywhere in the evidence pack. missing for 10: explicit JS/TS SDK documentation, parity of features across languages, and confirmation the generic SDK is multi-language rather than Python-only.

   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
   • [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
   • [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”

   proof ↗⚑ flag

Trace capture

1. developerTrace multi-step agent runs as nested spans grouped into sessions or threads

   weight 2 · round to Braintrust
   Braintrust~partialclaimed6/10

   Braintrust documents tracing that captures spans, logs, and full request detail, and lets users browse traces and individual spans via UI or CLI, supporting general span-based tracing. However, the evidence never explicitly describes nested multi-step spans grouped into sessions/threads for agent runs (no mention of parent-child span hierarchies or session/thread grouping semantics). Missing for 10: explicit documentation of nested span hierarchies for multi-step agent runs, explicit session/thread grouping constructs, and independent/hands-on confirmation of this specific structure.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with [`bt view logs`](/docs/reference/cli/view).”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”

   proof ↗⚑ flag

   Cekura—none0/10

   Cekura's observability evidence centers on capturing call-level logs (CallLog with transcript, recording, metadata) and LiveKit SDK tracing, plus dashboards and PII redaction, but nothing in the evidence describes a hierarchical trace model with nested spans grouped into sessions or threads for multi-step agent runs. Missing for 10: explicit span/trace data model, parent-child span nesting, and session/thread grouping documentation.

   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
   • [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”

   proof ↗⚑ flag

2. developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK

   weight 2 · round to Braintrust
   Braintrust~partialcommunity4/10

   Docs confirm a general instrumentation capability and state you can 'integrate with AI providers and frameworks to send traces to Braintrust,' but no evidence explicitly names LangChain, the OpenAI SDK, or the Vercel AI SDK as supported integrations. A community comment even calls integration options 'limited.' Missing for 10: explicit documentation or hands-on confirmation of dedicated LangChain, OpenAI SDK, and Vercel AI SDK integrations.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [community] “Braintrust: Interesting approach to evaluation but feels like an early-stage product. Documentation is sparse and integration options are li…”

   proof ↗⚑ flag

   Cekura~partialclaimed3/10

   Cekura documents a generic observability API/webhook (send-calls) and SDK/CLI that any agent could POST to, plus a named integration for LiveKit voice agents, but there is no documented integration for LangChain, the OpenAI SDK, or the Vercel AI SDK specifically as the story requests. missing for 10: named LangChain integration, named OpenAI SDK integration, named Vercel AI SDK integration.

   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”

   proof ↗⚑ flag

3. developerCapture multimodal payloads (images, audio, files) inside my traces

   weight 1 · round to Cekura
   Braintrust—none0/10

   The evidence pack describes tracing capturing inputs, outputs, model parameters, latency, token usage, and metadata, but there is no explicit mention of multimodal payload support (images, audio, files) within traces. Missing for 10: explicit documentation or examples showing image/audio/file capture in traces, SDK support for multimodal attachments, or any UI rendering of multimodal content in trace spans.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”

   proof ↗⚑ flag

   Cekura~partialclaimed5/10

   Cekura's observability pipeline explicitly captures audio (recording URL) and transcript/metadata as part of CallLog traces, and PII redaction applies to both transcripts and audio recordings, showing audio is a first-class multimodal payload. However, there is no evidence of support for images or generic file attachments within traces — the product is voice-agent focused and docs never mention image/file capture. Missing for 10: documented support for image payloads, documented support for arbitrary file attachments in traces.

   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”

   proof ↗⚑ flag

4. developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format

   weight 3 · round drawn
   Braintrust—none0/10

   No evidence pack item mentions OpenTelemetry or OTLP support; Braintrust's docs describe its own SDK/instrumentation and integrations but never reference OTLP-based trace ingestion/export. Missing for 10: any mention of OTLP endpoint, OpenTelemetry SDK compatibility, or OTel collector integration.

   ⚑ flag

   Cekura—none0/10

   Cekura's observability ingestion uses a proprietary POST endpoint (transcript, recording URL, metadata) and its own API/CLI/SDK, with no mention of OpenTelemetry or OTLP support anywhere in the evidence pack.

   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
   • [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”

   proof ↗⚑ flag

5. developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage

   weight 3 · round to Braintrust
   Braintrust✓fullclaimed8/10

   Braintrust docs explicitly state instrumentation captures inputs, outputs, model parameters, latency, token usage, and metadata for every LLM call, with SDK examples and CLI/UI browsing of traces and spans. missing for 10: independent hands-on verification of exact captured fields and no third-party benchmark of tracing accuracy/performance overhead.

   • [claimed-docs] “Instrumentation captures detailed traces from your AI application, recording inputs, outputs, model parameters, latency, token usage, and me…”
   • [claimed-docs] “Add tracing to your AI application to capture LLM calls, application logic, and user feedback — the foundation for observability and evaluat…”
   • [claimed-docs] “Integrate with AI providers and frameworks to send traces to Braintrust”
   • [claimed-docs] “Browse traces and individual spans, in the UI or from the terminal with `bt view logs`. ... Download logs as CSV or JSON, or pull them local…”
   • [github] “Eval("Say Hi Bot", { data: () => [ { input: "Foo", expected: "Hi Foo" }, ], task: (input) => "Hi " + input, scores: [Levenshtein…”

   proof ↗⚑ flag

   Cekura~partialprobed4/10

   Cekura's observability captures call-level data (transcript, recording URL, metadata) via its send-calls API and stores it as a CallLog, and offers 'enhanced observability' for LiveKit agents, but the docs never explicitly mention capturing per-LLM-call inputs/outputs, latency, or token usage — the focus is on transcript/audio-level call logging for voice-agent evaluation rather than granular LLM tracing. Missing for 10: explicit documentation of input/output payload capture per LLM call, latency metrics, and token usage tracking.

   • [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
   • [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
   • [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”

   proof ↗⚑ flag