LLM Evals & Observability Arena
W&B Weave vs Cekura
W&B Weave wins · 16–14 (18 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round drawnDirect probes confirm llms.txt (HTTP 200 with structured doc index) and a .md variant of docs pages exist and are crawlable, exactly matching the ask for agent-oriented docs, plus an OpenAPI spec and MCP server for further agent integration. Missing for 10: no independent/community confirmation that agents actually consume these docs successfully in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.wandb.ai/llms.txt # Weights & Biases Documentation - [Products (407 pages)](https://docs.wandb.ai/…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.wandb.ai/weave.md > ## Documentation Index > Fetch the complete documentation index at: https://docs…”
- [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
- [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
Cekura has a confirmed live llms.txt endpoint (HTTP 200) plus per-page .md documentation format and an index pointing agents to llms.txt, directly enabling agent-oriented doc consumption; this is corroborated by both docs and live probes. Missing for 10: independent third-party confirmation of an agent actually consuming llms.txt successfully in the wild.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.cekura.ai/documentation/introduction.md > ## Documentation Index > Fetch the complete documentation …”
- [claimed-docs] “create a comprehensive plan to integrate cekura observability”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to CekuraWeave's SDK (weave.op, weave.init) and Service API/REST endpoints allow programmatic, non-UI instrumentation and evaluation that can run in scripts or CI pipelines, and the OTel-compatible ingestion endpoint supports headless trace collection. However, there is no explicit documentation of a CI-specific workflow, headless auth/config for pipelines, or a dedicated CLI/automation example confirming CI usage. missing for 10: explicit CI/headless setup guide, documented non-interactive auth flow for automated pipelines, concrete CI example (e.g. GitHub Actions integration).
- [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
- [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
- [github] “Log and debug language model inputs, outputs, and traces”
Cekura documents a CLI/SDK for programmatic and terminal-based agent/scenario/run management, a REST API with OpenAPI spec and API keys, cron job scheduling for automated evaluation workflows, and explicit GitHub Actions CI/CD integration for testing agents on every change—together these clearly support headless/CI automation. Missing for 10: independent hands-on confirmation of the GitHub Actions or cron setup actually working in a real pipeline beyond docs.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
ai-native userConnect an agent via an official MCP server
weight 3 · round drawnW&B ships an official MCP server (wandb-mcp-server) enabling coding agents like Claude Code to connect to Weave, read live production data, run evaluations, and execute iteration loops autonomously — this is documented both on the product site and via a dedicated GitHub repo. Missing for 10: deeper documentation of MCP server setup/configuration and independent hands-on corroboration beyond vendor claims.
- [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
- [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
Cekura is not itself an agent but a testing/observability platform, so the axis applies; it documents an official MCP server ("Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations") confirmed live via probe. Missing for 10: independent hands-on verification of the MCP server working end-to-end and more detail on supported tool/resource surface.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
ai-native userUse an official CLI
weight 2 · round to CekuraThere is evidence of an official W&B CLI (docs.wandb.ai/models/ref/cli), but this CLI is documented under the Models product, not specifically Weave, and no evidence details Weave-specific CLI commands (e.g., managing traces, evaluations, or ops from the terminal) or AI-native/agentic use of it. Missing for 10: Weave-specific CLI command reference, evidence of agentic/programmatic use of the CLI, independent hands-on confirmation.
- [probe] “official CLI documented at https://docs.wandb.ai/models/ref/cli”
Cekura documents an official CLI ('Manage agents, scenarios, runs, and call data from your terminal') alongside an SDK, confirmed by both docs and probe evidence. Missing for 10: independent hands-on review of the CLI's usability/completeness beyond first-party docs.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
ai-native userDrive the product through a documented public API
weight 3 · round to CekuraWeave documents a public REST Service API for programmatic access, an openapi.json spec, Python/TypeScript SDKs with @weave.op decorators, and an official MCP server enabling agent-driven interaction with live data and evaluations. missing for 10: independent third-party validation of API stability/versioning and rate-limit documentation beyond first-party docs.
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
- [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
- [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
- [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
Cekura publishes a documented public API with OpenAPI spec (probe confirms live openapi.json), API key auth, dedicated API reference endpoints (e.g. send-calls), plus SDKs (sync/async) and a CLI for programmatic/terminal control—clearly enabling an AI-native user to drive the product via documented API access. Missing for 10: independent third-party review/hands-on validation of the API beyond vendor docs and probes.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to CekuraW&B Weavenone0/10W&B Weave is an LLM observability/evaluation tool; the evidence pack covers tracing, evaluations, cost tracking, and an MCP/skills integration, but there is no mention of scoped or least-privilege API credential issuance for agents. Missing for 10: any documentation of API key scoping, permission granularity, or credential management for agent access.
Cekura documents API key creation restricted to project-level scope, showing some credential scoping capability, but there's no evidence of finer-grained least-privilege controls like read/write permission tiers, resource-specific scoping, or per-agent credential issuance. missing for 10: granular permission scopes (read-only vs write), per-agent or per-resource credential issuance, role-based access control documentation.
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userBuild against official SDKs
weight 2 · round drawnWeave offers official Python and TypeScript SDKs with decorator-based tracing (@weave.op), a REST Service API, OTel-compatible SDK, and a documented CLI/MCP server, all backed by first-party docs and public GitHub repo. Missing for 10: independent third-party benchmarking or hands-on developer reviews validating SDK stability/completeness beyond vendor docs.
- [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
- [github] “You can trace any function using `weave.op` - from api calls to OpenAI, Anthropic, Google AI Studio etc to generation calls from Hugging Fac…”
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
- [claimed-docs] “Trace and collect metrics about your agents built with popular SDKs and harnesses using Weave OTel-compatible SDK”
- [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
- [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
- [probe] “official CLI documented at https://docs.wandb.ai/models/ref/cli”
Cekura documents official sync/async SDKs and a CLI for programmatic access ('Manage agents, scenarios, runs, and call data from your terminal', 'sync and async clients for application code'), plus an OpenAPI spec, backed by probe confirmation of live docs and endpoints. missing for 10: independent hands-on developer reports specifically about SDK usage (only community praise for the platform generally, not SDK-specific).
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userSubscribe to events via webhooks
weight 2 · round to W&B WeaveWeave documents alerts that 'trigger webhook automations' from production insights, indicating some outbound webhook mechanism exists, but there is no documentation of a subscription API, event types, payload schema, or configuration steps for webhooks. missing for 10: documented webhook subscription/configuration API, list of subscribable event types, payload format, independent/hands-on confirmation.
- [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
Cekuranone0/10Evidence shows Cekura receiving inbound webhooks (agent's provider posts call data to Cekura) but nothing about Cekura emitting outbound webhooks that a user could subscribe to for events like evaluation completion or drift detection. Missing for 10: documented outbound webhook/event subscription mechanism, webhook configuration UI/API, event types list.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round drawnWeave ships built-in LLM-judge scorers (hallucination detection, summarization quality, embedding similarity) and Guardrails (toxicity, bias, PII detection) that automatically generate AI-based assessments of traced data, plus 'signals' that auto-classify agent interactions — all forms of AI-generated insight surfaced inside the product. However, these are narrow, pre-defined quality/safety classifiers rather than general proactive 'suggestions' or exploratory insight generation across arbitrary data, and most of the deeper analysis (custom scorers, evaluation criteria) requires user-authored code rather than the product generating novel suggestions on its own. Missing for 10: evidence of open-ended AI-generated recommendations/next-step suggestions (not just fixed scorer categories), and independent/hands-on confirmation these signals surface meaningfully useful insights in practice.
- [claimed-docs] “Weave comes with predefined scorers and local SLM scorers that you can use right away, including: Hallucination detection, Summarization qua…”
- [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
- [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”
Cekura provides LLM-judge evaluation metrics, live drift detection, and auto-generated fix suggestions ('flags issues → reproduces in simulation → suggests fixes automatically'), showing AI-generated insights derived from call/agent data. However, this is narrowly scoped to voice-agent testing/observability rather than broad, ad-hoc AI-generated insights across arbitrary product data, and there's no independent hands-on evidence of the 'suggests fixes automatically' claim beyond marketing copy. Missing for 10: independent/hands-on verification of the auto-fix-suggestion feature, broader insight/analytics generation beyond voice-agent metrics, and detail on how suggestions are surfaced to users.
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Live drift detection across every call.”
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
ai-native userSet up automations that run autonomously in the background
weight 2 · round to CekuraWeave documents automated background signals and alerting (built-in/custom signals classify agent interactions and trigger Slack/webhook automations) and mentions agents connecting via MCP to 'execute automatic iteration loops on their own,' which suggests some autonomous background automation. However, Weave is primarily a tracing/evaluation/observability tool, not a scheduler or workflow-automation platform, and there's no dedicated docs on setting up persistent background jobs or scheduled autonomous runs beyond alert-triggered webhooks. Missing for 10: dedicated automation/scheduling feature docs, evidence of persistent autonomous background jobs beyond alert webhooks, independent corroboration of the MCP-driven 'automatic iteration loops' claim.
- [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
- [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
Cekura supports autonomous background automation via cron jobs for scheduled testing/evaluation runs, GitHub Actions CI/CD integration, and live drift detection across calls, plus automatic flagging/reproduction/fix-suggestion pipelines. However, evidence is entirely first-party docs describing scheduling/triggering mechanisms rather than a broad general-purpose 'automation builder' for arbitrary agentic workflows. missing for 10: independent/hands-on confirmation that cron/CI automations run reliably unattended, evidence of more flexible trigger types beyond cron/CI/webhooks, and detail on monitoring/error-handling for long-running autonomous automations.
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Live drift detection across every call.”
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
ai-native userOperate the product with natural-language commands
weight 2 · round drawnWeave itself is an observability/eval dashboard with no native chat-command interface, but an official MCP server lets AI coding agents like Claude Code read production data, run evaluations, and iterate automatically using natural-language instructions relayed through MCP tools. This gives indirect NL-driven operation rather than a first-party conversational control surface. Missing for 10: a built-in Weave chat/NL console, independent hands-on verification of the MCP-driven workflow, and broader agent support beyond Claude Code.
- [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
- [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
Cekura ships an MCP server and Skills package so an AI assistant can design, run, and improve voice-agent evaluations via natural-language commands, plus an LLM-judge metric for natural-language evaluation criteria; this supports agentic/NL operation for a subset of workflows. missing for 10: independent hands-on evidence of natural-language command execution beyond vendor docs, and confirmation that the full breadth of platform actions (not just evaluation/testing tasks) can be driven purely by NL commands.
- [claimed-docs] “npx skills add cekura-ai/cekura-skills --all”
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [github] “13 Skills, 14 Commands in one plugin”
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round to CekuraThe Weave Service API and an OpenAPI spec (openapi.json) exist, suggesting some form of structured API reference, but there is no evidence of an interactive, in-browser reference with runnable/executable examples (e.g., a Swagger/try-it-out console or live code sandbox). missing for 10: confirmation of an interactive UI with 'try it now' runnable examples, evidence of live execution from the docs, and any independent confirmation of this feature.
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
Cekura publishes an OpenAPI spec and structured API-reference docs (e.g., send-calls, create-api-key endpoints) confirming a machine-readable interactive reference exists, but no evidence shows runnable 'try it' examples or a sandboxed console in the docs. Missing for 10: explicit runnable/try-it example UI, evidence of live request execution, independent confirmation of interactivity.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round drawnA probe confirms an OpenAPI spec is served at https://docs.wandb.ai/openapi.json (HTTP 200, contains an 'openapi' key), and docs also describe a Service API providing REST endpoints for programmatic access. missing for 10: no independent third-party confirmation that the spec is actively used/maintained beyond the probe check.
- [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
A direct probe confirms an OpenAPI spec is publicly downloadable at https://docs.cekura.ai/openapi.json, returning HTTP 200 with a valid "openapi" key, corroborated by API reference docs built from it. Missing for 10: no independent third-party confirmation of spec completeness/versioning beyond the probe.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to CekuraWeave's Evaluation framework lets users test against curated Datasets/test examples rather than live production data, and the Playground lets you test prompts/models interactively, which implicitly avoids touching production traffic. However, there is no explicit 'sandbox environment' feature, and other docs (e.g., testing against production traces) actually emphasize using real production data rather than isolating from it. Missing for 10: dedicated sandbox/staging environment concept, explicit data isolation guarantees, and evidence separating test vs production data paths.
- [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
- [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
- [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”
Cekura explicitly supports testing voice agents via synthetic/simulated conversations 'before go-live' and running scenarios without external API keys, which implies a sandbox-like testing flow separate from live production calls. However, there is no explicit documentation of a dedicated sandbox environment, data isolation guarantees, or explicit statements that test runs never touch production data/logs. missing for 10: explicit sandbox/staging environment documentation, data isolation guarantees between test and production calls, independent confirmation that simulated runs are fully separated from production data.
- [claimed-docs] “Learn how to quickly create and test voice agents on Cekura without requiring any external API keys or integrations.”
- [claimed-docs] “Run thousands of synthetic conversations before go-live.”
- [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnW&B Weavenone0/10No evidence of a versioned API scheme or documented deprecation policy for Weave's SDK/Service API; only an OpenAPI spec presence is shown, not versioning/deprecation commitments. missing for 10: versioning scheme documentation, deprecation policy/notice process, changelog or migration guides for breaking changes.
- [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
Cekuranone0/10There is an OpenAPI spec and API-key auth docs, but no evidence of API versioning scheme or a documented deprecation policy anywhere in the pack. missing for 10: versioning scheme documentation, deprecation policy, changelog/migration guides.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
Alerting dashboards — stories about alerting dashboards in this arenaAlerting dashboards
Stories about alerting dashboards in this arena
Monitoring
ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics
weight 2 · round drawnWeave tracks latency/traces, cost (automatic and custom cost tracking), and eval scores (via Evaluation objects, scorers, leaderboards), and supports alerting via Slack/webhooks on signals — covering most of the metrics named in the story. However, there is no explicit evidence of a customizable dashboard-building UI (e.g., drag-and-drop widgets, custom charts combining these metrics side-by-side) beyond the built-in calls table, trace tree, and leaderboards. missing for 10: explicit custom dashboard/visualization builder evidence, unified view combining latency+error+cost+eval-score in one configurable dashboard, independent/hands-on confirmation of dashboard flexibility.
- [claimed-docs] “Automatic cost tracking: For supported integrations, Weave captures token usage from the API response and applies built-in pricing for the m…”
- [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
- [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
- [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
- [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”
- [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
Cekura docs explicitly describe building custom dashboards with widgets to visualize call data, metrics, and metadata, which covers eval-score and general metric visualization for an ML engineer's needs. However, the docs don't explicitly enumerate latency, error rate, or cost as first-class dashboard metrics, and there's no independent/hands-on corroboration of dashboard flexibility. Missing for 10: explicit mention of latency/error/cost metric widgets, and third-party evidence of dashboard customization in practice.
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email
weight 2 · round to W&B WeaveWeave marketing docs mention built-in/custom 'signals' that capture and classify agent interactions, with alerts routed via Slack notifications and webhook automations, which could plausibly trigger error-rate or eval-score alerts. However, there is no explicit mention of cost-spike alerts, no native PagerDuty or email notification channels (only Slack + generic webhooks), and no detail on how alert thresholds are configured for error rates or eval-score drops specifically. missing for 10: native PagerDuty integration, native email notification channel, explicit documentation of alert types (error rate, cost spike, eval-score drop) and threshold configuration.
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”
- [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to CekuraWeave's Evaluation object runs scoring across an entire Dataset of many test examples in one call, and the Service API exposes REST endpoints for programmatic access, which together support batch-style automation over many items. However, there is no explicit evidence of bulk trace management operations (bulk delete, bulk tagging, bulk export/update of many logged calls) that a fully bulk-operations story would require. Missing for 10: documented bulk edit/delete/export APIs for traces or datasets, and independent confirmation of large-scale batch throughput.
- [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
- [claimed-docs] “The core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples.”
- [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
- [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”
Cekura's docs advertise running thousands of synthetic conversations/scenarios at once and provide a CLI/SDK to manage agents, scenarios, runs, and call data programmatically, which supports bulk test execution and automation via cron/CI. However, there's no explicit documentation of bulk edit/delete/update operations across many existing items (e.g., batch API endpoints for updating many scenarios or calls at once). Missing for 10: explicit bulk CRUD API/CLI commands operating on many items simultaneously, and any hands-on evidence of large-scale batch throughput.
- [claimed-docs] “Run thousands of synthetic conversations before go-live.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to CekuraWeave's marketing docs mention custom signals that classify agent interactions and alerts that route to Slack or trigger webhook automations, which matches the idea of rule-based triggers on events. However, this is only briefit is only referenced on the marketing page, with no dedicated guide, configuration reference, or independent confirmation of how rules are authored or scoped. Missing for 10: a dedicated docs page detailing rule/condition syntax, examples of trigger configuration, and independent/hands-on verification that these automations work as described.
- [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”
Cekura supports automation triggers like scheduled cron jobs for test workflows, GitHub Actions CI/CD hooks that fire tests on code changes, and an automatic flag→reproduce→suggest-fix pipeline triggered by detected issues, which are event-driven automations. However, there's no evidence of a general-purpose user-defined rules engine (e.g., 'if X metric drops, then do Y') beyond these fixed automation patterns. Missing for 10: a documented custom rule/condition builder for arbitrary events, and independent confirmation these automations behave as advertised.
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Live drift detection across every call.”
ai-native userVersion, review, and roll back my automations
weight 1 · round drawnWeave documents automatic versioning of traced functions, datasets, and evaluation objects (weave.op(), Evaluation objects) and provides alert/webhook 'automations' for production signals, but there is no evidence of a review or rollback mechanism for these automations/versions. Missing for 10: explicit rollback UI/API for automations, version-history browsing/restore workflow, and evidence tying versioning to the alert/webhook automations themselves.
- [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
- [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
- [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications a…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
Cekura supports comparing two versions of an agent side-by-side (A/B testing) and provides dashboards/metrics for reviewing agent performance, which touches 'version' and 'review' aspects, but there is no evidence of a rollback mechanism to revert an automation/agent to a prior version. Missing for 10: explicit version history/versioning system, rollback/undo capability, and any documented workflow for reverting agent configurations.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
Cost monitoring — stories about cost monitoring in this arenaCost monitoring
Stories about cost monitoring in this arena
Cost tracking
developerAttribute cost and usage to users, sessions, and features via custom metadata
weight 2 · round to W&B WeaveWeave captures call metadata via weave.op, tracks token usage/cost automatically or via custom cost entries, and groups related calls into Threads (sessions), which together enable some cost/usage attribution. However, there is no explicit documentation of tagging calls with custom user/feature metadata or aggregating/filtering cost by such tags. Missing for 10: explicit custom-attribute tagging API (e.g., user_id/feature tags) and evidence of cost rollups/dashboards filtered by those custom dimensions.
- [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
- [claimed-docs] “Threads are collections of Traces related to a single session or conversation. You can use Threads to analyze or score the entire conversati…”
- [claimed-docs] “Automatic cost tracking: For supported integrations, Weave captures token usage from the API response and applies built-in pricing for the m…”
- [claimed-docs] “Add a custom cost with the `add_cost` method. The three required fields are `llm_id`, `prompt_token_cost`, and `completion_token_cost`.”
- [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
- [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
- [claimed-docs] “Use custom cost tracking when automatic cost tracking isn’t available, such as with fine-tuned models, self-hosted models, or providers that…”
Cekuranone0/10Cekura's docs show metadata being sent with calls and custom dashboards for visualizing metrics/metadata, but there is no evidence of cost tracking, cost attribution, or usage-cost breakdown by user/session/feature — the platform is framed purely around voice-agent evaluation and observability, not cost accounting.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
developerSee cost and token usage per request, model, and time period in dashboards
weight 3 · round to W&B WeaveWeave automatically tracks token usage and cost per call using built-in pricing for supported integrations, with costs shown in the trace tree and calls table in the Weave UI, plus custom cost support for unsupported models; calls table is filterable/aggregable by model and time via the trace UI. missing for 10: explicit documentation of pre-built cost dashboards aggregating by time period across the whole org, and independent/hands-on confirmation beyond vendor docs.
- [claimed-docs] “Automatic cost tracking: For supported integrations, Weave captures token usage from the API response and applies built-in pricing for the m…”
- [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
- [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
- [claimed-docs] “Use custom cost tracking when automatic cost tracking isn’t available, such as with fine-tuned models, self-hosted models, or providers that…”
- [claimed-docs] “Add a custom cost with the `add_cost` method. The three required fields are `llm_id`, `prompt_token_cost`, and `completion_token_cost`.”
Cekuranone0/10Cekura documents customizable dashboards for call data, metrics, and metadata (cekura-docs-11), but nothing in the evidence pack mentions cost or token usage tracking per request, model, or time period. As an observability platform for voice agents, this axis plausibly applies, but no evidence supports the specific cost/token dashboard capability.
Data access export — stories about data access export in this arenaData access export
Stories about data access export in this arena
Data export
developerBulk-export traces and datasets to blob storage or my data warehouse
weight 2 · round drawnW&B Weavenone0/10Evidence shows Weave has a Service API for programmatic access and OTel import for bringing trace data in, but nothing documents bulk export of traces/datasets to blob storage (S3/GCS) or a data warehouse (Snowflake/BigQuery), which is a reasonable ask for an observability/eval platform.
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
- [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
Cekuranone0/10Evidence shows Cekura has an API, CLI, SDK, and dashboards for accessing call data, but there is no mention of bulk-export functionality to blob storage (S3/GCS) or data warehouse integrations (Snowflake, BigQuery, etc). Missing for 10: bulk export feature, blob storage integration, data warehouse connectors/sinks, any documentation or API endpoint for exporting datasets in bulk.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets
Measuring quality — datasets, eval runs, regression tracking
Ai eval ops
ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results
weight 2 · round drawnWeave provides a programmatic Evaluation API with Dataset objects and scoring functions (docs-6, docs-19, docs-25, docs-31), a Service API with REST endpoints for programmatic access to results (docs-15), and an official MCP server explicitly enabling coding agents to 'read live production data, run evaluations, and execute automatic iteration loops on their own' (docs-20, probe-4) — directly matching the agent-driven create-dataset/trigger-eval/read-results workflow. Missing for 10: independent/hands-on confirmation of an agent autonomously completing this full loop end-to-end, and explicit example code showing dataset creation + eval trigger + result read-back in one flow.
- [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
- [claimed-docs] “Measure every improvement to your harnesses and models with confidence with a flexible imperative evaluation API. Weave provides powerful ev…”
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
- [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
- [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
- [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
Cekura ships a CLI/SDK explicitly for managing agents, scenarios (datasets), runs, and call data from code, plus an official MCP server letting an AI assistant 'design, run, and improve' evaluations, backed by a documented OpenAPI spec and API-key auth for programmatic access — together covering dataset creation, triggering eval runs, and reading back results. missing for 10: independent/hands-on confirmation that an agent using the MCP/CLI actually completes the full create-dataset-to-trigger-run-to-read-results loop end-to-end.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
Human review
ml engineerRoute outputs to human annotation queues for review and labeling
weight 2 · round drawnW&B Weavenone0/10Weave's evidence covers tracing, evaluation, scoring, cost tracking, and feedback collection, but there is no mention of routing outputs to human annotation/labeling queues or any human-in-the-loop review workflow tooling.
Cekuranone0/10Cekura's evidence pack covers voice-agent testing, evaluation metrics (LLM-judge, Python), dashboards, PII redaction, and observability, but there is no mention of routing outputs to human annotation queues, labeling workflows, or human-in-the-loop review pipelines. Absence of evidence for this applicable ML-eval capability yields none. Missing for 10: any mention of human annotation queues, labeling UI/workflow, or reviewer assignment features.
Offline evals
developerRun evals in CI and gate deployments on their results
weight 2 · round to CekuraWeave's imperative Evaluation API and Service API (REST) mean evals can be scripted and run programmatically, which a team could wire into a CI job, but the evidence never documents a CI/CD integration, pipeline templates, or a mechanism for gating/blocking deployments based on eval results. Missing for 10: explicit CI/CD integration guides (e.g., GitHub Actions), exit-code/threshold-based gating support, and any documented deployment-blocking workflow.
- [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
- [claimed-docs] “Measure every improvement to your harnesses and models with confidence with a flexible imperative evaluation API. Weave provides powerful ev…”
- [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
Cekura provides a dedicated GitHub Actions CI/CD guide plus CLI/SDK and cron job automation for testing agents on each change, indicating CI integration is a first-class capability. However, the evidence pack only shows the guide's title/description, not concrete details on exit-code/gating behavior for deployments, and there is no independent corroboration of this specific workflow. Missing for 10: explicit documentation of deployment-gating logic (pass/fail thresholds blocking merges/deploys), and independent/hands-on confirmation of the GitHub Actions integration.
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ml engineerWrite custom code-based scorers and metrics for my evaluations
weight 2 · round to W&B WeaveWeave's Evaluation object explicitly supports custom scoring functions, and dedicated docs on Custom Scorers describe encoding use-case-specific evaluation criteria beyond built-in scorers, backed by predefined scorers as a baseline. This directly matches writing code-based scorers/metrics for evaluations. Missing for 10: independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
- [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”
- [claimed-docs] “The core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples.”
- [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
- [github] “Build rigorous, apples-to-apples evaluations for language model use cases”
Cekura explicitly documents Python metrics that let engineers 'write custom evaluation logic in Python to evaluate your AI agent's performance with complete control', alongside LLM-judge metrics as an alternative, giving ML engineers code-based scorer capability. missing for 10: no independent hands-on example of a custom Python metric in use, and no detail on SDK/CLI invocation of custom scorers beyond the docs page.
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
ml engineerCompare eval runs side by side to catch regressions between prompt or model versions
weight 3 · round to W&B WeaveWeave's Evaluation object plus scorers explicitly support comparing runs over time to catch regressions, and docs state comparisons/visualizations exist to 'catch regressions before they reach users,' with leaderboards to aggregate and compare evaluations across versions. missing for 10: no independent/hands-on corroboration of side-by-side UI comparison workflow beyond vendor docs.
- [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
- [claimed-docs] “Measure every improvement to your harnesses and models with confidence with a flexible imperative evaluation API. Weave provides powerful ev…”
- [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
- [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”
- [github] “Build rigorous, apples-to-apples evaluations for language model use cases”
Cekura has an explicit A/B testing feature to compare two agent versions side-by-side to measure prompt, model, or configuration changes, plus dashboards for visualizing metrics/call data, which supports regression detection between versions. However, the story specifically frames this as an ML-engineer 'eval run' comparison workflow, and evidence is limited to a single docs page description without detail on run history diffing, metric-level regression flags, or independent hands-on confirmation. Missing for 10: detailed UI/workflow evidence of comparing full eval-run history (not just two agent versions), regression-flagging mechanics, and independent/community corroboration of the comparison feature itself.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
- [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”
ml engineerScore outputs with configurable LLM-as-a-judge evaluators
weight 3 · round to W&B WeaveWeave provides a first-class Evaluation object with scoring functions, built-in LLM-judge scorers (hallucination, summarization quality, etc.), and explicit support for custom scorers to encode use-case-specific criteria, plus Guardrails pre-built safety/quality scorers. Missing for 10: independent/hands-on third-party corroboration beyond vendor docs.
- [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
- [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”
- [claimed-docs] “Weave comes with predefined scorers and local SLM scorers that you can use right away, including: Hallucination detection, Summarization qua…”
- [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
- [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
- [github] “Build rigorous, apples-to-apples evaluations for language model use cases”
Cekura documents an LLM-judge metric type that evaluates calls using natural language criteria, alongside custom Python metrics for full control, giving configurable LLM-as-a-judge evaluators for scoring agent outputs. This is corroborated by dashboards/A-B testing docs showing these metrics feed into broader evaluation workflows. Missing for 10: no independent hands-on benchmark of judge accuracy/configurability, and no detail on rubric customization depth or multi-model judge support.
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
ml engineerCurate datasets from production traces and run offline evaluations against them
weight 3 · round to W&B WeaveWeave supports capturing production traces via @weave.op instrumentation, and explicitly supports building Datasets from these traces for use in its Evaluation object, which runs scoring functions/LLM judges against test examples; docs also mention testing new LLMs/custom models against production traces (offline evaluation). missing for 10: no explicit hands-on/independent example walking through 'export trace → dataset → evaluation' end-to-end, and no third-party corroboration of this specific workflow.
- [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
- [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”
- [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
- [claimed-docs] “Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”
- [claimed-docs] “Weave comes with predefined scorers and local SLM scorers that you can use right away, including: Hallucination detection, Summarization qua…”
- [github] “Build rigorous, apples-to-apples evaluations for language model use cases”
- [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
Cekura stores production call data as CallLog objects via its observability API and lets teams run metric evaluations (Python/LLM-judge) and scenario-based tests, which supports building and evaluating datasets from real traces, but there is no explicit documentation of a 'dataset curation' workflow (filtering/tagging/versioning production traces into an eval set) or an offline batch-eval mode distinct from live/simulated runs. missing for 10: dedicated dataset-curation/versioning feature for production traces, explicit offline-evaluation-on-curated-dataset workflow, independent evidence of this specific use case.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
Online evals
ml engineerRun evaluators continuously on live production traffic, not just offline datasets
weight 2 · round to CekuraWeave supports testing against production traces (docs-21), monitoring live agent interactions with signals/alerts (docs-16/17/28/36), and Guardrails scorers can presumably run on live traffic, plus custom/predefined scorers (docs-7,8,22). However, the core Evaluation workflow is explicitly framed around Datasets/lists of test examples run offline (docs-6, docs-31), and there's no explicit documentation of a continuous/streaming online-evaluation pipeline that automatically scores all live production calls in real time as they occur. Missing for 10: explicit documentation of automated/continuous scoring pipelines applied to every live production call (not just ad-hoc production trace sampling), and independent/hands-on confirmation of this online-evaluation mode.
- [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”
- [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
- [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
- [claimed-docs] “the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more sco…”
- [claimed-docs] “This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable tes…”
- [claimed-docs] “Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts r…”
Cekura's observability pipeline explicitly ingests live production calls via webhook/API, stores them as CallLogs, and 'schedules metric evaluation' automatically, with marketing copy specifically citing 'Live drift detection across every call' as a continuous production-monitoring feature distinct from offline simulation runs. missing for 10: independent/hands-on confirmation of evaluator behavior on live traffic at scale (only vendor docs/marketing cited).
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Live drift detection across every call.”
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to CekuraWeave exposes a Service API with REST endpoints for programmatic access, plus SDK-level ops for tracing, evaluations, and cost tracking, and an OpenAPI spec is served, indicating broad API coverage. However, some UI-centric features (Playground model comparison/testing, leaderboards, Slack alert configuration) are documented mainly as UI workflows without explicit evidence that every one of these is fully API-exposed. missing for 10: explicit documentation confirming Playground, leaderboards, and alerting/webhook configuration are all fully controllable via the API/SDK rather than just the UI.
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
- [claimed-docs] “Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”
- [claimed-docs] “Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteratio…”
Cekura ships a full OpenAPI spec, CLI, and SDK that cover core workflows—managing agents, scenarios, runs, call data, sending calls, and metrics—giving strong programmatic parity with the UI (cekura-docs-4, cekura-docs-5, cekura-docs-6, cekura-probe-3, cekura-probe-5). However, some UI-specific features like custom dashboard building (cekura-docs-11) and cron/CI setup aren't clearly confirmed as fully API-driven rather than UI-configured. Missing for 10: explicit confirmation that dashboard creation and all UI-configurable settings are exposed via API/CLI, and independent verification of complete feature parity.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
ai-native userExport all of my data in open formats and leave
weight 3 · round to W&B WeaveWeave documents a REST Service API for 'programmatic access to Weave functionality' and OTel-compatible trace import/export interoperability, which could theoretically be used to pull data out in an open format, but there is no explicit documentation of a bulk 'export all my data' feature or guidance for migrating off the platform entirely. missing for 10: explicit bulk export/download feature, documented open-format export (e.g., JSON/OTLP dump of all traces/evals), and any guidance for full data portability or platform exit.
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
- [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
- [claimed-docs] “Use this integration when you want to instrument your application with the OpenTelemetry standard and have those traces appear alongside you…”
ai-native userRead the product's source under an open license
weight 2 · round drawnW&B Weavenone0/10The evidence pack confirms a public GitHub repository (github.com/wandb/weave) exists with descriptions of its tracing/evaluation code, but none of the citations mention an open-source license (e.g., Apache-2.0/MIT) or any licensing terms at all, so there is no evidence the source is available under an open license.
- [github] “You can trace any function using `weave.op` - from api calls to OpenAI, Anthropic, Google AI Studio etc to generation calls from Hugging Fac…”
- [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”
- [github] “Log and debug language model inputs, outputs, and traces”
- [github] “Build rigorous, apples-to-apples evaluations for language model use cases”
Cekuranone0/10Cekura is a closed SaaS platform with a GitHub repo only for a skills/plugin package (cekura-skills), not the product's core source code, and no evidence of an open-source license for the platform itself. No license file or open-source repo for the core product is mentioned anywhere in the evidence.
ai-native userSelf-host the core product
weight 3 · round drawnW&B Weavenone0/10Weave is documented as a hosted SaaS platform (weave.init() connecting to W&B's cloud) with no evidence pack mentions of a self-hosted or on-prem deployment option for the core Weave product itself; only W&B Models/Platform is known to have enterprise self-hosting but that's not evidenced here for Weave specifically.
Cekuranone0/10Cekura is presented as a hosted SaaS platform (API keys, cloud dashboards, webhooks) with no mention of a self-hosted or on-premises deployment option anywhere in the docs, CLI, SDK, or website copy. No evidence of Docker images, self-hosting instructions, or open-source core.
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [claimed-docs] “create a comprehensive plan to integrate cekura observability”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnW&B Weavenone0/10No evidence of region/residency data storage controls for Weave; the pack covers tracing, evaluation, cost tracking, and integrations only, with no mention of self-hosting, EU/US data residency, or region selection options. Missing for 10: any documentation of regional data storage, residency guarantees, or self-hosted/on-prem deployment options.
ai-native userPrevent my data from being used to train AI models
weight 3 · round drawnW&B Weavenone0/10No evidence in the pack addresses data usage policies, opt-out of training, or any privacy commitment regarding customer data being used to train models; the evidence pack focuses entirely on tracing, evaluation, and observability features. Missing for 10: any privacy policy statement, training opt-out mechanism, or data usage terms documentation.
ai-native userControl data retention and deletion
weight 2 · round drawnW&B Weavenone0/10The evidence pack covers tracing, evaluation, cost tracking, OTel import, and playground features, but there is no mention of data retention policies, deletion controls, or data lifecycle management for logged traces/data. missing for 10: documentation on data retention periods, user-initiated deletion of traces/projects/data, GDPR/CCPA compliance controls, or any retention configuration options.
Cekuranone0/10Cekura documents PII redaction to remove sensitive info from transcripts, but there is no evidence of user-facing data retention controls or deletion policies/APIs (e.g., data export, retention windows, right-to-delete). missing for 10: documented retention policy, deletion/export mechanism, user control over stored call data lifecycle.
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnW&B Weavenone0/10The evidence pack contains no mention of a telemetry opt-out, privacy settings, or usage-tracking controls for Weave itself; all evidence concerns tracing/evaluation features that Weave provides for users' LLM apps, not W&B's own telemetry collection. Since Weave is a SaaS-style observability tool where such an axis plausibly applies, absence of evidence yields 'none'.
Cekuranone0/10No evidence of any telemetry opt-out or usage tracking controls; Cekura's docs focus on PII redaction for customer call data, not on the vendor's own product telemetry. Missing for 10: any mention of telemetry/analytics collection, opt-out mechanism, or privacy settings for usage data.
Prompt management — stories about prompt management in this arenaPrompt management
Stories about prompt management in this arena
Prompt workflow
developerIterate on prompts in a playground against real models and variables
weight 2 · round to W&B WeaveWeave's Playground explicitly supports prompt editing, message retrying, model comparison, and testing custom/OpenAI-compatible endpoints against real models, plus saved model presets for reusable variable configs, directly matching the story. Missing for 10: independent/hands-on corroboration beyond vendor docs, and explicit detail on templated variable substitution within prompts.
- [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
- [claimed-docs] “Custom runtimes: Test OpenAI-compatible API endpoints for custom models.”
- [claimed-docs] “Saved models: Create and configure a reusable model preset for your workflow.”
- [claimed-docs] “You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”
Cekuranone0/10Cekura's docs describe testing/evaluation workflows (scenario runs, A/B testing agent versions, synthetic conversations) but there is no evidence of an interactive playground where a developer iterates on prompt text against real models with variables in real time — the closest items (ab-testing, testing-agents/overview) describe comparing pre-built agent versions, not a live prompt-editing sandbox.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [claimed-docs] “Learn how to quickly create and test voice agents on Cekura without requiring any external API keys or integrations.”
- [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”
Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation
Instrumenting code and tracing requests end to end
Ai observability
ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app
weight 3 · round to W&B WeaveWeave provides a documented Service API (REST) for programmatic access to traces/evals, plus an official MCP server (wandb-mcp-server) explicitly described as letting coding agents like Claude Code 'read live production data, run evaluations, and execute automatic iteration loops on their own.' This directly matches the story of an agent querying traces/metrics/evals to debug an app. missing for 10: independent/hands-on corroboration of the MCP server in real debugging workflows beyond vendor docs.
- [claimed-docs] “Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
- [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
- [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
- [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
Cekura has documented API access (OpenAPI spec, send-calls, SDK/CLI for agents/scenarios/runs/call data) and an official MCP server, but the MCP server's documented purpose is to 'design, run, and improve voice-agent evaluations' rather than explicitly querying traces/metrics/eval results for debugging — no direct evidence the MCP tools expose trace/metric/eval query endpoints for agent-driven debugging. missing for 10: explicit MCP tool list showing trace/metric/eval query capabilities, independent hands-on confirmation of an agent using MCP or API to debug via traces/metrics/evals.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
Data controls
developerMask or redact sensitive data before it is stored in traces
weight 2 · round to CekuraW&B Weavenone0/10The evidence describes tracing, evaluation, cost tracking, and PII *detection* via Guardrails scorers (wandb-weave-docs-22), but nothing about masking or redacting sensitive data before it is written into stored traces. This is a fair capability to expect from a tracing/instrumentation product, so absence of evidence means 'none' rather than 'na'.
- [claimed-docs] “Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detec…”
- [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
- [claimed-docs] “When you decorate a function with @weave.op() (Python) or wrap it with weave.op() (TypeScript), Weave automatically captures its code, input…”
Cekura has a dedicated PII redaction feature that automatically detects and removes sensitive information from transcripts and audio recordings before/at storage, directly addressing masking/redaction in traces. missing for 10: no independent/hands-on corroboration of redaction accuracy or configurability details beyond the doc summary.
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
Sdk coverage
developerInstrument apps in both Python and JS/TS with officially supported SDKs
weight 2 · round to W&B WeaveDocs explicitly confirm both Python (@weave.op() decorator) and TypeScript (weave.op() wrap) SDK support for instrumenting functions and LLM calls, with consistent API design across languages. Missing for 10: independent/third-party corroboration of TS SDK parity and maturity, and more detail on JS/TS-specific setup/init beyond the single mention.
- [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
- [claimed-docs] “When you decorate a function with @weave.op() (Python) or wrap it with weave.op() (TypeScript), Weave automatically captures its code, input…”
- [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”
- [claimed-docs] “Manually instrument your application’s LLM calls and arbitrary functions to trace, version, and collect feedback about your application”
Cekura documents a general SDK ('sync and async clients for application code') and a Python-specific custom metric guide, implying Python SDK support, but there is no explicit mention of an officially supported JS/TS SDK anywhere in the evidence pack. missing for 10: explicit JS/TS SDK documentation, parity of features across languages, and confirmation the generic SDK is multi-language rather than Python-only.
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
Trace capture
developerTrace multi-step agent runs as nested spans grouped into sessions or threads
weight 2 · round to W&B WeaveWeave supports automatic nested span capture via @weave.op() producing trace trees, plus first-class grouping into Threads/sessions/turns/sub-agents for multi-step agent runs, explicitly designed to navigate agent sessions as executed. Missing for 10: independent hands-on corroboration beyond vendor docs.
- [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
- [claimed-docs] “Threads are collections of Traces related to a single session or conversation. You can use Threads to analyze or score the entire conversati…”
- [claimed-docs] “Weave now brings sessions, turns, steps, tools, and sub-agents as first-class concepts, making it much easier to navigate an agent session t…”
- [claimed-docs] “Weave organizes traces into sessions and turns from the ground up.”
- [claimed-docs] “Weave organizes traces into sessions and turns from the ground up. That structure, paired with native analytics tools, makes it easy to trac…”
- [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”
Cekuranone0/10Cekura's observability evidence centers on capturing call-level logs (CallLog with transcript, recording, metadata) and LiveKit SDK tracing, plus dashboards and PII redaction, but nothing in the evidence describes a hierarchical trace model with nested spans grouped into sessions or threads for multi-step agent runs. Missing for 10: explicit span/trace data model, parent-child span nesting, and session/thread grouping documentation.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK
weight 2 · round to W&B WeaveWeave clearly supports auto-instrumentation for OpenAI (and Anthropic, Cohere, Mistral, Hugging Face) via weave.op() decorators/wrappers and has a TypeScript wrap() function, satisfying the OpenAI-SDK part of the story, and it also supports generic OTel-based instrumentation for 'popular SDKs and harnesses.' However, no evidence pack citation explicitly names a LangChain integration or a Vercel AI SDK integration, so those specific framework integrations are unconfirmed. missing for 10: explicit documentation of a first-party LangChain integration, explicit documentation of a Vercel AI SDK integration.
- [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
- [github] “You can trace any function using `weave.op` - from api calls to OpenAI, Anthropic, Google AI Studio etc to generation calls from Hugging Fac…”
- [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
- [claimed-docs] “Trace and collect metrics about your agents built with popular SDKs and harnesses using Weave OTel-compatible SDK”
- [claimed-docs] “Use this integration when you want to instrument your application with the OpenTelemetry standard and have those traces appear alongside you…”
Cekura documents a generic observability API/webhook (send-calls) and SDK/CLI that any agent could POST to, plus a named integration for LiveKit voice agents, but there is no documented integration for LangChain, the OpenAI SDK, or the Vercel AI SDK specifically as the story requests. missing for 10: named LangChain integration, named OpenAI SDK integration, named Vercel AI SDK integration.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
developerCapture multimodal payloads (images, audio, files) inside my traces
weight 1 · round to CekuraW&B Weavenone0/10The evidence describes Weave's tracing capturing function inputs/outputs, code, and metadata via @weave.op(), but none of the docs or GitHub excerpts mention support for images, audio, or file attachments within traces. Missing for 10: any explicit mention of multimodal payload types (images, audio, files) being captured, rendered, or stored in trace data.
Cekura's observability pipeline explicitly captures audio (recording URL) and transcript/metadata as part of CallLog traces, and PII redaction applies to both transcripts and audio recordings, showing audio is a first-class multimodal payload. However, there is no evidence of support for images or generic file attachments within traces — the product is voice-agent focused and docs never mention image/file capture. Missing for 10: documented support for image payloads, documented support for arbitrary file attachments in traces.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format
weight 3 · round to W&B WeaveWeave documents a dedicated OTLP import endpoint and an OTel-compatible SDK so external OpenTelemetry traces can be sent in and appear alongside native Weave traces, rather than requiring the proprietary weave.op format exclusively. However, this is framed as import-only alongside existing pipelines, not a full bidirectional replace-your-format OTLP transport, and there's no evidence of receiving/exporting traces out via OTLP or independent corroboration of interoperability. Missing for 10: evidence of OTLP export/round-trip, independent hands-on confirmation, and clarity that OTel is a full alternative rather than a supplementary ingestion path.
- [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
- [claimed-docs] “Trace and collect metrics about your agents built with popular SDKs and harnesses using Weave OTel-compatible SDK”
- [claimed-docs] “Use this integration when you want to instrument your application with the OpenTelemetry standard and have those traces appear alongside you…”
Cekuranone0/10Cekura's observability ingestion uses a proprietary POST endpoint (transcript, recording URL, metadata) and its own API/CLI/SDK, with no mention of OpenTelemetry or OTLP support anywhere in the evidence pack.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage
weight 3 · round to W&B WeaveWeave's @weave.op() decorator automatically captures code, inputs, outputs, and execution metadata for LLM calls, with automatic token usage and cost tracking recorded per call and displayed in the trace tree/UI. Latency is inherently part of the captured trace/execution metadata; OTel-compatible import and GitHub docs corroborate first-party and independent-style evidence. Missing for 10: explicit standalone documentation calling out latency capture by name, and independent (non-vendor) hands-on validation.
- [claimed-docs] “When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, i…”
- [claimed-docs] “Weave automatically records token usage and calculates cost for each call. Costs appear in the trace tree and in the calls table in the Weav…”
- [claimed-docs] “When you call weave.init() and use a supported LLM integration such as OpenAI, Anthropic, Cohere, or Mistral, Weave automatically records to…”
- [github] “Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”
- [github] “Log and debug language model inputs, outputs, and traces”
- [claimed-docs] “Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
Cekura's observability captures call-level data (transcript, recording URL, metadata) via its send-calls API and stores it as a CallLog, and offers 'enhanced observability' for LiveKit agents, but the docs never explicitly mention capturing per-LLM-call inputs/outputs, latency, or token usage — the focus is on transcript/audio-level call logging for voice-agent evaluation rather than granular LLM tracing. Missing for 10: explicit documentation of input/output payload capture per LLM call, latency metrics, and token usage tracking.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
Not comparable on these axes
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · not comparableW&B Weaven/aWeave is an LLM observability/tracing/evaluation platform, not an agent that itself performs tasks using external tools — the 'plug MCP servers in so it can use their tools' story is a category mismatch for this kind of product. The only MCP-related evidence (wandb-weave-docs-20, wandb-weave-probe-4) shows the opposite direction: Weave exposes its own MCP server so other coding agents (e.g., Claude Code) can connect to and use Weave's data/tools, not Weave consuming external MCP servers as a client.
- [claimed-docs] “Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, a…”
- [probe] “official MCP server documented at https://github.com/wandb/wandb-mcp-server”
Cekuranone0/10All Cekura MCP evidence describes Cekura exposing its own MCP server so external AI assistants can call Cekura's tools (docs-3, probe-4), not Cekura itself acting as an MCP client that consumes third-party MCP servers' tools. No evidence shows Cekura can plug in and use external MCP servers.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · not comparableW&B Weaven/aW&B Weave is an LLM observability/evaluation platform, not an AI assistant product; it's a tool that other agents/apps instrument and connect to (e.g., via MCP), not a built-in assistant that a user delegates tasks to within Weave itself. This is a category mismatch for the 'built-in AI assistant' story.
Cekuranone0/10Cekura's AI-assistant integrations (Skills, MCP, CLI) are designed so external AI assistants like Claude or Cursor can drive Cekura's testing/evaluation features — this is the reverse relationship of an AI-native user delegating tasks to a built-in assistant inside Cekura itself. The 'flags issues → reproduces → suggests fixes automatically' pipeline is an automated feature, not a delegatable in-product assistant, so there is no evidence of a built-in AI assistant you can converse with or delegate open-ended tasks to.
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [claimed-docs] “npx skills add cekura-ai/cekura-skills --all”
ai-native userSchedule recurring jobs or workflows
weight 2 · not comparableW&B Weaven/aW&B Weave is an LLM observability/evaluation platform for tracing, evaluating, and monitoring AI applications, not a job scheduler or workflow orchestrator; scheduling recurring jobs is outside its product category and category-adjacent tools (alerts, webhooks) only react to events rather than run on a schedule.
Cekura documents dedicated cron job support for scheduling automated testing/evaluation workflows (cekura-docs-13), plus GitHub Actions integration for triggering workflows on changes (cekura-docs-14), giving AI-native users concrete recurring automation options via CLI/SDK/API. Missing for 10: independent/hands-on confirmation that cron scheduling works reliably in production and more detail on scheduling granularity/limits.
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
developerVersion prompts and deploy changes to production without shipping code
weight 3 · not comparableW&B Weavenone0/10The evidence pack covers tracing, evaluation, cost tracking, and a Playground for prompt editing/model comparison, but nothing describes a prompt versioning/registry system or a mechanism to push prompt changes to production without redeploying code. Playground's 'prompt editing' (wandb-weave-docs-9) and 'Saved models' preset (wandb-weave-docs-11) are experimentation tools, not a production deployment/versioning workflow for prompts decoupled from code.
- [claimed-docs] “With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
- [claimed-docs] “Saved models: Create and configure a reusable model preset for your workflow.”
Cekuran/aCekura is a testing/observability platform for voice AI agents (simulated calls, metrics, A/B comparison of agent versions), not a prompt-management or deployment system. Prompt versioning and production deployment without code changes belongs to the underlying agent-building platform, not to a third-party evaluation/observability tool like Cekura, making this axis a category error for this product type.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”