LLM Evals & Observability Arena
Galileo vs Cekura
Cekura wins · 10–25 (14 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to CekuraDirect probes confirm llms.txt returns HTTP 200 with a documentation index, and individual doc pages provide .md versions with pointers back to llms.txt, making the docs agent-consumable as claimed. missing for 10: no independent/third-party confirmation of an agent actually consuming these files successfully, and no evidence of broader machine-readable spec coverage (e.g., OpenAPI probe returned 404s).
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.galileo.ai/llms.txt # Galileo - [What Is Galileo?](https://docs.galileo.ai/what-is-galileo.md) - […”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.galileo.ai/what-is-galileo.md > ## Documentation Index > Fetch the complete documentation index at: …”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Cekura has a confirmed live llms.txt endpoint (HTTP 200) plus per-page .md documentation format and an index pointing agents to llms.txt, directly enabling agent-oriented doc consumption; this is corroborated by both docs and live probes. Missing for 10: independent third-party confirmation of an agent actually consuming llms.txt successfully in the wild.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.cekura.ai/documentation/introduction.md > ## Documentation Index > Fetch the complete documentation …”
- [claimed-docs] “create a comprehensive plan to integrate cekura observability”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to CekuraGalileo ships a Python SDK (with `@log` decorators, OpenTelemetry distributed tracing, and experiment/dataset APIs) that can be invoked programmatically without the console UI, implying headless/CI usage is possible. However, the evidence never explicitly documents a CI/CD pipeline example, a CLI, or headless-run guidance—experiments are largely framed around the console UI's 'Create Experiment' button. Missing for 10: explicit CI/CD integration docs or examples, a documented CLI/headless entrypoint, and confirmation that experiments can be fully triggered/scored outside the UI.
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
Cekura documents a CLI/SDK for programmatic and terminal-based agent/scenario/run management, a REST API with OpenAPI spec and API keys, cron job scheduling for automated evaluation workflows, and explicit GitHub Actions CI/CD integration for testing agents on every change—together these clearly support headless/CI automation. Missing for 10: independent hands-on confirmation of the GitHub Actions or cron setup actually working in a real pipeline beyond docs.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
ai-native userConnect an agent via an official MCP server
weight 3 · round drawnGalileo, as an observability/evaluation platform (not itself an agent), documents an official MCP server that lets users access dataset management, experiments, and prompt templates directly from their dev environment, confirmed live via docs page. Missing for 10: independent/hands-on verification beyond first-party docs and details on broader client compatibility.
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
Cekura is not itself an agent but a testing/observability platform, so the axis applies; it documents an official MCP server ("Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations") confirmed live via probe. Missing for 10: independent hands-on verification of the MCP server working end-to-end and more detail on supported tool/resource surface.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
ai-native userUse an official CLI
weight 2 · round to CekuraGalileonone0/10Evidence shows Python SDK, MCP server integration, and REST-like console features, but no mention of an official CLI tool for Galileo. missing for 10: any documentation of a dedicated CLI binary/command, install instructions, or command reference.
Cekura documents an official CLI ('Manage agents, scenarios, runs, and call data from your terminal') alongside an SDK, confirmed by both docs and probe evidence. Missing for 10: independent hands-on review of the CLI's usability/completeness beyond first-party docs.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
ai-native userDrive the product through a documented public API
weight 3 · round to CekuraGalileo provides a Python SDK (galileo-python) with decorators for logging traces/spans, an MCP server for programmatic access to datasets/experiments, and OpenTelemetry-based distributed tracing support, indicating a documented API surface for AI-native/agentic use. However, no formal public REST/OpenAPI reference was found (openapi probe returned 404s across all candidate paths), so the API's full documented surface and versioning/auth details are unclear. missing for 10: a discoverable OpenAPI/REST API spec, formal API reference docs beyond SDK/MCP usage, and independent confirmation of API completeness.
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
Cekura publishes a documented public API with OpenAPI spec (probe confirms live openapi.json), API key auth, dedicated API reference endpoints (e.g. send-calls), plus SDKs (sync/async) and a CLI for programmatic/terminal control—clearly enabling an AI-native user to drive the product via documented API access. Missing for 10: independent third-party review/hands-on validation of the API beyond vendor docs and probes.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to CekuraGalileonone0/10Galileo is an AI observability/evaluation platform; evidence covers tracing, metrics, experiments, and MCP integration, but there is no mention of scoped or least-privilege API credential/key management for agents.
Cekura documents API key creation restricted to project-level scope, showing some credential scoping capability, but there's no evidence of finer-grained least-privilege controls like read/write permission tiers, resource-specific scoping, or per-agent credential issuance. missing for 10: granular permission scopes (read-only vs write), per-agent or per-resource credential issuance, role-based access control documentation.
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userBuild against official SDKs
weight 2 · round to CekuraGalileo has an official Python SDK (galileo-python) with decorators/logging APIs referenced in GitHub docs, plus MCP server integration for accessing Galileo capabilities from dev environments, supporting AI-native/agentic workflows. However, evidence lacks details on multi-language SDK coverage, versioning/stability, or independent developer corroboration beyond first-party docs, and OpenAPI spec probes all 404'd. Missing for 10: multi-language SDK evidence, independent hands-on validation, public API reference/OpenAPI spec.
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Cekura documents official sync/async SDKs and a CLI for programmatic access ('Manage agents, scenarios, runs, and call data from your terminal', 'sync and async clients for application code'), plus an OpenAPI spec, backed by probe confirmation of live docs and endpoints. missing for 10: independent hands-on developer reports specifically about SDK usage (only community praise for the platform generally, not SDK-specific).
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userSubscribe to events via webhooks
weight 2 · round drawnGalileonone0/10There is a mention of alerting (galileo-docs-9) but no evidence of webhook subscription support; the OpenAPI/API endpoints probe returned 404s and nothing in the evidence pack describes webhooks or event subscription mechanisms.
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Cekuranone0/10Evidence shows Cekura receiving inbound webhooks (agent's provider posts call data to Cekura) but nothing about Cekura emitting outbound webhooks that a user could subscribe to for events like evaluation completion or drift detection. Missing for 10: documented outbound webhook/event subscription mechanism, webhook configuration UI/API, event types list.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round to CekuraGalileo offers LLM-as-a-judge and custom metrics that can evaluate outputs, natural-language feedback loops that auto-improve metric alignment, and alerts on anomalies, which imply some AI-assisted analysis of data — but there is no explicit documentation of a dedicated 'insights/suggestions' feature that proactively surfaces AI-generated recommendations to users. Missing for 10: a clear insights/suggestions UI or feature description, independent examples of such AI-generated recommendations in use, and confirmation this goes beyond metric scoring to actionable suggestions.
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
- [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”
- [claimed-docs] “Agentic metrics help you measure how well your AI agents perform complex, multi-step tasks—especially when those agents need to use tools, m…”
Cekura provides LLM-judge evaluation metrics, live drift detection, and auto-generated fix suggestions ('flags issues → reproduces in simulation → suggests fixes automatically'), showing AI-generated insights derived from call/agent data. However, this is narrowly scoped to voice-agent testing/observability rather than broad, ad-hoc AI-generated insights across arbitrary product data, and there's no independent hands-on evidence of the 'suggests fixes automatically' claim beyond marketing copy. Missing for 10: independent/hands-on verification of the auto-fix-suggestion feature, broader insight/analytics generation beyond voice-agent metrics, and detail on how suggestions are surfaced to users.
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Live drift detection across every call.”
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
ai-native userSet up automations that run autonomously in the background
weight 2 · round to CekuraGalileo documents background alerting ('get alerted whenever unexpected things happen') and continuous automatic capture of traces/spans, which are autonomous background processes, but there is no evidence of a general-purpose automation/scheduling system for agentic workflows that a user configures to run independently. missing for 10: explicit automation/workflow scheduler, triggers/conditions configuration, evidence of autonomous multi-step agent execution beyond monitoring/alerts.
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
Cekura supports autonomous background automation via cron jobs for scheduled testing/evaluation runs, GitHub Actions CI/CD integration, and live drift detection across calls, plus automatic flagging/reproduction/fix-suggestion pipelines. However, evidence is entirely first-party docs describing scheduling/triggering mechanisms rather than a broad general-purpose 'automation builder' for arbitrary agentic workflows. missing for 10: independent/hands-on confirmation that cron/CI automations run reliably unattended, evidence of more flexible trigger types beyond cron/CI/webhooks, and detail on monitoring/error-handling for long-running autonomous automations.
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Live drift detection across every call.”
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round drawnGalileonone0/10Galileo's evidence covers evaluating and monitoring external AI agents (agentic metrics, tracing, MCP access to its own capabilities from a dev environment) but nothing about a built-in assistant inside Galileo's own product that a user can delegate tasks to.
Cekuranone0/10Cekura's AI-assistant integrations (Skills, MCP, CLI) are designed so external AI assistants like Claude or Cursor can drive Cekura's testing/evaluation features — this is the reverse relationship of an AI-native user delegating tasks to a built-in assistant inside Cekura itself. The 'flags issues → reproduces → suggests fixes automatically' pipeline is an automated feature, not a delegatable in-product assistant, so there is no evidence of a built-in AI assistant you can converse with or delegate open-ended tasks to.
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [claimed-docs] “npx skills add cekura-ai/cekura-skills --all”
ai-native userOperate the product with natural-language commands
weight 2 · round to CekuraGalileo ships an official MCP server that lets AI assistants create/manage datasets, run experiments, and set up prompt templates from a dev environment using natural language, and it supports continuous natural-language feedback to refine metrics — both are concrete NL-driven operation paths. However, there's no evidence of a native chat/NL command interface within the Galileo console itself; missing for 10: first-party in-app NL command console, broader coverage of all product actions via NL, and independent hands-on confirmation of the MCP NL workflow.
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
Cekura ships an MCP server and Skills package so an AI assistant can design, run, and improve voice-agent evaluations via natural-language commands, plus an LLM-judge metric for natural-language evaluation criteria; this supports agentic/NL operation for a subset of workflows. missing for 10: independent hands-on evidence of natural-language command execution beyond vendor docs, and confirmation that the full breadth of platform actions (not just evaluation/testing tasks) can be driven purely by NL commands.
- [claimed-docs] “npx skills add cekura-ai/cekura-skills --all”
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [github] “13 Skills, 14 Commands in one plugin”
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round to CekuraGalileonone0/10No evidence of an interactive API reference with runnable examples; openapi probes all returned 404, and no Swagger/Redoc-style playground is mentioned anywhere in the docs pack.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Cekura publishes an OpenAPI spec and structured API-reference docs (e.g., send-calls, create-api-key endpoints) confirming a machine-readable interactive reference exists, but no evidence shows runnable 'try it' examples or a sandboxed console in the docs. Missing for 10: explicit runnable/try-it example UI, evidence of live request execution, independent confirmation of interactivity.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round to CekuraGalileonone0/10A direct probe for OpenAPI/Swagger spec files at all standard locations returned 404, and no documentation references a downloadable machine-readable API spec; only an llms.txt index and MCP server exist, neither of which is an OpenAPI spec.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
A direct probe confirms an OpenAPI spec is publicly downloadable at https://docs.cekura.ai/openapi.json, returning HTTP 200 with a valid "openapi" key, corroborated by API reference docs built from it. Missing for 10: no independent third-party confirmation of spec completeness/versioning beyond the probe.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to CekuraGalileo's Experiments feature lets users evaluate prompts/models against well-defined inputs and datasets can be built from 'synthetic, development, and live production data,' implying some separation between test and production data, but there is no explicit sandbox/staging environment concept described. missing for 10: explicit sandbox/staging environment docs, isolation guarantees from production data, and independent confirmation of non-production testing workflow.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
Cekura explicitly supports testing voice agents via synthetic/simulated conversations 'before go-live' and running scenarios without external API keys, which implies a sandbox-like testing flow separate from live production calls. However, there is no explicit documentation of a dedicated sandbox environment, data isolation guarantees, or explicit statements that test runs never touch production data/logs. missing for 10: explicit sandbox/staging environment documentation, data isolation guarantees between test and production calls, independent confirmation that simulated runs are fully separated from production data.
- [claimed-docs] “Learn how to quickly create and test voice agents on Cekura without requiring any external API keys or integrations.”
- [claimed-docs] “Run thousands of synthetic conversations before go-live.”
- [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnGalileonone0/10No evidence of API versioning scheme or a documented deprecation policy; OpenAPI spec probes returned 404 and no changelog/versioning docs are present in the evidence pack.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Cekuranone0/10There is an OpenAPI spec and API-key auth docs, but no evidence of API versioning scheme or a documented deprecation policy anywhere in the pack. missing for 10: versioning scheme documentation, deprecation policy, changelog/migration guides.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
Alerting dashboards — stories about alerting dashboards in this arenaAlerting dashboards
Stories about alerting dashboards in this arena
Monitoring
ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics
weight 2 · round to CekuraGalileonone0/10Evidence covers logging/tracing (latency, spans), custom metrics, LLM-as-judge evals, and alerting, but no documentation describes a dashboard-building UI or customizable visualization layer combining latency, error, cost, and eval-score metrics. missing for 10: dashboard/widget customization UI, evidence of combining metrics types into a single view, cost-metric tracking, independent/hands-on confirmation of dashboarding.
Cekura docs explicitly describe building custom dashboards with widgets to visualize call data, metrics, and metadata, which covers eval-score and general metric visualization for an ML engineer's needs. However, the docs don't explicitly enumerate latency, error rate, or cost as first-class dashboard metrics, and there's no independent/hands-on corroboration of dashboard flexibility. Missing for 10: explicit mention of latency/error/cost metric widgets, and third-party evidence of dashboard customization in practice.
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email
weight 2 · round to GalileoGalileo docs confirm a general alerting capability ("get alerted whenever unexpected things happen" via a 'set up alerts on logs' guide), but the evidence pack contains no detail on which triggers (error rate, cost spike, eval-score drop) are supported nor which notification channels (Slack, PagerDuty, email) are integrated. Missing for 10: documented list of supported alert conditions, confirmation of Slack/PagerDuty/email integrations, and any hands-on/independent verification of alert delivery.
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to CekuraGalileo's Experiments feature runs evaluations across datasets of many inputs at once, and MCP/SDK access lets users programmatically create and manage datasets and run experiments in bulk from code rather than one item at a time (galileo-docs-2, galileo-docs-3, galileo-docs-4, galileo-docs-11). However, there is no explicit documentation of bulk edit/delete/tag/annotate operations across arbitrary large sets of existing items in the console or API. Missing for 10: explicit bulk CRUD operations (batch edit/delete/tag) across items, batch API endpoints/rate-limit guidance for large-scale automation, and independent confirmation of bulk-scale reliability.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
Cekura's docs advertise running thousands of synthetic conversations/scenarios at once and provide a CLI/SDK to manage agents, scenarios, runs, and call data programmatically, which supports bulk test execution and automation via cron/CI. However, there's no explicit documentation of bulk edit/delete/update operations across many existing items (e.g., batch API endpoints for updating many scenarios or calls at once). Missing for 10: explicit bulk CRUD API/CLI commands operating on many items simultaneously, and any hands-on evidence of large-scale batch throughput.
- [claimed-docs] “Run thousands of synthetic conversations before go-live.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to CekuraGalileo docs mention that users can set up alerts on logs to be notified of unexpected events, which is a basic rule-trigger-on-event capability, but there is no detail on defining custom rule logic, condition types, or automated actions beyond alerting (e.g., webhooks, workflow triggers, remediation actions). missing for 10: rule definition UI/API details, supported trigger conditions, and evidence of automated actions beyond simple alert notifications.
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
Cekura supports automation triggers like scheduled cron jobs for test workflows, GitHub Actions CI/CD hooks that fire tests on code changes, and an automatic flag→reproduce→suggest-fix pipeline triggered by detected issues, which are event-driven automations. However, there's no evidence of a general-purpose user-defined rules engine (e.g., 'if X metric drops, then do Y') beyond these fixed automation patterns. Missing for 10: a documented custom rule/condition builder for arbitrary events, and independent confirmation these automations behave as advertised.
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Live drift detection across every call.”
ai-native userSchedule recurring jobs or workflows
weight 2 · round to CekuraGalileonone0/10Galileo is an LLM observability/evaluation platform with experiments, alerts, and MCP integration, but no evidence describes scheduling recurring jobs or workflows (e.g., cron-like automation, scheduled evaluation runs). Alerts are reactive, not scheduled, and no scheduler feature is documented.
Cekura documents dedicated cron job support for scheduling automated testing/evaluation workflows (cekura-docs-13), plus GitHub Actions integration for triggering workflows on changes (cekura-docs-14), giving AI-native users concrete recurring automation options via CLI/SDK/API. Missing for 10: independent/hands-on confirmation that cron scheduling works reliably in production and more detail on scheduling granularity/limits.
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
Cost monitoring — stories about cost monitoring in this arenaCost monitoring
Stories about cost monitoring in this arena
Cost tracking
developerAttribute cost and usage to users, sessions, and features via custom metadata
weight 2 · round to GalileoGalileo's logging captures sessions, traces, and spans (galileo-docs-6) and supports custom metrics (galileo-docs-8), and mentions Luna models monitoring traffic at lower cost (galileo-docs-12), implying some usage/cost tracking infrastructure exists. However, there is no explicit documentation of tagging traces/sessions with custom metadata fields (e.g., user ID, feature name) for cost attribution or cost breakdown by dimension. missing for 10: explicit custom metadata tagging API/fields for user/session/feature attribution, cost-per-tag reporting or dashboards, and any hands-on example of cost attribution via metadata.
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
- [claimed-docs] “Distill your optimized evals into Luna models that monitor 100% of your traffic at 96% lower cost.”
Cekuranone0/10Cekura's docs show metadata being sent with calls and custom dashboards for visualizing metrics/metadata, but there is no evidence of cost tracking, cost attribution, or usage-cost breakdown by user/session/feature — the platform is framed purely around voice-agent evaluation and observability, not cost accounting.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
developerSee cost and token usage per request, model, and time period in dashboards
weight 3 · round drawnGalileonone0/10The evidence pack covers tracing, experiments, metrics, and alerts, but contains no mention of cost or token usage tracking, nor dashboards broken down by request, model, or time period. This is a plausible axis for an LLM observability platform, so absence of evidence yields 'none' rather than 'na'.
Cekuranone0/10Cekura documents customizable dashboards for call data, metrics, and metadata (cekura-docs-11), but nothing in the evidence pack mentions cost or token usage tracking per request, model, or time period. As an observability platform for voice agents, this axis plausibly applies, but no evidence supports the specific cost/token dashboard capability.
Data access export — stories about data access export in this arenaData access export
Stories about data access export in this arena
Data export
developerBulk-export traces and datasets to blob storage or my data warehouse
weight 2 · round drawnGalileonone0/10No evidence of any bulk-export capability to blob storage or a data warehouse; docs cover logging, tracing, experiments, and MCP dataset management but never mention exporting data out to S3/GCS/BigQuery/Snowflake, and the OpenAPI probe returned 404s with no export endpoint mentioned.
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Cekuranone0/10Evidence shows Cekura has an API, CLI, SDK, and dashboards for accessing call data, but there is no mention of bulk-export functionality to blob storage (S3/GCS) or data warehouse integrations (Snowflake, BigQuery, etc). Missing for 10: bulk export feature, blob storage integration, data warehouse connectors/sinks, any documentation or API endpoint for exporting datasets in bulk.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets
Measuring quality — datasets, eval runs, regression tracking
Ai eval ops
ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results
weight 2 · round to CekuraGalileo's official MCP server explicitly exposes dataset creation, experiment (eval) running, and prompt template management directly from an agent's dev environment, and separate SDK/decorator logging plus experiment docs confirm results are captured and queryable. Missing for 10: no hands-on/independent confirmation of an agent actually reading back structured eval results via MCP, and no explicit example showing the full create-dataset→run-eval→read-results loop end-to-end.
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
Cekura ships a CLI/SDK explicitly for managing agents, scenarios (datasets), runs, and call data from code, plus an official MCP server letting an AI assistant 'design, run, and improve' evaluations, backed by a documented OpenAPI spec and API-key auth for programmatic access — together covering dataset creation, triggering eval runs, and reading back results. missing for 10: independent/hands-on confirmation that an agent using the MCP/CLI actually completes the full create-dataset-to-trigger-run-to-read-results loop end-to-end.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
Human review
ml engineerRoute outputs to human annotation queues for review and labeling
weight 2 · round to GalileoGalileo docs mention capturing 'subject matter expert annotations' to build datasets and using natural-language feedback to align metrics, implying some human-in-the-loop capability, but there is no explicit documentation of a dedicated annotation queue, review workflow, task assignment, or labeling UI for routing outputs to human reviewers. missing for 10: dedicated annotation queue/workflow feature, reviewer assignment mechanism, labeling UI documentation, independent corroboration of human review routing.
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
- [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”
Cekuranone0/10Cekura's evidence pack covers voice-agent testing, evaluation metrics (LLM-judge, Python), dashboards, PII redaction, and observability, but there is no mention of routing outputs to human annotation queues, labeling workflows, or human-in-the-loop review pipelines. Absence of evidence for this applicable ML-eval capability yields none. Missing for 10: any mention of human annotation queues, labeling UI/workflow, or reviewer assignment features.
Offline evals
developerRun evals in CI and gate deployments on their results
weight 2 · round to CekuraGalileonone0/10Evidence shows experiments/evals can be run via console or SDK, but there is no mention of a CI integration, CLI exit codes, or gating deployments based on eval results. missing for 10: CI/CD integration docs, pass/fail thresholds for gating, pipeline examples (GitHub Actions, Jenkins, etc.), any mention of 'CI' or 'gate' in evidence.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
Cekura provides a dedicated GitHub Actions CI/CD guide plus CLI/SDK and cron job automation for testing agents on each change, indicating CI integration is a first-class capability. However, the evidence pack only shows the guide's title/description, not concrete details on exit-code/gating behavior for deployments, and there is no independent corroboration of this specific workflow. Missing for 10: explicit documentation of deployment-gating logic (pass/fail thresholds blocking merges/deploys), and independent/hands-on confirmation of the GitHub Actions integration.
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ml engineerWrite custom code-based scorers and metrics for my evaluations
weight 2 · round to CekuraDocs explicitly mention custom code-based metrics as an extension to built-in metrics, alongside LLM-as-a-judge metrics, confirming the capability exists. However, evidence lacks concrete SDK examples, API reference, or hands-on walkthroughs showing how to author and register a custom scorer function. Missing for 10: code samples/API reference for writing custom scorers, independent corroboration of usage, details on scorer registration/execution lifecycle.
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
Cekura explicitly documents Python metrics that let engineers 'write custom evaluation logic in Python to evaluate your AI agent's performance with complete control', alongside LLM-judge metrics as an alternative, giving ML engineers code-based scorer capability. missing for 10: no independent hands-on example of a custom Python metric in use, and no detail on SDK/CLI invocation of custom scorers beyond the docs page.
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
ml engineerCompare eval runs side by side to catch regressions between prompt or model versions
weight 3 · round to CekuraGalileo's Experiments feature lets you evaluate prompts, models, and code against chosen metrics, and the console provides a way to create and add experiments to a project, implying some run-to-run evaluation tracking. However, no evidence explicitly describes a side-by-side comparison view or regression-detection UI/workflow between prompt or model versions. missing for 10: explicit side-by-side comparison UI, diffing/regression alerts between experiment runs, independent user confirmation of comparison workflow.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
Cekura has an explicit A/B testing feature to compare two agent versions side-by-side to measure prompt, model, or configuration changes, plus dashboards for visualizing metrics/call data, which supports regression detection between versions. However, the story specifically frames this as an ML-engineer 'eval run' comparison workflow, and evidence is limited to a single docs page description without detail on run history diffing, metric-level regression flags, or independent hands-on confirmation. Missing for 10: detailed UI/workflow evidence of comparing full eval-run history (not just two agent versions), regression-flagging mechanics, and independent/community corroboration of the comparison feature itself.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
- [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”
ml engineerScore outputs with configurable LLM-as-a-judge evaluators
weight 3 · round drawnDocs explicitly describe LLM-as-a-judge as a configurable metric type alongside custom code-based metrics, plus continuous feedback loops to align metrics to domain needs, and experiments to run these metrics against outputs. Missing for 10: independent/hands-on corroboration beyond vendor docs and more detail on configuring specific judge prompts/models.
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
- [claimed-docs] “This allows you to continuously provide feedback in natural language that automatically improves the metrics to align better with your domai…”
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
Cekura documents an LLM-judge metric type that evaluates calls using natural language criteria, alongside custom Python metrics for full control, giving configurable LLM-as-a-judge evaluators for scoring agent outputs. This is corroborated by dashboards/A-B testing docs showing these metrics feed into broader evaluation workflows. Missing for 10: no independent hands-on benchmark of judge accuracy/configurability, and no detail on rubric customization depth or multi-model judge support.
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
ml engineerCurate datasets from production traces and run offline evaluations against them
weight 3 · round to GalileoGalileo explicitly supports building datasets from production/live traces (galileo-docs-11), capturing traces/spans in production (galileo-docs-6, galileo-docs-7), and running offline evaluations/experiments against datasets with custom or LLM-as-judge metrics (galileo-docs-2, galileo-docs-8). This directly covers curating datasets from production traces and running offline evals. Missing for 10: independent/hands-on corroboration of the full production-trace-to-dataset-to-experiment workflow beyond vendor docs.
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
Cekura stores production call data as CallLog objects via its observability API and lets teams run metric evaluations (Python/LLM-judge) and scenario-based tests, which supports building and evaluating datasets from real traces, but there is no explicit documentation of a 'dataset curation' workflow (filtering/tagging/versioning production traces into an eval set) or an offline batch-eval mode distinct from live/simulated runs. missing for 10: dedicated dataset-curation/versioning feature for production traces, explicit offline-evaluation-on-curated-dataset workflow, independent evidence of this specific use case.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
Online evals
ml engineerRun evaluators continuously on live production traffic, not just offline datasets
weight 2 · round drawnGalileo's docs describe real-time capture of every session/trace/span, alerting on live logs, and distilled 'Luna' models that monitor 100% of production traffic at lower cost, which directly supports continuous evaluator execution on live traffic (not just offline datasets), complemented by custom/LLM-as-judge metrics and datasets built from live production data. missing for 10: independent/hands-on verification that evaluators run continuously in production (all evidence is vendor docs) and clearer detail on evaluator scheduling/latency at scale.
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “Galileo enables you to get alerted whenever unexpected things happen.”
- [claimed-docs] “Distill your optimized evals into Luna models that monitor 100% of your traffic at 96% lower cost.”
- [claimed-docs] “Build your datasets from synthetic, development, and live production data. Capture subject matter expert annotations to create a living asse…”
- [claimed-docs] “You can then expand these metrics with custom metrics, using LLM-as-a-judge, or custom code-based metrics.”
Cekura's observability pipeline explicitly ingests live production calls via webhook/API, stores them as CallLogs, and 'schedules metric evaluation' automatically, with marketing copy specifically citing 'Live drift detection across every call' as a continuous production-monitoring feature distinct from offline simulation runs. missing for 10: independent/hands-on confirmation of evaluator behavior on live traffic at scale (only vendor docs/marketing cited).
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Live drift detection across every call.”
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to CekuraGalileo exposes a Python SDK, decorators, and an MCP server that cover core workflows (tracing, experiments, datasets, prompt templates), suggesting many UI actions have API/SDK equivalents (galileo-docs-2, galileo-docs-4, galileo-gh-1). However, docs explicitly describe some actions (e.g., 'Create Experiment' button) as UI-only, and no public OpenAPI/swagger spec is discoverable (galileo-probe-3), so full API parity is unproven. Missing for 10: an explicit statement or spec confirming 1:1 API/UI feature parity, a discoverable OpenAPI reference, and evidence that console-only features (alerts setup, dashboards) have API equivalents.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [github] “You can also use the `@log` decorator to log spans.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Cekura ships a full OpenAPI spec, CLI, and SDK that cover core workflows—managing agents, scenarios, runs, call data, sending calls, and metrics—giving strong programmatic parity with the UI (cekura-docs-4, cekura-docs-5, cekura-docs-6, cekura-probe-3, cekura-probe-5). However, some UI-specific features like custom dashboard building (cekura-docs-11) and cron/CI setup aren't clearly confirmed as fully API-driven rather than UI-configured. Missing for 10: explicit confirmation that dashboard creation and all UI-configurable settings are exposed via API/CLI, and independent verification of complete feature parity.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnGalileonone0/10No evidence in the pack describes any data export feature, open-format export, or data portability mechanism for traces, datasets, or experiments — only ingestion, logging, and metric features are documented.
ai-native userRead the product's source under an open license
weight 2 · round to GalileoEvidence shows only a GitHub repo for the Python client SDK (galileo-python), with no license details or indication that the core Galileo platform/backend is open source. missing for 10: explicit open-source license text, evidence that the full product (not just a client SDK) is source-available, independent confirmation of license terms.
- [github] “You can also use the `@log` decorator to log spans.”
Cekuranone0/10Cekura is a closed SaaS platform with a GitHub repo only for a skills/plugin package (cekura-skills), not the product's core source code, and no evidence of an open-source license for the platform itself. No license file or open-source repo for the core product is mentioned anywhere in the evidence.
ai-native userSelf-host the core product
weight 3 · round drawnGalileonone0/10No evidence of a self-hostable/on-prem version of Galileo; all documentation points to a hosted console/SaaS product with SDKs and MCP integration, not a self-hosted deployment option. missing for 10: any mention of self-hosting, on-prem deployment, Docker/Helm packages, or enterprise private-cloud install instructions.
Cekuranone0/10Cekura is presented as a hosted SaaS platform (API keys, cloud dashboards, webhooks) with no mention of a self-hosted or on-premises deployment option anywhere in the docs, CLI, SDK, or website copy. No evidence of Docker images, self-hosting instructions, or open-source core.
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [claimed-docs] “create a comprehensive plan to integrate cekura observability”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnGalileonone0/10No evidence pack item mentions data residency, regional storage options, or compliance controls for data location; the evidence covers tracing, experiments, metrics, and MCP only. Since Galileo is a SaaS platform where data residency is a plausible and common enterprise concern, absence of evidence means 'none' rather than 'na'.
ai-native userPrevent my data from being used to train AI models
weight 3 · round drawnGalileonone0/10The evidence pack covers Galileo's tracing, experiments, metrics, and MCP features but contains no mention of data usage policies, opt-out of model training, or privacy controls regarding customer data being used to train AI models.
ai-native userControl data retention and deletion
weight 2 · round drawnGalileonone0/10The evidence pack covers tracing, experiments, metrics, and MCP integration but contains no mention of data retention policies, deletion controls, or privacy/compliance configuration options for AI-native users. No documentation cites retention windows, data deletion APIs, or export/purge capabilities.
Cekuranone0/10Cekura documents PII redaction to remove sensitive info from transcripts, but there is no evidence of user-facing data retention controls or deletion policies/APIs (e.g., data export, retention windows, right-to-delete). missing for 10: documented retention policy, deletion/export mechanism, user control over stored call data lifecycle.
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnGalileonone0/10No evidence pack items mention telemetry opt-out, usage tracking controls, or privacy settings for Galileo itself; the evidence only covers product features like tracing, experiments, and metrics.
Cekuranone0/10No evidence of any telemetry opt-out or usage tracking controls; Cekura's docs focus on PII redaction for customer call data, not on the vendor's own product telemetry. Missing for 10: any mention of telemetry/analytics collection, opt-out mechanism, or privacy settings for usage data.
Prompt management — stories about prompt management in this arenaPrompt management
Stories about prompt management in this arena
Prompt workflow
developerIterate on prompts in a playground against real models and variables
weight 2 · round to GalileoGalileo's Experiments feature lets users evaluate prompts and models against defined inputs and metrics via a console UI, and prompt templates can be set up through the MCP integration, which loosely maps to prompt iteration with variables. However, there's no evidence of a dedicated interactive 'playground' for live, real-time prompt testing against models — missing for 10: a documented playground UI, live model response preview, and variable substitution workflow distinct from formal experiment runs.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “In the Galileo console UI, "Create Experiment" buttons allow you to easily add experiments to a project.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
Cekuranone0/10Cekura's docs describe testing/evaluation workflows (scenario runs, A/B testing agent versions, synthetic conversations) but there is no evidence of an interactive playground where a developer iterates on prompt text against real models with variables in real time — the closest items (ab-testing, testing-agents/overview) describe comparing pre-built agent versions, not a live prompt-editing sandbox.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [claimed-docs] “Learn how to quickly create and test voice agents on Cekura without requiring any external API keys or integrations.”
- [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”
Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation
Instrumenting code and tracing requests end to end
Ai observability
ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app
weight 3 · round to CekuraGalileo has a documented official MCP server (galileo-docs-4, galileo-probe-4) enabling access to Galileo capabilities from a dev environment, but the explicitly listed MCP capabilities are creating/managing datasets, running experiments, and setting up prompt templates—not querying traces, metrics, or eval results directly. No OpenAPI/API reference was discoverable (galileo-probe-3 returned 404s), so there's no clear evidence an agent can query traces/metrics/eval results programmatically for debugging via API or MCP. missing for 10: explicit MCP/API support for querying traces and metrics, evidence of eval-result retrieval via MCP, and a discoverable REST/OpenAPI spec for programmatic trace queries.
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.galileo.ai/openapi.json, https://docs.galileo.ai/swagger.json, https://docs.galileo.ai/…”
Cekura has documented API access (OpenAPI spec, send-calls, SDK/CLI for agents/scenarios/runs/call data) and an official MCP server, but the MCP server's documented purpose is to 'design, run, and improve voice-agent evaluations' rather than explicitly querying traces/metrics/eval results for debugging — no direct evidence the MCP tools expose trace/metric/eval query endpoints for agent-driven debugging. missing for 10: explicit MCP tool list showing trace/metric/eval query capabilities, independent hands-on confirmation of an agent using MCP or API to debug via traces/metrics/evals.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
Data controls
developerMask or redact sensitive data before it is stored in traces
weight 2 · round to CekuraGalileonone0/10No evidence in the pack mentions masking, redaction, PII scrubbing, or any data sanitization mechanism applied before storing traces; documentation focuses on logging, tracing, metrics, and experiments without addressing sensitive data handling.
Cekura has a dedicated PII redaction feature that automatically detects and removes sensitive information from transcripts and audio recordings before/at storage, directly addressing masking/redaction in traces. missing for 10: no independent/hands-on corroboration of redaction accuracy or configurability details beyond the doc summary.
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
Sdk coverage
developerInstrument apps in both Python and JS/TS with officially supported SDKs
weight 2 · round to GalileoEvidence confirms a Python SDK (galileo-python) with decorator-based span logging and OTel-based distributed tracing, but no evidence of an official JS/TS SDK or its documentation. missing for 10: JS/TS SDK repo or docs, parity of instrumentation features (decorators, spans) between Python and JS/TS.
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
Cekura documents a general SDK ('sync and async clients for application code') and a Python-specific custom metric guide, implying Python SDK support, but there is no explicit mention of an officially supported JS/TS SDK anywhere in the evidence pack. missing for 10: explicit JS/TS SDK documentation, parity of features across languages, and confirmation the generic SDK is multi-language rather than Python-only.
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
Trace capture
developerTrace multi-step agent runs as nested spans grouped into sessions or threads
weight 2 · round to GalileoDocs explicitly describe capturing sessions, traces, and spans with structured logging, and distributed tracing docs show spans joined via shared trace IDs (nested spans under a trace) plus the @log decorator for span-level instrumentation. Missing for 10: explicit worked example showing session/thread grouping across multiple agent runs and independent hands-on corroboration beyond first-party docs.
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “Agentic metrics help you measure how well your AI agents perform complex, multi-step tasks—especially when those agents need to use tools, m…”
Cekuranone0/10Cekura's observability evidence centers on capturing call-level logs (CallLog with transcript, recording, metadata) and LiveKit SDK tracing, plus dashboards and PII redaction, but nothing in the evidence describes a hierarchical trace model with nested spans grouped into sessions or threads for multi-step agent runs. Missing for 10: explicit span/trace data model, parent-child span nesting, and session/thread grouping documentation.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK
weight 2 · round to CekuraGalileonone0/10The evidence pack shows Galileo's own Python SDK (decorator-based logging) and generic OpenTelemetry distributed tracing support, but contains no mention of pre-built integrations for LangChain, the OpenAI SDK, or the Vercel AI SDK specifically. Since this is a well-known, plausible capability for a tracing/observability product, absence of evidence means 'none' rather than 'na'.
- [github] “You can also use the `@log` decorator to log spans.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
Cekura documents a generic observability API/webhook (send-calls) and SDK/CLI that any agent could POST to, plus a named integration for LiveKit voice agents, but there is no documented integration for LangChain, the OpenAI SDK, or the Vercel AI SDK specifically as the story requests. missing for 10: named LangChain integration, named OpenAI SDK integration, named Vercel AI SDK integration.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
developerCapture multimodal payloads (images, audio, files) inside my traces
weight 1 · round to CekuraGalileonone0/10The evidence pack covers tracing, spans, metrics, alerts, and MCP integration, but contains no mention of multimodal payload capture (images, audio, files) within traces or spans.
Cekura's observability pipeline explicitly captures audio (recording URL) and transcript/metadata as part of CallLog traces, and PII redaction applies to both transcripts and audio recordings, showing audio is a first-class multimodal payload. However, there is no evidence of support for images or generic file attachments within traces — the product is voice-agent focused and docs never mention image/file capture. Missing for 10: documented support for image payloads, documented support for arbitrary file attachments in traces.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format
weight 3 · round to GalileoDocs confirm Galileo's SDK supports distributed tracing using OpenTelemetry's W3C traceparent header to propagate context and stitch spans into a single trace, showing OTel compatibility beyond a fully proprietary format. However, there's no evidence of a dedicated OTLP ingestion/export endpoint or explicit statement that Galileo accepts/emits OTLP-formatted traces from arbitrary OTel collectors. missing for 10: explicit OTLP endpoint/collector documentation, confirmation of both sending and receiving OTLP data, and independent verification of OTLP interoperability outside Galileo's own SDK.
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
- [github] “You can also use the `@log` decorator to log spans.”
Cekuranone0/10Cekura's observability ingestion uses a proprietary POST endpoint (transcript, recording URL, metadata) and its own API/CLI/SDK, with no mention of OpenTelemetry or OTLP support anywhere in the evidence pack.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage
weight 3 · round to GalileoGalileo's docs describe capturing sessions, traces, and spans once instrumented, with a `@log` decorator for logging spans and OTel-based distributed tracing joining spans by trace ID, forming a structured real-time data stream. However, explicit confirmation that captured traces include latency and token usage fields specifically is not directly quoted, only implied via 'structured stream of real-time data.' Missing for 10: explicit documentation/screenshot confirming latency and token-usage fields are captured per trace, and independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “Create and run your first trace in less than 5 minutes.”
- [claimed-docs] “Once instrumented, Galileo captures every session, trace, and span, producing a structured stream of real-time data.”
- [claimed-docs] “OpenTelemetry's W3C `traceparent` header carries the trace context across the wire. Galileo joins all spans that share a trace ID into a sin…”
- [github] “You can also use the `@log` decorator to log spans.”
Cekura's observability captures call-level data (transcript, recording URL, metadata) via its send-calls API and stores it as a CallLog, and offers 'enhanced observability' for LiveKit agents, but the docs never explicitly mention capturing per-LLM-call inputs/outputs, latency, or token usage — the focus is on transcript/audio-level call logging for voice-agent evaluation rather than granular LLM tracing. Missing for 10: explicit documentation of input/output payload capture per LLM call, latency metrics, and token usage tracking.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
Not comparable on these axes
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · not comparableGalileon/aGalileo is an LLM observability/evaluation platform, not an agentic tool-using product; the MCP evidence shows Galileo exposing its own capabilities via an MCP server for other clients (dev environments) to consume, not Galileo itself consuming external MCP servers to gain new tool capabilities. This 'plug servers in so it can use their tools' axis is a category mismatch for this kind of product.
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
- [probe] “official MCP server documented at https://docs.galileo.ai/getting-started/mcp/setup-galileo-mcp”
Cekuranone0/10All Cekura MCP evidence describes Cekura exposing its own MCP server so external AI assistants can call Cekura's tools (docs-3, probe-4), not Cekura itself acting as an MCP client that consumes third-party MCP servers' tools. No evidence shows Cekura can plug in and use external MCP servers.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
ai-native userVersion, review, and roll back my automations
weight 1 · not comparableGalileon/aGalileo is an LLM evaluation/observability platform (traces, experiments, metrics, alerts) — it does not build or run 'automations' as workflows that would need versioning, review, or rollback; this axis targets a different product category.
Cekura supports comparing two versions of an agent side-by-side (A/B testing) and provides dashboards/metrics for reviewing agent performance, which touches 'version' and 'review' aspects, but there is no evidence of a rollback mechanism to revert an automation/agent to a prior version. Missing for 10: explicit version history/versioning system, rollback/undo capability, and any documented workflow for reverting agent configurations.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
developerVersion prompts and deploy changes to production without shipping code
weight 3 · not comparableGalileonone0/10Evidence shows Galileo supports experiments for evaluating prompts and mentions 'setting up prompt templates' via MCP, but there is no documentation of prompt versioning, a prompt registry, or a mechanism to deploy prompt changes to production independent of code deploys.
- [claimed-docs] “Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.”
- [claimed-docs] “With MCP, you can access Galileo's capabilities directly from your development environment, including: Creating and managing datasets, Runni…”
Cekuran/aCekura is a testing/observability platform for voice AI agents (simulated calls, metrics, A/B comparison of agent versions), not a prompt-management or deployment system. Prompt versioning and production deployment without code changes belongs to the underlying agent-building platform, not to a third-party evaluation/observability tool like Cekura, making this axis a category error for this product type.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”