LLM Evals & Observability Arena
Helicone vs Cekura
Helicone
Helicone, Inc.
Cekura wins · 17–21 (7 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to CekuraHelicone has a confirmed llms.txt at docs.helicone.ai/llms.txt (HTTP 200) with an index of docs, plus per-page .md versions and an OpenAPI/swagger.json spec, all directly probeable and agent-consumable. missing for 10: no independent/community confirmation that agents actually consume this llms.txt successfully in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.helicone.ai/llms.txt # Helicone OSS LLM Observability - [Quickstart](https://docs.helicone.ai/gett…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.helicone.ai/getting-started/quick-start.md > ## Documentation Index > Fetch the complete documentati…”
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
Cekura has a confirmed live llms.txt endpoint (HTTP 200) plus per-page .md documentation format and an index pointing agents to llms.txt, directly enabling agent-oriented doc consumption; this is corroborated by both docs and live probes. Missing for 10: independent third-party confirmation of an agent actually consuming llms.txt successfully in the wild.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.cekura.ai/documentation/introduction.md > ## Documentation Index > Fetch the complete documentation …”
- [claimed-docs] “create a comprehensive plan to integrate cekura observability”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to CekuraHelicone is fundamentally an API/SDK-based proxy and observability layer — one-line SDK integration, REST API for queries, webhooks, and self-hosted Docker deployment all imply it can run without any UI, i.e., headlessly in scripts or pipelines. However, there is no explicit documentation or example of CI/CD usage, automated test pipelines, or headless agent orchestration specifically. Missing for 10: explicit CI/CD integration guide or example, documented non-interactive/automation-focused workflow, and independent confirmation of headless operation in production pipelines.
- [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
Cekura documents a CLI/SDK for programmatic and terminal-based agent/scenario/run management, a REST API with OpenAPI spec and API keys, cron job scheduling for automated evaluation workflows, and explicit GitHub Actions CI/CD integration for testing agents on every change—together these clearly support headless/CI automation. Missing for 10: independent hands-on confirmation of the GitHub Actions or cron setup actually working in a real pipeline beyond docs.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
ai-native userConnect an agent via an official MCP server
weight 3 · round to CekuraHeliconenone0/10Helicone is an LLM observability/gateway platform, and this axis (offering an official MCP server so agents can connect) plausibly applies since it has an ecosystem of integrations, docs, and APIs. However, no evidence in the pack mentions an MCP server, MCP integration, or any agent-connection protocol offered by Helicone.
Cekura is not itself an agent but a testing/observability platform, so the axis applies; it documents an official MCP server ("Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations") confirmed live via probe. Missing for 10: independent hands-on verification of the MCP server working end-to-end and more detail on supported tool/resource surface.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
ai-native userUse an official CLI
weight 2 · round to CekuraHeliconenone0/10No evidence of an official Helicone CLI tool; integration is via SDKs, API keys, gateway, and REST/OpenAPI, but no CLI is mentioned anywhere in docs, GitHub, or community sources.
Cekura documents an official CLI ('Manage agents, scenarios, runs, and call data from your terminal') alongside an SDK, confirmed by both docs and probe evidence. Missing for 10: independent hands-on review of the CLI's usability/completeness beyond first-party docs.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
ai-native userDrive the product through a documented public API
weight 3 · round to CekuraHelicone publishes a documented REST API (swagger/OpenAPI spec, point-query endpoints, HQL) plus SDK/gateway APIs, all accessible via public docs and confirmed by a live OpenAPI probe. This gives AI-native users a documented programmatic surface beyond the UI. Missing for 10: deeper docs/examples of full CRUD API coverage and independent third-party corroboration of API usage beyond ingestion/logging.
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.helicone.ai/llms.txt # Helicone OSS LLM Observability - [Quickstart](https://docs.helicone.ai/gett…”
- [github] “AI Gateway: Access 100+ AI models with 1 API key through the OpenAI API with intelligent routing and automatic fallbacks.”
Cekura publishes a documented public API with OpenAPI spec (probe confirms live openapi.json), API key auth, dedicated API reference endpoints (e.g. send-calls), plus SDKs (sync/async) and a CLI for programmatic/terminal control—clearly enabling an AI-native user to drive the product via documented API access. Missing for 10: independent third-party review/hands-on validation of the API beyond vendor docs and probes.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to CekuraHeliconenone0/10No evidence Helicone supports issuing scoped or least-privilege API credentials/keys for agents; docs mention bringing your own provider keys or using Helicone's own key, but nothing about granular permission scoping. A community comment even flags concern about sending a full API key on every request, with no rebuttal describing scoped-key mitigation.
- [claimed-docs] “Want more control? You can bring your own provider keys instead.”
- [community] “Hmm, so to integrate I have to basically send my api key to you on every request? Not great”
Cekura documents API key creation restricted to project-level scope, showing some credential scoping capability, but there's no evidence of finer-grained least-privilege controls like read/write permission tiers, resource-specific scoping, or per-agent credential issuance. missing for 10: granular permission scopes (read-only vs write), per-agent or per-resource credential issuance, role-based access control documentation.
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userBuild against official SDKs
weight 2 · round drawnHelicone works by wrapping official/familiar SDKs (OpenAI SDK, Anthropic, LangChain, Gemini, Vercel AI SDK) with one-line integration, backed by docs, GitHub README, and independent hands-on community confirmation of easy SDK-based integration. missing for 10: no dedicated first-party Helicone SDK docs page enumerated (language coverage) and no independent benchmark of SDK completeness beyond community anecdotes.
- [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”
- [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
- [community] “Happy Helicone customer here. It's a dead simple setup. It's great to have the extra charts and logging to debug issues and make sure all is…”
- [community] “Your onboarding is impressive, one of the few products where 'get set up in one line of code' is true.”
Cekura documents official sync/async SDKs and a CLI for programmatic access ('Manage agents, scenarios, runs, and call data from your terminal', 'sync and async clients for application code'), plus an OpenAPI spec, backed by probe confirmation of live docs and endpoints. missing for 10: independent hands-on developer reports specifically about SDK usage (only community praise for the platform generally, not SDK-specific).
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userSubscribe to events via webhooks
weight 2 · round to HeliconeHelicone documents a dedicated webhooks feature that sends instant notifications on LLM request completion, supports property-based filtering, and enables automation/scoring/integration with external systems, directly matching the story's subscribe-to-events use case. missing for 10: independent hands-on confirmation of webhook reliability/usage and richer docs on payload schema or delivery guarantees.
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Only requests matching ALL specified properties will trigger webhooks.”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
Cekuranone0/10Evidence shows Cekura receiving inbound webhooks (agent's provider posts call data to Cekura) but nothing about Cekura emitting outbound webhooks that a user could subscribe to for events like evaluation completion or drift detection. Missing for 10: documented outbound webhook/event subscription mechanism, webhook configuration UI/API, event types list.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round to CekuraHelicone advertises 'real-time evaluation' that automatically scores and evaluates LLM responses for quality, safety, and relevance, which is a form of AI-generated insight on the user's own request data, but the evidence pack shows no broader AI-generated dashboard summaries, anomaly explanations, or proactive suggestions beyond this single scoring feature. missing for 10: evidence of AI-generated narrative insights/recommendations across usage/cost data, natural-language querying of analytics, or proactive suggestion features beyond automated response scoring.
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
Cekura provides LLM-judge evaluation metrics, live drift detection, and auto-generated fix suggestions ('flags issues → reproduces in simulation → suggests fixes automatically'), showing AI-generated insights derived from call/agent data. However, this is narrowly scoped to voice-agent testing/observability rather than broad, ad-hoc AI-generated insights across arbitrary product data, and there's no independent hands-on evidence of the 'suggests fixes automatically' claim beyond marketing copy. Missing for 10: independent/hands-on verification of the auto-fix-suggestion feature, broader insight/analytics generation beyond voice-agent metrics, and detail on how suggestions are surfaced to users.
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Live drift detection across every call.”
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round to HeliconeHelicone publishes a formal OpenAPI/swagger spec and REST API reference pages, and separately offers a 'Playground' for testing/iterating on prompts in its UI, suggesting some interactive tooling exists. However, there is no direct evidence of a Swagger-UI-style interactive reference page where a user can execute runnable request examples against the API from the docs themselves. Missing for 10: confirmation of an actual 'try it out' runnable console in the API reference docs, and evidence of code-sample generation across languages.
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [claimed-docs] “Get Requests (Point Queries)”
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
Cekura publishes an OpenAPI spec and structured API-reference docs (e.g., send-calls, create-api-key endpoints) confirming a machine-readable interactive reference exists, but no evidence shows runnable 'try it' examples or a sandboxed console in the docs. Missing for 10: explicit runnable/try-it example UI, evidence of live request execution, independent confirmation of interactivity.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round drawnHelicone exposes a machine-readable OpenAPI spec at /swagger.json (HTTP 200, valid 'openapi' key), and its REST endpoints are documented (e.g., request query endpoints). missing for 10: no independent/community confirmation of the OpenAPI spec's completeness or usage.
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [claimed-docs] “Get Requests (Point Queries)”
A direct probe confirms an OpenAPI spec is publicly downloadable at https://docs.cekura.ai/openapi.json, returning HTTP 200 with a valid "openapi" key, corroborated by API reference docs built from it. Missing for 10: no independent third-party confirmation of spec completeness/versioning beyond the probe.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to CekuraHelicone offers a Playground for testing/iterating on prompts (not necessarily isolated from production logging) and self-host Docker Compose options for local development, which could function as an ad-hoc sandbox, but there is no explicit documented 'sandbox environment' or test-mode that guarantees isolation from production data. Missing for 10: a dedicated sandbox/staging mode, explicit statement about not touching production data, and independent verification of data isolation.
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
Cekura explicitly supports testing voice agents via synthetic/simulated conversations 'before go-live' and running scenarios without external API keys, which implies a sandbox-like testing flow separate from live production calls. However, there is no explicit documentation of a dedicated sandbox environment, data isolation guarantees, or explicit statements that test runs never touch production data/logs. missing for 10: explicit sandbox/staging environment documentation, data isolation guarantees between test and production calls, independent confirmation that simulated runs are fully separated from production data.
- [claimed-docs] “Learn how to quickly create and test voice agents on Cekura without requiring any external API keys or integrations.”
- [claimed-docs] “Run thousands of synthetic conversations before go-live.”
- [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnHeliconenone0/10An OpenAPI spec exists (helicone-probe-3) confirming a REST API, but there is no evidence of API versioning scheme (e.g., v1/v2 paths with migration guides) or a documented deprecation policy for endpoints/models; nothing in the docs pack addresses lifecycle or backward-compatibility commitments.
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [claimed-docs] “Get Requests (Point Queries)”
Cekuranone0/10There is an OpenAPI spec and API-key auth docs, but no evidence of API versioning scheme or a documented deprecation policy anywhere in the pack. missing for 10: versioning scheme documentation, deprecation policy, changelog/migration guides.
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
Alerting dashboards — stories about alerting dashboards in this arenaAlerting dashboards
Stories about alerting dashboards in this arena
Monitoring
ml engineerBuild custom dashboards over latency, error, cost, and eval-score metrics
weight 2 · round drawnHelicone provides building blocks for custom analytics — HQL query language for ad-hoc queries, REST API point queries, one-line export to PostHog for custom dashboards, real-time eval scoring via webhooks, and alerts on error rates/cost — but evidence does not show a native in-product dashboard builder combining latency, error, cost, and eval-score metrics together; users must rely on external tools (PostHog) or raw queries (HQL) to assemble true custom dashboards. Missing for 10: evidence of a first-party customizable dashboard UI, direct proof eval scores can be plotted alongside latency/cost/error in one dashboard, and independent hands-on confirmation of dashboard customization.
- [claimed-docs] “HQL (Query Language)”
- [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
- [github] “Export to PostHog in one-line for custom dashboards”
- [claimed-docs] “Get Requests (Point Queries)”
- [community] “Discussion of Helicone's architecture allowing users to write SQL directly to a shared ClickHouse instance, with commenters noting noisy-nei…”
Cekura docs explicitly describe building custom dashboards with widgets to visualize call data, metrics, and metadata, which covers eval-score and general metric visualization for an ML engineer's needs. However, the docs don't explicitly enumerate latency, error rate, or cost as first-class dashboard metrics, and there's no independent/hands-on corroboration of dashboard flexibility. Missing for 10: explicit mention of latency/error/cost metric widgets, and third-party evidence of dashboard customization in practice.
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
developerSet alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email
weight 2 · round to HeliconeHelicone documents a dedicated Alerts feature for monitoring error rates and cost spikes (helicone-docs-11), and webhooks for real-time notifications tied to request events (helicone-docs-9, helicone-docs-23), but the evidence pack never mentions eval-score-drop alerts or specific delivery channels like Slack, PagerDuty, or email. Missing for 10: eval-score-based alert triggers, and documented integrations/notification channels (Slack, PagerDuty, email).
- [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to CekuraHelicone offers query/export mechanisms (HQL query language, REST 'point queries' API, one-line export to PostHog) that could be used to operate on many logged requests at once, but there is no explicit documentation of bulk actions like batch tagging, batch deletion, or batch updates across many items. missing for 10: dedicated bulk-edit/bulk-delete/batch-processing API, documentation of batch size limits, and independent confirmation of bulk operation reliability.
- [claimed-docs] “HQL (Query Language)”
- [claimed-docs] “Get Requests (Point Queries)”
- [github] “Export to PostHog in one-line for custom dashboards”
Cekura's docs advertise running thousands of synthetic conversations/scenarios at once and provide a CLI/SDK to manage agents, scenarios, runs, and call data programmatically, which supports bulk test execution and automation via cron/CI. However, there's no explicit documentation of bulk edit/delete/update operations across many existing items (e.g., batch API endpoints for updating many scenarios or calls at once). Missing for 10: explicit bulk CRUD API/CLI commands operating on many items simultaneously, and any hands-on evidence of large-scale batch throughput.
- [claimed-docs] “Run thousands of synthetic conversations before go-live.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to CekuraHelicone supports webhooks that fire on request-completion events and can be filtered so only requests matching ALL specified properties trigger them, plus alerts that monitor error rates/costs — both are event-triggered automations. However, this is a narrow, fixed set of trigger types (webhooks, alerts) rather than a general user-defined rules engine with configurable conditions and arbitrary actions. Missing for 10: evidence of a flexible rule-builder UI/API for custom conditions and actions beyond webhooks/alerts, and independent confirmation these automations work reliably in production.
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Only requests matching ALL specified properties will trigger webhooks.”
- [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
Cekura supports automation triggers like scheduled cron jobs for test workflows, GitHub Actions CI/CD hooks that fire tests on code changes, and an automatic flag→reproduce→suggest-fix pipeline triggered by detected issues, which are event-driven automations. However, there's no evidence of a general-purpose user-defined rules engine (e.g., 'if X metric drops, then do Y') beyond these fixed automation patterns. Missing for 10: a documented custom rule/condition builder for arbitrary events, and independent confirmation these automations behave as advertised.
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Live drift detection across every call.”
ai-native userVersion, review, and roll back my automations
weight 1 · round to HeliconeHelicone's prompts feature explicitly supports versioning and rollback ('Track every change, compare versions, and rollback instantly if something goes wrong') and instant deployment of prompt changes without redeploying, which covers version/rollback for prompt-based automations. However, this is scoped to prompts specifically, not broader automation workflows (e.g., webhooks, alerts pipelines), and there's no 'review' workflow (approvals, diffs, comments) documented, nor independent/community corroboration of the rollback feature working in practice. Missing for 10: review/approval workflow evidence, broader automation (not just prompts) versioning, independent hands-on confirmation of rollback reliability.
- [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”
- [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
- [claimed-docs] “Use your prompt instantly by referencing its ID in your AI Gateway. No code changes, no rebuilds.”
Cekura supports comparing two versions of an agent side-by-side (A/B testing) and provides dashboards/metrics for reviewing agent performance, which touches 'version' and 'review' aspects, but there is no evidence of a rollback mechanism to revert an automation/agent to a prior version. Missing for 10: explicit version history/versioning system, rollback/undo capability, and any documented workflow for reverting agent configurations.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
Cost monitoring — stories about cost monitoring in this arenaCost monitoring
Stories about cost monitoring in this arena
Cost tracking
developerAttribute cost and usage to users, sessions, and features via custom metadata
weight 2 · round to HeliconeHelicone's docs show a Sessions feature that groups related requests to trace agent flows (helicone-docs-5, helicone-docs-6) and per-user cache namespaces (helicone-docs-13), which support some usage segmentation, but there is no explicit evidence of a general custom-metadata/custom-properties system for tagging requests by user, feature, or session to attribute cost and usage in dashboards or queries. Missing for 10: explicit custom-properties/tagging API, cost breakdown by user/feature dashboards, and documentation showing metadata-based cost attribution reports.
- [claimed-docs] “Sessions group these related requests together, letting you trace the entire agent flow from initial user input to final response in one uni…”
- [claimed-docs] “Requests with the same path represent the same "type" of work, even if they happen at different times.”
- [claimed-docs] “Create separate cache namespaces for different users or contexts.”
Cekuranone0/10Cekura's docs show metadata being sent with calls and custom dashboards for visualizing metrics/metadata, but there is no evidence of cost tracking, cost attribution, or usage-cost breakdown by user/session/feature — the platform is framed purely around voice-agent evaluation and observability, not cost accounting.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
developerSee cost and token usage per request, model, and time period in dashboards
weight 3 · round to HeliconeCommunity feedback confirms cost/logging visibility ('extra charts and logging to debug issues', 'how much it costs', 'analytics/observability') and docs reference request query APIs and an HQL query language for analysis, but no first-party documentation explicitly describes dashboards broken down by request, model, and time period. Missing for 10: explicit docs/screenshots of dashboard UI showing cost/token breakdowns by model and time period, and independent confirmation of that specific breakdown granularity.
- [community] “Happy Helicone customer here. It's a dead simple setup. It's great to have the extra charts and logging to debug issues and make sure all is…”
- [community] “When your chains get long/complex enough in LangChain, it's really hard to understand from debug output what's final prompt that actually be…”
- [community] “justin - helicone's great. I login in ~10 times a day. Great for debugging - I'm mostly using for analytics / observability.”
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
Cekuranone0/10Cekura documents customizable dashboards for call data, metrics, and metadata (cekura-docs-11), but nothing in the evidence pack mentions cost or token usage tracking per request, model, or time period. As an observability platform for voice agents, this axis plausibly applies, but no evidence supports the specific cost/token dashboard capability.
Data access export — stories about data access export in this arenaData access export
Stories about data access export in this arena
Data export
developerBulk-export traces and datasets to blob storage or my data warehouse
weight 2 · round to HeliconeHelicone offers a REST API for point queries, an HQL query language for direct SQL access to logged data, and a one-line PostHog export integration, which give some programmatic data-access/export capability. However, no evidence documents a dedicated bulk-export feature to blob storage (S3/GCS) or to a data warehouse (Snowflake/BigQuery) specifically for traces/datasets. Missing for 10: explicit blob-storage export, data-warehouse connector/sync, bulk/batch export tooling or docs beyond point queries and third-party PostHog export.
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
- [github] “Export to PostHog in one-line for custom dashboards”
- [community] “Discussion of Helicone's architecture allowing users to write SQL directly to a shared ClickHouse instance, with commenters noting noisy-nei…”
Cekuranone0/10Evidence shows Cekura has an API, CLI, SDK, and dashboards for accessing call data, but there is no mention of bulk-export functionality to blob storage (S3/GCS) or data warehouse integrations (Snowflake, BigQuery, etc). Missing for 10: bulk export feature, blob storage integration, data warehouse connectors/sinks, any documentation or API endpoint for exporting datasets in bulk.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets
Measuring quality — datasets, eval runs, regression tracking
Ai eval ops
ai-native userHave an agent create a dataset, trigger an eval run programmatically, and read back the results
weight 2 · round to CekuraHeliconenone0/10Evidence shows observability features (sessions, webhooks, HQL query, REST API for point queries) and a vague mention of 'real-time evaluation' scoring, but there is no documentation of a dataset-creation API, a way to programmatically trigger an eval run, or an API to read back eval results — the core building blocks of this story are absent from the evidence pack.
Cekura ships a CLI/SDK explicitly for managing agents, scenarios (datasets), runs, and call data from code, plus an official MCP server letting an AI assistant 'design, run, and improve' evaluations, backed by a documented OpenAPI spec and API-key auth for programmatic access — together covering dataset creation, triggering eval runs, and reading back results. missing for 10: independent/hands-on confirmation that an agent using the MCP/CLI actually completes the full create-dataset-to-trigger-run-to-read-results loop end-to-end.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
Human review
ml engineerRoute outputs to human annotation queues for review and labeling
weight 2 · round drawnHeliconenone0/10Helicone's evidence covers observability, logging, prompts, caching, webhooks, and real-time automated scoring, but there is no mention of routing outputs to human annotation queues or workflows for manual review/labeling by human annotators.
Cekuranone0/10Cekura's evidence pack covers voice-agent testing, evaluation metrics (LLM-judge, Python), dashboards, PII redaction, and observability, but there is no mention of routing outputs to human annotation queues, labeling workflows, or human-in-the-loop review pipelines. Absence of evidence for this applicable ML-eval capability yields none. Missing for 10: any mention of human annotation queues, labeling UI/workflow, or reviewer assignment features.
Offline evals
developerRun evals in CI and gate deployments on their results
weight 2 · round to CekuraHeliconenone0/10Helicone offers observability, webhooks, real-time scoring, caching, and prompt versioning, but there is no evidence of a CI-integrated eval runner, test suite, or deployment gating mechanism tied to eval results. Nothing in the evidence pack mentions CI pipelines, exit codes, or blocking deploys based on evaluation scores.
Cekura provides a dedicated GitHub Actions CI/CD guide plus CLI/SDK and cron job automation for testing agents on each change, indicating CI integration is a first-class capability. However, the evidence pack only shows the guide's title/description, not concrete details on exit-code/gating behavior for deployments, and there is no independent corroboration of this specific workflow. Missing for 10: explicit documentation of deployment-gating logic (pass/fail thresholds blocking merges/deploys), and independent/hands-on confirmation of the GitHub Actions integration.
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
ml engineerWrite custom code-based scorers and metrics for my evaluations
weight 2 · round to CekuraHelicone's webhooks feature mentions enabling users to 'score responses' and lists 'Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance', implying some custom scoring hook exists, but there is no documented SDK, API, or framework for writing custom code-based scorers/metrics as part of an evals workflow. missing for 10: dedicated evals/scorer API or SDK docs, examples of custom metric code, integration with a datasets/evals product surface.
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
Cekura explicitly documents Python metrics that let engineers 'write custom evaluation logic in Python to evaluate your AI agent's performance with complete control', alongside LLM-judge metrics as an alternative, giving ML engineers code-based scorer capability. missing for 10: no independent hands-on example of a custom Python metric in use, and no detail on SDK/CLI invocation of custom scorers beyond the docs page.
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
ml engineerCompare eval runs side by side to catch regressions between prompt or model versions
weight 3 · round to CekuraHelicone documents prompt version tracking/comparison and rollback (docs-18) plus real-time automated scoring via webhooks (docs-23) and a playground for iterating on prompts (gh-3/gh-6), which are adjacent building blocks for regression detection, but there is no explicit evidence of a dedicated 'eval run' comparison view or side-by-side regression report across prompt/model versions. Missing for 10: a documented eval-run comparison UI, dataset-based batch evaluation runs, and any hands-on/community confirmation of side-by-side regression catching.
- [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
Cekura has an explicit A/B testing feature to compare two agent versions side-by-side to measure prompt, model, or configuration changes, plus dashboards for visualizing metrics/call data, which supports regression detection between versions. However, the story specifically frames this as an ML-engineer 'eval run' comparison workflow, and evidence is limited to a single docs page description without detail on run history diffing, metric-level regression flags, or independent hands-on confirmation. Missing for 10: detailed UI/workflow evidence of comparing full eval-run history (not just two agent versions), regression-flagging mechanics, and independent/community corroboration of the comparison feature itself.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
- [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”
ml engineerScore outputs with configurable LLM-as-a-judge evaluators
weight 3 · round to CekuraHelicone's docs mention webhooks enabling 'real-time evaluation' to automatically score responses for quality/safety/relevance, which points toward LLM-as-judge scoring, but there's no evidence of a configurable evaluator framework, built-in judge models, or dataset-based eval workflows that an ML engineer could set up and customize. missing for 10: dedicated evals/scoring product docs, configurable judge model selection, dataset-linked evaluation runs, independent validation of eval accuracy.
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
Cekura documents an LLM-judge metric type that evaluates calls using natural language criteria, alongside custom Python metrics for full control, giving configurable LLM-as-a-judge evaluators for scoring agent outputs. This is corroborated by dashboards/A-B testing docs showing these metrics feed into broader evaluation workflows. Missing for 10: no independent hands-on benchmark of judge accuracy/configurability, and no detail on rubric customization depth or multi-model judge support.
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
ml engineerCurate datasets from production traces and run offline evaluations against them
weight 3 · round to CekuraHelicone provides adjacent capabilities—HQL query language and point-query APIs to pull production request data (helicone-docs-15, helicone-docs-16), a Playground to test/iterate on prompts, sessions and traces (helicone-gh-3, helicone-gh-6), and webhook-driven 'real-time evaluation' scoring (helicone-docs-23)—but there is no explicit mention of a dataset-curation workflow or a dedicated offline/batch evaluation feature run against saved datasets. missing for 10: explicit dataset creation/curation from traces, offline evaluation harness or benchmark-running against curated sets, and independent confirmation these workflows work end-to-end.
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
Cekura stores production call data as CallLog objects via its observability API and lets teams run metric evaluations (Python/LLM-judge) and scenario-based tests, which supports building and evaluating datasets from real traces, but there is no explicit documentation of a 'dataset curation' workflow (filtering/tagging/versioning production traces into an eval set) or an offline batch-eval mode distinct from live/simulated runs. missing for 10: dedicated dataset-curation/versioning feature for production traces, explicit offline-evaluation-on-curated-dataset workflow, independent evidence of this specific use case.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
Online evals
ml engineerRun evaluators continuously on live production traffic, not just offline datasets
weight 2 · round to CekuraHelicone offers webhooks for real-time scoring/evaluation of production requests as they complete ('Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance') plus alerts on error rates/costs, which supports continuous evaluation on live traffic. However, this is implemented via webhook-triggered external scoring rather than a documented native 'evaluator' framework applied continuously to production traffic, and there's no evidence of built-in evaluator templates, dashboards specifically for eval results over time, or comparison against offline dataset evals. Missing for 10: dedicated evaluator/eval-suite feature applied to live traffic, evidence of eval results surfaced in-product over time, and independent/hands-on confirmation that this works as described.
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
- [claimed-docs] “Only requests matching ALL specified properties will trigger webhooks.”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
- [claimed-docs] “Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
Cekura's observability pipeline explicitly ingests live production calls via webhook/API, stores them as CallLogs, and 'schedules metric evaluation' automatically, with marketing copy specifically citing 'Live drift detection across every call' as a continuous production-monitoring feature distinct from offline simulation runs. missing for 10: independent/hands-on confirmation of evaluator behavior on live traffic at scale (only vendor docs/marketing cited).
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Live drift detection across every call.”
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to CekuraHelicone exposes a REST API and OpenAPI spec (swagger.json) with query endpoints, and HQL for querying logs, plus API-driven prompt versioning/webhooks/caching that mirror UI features. However, some UI-centric features like the interactive Playground for testing prompts/sessions are explicitly UI-only, and there's no evidence of full API parity for all dashboard/analytics/alert configuration workflows. missing for 10: explicit documentation of API endpoints for alerts configuration, full analytics dashboard equivalents, and Playground-equivalent API/CLI tooling.
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
- [claimed-docs] “Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI act…”
Cekura ships a full OpenAPI spec, CLI, and SDK that cover core workflows—managing agents, scenarios, runs, call data, sending calls, and metrics—giving strong programmatic parity with the UI (cekura-docs-4, cekura-docs-5, cekura-docs-6, cekura-probe-3, cekura-probe-5). However, some UI-specific features like custom dashboard building (cekura-docs-11) and cron/CI setup aren't clearly confirmed as fully API-driven rather than UI-configured. Missing for 10: explicit confirmation that dashboard creation and all UI-configurable settings are exposed via API/CLI, and independent verification of complete feature parity.
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
ai-native userExport all of my data in open formats and leave
weight 3 · round to HeliconeHelicone offers a REST API for querying/retrieving request data (JSON) and one-line export to PostHog, plus self-hosting options that let users keep their own data in open infra (Postgres/ClickHouse). However there's no explicit documented bulk 'export all your data' or account-portability feature, and the 'open source' framing itself was disputed by the community over licensing (Commons Clause), which weakens the openness narrative. Missing for 10: an explicit full-data export/download tool, documented data-portability guarantees, and independent confirmation that self-hosted/exported data is complete and usable outside Helicone.
- [claimed-docs] “Get Requests (Point Queries)”
- [github] “Export to PostHog in one-line for custom dashboards”
- [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
- [community] “Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typi…”
ai-native userRead the product's source under an open license
weight 2 · round to HeliconeHeliconedisputedcontradicted5/10Helicone's source is publicly viewable on GitHub and marketed as open-source/self-hostable (helicone-docs-14, helicone-docs-20, helicone-docs-21), but a community commenter concretely disputes the 'open source' framing, noting the project uses the Commons Clause, which is not an OSI-recognized open license (helicone-comm-5). This is a direct contradiction of the openness claim rather than mere skepticism. Missing for 10: clear documentation of the actual license terms, confirmation of unrestricted OSI-approved licensing, and resolution of the Commons Clause concern.
- [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
- [claimed-docs] “Helicone gives you more provider flexibility, is open-source, and scales more cost-effectively.”
- [community] “Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typi…”
Cekuranone0/10Cekura is a closed SaaS platform with a GitHub repo only for a skills/plugin package (cekura-skills), not the product's core source code, and no evidence of an open-source license for the platform itself. No license file or open-source repo for the core product is mentioned anywhere in the evidence.
ai-native userSelf-host the core product
weight 3 · round to HeliconeHelicone documents multiple self-host deployment methods (e.g., Docker Compose) via a dedicated self-host overview page, and claims to be open-source with provider flexibility, supporting AI-native self-hosting. However, community evidence disputes the 'open source' label since it uses the Commons Clause license, and there's noted risk around shared ClickHouse infra suggesting the hosted version may differ architecturally from a fully self-hostable stack. Missing for 10: independent verification of a full self-hosted deployment working end-to-end, clarity on license restrictions for self-hosting, and confirmation that all core features (AI Gateway, sessions, caching) work identically when self-hosted.
- [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
- [claimed-docs] “Helicone gives you more provider flexibility, is open-source, and scales more cost-effectively.”
- [community] “Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typi…”
- [community] “Discussion of Helicone's architecture allowing users to write SQL directly to a shared ClickHouse instance, with commenters noting noisy-nei…”
Cekuranone0/10Cekura is presented as a hosted SaaS platform (API keys, cloud dashboards, webhooks) with no mention of a self-hosted or on-premises deployment option anywhere in the docs, CLI, SDK, or website copy. No evidence of Docker images, self-hosting instructions, or open-source core.
- [claimed-docs] “Create a new API key. When using API key authentication, only project-level API keys can be created.”
- [claimed-docs] “create a comprehensive plan to integrate cekura observability”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round to HeliconeHelicone's self-hosting options (Docker Compose, multiple deployment methods) let a user control where their own infrastructure runs, which indirectly enables data residency, but there is no explicit region-selection feature or documented data-residency/compliance options for the hosted SaaS offering. missing for 10: explicit region/residency selection UI or docs, compliance certifications (SOC2/GDPR data-location guarantees), and any mention of choosing storage location within the hosted product.
- [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
ai-native userControl data retention and deletion
weight 2 · round drawnHeliconenone0/10The evidence pack has no mention of data retention settings, deletion controls, TTL policies, or privacy/compliance controls for stored logs and cached data; it only covers logging, caching, prompts, and observability features. Self-hosting is mentioned but not tied to retention/deletion controls specifically.
Cekuranone0/10Cekura documents PII redaction to remove sensitive info from transcripts, but there is no evidence of user-facing data retention controls or deletion policies/APIs (e.g., data export, retention windows, right-to-delete). missing for 10: documented retention policy, deletion/export mechanism, user control over stored call data lifecycle.
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
ai-native userOpt out of telemetry and usage tracking
weight 2 · round to HeliconeHelicone's docs describe self-hosting options (Docker Compose, multiple deployment methods) which would let an AI-native user avoid sending usage data to Helicone's cloud entirely, but there is no explicit documented telemetry/analytics opt-out toggle within the product itself for cloud users. missing for 10: explicit opt-out setting for product usage telemetry, documentation of what data Helicone collects about its own users, and confirmation that self-hosting fully disables any phone-home telemetry.
- [claimed-docs] “Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
- [claimed-docs] “Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
Cekuranone0/10No evidence of any telemetry opt-out or usage tracking controls; Cekura's docs focus on PII redaction for customer call data, not on the vendor's own product telemetry. Missing for 10: any mention of telemetry/analytics collection, opt-out mechanism, or privacy settings for usage data.
Prompt management — stories about prompt management in this arenaPrompt management
Stories about prompt management in this arena
Prompt workflow
developerIterate on prompts in a playground against real models and variables
weight 2 · round to HeliconeGitHub docs explicitly advertise a Playground for rapidly testing and iterating on prompts, sessions and traces in the UI, and prompt docs mention instant testing/deploying of prompt changes and version tracking. However, there's no explicit evidence of variable/templating support within the playground UI, no hands-on/independent confirmation of the playground experience, and no detail on live model comparison or variable injection workflow. Missing for 10: independent/hands-on validation of the playground, explicit documentation of variable substitution in playground testing, and detail on comparing multiple models side-by-side in that UI.
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
- [claimed-docs] “Use your prompt instantly by referencing its ID in your AI Gateway. No code changes, no rebuilds.”
- [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”
Cekuranone0/10Cekura's docs describe testing/evaluation workflows (scenario runs, A/B testing agent versions, synthetic conversations) but there is no evidence of an interactive playground where a developer iterates on prompt text against real models with variables in real time — the closest items (ab-testing, testing-agents/overview) describe comparing pre-built agent versions, not a live prompt-editing sandbox.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [claimed-docs] “Learn how to quickly create and test voice agents on Cekura without requiring any external API keys or integrations.”
- [claimed-docs] “Run the same scenarios across platforms and models. Pick the one that actually performs.”
Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation
Instrumenting code and tracing requests end to end
Ai observability
ai-native userHave an agent query my traces, metrics, and eval results through an API or MCP server to debug my app
weight 3 · round to CekuraHelicone documents a REST API for querying requests (helicone-docs-15) and an HQL query language (helicone-docs-16), plus a discoverable OpenAPI spec (helicone-probe-3) and llms.txt for AI-agent-friendly docs (helicone-probe-1), enabling programmatic access to traces/sessions/metrics for debugging. However, there is no evidence of a dedicated MCP server offering, nor explicit documentation of querying eval results via API, so an agent's ability to pull all three (traces, metrics, evals) through a single API/MCP interface is only partially substantiated. missing for 10: explicit MCP server documentation, API endpoints/examples for querying eval results specifically, and independent confirmation of agent-based API usage for debugging.
- [claimed-docs] “Get Requests (Point Queries)”
- [claimed-docs] “HQL (Query Language)”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.helicone.ai/llms.txt # Helicone OSS LLM Observability - [Quickstart](https://docs.helicone.ai/gett…”
- [probe] “PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
- [claimed-docs] “Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”
Cekura has documented API access (OpenAPI spec, send-calls, SDK/CLI for agents/scenarios/runs/call data) and an official MCP server, but the MCP server's documented purpose is to 'design, run, and improve voice-agent evaluations' rather than explicitly querying traces/metrics/eval results for debugging — no direct evidence the MCP tools expose trace/metric/eval query endpoints for agent-driven debugging. missing for 10: explicit MCP tool list showing trace/metric/eval query capabilities, independent hands-on confirmation of an agent using MCP or API to debug via traces/metrics/evals.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [probe] “PROBE openapi: HTTP 200 at https://docs.cekura.ai/openapi.json — contains "openapi" key”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
Data controls
developerMask or redact sensitive data before it is stored in traces
weight 2 · round to CekuraHeliconenone0/10No evidence in the pack mentions data masking, redaction, PII scrubbing, or any privacy-preserving controls applied before storing traces; the documentation excerpts cover gateway, sessions, caching, prompts, webhooks, and pricing but none address sensitive data redaction. missing for 10: any mention of masking/redaction feature, PII detection, or configuration options to omit/obfuscate sensitive fields before storage.
Cekura has a dedicated PII redaction feature that automatically detects and removes sensitive information from transcripts and audio recordings before/at storage, directly addressing masking/redaction in traces. missing for 10: no independent/hands-on corroboration of redaction accuracy or configurability details beyond the doc summary.
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
Sdk coverage
developerInstrument apps in both Python and JS/TS with officially supported SDKs
weight 2 · round to HeliconeEvidence shows one-line integration via the OpenAI SDK and support for multiple frameworks (OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK) which span both Python and JS/TS ecosystems, and community reports confirm the same style easy setup in real projects. However, the pack never explicitly names distinct 'Helicone Python SDK' and 'Helicone JS/TS SDK' packages or shows side-by-side code samples for each language. Missing for 10: explicit dedicated Python SDK docs/package reference, explicit dedicated JS/TS SDK docs/package reference, and independent hands-on confirmation for each language separately.
- [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”
- [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
- [community] “Your onboarding is impressive, one of the few products where 'get set up in one line of code' is true.”
Cekura documents a general SDK ('sync and async clients for application code') and a Python-specific custom metric guide, implying Python SDK support, but there is no explicit mention of an officially supported JS/TS SDK anywhere in the evidence pack. missing for 10: explicit JS/TS SDK documentation, parity of features across languages, and confirmation the generic SDK is multi-language rather than Python-only.
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Write custom evaluation logic in Python to evaluate your AI agent's performance with complete control”
- [probe] “official CLI documented at https://docs.cekura.ai/cli-sdk/cli”
Trace capture
developerTrace multi-step agent runs as nested spans grouped into sessions or threads
weight 2 · round to HeliconeHelicone's Sessions feature explicitly groups related requests together to trace an entire agent flow in one unified view, with path-based grouping to identify recurring 'types' of work, and the Playground supports testing/iterating on prompts, sessions and traces. This directly matches nested/grouped multi-step agent tracing, though evidence lacks detail on explicit parent/child span nesting depth or independent hands-on verification of the sessions UI. missing for 10: independent/hands-on corroboration of nested span hierarchy depth, more detail on thread-level grouping beyond path-based sessions.
- [claimed-docs] “Sessions group these related requests together, letting you trace the entire agent flow from initial user input to final response in one uni…”
- [claimed-docs] “Requests with the same path represent the same "type" of work, even if they happen at different times.”
- [github] “Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
- [github] “Rapidly test and iterate on prompts, sessions and traces in our UI.”
Cekuranone0/10Cekura's observability evidence centers on capturing call-level logs (CallLog with transcript, recording, metadata) and LiveKit SDK tracing, plus dashboards and PII redaction, but nothing in the evidence describes a hierarchical trace model with nested spans grouped into sessions or threads for multi-step agent runs. Missing for 10: explicit span/trace data model, parent-child span nesting, and session/thread grouping documentation.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
- [claimed-docs] “Build custom dashboards with widgets to visualize and analyze your call data, metrics, and metadata.”
developerInstrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK
weight 2 · round to HeliconeGitHub docs explicitly advertise one-line integration for OpenAI, Anthropic, LangChain, Gemini, and Vercel AI SDK, and community testimonials corroborate easy OpenAI SDK/LangChain integration and quick onboarding. Missing for 10: dedicated docs pages per-framework (e.g., a LangChain-specific integration guide) and independent hands-on confirmation for Vercel AI SDK specifically.
- [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
- [community] “When your chains get long/complex enough in LangChain, it's really hard to understand from debug output what's final prompt that actually be…”
- [community] “Your onboarding is impressive, one of the few products where 'get set up in one line of code' is true.”
- [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”
Cekura documents a generic observability API/webhook (send-calls) and SDK/CLI that any agent could POST to, plus a named integration for LiveKit voice agents, but there is no documented integration for LangChain, the OpenAI SDK, or the Vercel AI SDK specifically as the story requests. missing for 10: named LangChain integration, named OpenAI SDK integration, named Vercel AI SDK integration.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
developerCapture multimodal payloads (images, audio, files) inside my traces
weight 1 · round to CekuraHeliconenone0/10The evidence pack covers logging, sessions, prompts, webhooks, caching, and deployment options, but contains no mention of capturing images, audio, or file payloads within traces—no documentation, GitHub feature, or community report addresses multimodal payload capture.
Cekura's observability pipeline explicitly captures audio (recording URL) and transcript/metadata as part of CallLog traces, and PII redaction applies to both transcripts and audio recordings, showing audio is a first-class multimodal payload. However, there is no evidence of support for images or generic file attachments within traces — the product is voice-agent focused and docs never mention image/file capture. Missing for 10: documented support for image payloads, documented support for arbitrary file attachments in traces.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Automatically detect and remove sensitive information from transcripts and audio recordings”
developerSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format
weight 3 · round drawnHeliconenone0/10No evidence pack items mention OpenTelemetry, OTLP, or any standard tracing protocol support; Helicone's docs describe proprietary logging via SDK integration, sessions, and REST API, not OTLP ingestion/export.
Cekuranone0/10Cekura's observability ingestion uses a proprietary POST endpoint (transcript, recording URL, metadata) and its own API/CLI/SDK, with no mention of OpenTelemetry or OTLP support anywhere in the evidence pack.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
developerCapture traces of my LLM calls with inputs, outputs, latency, and token usage
weight 3 · round to HeliconeHelicone's core product is LLM request logging with one-line integration capturing inputs/outputs, latency, tokens, and cost, corroborated by both docs and independent HN community reports of debugging and analytics use. Missing for 10: no explicit first-party doc snippet enumerating latency/token-usage fields captured per trace, relying on inference from 'automatic logging, observability' and community mentions of cost/debugging.
- [claimed-docs] “Get your first LLM request logged with Helicone in under 2 minutes using the AI Gateway.”
- [claimed-docs] “Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and …”
- [github] “Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [github] “One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
- [community] “We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why re…”
- [community] “Happy Helicone customer here. It's a dead simple setup. It's great to have the extra charts and logging to debug issues and make sure all is…”
- [community] “When your chains get long/complex enough in LangChain, it's really hard to understand from debug output what's final prompt that actually be…”
- [community] “justin - helicone's great. I login in ~10 times a day. Great for debugging - I'm mostly using for analytics / observability.”
Cekura's observability captures call-level data (transcript, recording URL, metadata) via its send-calls API and stores it as a CallLog, and offers 'enhanced observability' for LiveKit agents, but the docs never explicitly mention capturing per-LLM-call inputs/outputs, latency, or token usage — the focus is on transcript/audio-level call logging for voice-agent evaluation rather than granular LLM tracing. Missing for 10: explicit documentation of input/output payload capture per LLM call, latency metrics, and token usage tracking.
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
- [claimed-docs] “Enhanced observability for LiveKit agents using the Cekura SDK”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
Not comparable on these axes
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · not comparableHeliconen/aHelicone is an LLM observability/gateway product, not an agent; the evidence pack shows no MCP client role or capability to plug in MCP servers to use their tools—this axis doesn't apply to this product category.
Cekuranone0/10All Cekura MCP evidence describes Cekura exposing its own MCP server so external AI assistants can call Cekura's tools (docs-3, probe-4), not Cekura itself acting as an MCP client that consumes third-party MCP servers' tools. No evidence shows Cekura can plug in and use external MCP servers.
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
ai-native userSet up automations that run autonomously in the background
weight 2 · not comparableHeliconen/aHelicone is an LLM observability/gateway platform for logging, monitoring, and debugging LLM requests, not an automation/agent-runner product; it has no evidence of scheduled or autonomous background task execution, and this axis is a category error for its product type (webhooks/alerts are reactive, not autonomous automations).
Cekura supports autonomous background automation via cron jobs for scheduled testing/evaluation runs, GitHub Actions CI/CD integration, and live drift detection across calls, plus automatic flagging/reproduction/fix-suggestion pipelines. However, evidence is entirely first-party docs describing scheduling/triggering mechanisms rather than a broad general-purpose 'automation builder' for arbitrary agentic workflows. missing for 10: independent/hands-on confirmation that cron/CI automations run reliably unattended, evidence of more flexible trigger types beyond cron/CI/webhooks, and detail on monitoring/error-handling for long-running autonomous automations.
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Live drift detection across every call.”
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Your agent (or its provider's post-call webhook) POSTs here with the transcript, recording URL, and metadata; Cekura stores it as a `CallLog…”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · not comparableHeliconen/aHelicone is an LLM observability/gateway platform for logging, monitoring, prompts, caching, and analytics of LLM requests — it has no built-in AI assistant/agent persona for users to delegate tasks to; this axis is a category error for this product type.
Cekuranone0/10Cekura's AI-assistant integrations (Skills, MCP, CLI) are designed so external AI assistants like Claude or Cursor can drive Cekura's testing/evaluation features — this is the reverse relationship of an AI-native user delegating tasks to a built-in assistant inside Cekura itself. The 'flags issues → reproduces → suggests fixes automatically' pipeline is an automated feature, not a delegatable in-product assistant, so there is no evidence of a built-in AI assistant you can converse with or delegate open-ended tasks to.
- [claimed-docs] “Cekura flags issues → reproduces in simulation → suggests fixes automatically.”
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [claimed-docs] “npx skills add cekura-ai/cekura-skills --all”
ai-native userOperate the product with natural-language commands
weight 2 · not comparableHeliconen/aHelicone is an LLM observability/gateway platform for developers, not a conversational or command-driven interface; there's no evidence (or plausible expectation) of natural-language command operation as an axis for this product type.
Cekura ships an MCP server and Skills package so an AI assistant can design, run, and improve voice-agent evaluations via natural-language commands, plus an LLM-judge metric for natural-language evaluation criteria; this supports agentic/NL operation for a subset of workflows. missing for 10: independent hands-on evidence of natural-language command execution beyond vendor docs, and confirmation that the full breadth of platform actions (not just evaluation/testing tasks) can be driven purely by NL commands.
- [claimed-docs] “npx skills add cekura-ai/cekura-skills --all”
- [claimed-docs] “Install Cekura Skills and MCP so your AI assistant can design, run, and improve voice-agent evaluations.”
- [github] “13 Skills, 14 Commands in one plugin”
- [claimed-docs] “Evaluate AI voice agent calls using natural language criteria with LLM-powered judging”
- [probe] “official MCP server documented at https://docs.cekura.ai/mcp/overview”
ai-native userSchedule recurring jobs or workflows
weight 2 · not comparableHeliconen/aHelicone is an LLM observability/gateway platform for logging, monitoring, caching, and prompt management—not a workflow/job scheduler or automation orchestrator. Scheduling recurring jobs is outside its product category (wrong axis) rather than a missing feature.
Cekura documents dedicated cron job support for scheduling automated testing/evaluation workflows (cekura-docs-13), plus GitHub Actions integration for triggering workflows on changes (cekura-docs-14), giving AI-native users concrete recurring automation options via CLI/SDK/API. Missing for 10: independent/hands-on confirmation that cron scheduling works reliably in production and more detail on scheduling granularity/limits.
- [claimed-docs] “Learn how to set up and manage automated cron jobs for your AI agent testing and evaluation workflows.”
- [claimed-docs] “Learn how to set up GitHub Actions workflows to automatically test your agents with each change”
- [claimed-docs] “Manage agents, scenarios, runs, and call data from your terminal”
- [claimed-docs] “Programmatic access to Cekura — sync and async clients for application code”
ai-native userPrevent my data from being used to train AI models
weight 3 · not comparableHeliconen/aHelicone is an LLM observability/gateway tool for developers logging their own API requests; it has no relationship to controlling whether end-user data is used to train third-party AI models. This story concerns AI-native consumer privacy posture, which is a wrong axis for this developer infrastructure product.
developerVersion prompts and deploy changes to production without shipping code
weight 3 · not comparableHelicone's Prompts feature explicitly supports versioning ('Track every change, compare versions, and rollback instantly') and production deployment without code changes ('Test and deploy prompt changes instantly without rebuilding or redeploying your application' by referencing a prompt ID in the AI Gateway). This directly matches the story of versioning and deploying prompt changes without shipping code. Missing for 10: independent/hands-on community corroboration specifically of the prompt-versioning workflow (community evidence covers observability/logging, not prompt deployment) and detail on rollback UX.
- [claimed-docs] “Test and deploy prompt changes instantly without rebuilding or redeploying your application”
- [claimed-docs] “Use your prompt instantly by referencing its ID in your AI Gateway. No code changes, no rebuilds.”
- [claimed-docs] “Track every change, compare versions, and rollback instantly if something goes wrong”
Cekuran/aCekura is a testing/observability platform for voice AI agents (simulated calls, metrics, A/B comparison of agent versions), not a prompt-management or deployment system. Prompt versioning and production deployment without code changes belongs to the underlying agent-building platform, not to a third-party evaluation/observability tool like Cekura, making this axis a category error for this product type.
- [claimed-docs] “Compare two versions of your agent side-by-side to measure prompt, model, or configuration changes.”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.cekura.ai/llms.txt # Cekura > Cekura is the testing and observability platform for voice AI agents…”