ElevenLabs Agents vs Pipecat

Agent ops

1. ai-native userMy coding agent can provision a complete voice agent end to end — create the agent, attach a number, and place a call — through the API, CLI, or MCP without touching the dashboard

   weight 3 · round to ElevenLabs Agents
   ElevenLabs Agents~partialprobed6/10

   Docs and runtime probes confirm agents can be created and managed via API, a real CLI (@elevenlabs/cli, verified live on npm), and a hosted MCP server (verified live with OAuth), all without the dashboard (docs-1,3,4,27,28,38; probe-rt-1,2). Phone-number attachment is also supported via SIP trunking to route calls to agents (docs-12,36), but there is no explicit documentation or evidence of actually placing/initiating an outbound call end-to-end through the API/CLI/MCP — only routing inbound infrastructure integration is described. Missing for 10: explicit API/CLI/MCP call to place an outbound call, and a concrete end-to-end walkthrough combining create+attach+call in one non-dashboard flow.

   • [claimed-docs] “ElevenLabs Agents are managed either through the ElevenAgents dashboard, the ElevenLabs API, the Agents CLI or the hosted MCP server.”
   • [claimed-docs] “Use the ElevenLabs agents skill to build and manage voice agents from your AI coding assistant: npx skills add elevenlabs/skills --skill age…”
   • [claimed-docs] “an assistant such as Claude can create, configure, and manage the agents in your workspace through natural language, with nothing to install…”
   • [claimed-docs] “This integration enables all customers to use their existing phone systems while leveraging ElevenLabs' advanced voice AI capabilities.”
   • [claimed-docs] “Route calls to AI agents without changing your existing phone infrastructure”
   • [claimed-docs] “connect the hosted MCP server to Claude or any other MCP client to create and manage agents through natural language”
   • [probe] “PROBE runtime (recorded 2026-09-05): `npx -y @elevenlabs/cli --version` printed `elevenlabs 1.1.0` keylessly — the official ElevenLabs CLI (…”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.elevenlabs.io/v1/mcp returned HTT…”

   proof ↗⚑ flag

   Pipecat~partialprobed3/10

   Pipecat provides a CLI (`pipecat init`, `pipecat cloud`, `pipecat context-hub`) and code-level telephony hooks (`python bot.py -t twilio -x ngrok...`) that let a coding agent scaffold and run a bot, and the context-hub can register an MCP server for coding-agent tooling, but there is no documented single API/CLI/MCP flow that creates the agent, provisions/attaches a phone number, and places a call — telephony setup still relies on the carrier's (Twilio/Telnyx) own console/API for number purchase and webhook config. missing for 10: a unified provisioning API or CLI command that attaches a phone number and places a call, MCP-driven agent creation (context-hub only builds a docs index, not agent provisioning), evidence of number attachment without touching a carrier dashboard.

   • [claimed-docs] “Telephony bots have a different shape than WebRTC bots. The session doesn't start because a client sent your dispatcher an HTTP request — it…”
   • [claimed-docs] “the carrier (Twilio, Telnyx, Plivo, Exotel, or your SIP provider) is calling your webhook to tell you there's an inbound call”
   • [claimed-docs] “python bot.py -t twilio -x your-name.ngrok.io”
   • [claimed-docs] “pipecat init quickstart # Change to the project directory cd pipecat-quickstart”
   • [claimed-docs] “pipecat init quickstart”
   • [claimed-docs] “pipecat context-hub install registers the hub as an MCP server with your coding agent and builds the local index”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”

   proof ↗⚑ flag

2. ai-native userThe platform's own AI helps me author agents — generating or improving prompts, flows, and test cases from a description

   weight 1 · round to ElevenLabs Agents
   ElevenLabs Agents~partialprobed5/10

   ElevenLabs documents that an external AI assistant (e.g., Claude) can create, configure, and manage agents through natural language via the hosted MCP server, and agent-testing docs describe turning real conversations into test cases for regression testing — both are AI-assisted authoring flows. However, this relies on a third-party AI client (Claude) rather than a built-in, platform-native 'describe your agent and we generate it' copilot for prompts/flows within the dashboard itself. Missing for 10: evidence of a first-party generative wizard that authors prompts/flows purely from a natural-language description inside the ElevenLabs UI, and evidence of AI-generated (not just conversation-derived) test-case authoring.

   • [claimed-docs] “an assistant such as Claude can create, configure, and manage the agents in your workspace through natural language, with nothing to install…”
   • [claimed-docs] “connect the hosted MCP server to Claude or any other MCP client to create and manage agents through natural language”
   • [claimed-docs] “Transform real conversations into test cases when you find an interaction where the agent underperformed.”
   • [claimed-docs] “Agent testing lets you verify conversational responses, tool usage, and full multi-turn outcomes before you deploy.”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.elevenlabs.io/v1/mcp returned HTT…”

   proof ↗⚑ flag

   Pipecat—none0/10

   Evidence shows Pipecat has a context-hub that registers as an MCP server so external coding agents (e.g. Claude) can get context about the framework, and an Evals system for testing agent behavior — but these rely on the user writing prompts/flows/tests and connecting third-party AI tools, not Pipecat's own AI generating or improving prompts, flows, or test cases from a description. No evidence of an in-platform AI assistant that authors agent configs.

   • [claimed-docs] “pipecat context-hub install registers the hub as an MCP server with your coding agent and builds the local index”
   • [claimed-docs] “pipecat context-hub install` registers the hub as an MCP server with your coding agent and builds the local index”
   • [claimed-docs] “Pipecat Evals is the framework's built-in system for testing agent behavior. You describe a conversation and the behavior you expect, and Pi…”
   • [claimed-docs] “You describe a conversation and the behavior you expect, and Pipecat runs it against your real agent (the same pipeline, the same services, …”

   proof ↗⚑ flag

Build

1. developerBuild a working phone voice agent — prompt, voice, and phone number — and take my first live call within an hour

   weight 3 · round drawn
   ElevenLabs Agents~partialclaimed7/10

   Docs show a clear quickstart path — create an agent via dashboard/API, pick from 5k+ voices, and connect phone numbers via SIP trunking to route calls to the agent without changing existing phone infrastructure — which together cover the prompt+voice+phone number pieces of the story. However, there's no explicit end-to-end 'first call in under an hour' walkthrough or independent timing evidence, and phone-number provisioning specifics (vs. just SIP integration) are thin. Missing for 10: an explicit timed/complete quickstart-to-live-call walkthrough, first-party or independent evidence of the actual time-to-first-call, and clearer native phone-number purchase/setup docs beyond SIP trunking.

   • [claimed-docs] “ElevenLabs Agents are managed either through the ElevenAgents dashboard, the ElevenLabs API, the Agents CLI or the hosted MCP server.”
   • [claimed-docs] “Select from 5k+ voices across 31 languages with customization options”
   • [claimed-docs] “This integration enables all customers to use their existing phone systems while leveraging ElevenLabs' advanced voice AI capabilities.”
   • [claimed-docs] “Route calls to AI agents without changing your existing phone infrastructure”
   • [claimed-docs] “we'll start by creating an agent via the API or the web dashboard”
   • [claimed-docs] “This assistant can be embedded into your website or app to provide real-time support to your customers”

   proof ↗⚑ flag

   Pipecat~partialprobed7/10

   Pipecat's quickstart gets a browser voice bot running in ~5 minutes with prompt+voice configurable across 200+ providers, and the telephony guide shows the same bot file working with real carriers (Twilio/Telnyx/Plivo) via a one-line command with ngrok tunneling, so all three pieces (prompt, voice, phone number) are documented and connectable to the same codebase. However, the telephony path is a separate 'production' doc requiring carrier account setup/ngrok tunneling not folded into the 5-minute quickstart, and there's no first-party or independent account of someone going from zero to a live phone call within an hour. Missing for 10: an end-to-end timed walkthrough combining phone-number provisioning with the quickstart, and independent hands-on confirmation of the 'first live call within an hour' claim.

   • [claimed-docs] “Build and run your first Pipecat voice AI bot in under 5 minutes”
   • [claimed-docs] “Build and run your first Pipecat voice AI bot in under 5 minutes: a simple conversational agent you can talk to in a browser.”
   • [claimed-docs] “python bot.py -t twilio -x your-name.ngrok.io”
   • [claimed-docs] “the carrier (Twilio, Telnyx, Plivo, Exotel, or your SIP provider) is calling your webhook to tell you there's an inbound call”
   • [claimed-docs] “Telephony bots have a different shape than WebRTC bots. The session doesn't start because a client sent your dispatcher an HTTP request — it…”
   • [claimed-docs] “Swap speech, language, and vision services from any of 200+ integrated providers and services — usually one line of code.”
   • [claimed-docs] “Any Transport: WebRTC, SIP, PSTN”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”
   • [probe] “PROBE runtime (recorded 2026-09-05): pypi pipecat-ai 1.8.1 (BSD-2 OSS) installs and `import pipecat` succeeds with no key, printing its star…”

   proof ↗⚑ flag

2. developerRun conversations in multiple languages, including detecting and switching language mid-call

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents~partialclaimed4/10

   Docs confirm broad multi-language support (70+ languages, language-specific voice settings) and voice-switching for multi-character use, but there is no explicit documentation of automatic language detection or switching languages mid-call. Missing for 10: explicit language-detection/mid-call switching feature docs, and any hands-on demonstration of a live language switch during a single conversation.

   • [claimed-docs] “Configure, deploy and monitor natural, human-sounding agents in 70+ languages with leading accuracy and ultra-low latency across voice or ch…”
   • [claimed-docs] “You can customize various aspects of your AI agent's voice to create a more natural and engaging conversation experience. This includes cont…”
   • [claimed-docs] “Enable your agent to switch between different voices for multi-character conversations, storytelling, and language tutoring.”
   • [claimed-docs] “Select from 5k+ voices across 31 languages with customization options”

   proof ↗⚑ flag

   Pipecat—none0/10

   The evidence mentions swapping speech/language services across providers (one line of code) but nothing about detecting a spoken language mid-call or dynamically switching STT/TTS/LLM language settings within an active conversation. No docs, community reports, or runtime probes address multilingual or language-switching behavior. missing for 10: language detection mechanism, mid-call language switching mechanism, any docs/demo showing multilingual conversation handling.

   • [claimed-docs] “Swap speech, language, and vision services from any of 200+ integrated providers and services — usually one line of code.”

   proof ↗⚑ flag

3. founderDesign multi-step conversation flows in a visual builder with branching, states, and handoffs without writing code

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents~partialclaimed6/10

   Docs explicitly claim a 'visual workflow builder' for multi-step workflows, plus dashboard-based agent creation, conversation-flow settings (turn-taking, interruptions, timeouts), and tools/handoff-like actions (webhooks, client tools) — supporting the no-code, branching, multi-step story. However, evidence never details branching logic, explicit 'states' or agent-to-agent 'handoffs' within the builder, nor independent hands-on confirmation of the visual builder's depth. Missing for 10: concrete detail/screenshots of branching and state logic in the builder, explicit handoff-between-agents feature description, and independent/third-party corroboration of the builder's no-code sufficiency.

   • [claimed-docs] “Build multi-step workflows with visual workflow builder”
   • [claimed-docs] “Conversation flow settings determine how your assistant handles periods of user silence, interruptions during speech, and turn-taking behavi…”
   • [claimed-docs] “Configure turn-taking, interruptions, and timeout settings”
   • [claimed-docs] “we'll start by creating an agent via the API or the web dashboard”
   • [claimed-docs] “Enable agents to call clients & APIs to perform actions”

   proof ↗⚑ flag

   Pipecat—none0/10

   Pipecat Flows lets you define branching conversation states via Python functions/config (docs-6, docs-22, docs-39), but this is a code-first framework with a CLI and Python SDK, not a visual drag-and-drop builder — no evidence pack item mentions any GUI/no-code flow designer. Since competing voice-agent platforms do offer visual builders, this is a fair axis to ask about, and Pipecat's evidence shows the opposite (functions/config authored in code).

   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data ... 2. Progress the conversation by transitioning between nodes”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: Process data by interfacing with external systems and APIs...Progress the conversation by…”
   • [claimed-docs] “Flows auto-derives the function's metadata — name, description, parameter properties (with their descriptions), and which parameters are req…”
   • [claimed-docs] “Hand off to subagents for long-running tools and complex tasks, and use Pipecat Flows when a conversation needs to follow a defined path.”

   proof ↗⚑ flag

Personalization

1. developerInject dynamic variables and per-caller context at call time so each conversation is personalized

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents✓fullclaimed8/10

   Docs explicitly state 'Use dynamic variables and overrides for per-conversation customization' as a core build feature, directly matching the story of injecting per-caller context at call time. Missing for 10: no independent/hands-on example showing a real dynamic-variable injection call in code, and no detail on variable scoping/limits beyond the single doc mention.

   • [claimed-docs] “Use dynamic variables and overrides for per-conversation customization”

   proof ↗⚑ flag

   Pipecat—none0/10

   Evidence shows Pipecat Flows functions can process data and transition conversation state, and telephony bots receive caller info via webhook, but nothing documents a mechanism for injecting per-caller dynamic variables/context into the bot's prompt or state at call start for personalization. This is a fair capability to expect from a voice-agent framework, but no citation directly demonstrates it.

   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data ... 2. Progress the conversation by transitioning between nodes”
   • [claimed-docs] “Telephony bots have a different shape than WebRTC bots. The session doesn't start because a client sent your dispatcher an HTTP request — it…”
   • [claimed-docs] “Flows auto-derives the function's metadata — name, description, parameter properties (with their descriptions), and which parameters are req…”

   proof ↗⚑ flag

2. developerGround the agent on my documents with a built-in knowledge base or RAG so it answers from my content

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents✓fullclaimed8/10

   Docs explicitly state agents support uploading documents and enabling RAG for grounded responses, directly matching the story's built-in knowledge base/RAG capability. Missing for 10: independent/hands-on corroboration of RAG quality and details on knowledge base management (chunking, size limits, sources).

   • [claimed-docs] “Upload documents and enable RAG for grounded responses”

   proof ↗⚑ flag

   Pipecat—none0/10

   Pipecat's docs describe voice/multimodal pipeline orchestration, function calling via Flows, telephony, evals, and a context-hub for coding-agent tooling — but nothing about a built-in knowledge base or RAG mechanism for grounding the agent on user documents. The closest hit (context-hub) indexes Pipecat's own docs/API for coding assistants, not user content for the voice agent.

   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data ... 2. Progress the conversation by transitioning between nodes”
   • [claimed-docs] “pipecat context-hub install registers the hub as an MCP server with your coding agent and builds the local index”
   • [claimed-docs] “pipecat context-hub install` registers the hub as an MCP server with your coding agent and builds the local index”

   proof ↗⚑ flag

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round drawn
   ElevenLabs Agents✓fullprobed9/10

   ElevenLabs publishes a working llms.txt at the root (HTTP 200) and structures its docs as machine-readable .md pages with an explicit pointer to a full docs/llms.txt index, exactly matching the ai-native 'point an agent at llms.txt' pattern; this was independently verified via live probes rather than just claimed docs. Missing for 10: no evidence of a hands-on agent actually consuming the file end-to-end to complete a task (only that the endpoint/format is live and documented).

   • [probe] “PROBE llms.txt: HTTP 200 at https://elevenlabs.io/llms.txt # ElevenLabs ElevenLabs is an AI research and product company transforming how w…”
   • [probe] “PROBE docs-md: HTTP 200 at https://elevenlabs.io/docs/eleven-agents/overview.md > This is a page from the ElevenLabs documentation. For a co…”

   proof ↗⚑ flag

   Pipecat✓fullprobed9/10

   Pipecat provides a working llms.txt endpoint (confirmed HTTP 200 with agent-oriented summary) plus an explicit context-hub CLI feature that registers docs as an MCP server for coding agents and builds a local index, directly enabling agents to consume its docs programmatically. missing for 10: no independent/community confirmation of an agent successfully using llms.txt or context-hub end-to-end.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.pipecat.ai/llms.txt # Pipecat > Pipecat is an open source ecosystem for building voice and multimo…”
   • [claimed-docs] “pipecat context-hub install registers the hub as an MCP server with your coding agent and builds the local index”
   • [claimed-docs] “pipecat context-hub install` registers the hub as an MCP server with your coding agent and builds the local index”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round drawn
   ElevenLabs Agents~partialprobed6/10

   ElevenLabs Agents can be managed via a full REST API and an official CLI (verified runtime install/run), enabling scripted/headless creation and configuration of agents outside the dashboard, which supports CI-style automation. However, evidence is docs/probe-based with no CI pipeline examples, no first-class 'run in CI' guidance, and Agent testing is described as pre-deploy validation without confirmation it can be triggered non-interactively/headlessly. Missing for 10: explicit CI/CD integration examples, headless testing invocation docs, and independent hands-on confirmation of automated pipelines.

   • [claimed-docs] “ElevenLabs Agents are managed either through the ElevenAgents dashboard, the ElevenLabs API, the Agents CLI or the hosted MCP server.”
   • [claimed-docs] “we'll start by creating an agent via the API or the web dashboard”
   • [claimed-docs] “Agent testing lets you verify conversational responses, tool usage, and full multi-turn outcomes before you deploy.”
   • [probe] “PROBE runtime (recorded 2026-09-05): `npx -y @elevenlabs/cli --version` printed `elevenlabs 1.1.0` keylessly — the official ElevenLabs CLI (…”

   proof ↗⚑ flag

   Pipecat~partialprobed6/10

   Pipecat is a Python framework that is pip-installable with no key required and runs bots as scripts (bot.py) with a CLI supporting init, cloud deploy, and eval subcommands, which supports headless/CI-friendly automation; the Evals system explicitly runs conversations against the real agent programmatically, suitable for CI. However there is no explicit CI/CD example, no documented headless flag or non-interactive mode guidance, and the development runner is framed around local dev/demo UI rather than CI. missing for 10: explicit CI pipeline documentation/example, confirmation that pipecat.runner.run or bot.py can run fully headless without any UI/browser dependency, independent hands-on report of running Pipecat in an automated CI environment.

   • [claimed-docs] “Pipecat Evals is the framework's built-in system for testing agent behavior. You describe a conversation and the behavior you expect, and Pi…”
   • [claimed-docs] “You describe a conversation and the behavior you expect, and Pipecat runs it against your real agent (the same pipeline, the same services, …”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”
   • [probe] “PROBE runtime (recorded 2026-09-05): pypi pipecat-ai 1.8.1 (BSD-2 OSS) installs and `import pipecat` succeeds with no key, printing its star…”
   • [claimed-docs] “Pipecat ships a built-in development runner (pipecat.runner.run) that handles the server-side glue most bots need during development: creati…”
   • [claimed-docs] “python bot.py -t twilio -x your-name.ngrok.io”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to ElevenLabs Agents
   ElevenLabs Agents✓fullclaimed8/10

   Docs explicitly describe connecting conversational agents to external MCP servers so agents can access data sources and tools via MCP, directly matching the story, and this is corroborated by dedicated documentation pages (mcp.md) rather than just passing mention. Missing for 10: independent/hands-on verification of an actual third-party MCP server being plugged in and tool-calls succeeding in production (only docs and a runtime probe on the separate hosted-server feature are present).

   • [claimed-docs] “ElevenLabs allows you to connect your conversational agents to external MCP servers. This enables your agents to: Access and process informa…”
   • [claimed-docs] “ElevenLabs allows you to connect your conversational agents to external MCP servers.”
   • [claimed-docs] “Access and process information from various data sources via the MCP server”

   proof ↗⚑ flag

   Pipecat—none0/10

   The evidence only shows Pipecat's context-hub CLI *exposing itself* as an MCP server to a coding agent (docs-11/38) — the reverse of what the story asks (Pipecat consuming external MCP servers to gain their tools inside its voice-agent pipelines). No evidence shows Pipecat's Flows/function-calling system connecting to arbitrary MCP servers as a client.

   • [claimed-docs] “pipecat context-hub install registers the hub as an MCP server with your coding agent and builds the local index”
   • [claimed-docs] “pipecat context-hub install` registers the hub as an MCP server with your coding agent and builds the local index”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round to ElevenLabs Agents
   ElevenLabs Agents✓fullprobed9/10

   ElevenLabs explicitly documents and operates a hosted MCP server (api.elevenlabs.io/v1/mcp) that lets external MCP clients like Claude connect to create, configure, and manage agents through natural language, with a runtime probe confirming the live endpoint correctly responds with the MCP OAuth challenge. This is a first-party 'official MCP server' offering, not just client-side MCP consumption, matching the story precisely. Missing for 10: independent/community hands-on confirmation beyond ElevenLabs' own docs and the single runtime probe.

   • [claimed-docs] “ElevenLabs Agents are managed either through the ElevenAgents dashboard, the ElevenLabs API, the Agents CLI or the hosted MCP server.”
   • [claimed-docs] “an assistant such as Claude can create, configure, and manage the agents in your workspace through natural language, with nothing to install…”
   • [claimed-docs] “connect the hosted MCP server to Claude or any other MCP client to create and manage agents through natural language”
   • [claimed-docs] “create, configure, and manage the agents in your workspace through natural language, with nothing to install or run locally”
   • [probe] “official MCP server documented at https://elevenlabs.io/docs/eleven-agents/operate/hosted-mcp”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.elevenlabs.io/v1/mcp returned HTT…”

   proof ↗⚑ flag

   Pipecat~partialprobed6/10

   Pipecat ships an official `pipecat context-hub install` CLI command that registers itself as an MCP server with a coding agent, building a local docs/examples index — confirmed both in docs and via a runtime CLI probe showing the `context-hub` subcommand exists. This is a genuine official MCP server for agent connection, but it's scoped narrowly to indexing Pipecat's own docs/examples for coding assistants rather than exposing Pipecat's runtime voice-agent capabilities to arbitrary MCP clients. Missing for 10: independent/hands-on confirmation that the MCP server actually functions end-to-end (only CLI help text was probed, not live MCP protocol behavior), and broader documentation of what tools/resources the MCP server exposes.

   • [claimed-docs] “pipecat context-hub install registers the hub as an MCP server with your coding agent and builds the local index”
   • [claimed-docs] “pipecat context-hub install` registers the hub as an MCP server with your coding agent and builds the local index”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round drawn
   ElevenLabs Agents✓fullprobed9/10

   ElevenLabs documents an official Agents CLI for managing agents, and runtime probing confirms `npx @elevenlabs/cli` installs and runs keylessly from npm, corroborating the docs claim with hands-on evidence. Missing for 10: deeper documentation of full CLI command surface/subcommands beyond basic install confirmation.

   • [claimed-docs] “ElevenLabs Agents are managed either through the ElevenAgents dashboard, the ElevenLabs API, the Agents CLI or the hosted MCP server.”
   • [claimed-docs] “Use the ElevenLabs agents skill to build and manage voice agents from your AI coding assistant: npx skills add elevenlabs/skills --skill age…”
   • [claimed-docs] “Use the ElevenLabs agents skill to build and manage voice agents from your AI coding assistant”
   • [probe] “PROBE runtime (recorded 2026-09-05): `npx -y @elevenlabs/cli --version` printed `elevenlabs 1.1.0` keylessly — the official ElevenLabs CLI (…”

   proof ↗⚑ flag

   Pipecat✓fullprobed9/10

   Pipecat ships an official CLI (pip-installable pipecat-ai[cli], confirmed via runtime probe) supporting init/scaffolding, cloud deploy, eval, and context-hub (MCP registration for coding agents), well documented at docs.pipecat.ai/api-reference/cli/overview. This directly supports AI-native agentic workflows (context-hub install registers as MCP server, writes AGENTS.md/CLAUDE.md for coding agents). missing for 10: independent/third-party review of the CLI's robustness beyond vendor docs and a single runtime probe.

   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”
   • [probe] “official CLI documented at https://docs.pipecat.ai/api-reference/cli/overview”
   • [claimed-docs] “pipecat context-hub install registers the hub as an MCP server with your coding agent and builds the local index”
   • [claimed-docs] “Install the Pipecat CLI and scaffold the quickstart project (also writes AGENTS.md + CLAUDE.md)”
   • [claimed-docs] “pipecat context-hub install` registers the hub as an MCP server with your coding agent and builds the local index”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to ElevenLabs Agents
   ElevenLabs Agents✓fullprobed9/10

   ElevenLabs Agents ships a documented public API alongside a dashboard, CLI, and hosted MCP server for programmatic agent creation/management, with quickstart docs explicitly noting 'creating an agent via the API' and full customization (tools, voice, analysis, testing) accessible programmatically. Runtime probes confirm the CLI installs keylessly and the hosted MCP endpoint is live and functional. Missing for 10: no independent third-party API reference/SDK usage examples or community corroboration beyond first-party docs.

   • [claimed-docs] “ElevenLabs Agents are managed either through the ElevenAgents dashboard, the ElevenLabs API, the Agents CLI or the hosted MCP server.”
   • [claimed-docs] “we'll start by creating an agent via the API or the web dashboard”
   • [claimed-docs] “Enable agents to call clients & APIs to perform actions”
   • [probe] “PROBE runtime (recorded 2026-09-05): `npx -y @elevenlabs/cli --version` printed `elevenlabs 1.1.0` keylessly — the official ElevenLabs CLI (…”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.elevenlabs.io/v1/mcp returned HTT…”

   proof ↗⚑ flag

   Pipecat~partialprobed7/10

   Pipecat exposes a well-documented programmatic surface — a Python framework API (frame processors, Flows functions with auto-derived metadata), a runner API, and an official CLI (init/cloud/eval/context-hub) confirmed to work keylessly via runtime probe — giving AI-native users a documented, scriptable way to drive the product. However, probing shows no REST/OpenAPI spec (all openapi.json candidates 404'd), so there's no conventional network-callable public API, only the code-level Python/CLI surface. Missing for 10: a formal REST/OpenAPI-documented HTTP API, and independent hands-on confirmation of driving a full pipeline purely via that public API surface.

   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data ... 2. Progress the conversation by transitioning between nodes”
   • [claimed-docs] “Pipecat ships a built-in development runner (\`pipecat.runner.run\`) that handles the server-side glue most bots need during development”
   • [claimed-docs] “Pipecat ships a built-in development runner (pipecat.runner.run) that handles the server-side glue most bots need during development: creati…”
   • [claimed-docs] “Flows auto-derives the function's metadata — name, description, parameter properties (with their descriptions), and which parameters are req…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.pipecat.ai/openapi.json, https://docs.pipecat.ai/swagger.json, https://docs.pipecat.ai/…”
   • [probe] “official CLI documented at https://docs.pipecat.ai/api-reference/cli/overview”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   ElevenLabs Agents—none0/10

   Evidence covers agent-side authentication (protecting inbound access to a deployed agent) and OAuth for the hosted MCP server, but there is no documentation of issuing scoped or least-privilege API keys/tokens that limit what an agent itself can do via the ElevenLabs API. Missing for 10: any docs on API key permission scopes, workspace role-based tokens, or restricted-credential issuance for agents.

   • [claimed-docs] “Secure agent access | Authentication | Implement custom authentication for protected agent access”
   • [claimed-docs] “Implement custom authentication for protected agent access”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.elevenlabs.io/v1/mcp returned HTT…”

   proof ↗⚑ flag

   Pipecat—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Pipecat
   ElevenLabs Agents~partialprobed4/10

   Docs confirm programmatic access via the ElevenLabs API and an official CLI (`@elevenlabs/cli`, verified runtime) for building agents, but no evidence pack item explicitly documents a first-party language SDK (Python/Node/etc.) for Agents beyond the generic API reference. Missing for 10: explicit official SDK docs/repos, SDK code examples, and independent confirmation of SDK feature parity with the API.

   • [claimed-docs] “ElevenLabs Agents are managed either through the ElevenAgents dashboard, the ElevenLabs API, the Agents CLI or the hosted MCP server.”
   • [claimed-docs] “we'll start by creating an agent via the API or the web dashboard”
   • [probe] “PROBE runtime (recorded 2026-09-05): `npx -y @elevenlabs/cli --version` printed `elevenlabs 1.1.0` keylessly — the official ElevenLabs CLI (…”

   proof ↗⚑ flag

   Pipecat✓fullprobed9/10

   Pipecat is itself an official open-source Python SDK/framework (pip-installable, verified via runtime probe to install and import cleanly), and it also ships a family of official 'Pipecat Client' SDKs for web/mobile integration plus pre-built React components, all documented on the official site. Missing for 10: independent/third-party confirmation of the client SDKs' quality beyond docs (only the core Python framework was hands-on verified).

   • [claimed-docs] “Pipecat Clients are a family of SDKs that connect users to your Pipecat agents through web and mobile applications.”
   • [claimed-docs] “Pipecat Clients are a family of SDKs that connect users to your Pipecat agents through web and mobile applications. They handle real-time au…”
   • [claimed-docs] “Pre-built React components for voice AI interfaces.”
   • [claimed-docs] “Open source Python framework for building voice and multimodal AI pipelines. Orchestrate 150+ AI services with ultra-low latency.”
   • [probe] “PROBE runtime (recorded 2026-09-05): pypi pipecat-ai 1.8.1 (BSD-2 OSS) installs and `import pipecat` succeeds with no key, printing its star…”
   • [github] “Build a single voice agent or a full multi-agent system where specialists hand off, fan out in parallel, and coordinate over a shared bus, l…”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   ElevenLabs Agents—none0/10

   The evidence describes 'webhook tools' that let an agent make outbound calls to external endpoints during a conversation (docs-14, docs-29, docs-31), which is the opposite of subscribing to platform-emitted events via webhooks. There is no documented mechanism for a developer to register a webhook URL to receive event notifications (e.g., conversation-ended, call-completed) from ElevenLabs Agents.

   • [claimed-docs] “Tools give your assistant the ability to make external function calls to third party apps so you can get real-time information.”
   • [claimed-docs] “Tools allow ElevenLabs agents to perform actions beyond generating text responses.”
   • [claimed-docs] “allow an assistant to trigger authenticated actions based on the conversation, like scheduling meetings or initiating order returns”

   proof ↗⚑ flag

   Pipecat—none0/10

   Webhooks appear only as an inbound mechanism (a telephony carrier calling Pipecat's webhook to signal an incoming call), not as an outbound event-subscription system that lets a user register a webhook to receive Pipecat's own events (e.g., call end, transcript ready, errors). No evidence of a webhook subscription API or configurable event notifications for consumers.

   • [claimed-docs] “Telephony bots have a different shape than WebRTC bots. The session doesn't start because a client sent your dispatcher an HTTP request — it…”
   • [claimed-docs] “the carrier (Twilio, Telnyx, Plivo, Exotel, or your SIP provider) is calling your webhook to tell you there's an inbound call”
   • [claimed-docs] “python bot.py -t twilio -x your-name.ngrok.io”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents~partialclaimed5/10

   Agent-analysis features (sentiment analysis, success evaluation, structured data extraction, semantic conversation search) generate AI-derived insights from the user's own conversation data, which partially satisfies the story. However, these are evaluation/reporting metrics rather than proactive AI-generated suggestions or recommendations for next actions, and there is no independent/hands-on corroboration of this analytics experience. Missing for 10: evidence of proactive suggestion generation (e.g., recommended actions/optimizations), a dashboard summarization view, and independent verification beyond first-party docs.

   • [claimed-docs] “Success evaluation: Define custom criteria to assess conversation quality, goal achievement, and customer satisfaction”
   • [claimed-docs] “Data collection: Extract specific data points from conversations such as contact information, issue details, or any structured information”
   • [claimed-docs] “Sentiment analysis: Understand user sentiment across completed conversations”
   • [claimed-docs] “Searching conversations: Find messages by keyword or meaning across your conversation history.”
   • [claimed-docs] “Define custom criteria to assess conversation quality, goal achievement, and customer satisfaction”
   • [claimed-docs] “Extract structured information from conversations such as contact details and business data”

   proof ↗⚑ flag

   Pipecat—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round drawn
   ElevenLabs Agents~partialclaimed4/10

   ElevenLabs Agents supports tool-calling, webhook actions, and multi-step workflows that let a voice agent perform actions (scheduling, API calls, MCP access) during a conversation, and agents can be created/managed via API for programmatic triggering, which give some automation flavor. But all documented action-taking is tied to a live conversational turn (voice/chat) rather than an autonomous background process that runs without a triggering conversation or human interaction, so the 'runs autonomously in the background' framing is only weakly supported. Missing for 10: evidence of scheduled/cron-triggered autonomous runs, evidence of multi-step tasks executing without a live conversation session, and independent confirmation of background automation outside the demoed conversational/tool-call flow.

   • [claimed-docs] “Build multi-step workflows with visual workflow builder”
   • [claimed-docs] “Enable agents to call clients & APIs to perform actions”
   • [claimed-docs] “Tools give your assistant the ability to make external function calls to third party apps so you can get real-time information.”
   • [claimed-docs] “allow an assistant to trigger authenticated actions based on the conversation, like scheduling meetings or initiating order returns”
   • [claimed-docs] “ElevenLabs allows you to connect your conversational agents to external MCP servers. This enables your agents to: Access and process informa…”
   • [claimed-docs] “we'll start by creating an agent via the API or the web dashboard”

   proof ↗⚑ flag

   Pipecat~partialclaimed4/10

   Pipecat supports multi-agent systems where 'specialists hand off, fan out in parallel, and coordinate over a shared bus, locally or distributed' and can 'hand off to subagents for long-running tools and complex tasks,' plus telephony bots that start autonomously when a carrier calls a webhook rather than a user request — suggesting some background/autonomous agent operation. But Pipecat's core model is a live, turn-based conversational pipeline tied to an active voice/video session, not a scheduler or cron-like background automation runner independent of a live interaction. Missing for 10: explicit scheduling/triggering mechanisms for unattended background jobs, evidence of agents running without any live session, and independent confirmation of autonomous multi-agent orchestration in production.

   • [github] “Build a single voice agent or a full multi-agent system where specialists hand off, fan out in parallel, and coordinate over a shared bus, l…”
   • [claimed-docs] “Hand off to subagents for long-running tools and complex tasks, and use Pipecat Flows when a conversation needs to follow a defined path.”
   • [claimed-docs] “Telephony bots have a different shape than WebRTC bots. The session doesn't start because a client sent your dispatcher an HTTP request — it…”
   • [claimed-docs] “the carrier (Twilio, Telnyx, Plivo, Exotel, or your SIP provider) is calling your webhook to tell you there's an inbound call”

   proof ↗⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round to ElevenLabs Agents
   ElevenLabs Agents~partialprobed5/10

   ElevenLabs documents a first-party hosted MCP server letting an assistant like Claude create, configure, and manage agents in the workspace via natural language, with nothing to install locally — a form of task delegation to an AI assistant for managing the product. However this is not a native in-product chat/assistant UI; it requires an external MCP client (e.g., Claude) to act as the assistant, and delegation is scoped to agent-management tasks rather than general product tasks. Missing for 10: a built-in assistant surface inside the ElevenLabs dashboard itself, evidence of broader task delegation beyond agent CRUD, and independent hands-on confirmation of the natural-language workflow.

   • [claimed-docs] “an assistant such as Claude can create, configure, and manage the agents in your workspace through natural language, with nothing to install…”
   • [claimed-docs] “an assistant such as Claude can create, configure, and manage the agents in your workspace through natural language”
   • [claimed-docs] “connect the hosted MCP server to Claude or any other MCP client to create and manage agents through natural language”
   • [claimed-docs] “create, configure, and manage the agents in your workspace through natural language, with nothing to install or run locally”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.elevenlabs.io/v1/mcp returned HTT…”

   proof ↗⚑ flag

   Pipecat—none0/10

   Pipecat's docs describe a framework/CLI for building voice AI agents, a context-hub that exposes Pipecat's own docs as an MCP server to external coding agents, and dev tooling (runner, evals, Flows) — but none of this is a built-in assistant inside Pipecat's own product surface that a user delegates tasks to. Missing for 10: any first-party 'chat with an assistant' feature in the CLI/dashboard/docs site, evidence of task delegation to an embedded assistant, or independent confirmation of such a feature.

   • [claimed-docs] “pipecat context-hub install` registers the hub as an MCP server with your coding agent and builds the local index”
   • [claimed-docs] “pipecat context-hub install registers the hub as an MCP server with your coding agent and builds the local index”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”

   proof ↗⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents✓fullprobed8/10

   ElevenLabs Agents ships a hosted MCP server explicitly designed so assistants like Claude can 'create, configure, and manage the agents in your workspace through natural language, with nothing to install or run locally,' and a runtime probe confirms the endpoint is live and speaks the MCP OAuth flow. This is a direct, first-party natural-language control surface, not just a client-side integration. Missing for 10: independent/community hands-on confirmation of a full natural-language agent-management session (only endpoint liveness was probed, not an actual successful NL task).

   • [claimed-docs] “an assistant such as Claude can create, configure, and manage the agents in your workspace through natural language, with nothing to install…”
   • [claimed-docs] “an assistant such as Claude can create, configure, and manage the agents in your workspace through natural language”
   • [claimed-docs] “connect the hosted MCP server to Claude or any other MCP client to create and manage agents through natural language”
   • [claimed-docs] “create, configure, and manage the agents in your workspace through natural language, with nothing to install or run locally”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.elevenlabs.io/v1/mcp returned HTT…”
   • [probe] “official MCP server documented at https://elevenlabs.io/docs/eleven-agents/operate/hosted-mcp”

   proof ↗⚑ flag

   Pipecat~partialprobed6/10

   Pipecat's entire raison d'être is building voice agents that end users control via natural spoken language — turn detection, interruption/barge-in, and Flows functions that progress conversations based on NL input (pipecat-docs-9, pipecat-docs-19, pipecat-docs-6/22/30) all support NL-driven operation of the *bots* it produces. However, the story is ambiguous about whether it's the bots or the Pipecat tooling itself; Pipecat's own developer-facing surfaces (CLI init/cloud/eval/context-hub) are traditional structured commands, not natural-language commands (pipecat-probe-rt-1, pipecat-docs-34/37). Missing for 10: evidence that the Pipecat framework/CLI itself (not just end-user bots) can be operated by an AI-native user via natural-language instructions rather than code/CLI syntax.

   • [claimed-docs] “Smart Turn Detection uses an advanced machine learning model to determine when a user has finished speaking and your bot should respond.”
   • [claimed-docs] “Interruptions (also called barge-in) let the user talk over the bot. When the user starts speaking while the bot is talking, the bot stops i…”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data ... 2. Progress the conversation by transitioning between nodes”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: Process data by interfacing with external systems and APIs...Progress the conversation by…”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data by interfacing with external systems and APIs... 2. Progress the conversa…”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”
   • [claimed-docs] “pipecat init quickstart # Change to the project directory cd pipecat-quickstart”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   ElevenLabs Agents—none0/10

   The evidence pack contains extensive markdown documentation for ElevenLabs Agents (quickstart, customization, tools, etc.) but nothing describes an interactive API reference page with runnable/'try it' examples — no mention of a Swagger/OpenAPI explorer, live code sandbox, or embedded runnable snippets. Since ElevenLabs Agents exposes an API, this axis is applicable, but no evidence supports the specific capability.

   ⚑ flag

   Pipecat—none0/10

   Pipecat's docs are static markdown pages (get-started, fundamentals, flows, CLI reference) with no evidence of an interactive API reference or runnable code examples; a direct probe for OpenAPI/Swagger endpoints returned 404 on all candidate paths, indicating no interactive API explorer exists.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.pipecat.ai/openapi.json, https://docs.pipecat.ai/swagger.json, https://docs.pipecat.ai/…”
   • [probe] “official CLI documented at https://docs.pipecat.ai/api-reference/cli/overview”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   ElevenLabs Agents—none0/10

   The evidence pack documents the API, CLI, dashboard, and hosted MCP server for ElevenLabs Agents, but nowhere mentions a downloadable OpenAPI/Swagger spec or machine-readable schema for the API. This axis clearly applies to an API-driven product, but no evidence shows it being delivered.

   ⚑ flag

   Pipecat—none0/10

   A direct probe for OpenAPI/swagger specs at all standard locations (openapi.json, swagger.json, etc.) on docs.pipecat.ai returned 404s, and no evidence pack item shows a downloadable machine-readable API spec despite Pipecat having an api-reference docs section and cloud service.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.pipecat.ai/openapi.json, https://docs.pipecat.ai/swagger.json, https://docs.pipecat.ai/…”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Pipecat
   ElevenLabs Agents~partialclaimed4/10

   Docs describe an 'agent testing' feature to verify conversational responses, tool usage, and multi-turn outcomes 'before you deploy', implying a pre-production test workflow, and tool execution can run in a 'sandboxed environment' on ElevenLabs' infrastructure. However there is no explicit documentation of a separate sandbox/staging environment or test data isolation from production conversation/analytics data. Missing for 10: explicit sandbox/staging environment separate from production data, documentation of test-vs-prod data isolation, independent confirmation that test runs don't affect production analytics or logs.

   • [claimed-docs] “Agent testing lets you verify conversational responses, tool usage, and full multi-turn outcomes before you deploy.”
   • [claimed-docs] “Agent testing lets you verify conversational responses, tool usage, and full multi-turn outcomes before you deploy”
   • [claimed-docs] “Custom JavaScript executed in a sandboxed environment on ElevenLabs' infrastructure.”
   • [claimed-docs] “Custom JavaScript executed in a sandboxed environment on ElevenLabs' infrastructure”

   proof ↗⚑ flag

   Pipecat~partialclaimed5/10

   Pipecat provides a local development runner (`pipecat.runner.run`) and CLI scaffolding that let developers run and iterate on bots entirely locally, plus a built-in Evals system that runs described conversations against 'your real agent' pipeline for behavior testing — both usable without deploying to Pipecat Cloud or touching a production instance. However, there's no explicit 'sandbox environment' concept with data isolation from production (e.g. no staging vs prod data separation, no test-data seeding, no mention of production data at all since Pipecat doesn't store conversational data centrally). Missing for 10: explicit sandbox/production data isolation, documented data classification for dev vs prod, hands-on confirmation that Evals or local runner never touch live production resources.

   • [claimed-docs] “Pipecat ships a built-in development runner (\`pipecat.runner.run\`) that handles the server-side glue most bots need during development”
   • [claimed-docs] “Pipecat ships a built-in development runner (pipecat.runner.run) that handles the server-side glue most bots need during development: creati…”
   • [claimed-docs] “Pipecat ships a built-in development runner (`pipecat.runner.run`) that handles the server-side glue most bots need during development: crea…”
   • [claimed-docs] “Pipecat Evals is the framework's built-in system for testing agent behavior. You describe a conversation and the behavior you expect, and Pi…”
   • [claimed-docs] “You describe a conversation and the behavior you expect, and Pipecat runs it against your real agent (the same pipeline, the same services, …”
   • [claimed-docs] “This is the property that makes the same bot file portable across the development runner, Pipecat Cloud, and most production self-hosting se…”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   ElevenLabs Agents—none0/10

   No evidence in the pack mentions API versioning scheme, version numbers, or any documented deprecation policy for ElevenLabs Agents' API/CLI/MCP interfaces. missing for 10: versioned API endpoints or version headers, changelog/release notes, explicit deprecation policy or sunset timelines, migration guides for breaking changes.

   ⚑ flag

   Pipecat—none0/10

   No evidence pack item discusses API versioning scheme, version numbers surfaced in docs, or any documented deprecation policy/timeline for Pipecat's SDKs or APIs; the only version datum (pip package 1.8.1) shows a version exists but nothing about a deprecation policy is documented.

   ⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Pipecat
   ElevenLabs Agents—none0/10

   The evidence documents API, CLI, and MCP-based management of agents, but nothing describes bulk/batch operations (e.g., batch-create, batch-update, or batch-delete across many agents or conversations at once). This axis is fair to ask given the product exposes an API and CLI, but no evidence confirms bulk-operation support.

   ⚑ flag

   Pipecat~partialclaimed3/10

   Pipecat supports 'unlimited concurrency' for running many simultaneous voice sessions and multi-agent fan-out ('fan out in parallel... coordinate over a shared bus'), which could support bulk-style parallel processing of many calls/agents at once. However, there is no explicit bulk-operation tooling (e.g., batch APIs, bulk data import/export, mass-edit of items) documented anywhere in the evidence pack. Missing for 10: explicit bulk/batch operation APIs or workflows, evidence of processing many discrete 'items' (not just concurrent voice sessions) in one operation, and any hands-on confirmation of this at scale.

   • [claimed-docs] “Pipecat Cloud supports unlimited concurrency.”
   • [github] “Build a single voice agent or a full multi-agent system where specialists hand off, fan out in parallel, and coordinate over a shared bus, l…”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round drawn
   ElevenLabs Agents~partialclaimed6/10

   ElevenLabs Agents supports tool-calling and webhook actions that are triggered automatically based on conversation events (e.g., 'trigger authenticated actions based on the conversation, like scheduling meetings or initiating order returns'), plus multi-step visual workflows and MCP-based external actions. This covers event-triggered automation within a conversation but the evidence doesn't show general rule-based triggers on arbitrary system events (e.g., scheduled/external triggers outside a live conversation) — missing for 10: documentation of standalone event/rule engine independent of a live conversation, examples of conditional branching logic beyond tool calls, and independent verification of automation reliability.

   • [claimed-docs] “Enable agents to call clients & APIs to perform actions”
   • [claimed-docs] “Tools give your assistant the ability to make external function calls to third party apps so you can get real-time information.”
   • [claimed-docs] “allow an assistant to trigger authenticated actions based on the conversation, like scheduling meetings or initiating order returns”
   • [claimed-docs] “Build multi-step workflows with visual workflow builder”
   • [claimed-docs] “ElevenLabs allows you to connect your conversational agents to external MCP servers. This enables your agents to: Access and process informa…”

   proof ↗⚑ flag

   Pipecat~partialclaimed6/10

   Pipecat's frame-based pipeline is inherently event-driven — interruption events automatically halt bot output when the user speaks, and Pipecat Flows functions can process data and automatically transition conversation nodes based on function-call events. This gives a real automation layer, but it's scoped to conversation/pipeline events rather than a general-purpose rules engine for arbitrary triggers/actions. Missing for 10: a documented generic rule-definition system (arbitrary trigger→action bindings outside conversation flow), and independent/hands-on evidence of custom event-rule automation beyond the built-in interruption/Flows mechanics.

   • [claimed-docs] “When the user starts speaking while the bot is talking, the bot stops immediately, in-flight work is cancelled, and the pipeline is ready fo…”
   • [claimed-docs] “Interruptions (also called barge-in) let the user talk over the bot. When the user starts speaking while the bot is talking, the bot stops i…”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data ... 2. Progress the conversation by transitioning between nodes”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: Process data by interfacing with external systems and APIs...Progress the conversation by…”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data by interfacing with external systems and APIs... 2. Progress the conversa…”
   • [claimed-docs] “Pipecat's turn events make it easy to collect both user and assistant messages as they occur.”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round drawn
   ElevenLabs Agents—none0/10

   The evidence pack covers multi-step workflow building, tool calling, MCP integration, and analysis, but no documentation mentions scheduling recurring jobs, cron-like triggers, or automated recurring workflow execution. This is a fair capability to expect from an automation-focused agent platform, but no evidence supports it.

   ⚑ flag

   Pipecat—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round drawn
   ElevenLabs Agents—none0/10

   The docs describe building, testing, and analyzing agents (agent-testing, agent-analysis) but there is no mention of version history, change review workflows, or rollback/revert capability for agent configurations. Missing for 10: version history UI, rollback/restore of prior agent configs, change-review/approval workflow, audit trail of edits.

   ⚑ flag

   Pipecat—none0/10

   Pipecat's docs describe building pipelines/Flows in Python code but no evidence of built-in versioning, review workflows, or rollback capabilities for automations (Flows or pipelines) — no changelog, diff, or rollback tooling is mentioned anywhere in the evidence pack.

   ⚑ flag

Compliance

1. founderMeet call-recording consent and disclosure obligations with per-call recording controls and configurable data retention

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents~partialclaimed4/10

   Docs confirm configurable data-retention settings for conversation transcripts and audio recordings, which addresses the retention half of the story, plus HIPAA/BAA support suggesting compliance posture. However, there is no evidence of per-call recording enable/disable controls or built-in consent/disclosure messaging mechanisms for calls. Missing for 10: per-call recording toggle, consent/disclosure prompt configuration, and any independent verification of retention enforcement.

   • [claimed-docs] “Retention settings allow you to configure how long your conversational agent stores conversation transcripts and audio recordings.”
   • [claimed-docs] “Retention settings allow you to configure how long your conversational agent stores conversation transcripts and audio recordings”
   • [claimed-docs] “ElevenLabs allows such customers to develop AI-powered voice agents for the handling Protected Health Information (PHI)”
   • [claimed-docs] “ElevenLabs Agents is one of ElevenLabs' HIPAA-eligible services, and we offer Business Associate Agreements (BAAs) to eligible customers”

   proof ↗⚑ flag

   Pipecat~partialclaimed3/10

   Pipecat provides an AudioBufferProcessor that developers can use to capture call recordings, which is a building block a founder could use to implement recording, but there's no documented feature for per-call consent/disclosure toggles or configurable data-retention policies — those would have to be built entirely by the developer. missing for 10: consent/disclosure mechanisms, retention-policy configuration, compliance-specific controls, any documentation tying recording to legal/consent obligations.

   • [claimed-docs] “Pipecat's \`AudioBufferProcessor\` makes it easy to capture high-quality audio recordings of both the user and bot during interactions.”
   • [claimed-docs] “Pipecat's AudioBufferProcessor makes it easy to capture high-quality audio recordings of both the user and bot during interactions.”
   • [claimed-docs] “Pipecat's `AudioBufferProcessor` makes it easy to capture high-quality audio recordings of both the user and bot during interactions.”

   proof ↗⚑ flag

2. platform-engineerRun regulated workloads with HIPAA/BAA support, SOC 2, and data-residency options

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents~partialclaimed5/10

   ElevenLabs Agents documents HIPAA eligibility and BAA availability for eligible customers, directly supporting regulated PHI workloads. However, there is no evidence in the pack of SOC 2 certification or data-residency options, both explicitly named in the story. Missing for 10: SOC 2 attestation/report evidence, data-residency region selection or documentation, independent audit corroboration of HIPAA/BAA claims.

   • [claimed-docs] “ElevenLabs allows such customers to develop AI-powered voice agents for the handling Protected Health Information (PHI)”
   • [claimed-docs] “ElevenLabs allows such customers to develop AI-powered voice agents for the handling Protected Health Information (PHI).”
   • [claimed-docs] “ElevenLabs Agents is one of ElevenLabs' HIPAA-eligible services, and we offer Business Associate Agreements (BAAs) to eligible customers”

   proof ↗⚑ flag

   Pipecat—none0/10

   No evidence in the pack mentions HIPAA/BAA, SOC 2 certification, or data-residency options for Pipecat or Pipecat Cloud; the pack covers pipeline features, telephony, CLI, and dev tooling but nothing on compliance/regulatory posture. missing for 10: HIPAA/BAA documentation, SOC 2 attestation, data-residency configuration options.

   ⚑ flag

Scale

1. platform-engineerSee documented concurrency limits and scale to many simultaneous calls without manual capacity begging

   weight 2 · round to Pipecat
   ElevenLabs Agents—none0/10

   No evidence pack item documents concurrency limits, per-account call caps, autoscaling behavior, or a process for scaling to many simultaneous calls without contacting sales/support; docs cover features like tools, voice customization, SIP trunking, HIPAA, and MCP but nothing on capacity/scale numbers or self-service scaling guarantees.

   ⚑ flag

   Pipecat~partialcommunity4/10

   Pipecat Cloud's pricing page asserts 'unlimited concurrency' and offers multiple agent profiles for compute needs, giving a vendor-level answer to scaling, but there is no documented capacity planning guide, per-instance concurrency limits, or benchmarks for self-hosted deployments. Community commentary (a developer porting Pipecat to avoid Python's GIL for 'real concurrency') suggests self-hosted scaling is not frictionless and may require manual work to reach high concurrency. Missing for 10: documented concurrency ceilings/benchmarks for self-hosted deployments, load-testing data, and detail behind the 'unlimited concurrency' claim beyond a pricing-page bullet.

   • [claimed-docs] “Pipecat Cloud supports unlimited concurrency.”
   • [claimed-docs] “Pipecat Cloud offers different agent profiles, to best support your use case and compute needs.”
   • [community] “I got tired of running Python servers for voice agents, so I started porting Pipecat over. The design is Pipecat's, frames and processors an…”

   proof ↗⚑ flag

Self host

1. platform-engineerSelf-host the voice agent runtime from open-source code on my own infrastructure

   weight 3 · round to Pipecat
   ElevenLabs Agents—none0/10

   ElevenLabs Agents is entirely a managed/hosted service — the CLI and MCP server are clients/interfaces to ElevenLabs' cloud infrastructure, not open-source runtime code that can be deployed on a platform-engineer's own servers. Nothing in the evidence pack mentions open-sourcing the runtime, self-hosting instructions, or on-prem deployment options; all references (dashboard, API, CLI, hosted MCP, SIP trunking) point to ElevenLabs-operated infrastructure.

   • [claimed-docs] “ElevenLabs Agents are managed either through the ElevenAgents dashboard, the ElevenLabs API, the Agents CLI or the hosted MCP server.”
   • [claimed-docs] “an assistant such as Claude can create, configure, and manage the agents in your workspace through natural language, with nothing to install…”
   • [claimed-docs] “create, configure, and manage the agents in your workspace through natural language, with nothing to install or run locally”
   • [probe] “PROBE runtime (recorded 2026-09-05): `npx -y @elevenlabs/cli --version` printed `elevenlabs 1.1.0` keylessly — the official ElevenLabs CLI (…”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.elevenlabs.io/v1/mcp returned HTT…”

   proof ↗⚑ flag

   Pipecat✓fullprobed9/10

   Pipecat is confirmed as a pip-installable, BSD-2 open-source Python framework that runs keylessly with no vendor lock-in, ships a development runner and CLI, and docs explicitly state bot files are portable across the dev runner, Pipecat Cloud, and 'most production self-hosting setups.' Runtime probes confirm the package installs and imports successfully outside any managed cloud, and community evidence corroborates real-world self-hosted usage (local inference via Ollama/Whisper, custom ports of the framework). Missing for 10: a first-party detailed production self-hosting deployment guide (e.g., Docker/Kubernetes reference architecture) beyond the telephony/local-runner docs.

   • [probe] “PROBE runtime (recorded 2026-09-05): pypi pipecat-ai 1.8.1 (BSD-2 OSS) installs and `import pipecat` succeeds with no key, printing its star…”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”
   • [claimed-docs] “This is the property that makes the same bot file portable across the development runner, Pipecat Cloud, and most production self-hosting se…”
   • [claimed-docs] “Open source Python framework for building voice and multimodal AI pipelines. Orchestrate 150+ AI services with ultra-low latency.”
   • [community] “Local inference is already supported via Pipecat, you can use ollama or any custom OpenAI endpoint. Local STT is also supported via whisper,…”
   • [community] “Yes, Pipecat already supports that natively, so this can be done easily with ollama... Also, check out any provider they support, and it can…”
   • [github] “Build a single voice agent or a full multi-agent system where specialists hand off, fan out in parallel, and coordinate over a shared bus, l…”

   proof ↗⚑ flag

Latency

1. platform-engineerSee documented end-to-end voice latency numbers or tuning guidance backing the platform's speed claims

   weight 3 · round to Pipecat
   ElevenLabs Agents~partialclaimed4/10

   ElevenLabs markets 'ultra-low latency' (docs-37) and does document conversation-flow tuning knobs like turn-taking, interruption handling and timeouts (docs-33, docs-42, docs-19/49/54), which counts as tuning guidance for latency/turn-taking behavior. However there are no documented end-to-end latency benchmark numbers (e.g., ms figures, percentile breakdowns) anywhere in the evidence pack. Missing for 10: concrete latency benchmark numbers/metrics, methodology for measuring end-to-end latency, and any independent corroboration of the 'ultra-low latency' marketing claim.

   • [claimed-docs] “Configure, deploy and monitor natural, human-sounding agents in 70+ languages with leading accuracy and ultra-low latency across voice or ch…”
   • [claimed-docs] “Conversation flow settings determine how your assistant handles periods of user silence, interruptions during speech, and turn-taking behavi…”
   • [claimed-docs] “Configure turn-taking, interruptions, and timeout settings”
   • [claimed-docs] “The Max conversation duration setting limits the total time a conversation can remain active. ... The default is 600 seconds (10 minutes).”

   proof ↗⚑ flag

   Pipecat~partialclaimed6/10

   Pipecat documents a dedicated STT-latency-tuning guide describing measured P99 latency values per service and a TTFS (Time To Final Segment) metric, plus a built-in metrics system for enabling latency instrumentation — real tuning guidance exists. However, this is scoped to STT stage latency rather than a comprehensive end-to-end voice round-trip latency benchmark, and no actual published numbers/tables or independent corroboration of the claimed P99 figures are shown. Missing for 10: published end-to-end latency benchmark numbers (not just STT), tuning guidance across the full voice pipeline (TTS/LLM/turn-taking), and independent verification of the claimed latency figures.

   • [claimed-docs] “Pipecat ships with measured P99 latency values for each supported service so that turn detection can account for this delay automatically.”
   • [claimed-docs] “Time To Final Segment (TTFS) measures how long it takes from the moment a user stops speaking until the STT service delivers the final trans…”
   • [claimed-docs] “Pipecat offers built-in metrics for both, which can be enabled with straightforward configuration options.”

   proof ↗⚑ flag

Turn taking

1. developerRely on the agent to handle interruptions (barge-in) gracefully — stopping speech, updating context, and recovering the turn

   weight 3 · round to Pipecat
   ElevenLabs Agents~partialclaimed6/10

   Docs explicitly document configuring turn-taking, interruptions, and timeout settings, and conversation-flow settings that 'determine how your assistant handles periods of user silence, interruptions during speech, and turn-taking behavior,' confirming barge-in handling is a first-class configurable feature. However, evidence lacks detail on how context is updated after an interruption or how turn recovery specifically behaves, and there's no hands-on/independent validation of graceful recovery in practice. Missing for 10: technical detail on context truncation/update after interruption, independent or hands-on testing confirming graceful recovery, and latency benchmarks for barge-in specifically.

   • [claimed-docs] “Conversation flow settings determine how your assistant handles periods of user silence, interruptions during speech, and turn-taking behavi…”
   • [claimed-docs] “Configure turn-taking, interruptions, and timeout settings”
   • [claimed-docs] “The Max conversation duration setting limits the total time a conversation can remain active. ... The default is 600 seconds (10 minutes).”

   proof ↗⚑ flag

   Pipecat✓fullclaimed8/10

   Pipecat has explicit, dedicated documentation on interruptions/barge-in describing immediate speech stop, cancellation of in-flight work, and pipeline readiness for new input, plus Smart Turn Detection for natural turn-taking and Krisp-based false-interruption suppression in Pipecat Cloud. Missing for 10: independent hands-on validation of graceful context/turn recovery specifically (community evidence is about debugging/latency tooling, not barge-in correctness) and no explicit documentation of how conversation context is updated/rolled back after an interruption.

   • [claimed-docs] “When the user starts speaking while the bot is talking, the bot stops immediately, in-flight work is cancelled, and the pipeline is ready fo…”
   • [claimed-docs] “Interruptions (also called barge-in) let the user talk over the bot. When the user starts speaking while the bot is talking, the bot stops i…”
   • [claimed-docs] “Smart Turn Detection uses an advanced machine learning model to determine when a user has finished speaking and your bot should respond.”
   • [claimed-docs] “Smart Turn Detection recognizes natural conversational cues like intonation patterns and linguistic signals for more natural conversations.”
   • [claimed-docs] “Eliminates background noise and voices that cause false interruptions, powered by Krisp.”

   proof ↗⚑ flag

2. developerEnable noise suppression or audio filtering so the agent stays coherent on noisy real-world calls

   weight 1 · round to Pipecat
   ElevenLabs Agents—none0/10

   The evidence pack covers voice customization, turn-taking/interruption settings, tools, testing, and telephony (SIP) integration, but nowhere mentions noise suppression, background noise handling, or audio filtering for noisy real-world calls. This is a fair capability to expect for a voice-agent platform handling phone calls, but no documentation or feature reference confirms it exists.

   • [claimed-docs] “Conversation flow settings determine how your assistant handles periods of user silence, interruptions during speech, and turn-taking behavi…”
   • [claimed-docs] “Configure turn-taking, interruptions, and timeout settings”
   • [claimed-docs] “This integration enables all customers to use their existing phone systems while leveraging ElevenLabs' advanced voice AI capabilities.”

   proof ↗⚑ flag

   Pipecat✓fullclaimed8/10

   Pipecat Cloud explicitly ships Krisp-powered noise/background-voice suppression to eliminate false interruptions, and the framework's Smart Turn Detection complements this by recognizing genuine speech cues, keeping the agent coherent on noisy calls. Missing for 10: no first-party docs on self-hosted (non-cloud) noise suppression configuration, and no independent hands-on validation of suppression quality in real noisy environments.

   • [claimed-docs] “Eliminates background noise and voices that cause false interruptions, powered by Krisp.”
   • [claimed-docs] “Smart Turn Detection uses an advanced machine learning model to determine when a user has finished speaking and your bot should respond.”
   • [claimed-docs] “Smart Turn Detection recognizes natural conversational cues like intonation patterns and linguistic signals for more natural conversations.”

   proof ↗⚑ flag

3. developerUse model-based end-of-turn detection beyond simple VAD silence timeouts so the agent doesn't talk over slow speakers

   weight 2 · round to Pipecat
   ElevenLabs Agents—none0/10

   Docs mention configurable turn-taking, interruption handling, and timeout settings (docs-33, docs-42), but nothing in the evidence describes a model-based end-of-turn detector that goes beyond a VAD/silence-timeout mechanism — no mention of semantic or ML-based turn-completion detection to avoid interrupting slow speakers. Missing for 10: any documentation of a dedicated end-of-turn prediction model, benchmarks or comparisons versus simple VAD timeouts, or hands-on evidence that the agent waits appropriately for slow speakers.

   • [claimed-docs] “Conversation flow settings determine how your assistant handles periods of user silence, interruptions during speech, and turn-taking behavi…”
   • [claimed-docs] “Configure turn-taking, interruptions, and timeout settings”
   • [claimed-docs] “The Max conversation duration setting limits the total time a conversation can remain active. ... The default is 600 seconds (10 minutes).”

   proof ↗⚑ flag

   Pipecat✓fullclaimed8/10

   Pipecat documents 'Smart Turn Detection' as an ML model that determines end-of-turn based on intonation and linguistic cues, going beyond simple VAD silence timeouts, directly matching the story (pipecat-docs-9, pipecat-docs-31). Additional STT latency-tuning docs show the framework accounts for provider-specific delays to avoid premature cutoffs (pipecat-docs-15, pipecat-docs-20). Missing for 10: independent/hands-on evidence validating the model's accuracy on slow speakers in production, and no third-party benchmark of false-interruption reduction.

   • [claimed-docs] “Smart Turn Detection uses an advanced machine learning model to determine when a user has finished speaking and your bot should respond.”
   • [claimed-docs] “Smart Turn Detection recognizes natural conversational cues like intonation patterns and linguistic signals for more natural conversations.”
   • [claimed-docs] “Pipecat ships with measured P99 latency values for each supported service so that turn detection can account for this delay automatically.”
   • [claimed-docs] “Time To Final Segment (TTFS) measures how long it takes from the moment a user stops speaking until the STT service delivers the final trans…”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents~partialprobed7/10

   Docs show agents can be created, configured, and managed via the API, CLI, dashboard, or hosted MCP server, and quickstart explicitly says agents can be created 'via the API or the web dashboard,' implying broad parity across surfaces (elevenlabs-agents-docs-1, docs-39, docs-44). However, there's no explicit feature-by-feature confirmation that every dashboard capability (e.g., visual workflow builder, analytics dashboards, testing UI) has a full API equivalent, nor independent verification of parity. Missing for 10: an explicit parity statement or changelog confirming all dashboard features (workflow builder, analytics, testing suite) are fully scriptable via API, and independent/hands-on confirmation of that parity.

   • [claimed-docs] “ElevenLabs Agents are managed either through the ElevenAgents dashboard, the ElevenLabs API, the Agents CLI or the hosted MCP server.”
   • [claimed-docs] “we'll start by creating an agent via the API or the web dashboard”
   • [claimed-docs] “create, configure, and manage the agents in your workspace through natural language, with nothing to install or run locally”
   • [claimed-docs] “Build multi-step workflows with visual workflow builder”
   • [probe] “PROBE runtime (recorded 2026-09-05): `npx -y @elevenlabs/cli --version` printed `elevenlabs 1.1.0` keylessly — the official ElevenLabs CLI (…”

   proof ↗⚑ flag

   Pipecat~partialprobed4/10

   Pipecat is fundamentally code-first: the CLI covers init/scaffolding, cloud deploy, evals, and context-hub (pipecat-probe-rt-1, pipecat-docs-16/34/37), and the framework itself is the primary interface. But evidence never explicitly confirms that Pipecat Cloud's web dashboard (agent profiles, concurrency settings, monitoring) exposes no exclusive UI-only functionality, nor does it document a full REST/OpenAPI surface (probe shows no discoverable OpenAPI spec, pipecat-probe-2) for programmatic parity with any cloud console. Missing for 10: explicit UI/API parity statement, documented OpenAPI/REST API for Pipecat Cloud console features, and confirmation no dashboard-only settings exist.

   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”
   • [claimed-docs] “Install the Pipecat CLI and scaffold the quickstart project (also writes AGENTS.md + CLAUDE.md)”
   • [claimed-docs] “pipecat init quickstart # Change to the project directory cd pipecat-quickstart”
   • [claimed-docs] “Pipecat Cloud offers different agent profiles, to best support your use case and compute needs.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.pipecat.ai/openapi.json, https://docs.pipecat.ai/swagger.json, https://docs.pipecat.ai/…”
   • [claimed-docs] “Pipecat ships a built-in development runner (pipecat.runner.run) that handles the server-side glue most bots need during development: creati…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to Pipecat
   ElevenLabs Agents—none0/10

   No evidence describes a data export feature, open-format data dumps, or account deletion/portability tooling; only conversation retention settings are documented, not export mechanisms. Missing for 10: any documented export API/CLI command, supported open export formats (e.g., JSON/CSV), and confirmation of full data portability/account closure process.

   ⚑ flag

   Pipecat~partialprobed5/10

   Pipecat is open-source (BSD-2) and self-hostable, meaning users inherently retain full control of their data (audio, transcripts, logs) rather than it being locked in a vendor's system — recordings via AudioBufferProcessor and transcripts via turn events are captured directly in the developer's own infrastructure. However, there's no explicit documentation of an 'export' feature, standardized open data formats, or migration tooling for leaving a hosted deployment (e.g., Pipecat Cloud). missing for 10: explicit data-export/format documentation, Pipecat Cloud data portability/exit guarantees, independent confirmation of format openness.

   • [claimed-docs] “Pipecat's \`AudioBufferProcessor\` makes it easy to capture high-quality audio recordings of both the user and bot during interactions.”
   • [claimed-docs] “Pipecat's turn events make it easy to collect both user and assistant messages as they occur.”
   • [probe] “PROBE runtime (recorded 2026-09-05): pypi pipecat-ai 1.8.1 (BSD-2 OSS) installs and `import pipecat` succeeds with no key, printing its star…”
   • [claimed-docs] “Open source Python framework for building voice and multimodal AI pipelines. Orchestrate 150+ AI services with ultra-low latency.”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round to Pipecat
   ElevenLabs Agents—none0/10

   ElevenLabs Agents is a closed, proprietary SaaS product; the CLI is open on npm but no evidence indicates the core Agents platform/backend source is available under an open license. No license file, GitHub repo, or open-source statement is present in the evidence pack.

   ⚑ flag

   Pipecat✓fullprobed9/10

   Pipecat is explicitly documented as an 'Open source Python framework' with a public GitHub repo, and a runtime probe confirms the pip package is BSD-2 licensed and installable/importable without any key, confirming genuine open-source source access. Missing for 10: no explicit citation of a LICENSE file or license text itself, only the probe's characterization.

   • [claimed-docs] “Open source Python framework for building voice and multimodal AI pipelines. Orchestrate 150+ AI services with ultra-low latency.”
   • [github] “Build a single voice agent or a full multi-agent system where specialists hand off, fan out in parallel, and coordinate over a shared bus, l…”
   • [probe] “PROBE runtime (recorded 2026-09-05): pypi pipecat-ai 1.8.1 (BSD-2 OSS) installs and `import pipecat` succeeds with no key, printing its star…”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round to Pipecat
   ElevenLabs Agents—none0/10

   ElevenLabs Agents is entirely a hosted SaaS product—managed via dashboard, API, CLI, and hosted MCP server, all pointing to ElevenLabs' own infrastructure; there is no evidence of an open-source release, downloadable server binary, or self-hosting option for the core voice-agent runtime. missing for 10: any open-source repo or self-hosted deployment package, docs describing running the core service on one's own infrastructure, independent confirmation of self-hosting.

   • [claimed-docs] “ElevenLabs Agents are managed either through the ElevenAgents dashboard, the ElevenLabs API, the Agents CLI or the hosted MCP server.”
   • [claimed-docs] “an assistant such as Claude can create, configure, and manage the agents in your workspace through natural language, with nothing to install…”
   • [claimed-docs] “create, configure, and manage the agents in your workspace through natural language, with nothing to install or run locally”

   proof ↗⚑ flag

   Pipecat✓fullprobed8/10

   Pipecat is an open-source Python framework (BSD-2 licensed), pip-installable and runnable with no key/cloud dependency, with a documented dev runner and explicit note that the same bot file is portable to self-hosting setups; local inference (Ollama, Whisper) further confirms self-hosting without vendor lock-in. missing for 10: no dedicated self-hosting/production deployment guide (e.g., Docker/Kubernetes reference architecture) or independent case study of a large-scale self-hosted deployment.

   • [probe] “PROBE runtime (recorded 2026-09-05): pypi pipecat-ai 1.8.1 (BSD-2 OSS) installs and `import pipecat` succeeds with no key, printing its star…”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”
   • [claimed-docs] “This is the property that makes the same bot file portable across the development runner, Pipecat Cloud, and most production self-hosting se…”
   • [community] “Local inference is already supported via Pipecat, you can use ollama or any custom OpenAI endpoint. Local STT is also supported via whisper,…”
   • [community] “Yes, Pipecat already supports that natively, so this can be done easily with ollama... Also, check out any provider they support, and it can…”

   proof ↗⚑ flag

Pricing

1. founderSee published per-minute or usage pricing and estimate cost per call before committing

   weight 2 · round drawn
   ElevenLabs Agents—none0/10

   No evidence item in the pack references pricing, per-minute rates, usage-based billing, or cost calculators for ElevenLabs Agents; all citations are product/docs feature descriptions. Since founders evaluating a voice-agent SaaS could reasonably expect published pricing, the axis applies but is entirely unaddressed in this evidence pack.

   ⚑ flag

   Pipecat—none0/10

   The evidence pack references a Pipecat Cloud pricing page (concurrency, agent profiles) but never shows actual per-minute/usage rates, a pricing table, or a cost calculator that would let a founder estimate cost per call; the core open-source framework itself is free/self-hosted with no billing model at all.

   • [claimed-docs] “Pipecat Cloud supports unlimited concurrency.”
   • [claimed-docs] “Pipecat Cloud offers different agent profiles, to best support your use case and compute needs.”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round to Pipecat
   ElevenLabs Agents—none0/10

   Evidence covers data retention duration settings, HIPAA/BAA eligibility, and infrastructure details, but nothing addresses selectable data region/residency options for storage location. missing for 10: any mention of region selection, data residency controls, or geographic storage options.

   • [claimed-docs] “Retention settings allow you to configure how long your conversational agent stores conversation transcripts and audio recordings.”
   • [claimed-docs] “Retention settings allow you to configure how long your conversational agent stores conversation transcripts and audio recordings”
   • [claimed-docs] “ElevenLabs Agents is one of ElevenLabs' HIPAA-eligible services, and we offer Business Associate Agreements (BAAs) to eligible customers”

   proof ↗⚑ flag

   Pipecat~partialprobed3/10

   Pipecat is an open-source, self-hostable framework (pip-installable, runs locally or on your own infrastructure), which implicitly lets an operator choose where data/compute resides by choosing their own deployment location; however there is no explicit documentation of region/residency selection controls, especially for the hosted Pipecat Cloud offering. Missing for 10: explicit region-selection UI/API for Pipecat Cloud, data-residency compliance documentation (GDPR/SOC2 region guarantees), and confirmation that self-hosting is the only path to residency control.

   • [probe] “PROBE runtime (recorded 2026-09-05): pypi pipecat-ai 1.8.1 (BSD-2 OSS) installs and `import pipecat` succeeds with no key, printing its star…”
   • [claimed-docs] “Pipecat ships a built-in development runner (pipecat.runner.run) that handles the server-side glue most bots need during development: creati…”
   • [claimed-docs] “This is the property that makes the same bot file portable across the development runner, Pipecat Cloud, and most production self-hosting se…”
   • [claimed-docs] “Pipecat Cloud offers different agent profiles, to best support your use case and compute needs.”

   proof ↗⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   ElevenLabs Agents—none0/10

   The evidence pack covers data retention settings (how long transcripts/audio are stored) and HIPAA/BAA compliance, but nowhere mentions an explicit opt-out or policy regarding use of customer data/conversations for AI model training. missing for 10: explicit training-data opt-out policy or setting, terms-of-service language on model training use, any statement distinguishing enterprise vs free-tier data usage for training.

   • [claimed-docs] “Retention settings allow you to configure how long your conversational agent stores conversation transcripts and audio recordings.”
   • [claimed-docs] “Retention settings allow you to configure how long your conversational agent stores conversation transcripts and audio recordings”
   • [claimed-docs] “ElevenLabs Agents is one of ElevenLabs' HIPAA-eligible services, and we offer Business Associate Agreements (BAAs) to eligible customers”

   proof ↗⚑ flag

   Pipecat—none0/10

   The evidence pack contains no mention of data-training opt-outs, data-usage policies, or privacy commitments regarding customer data and AI model training for either the open-source Pipecat framework or the hosted Pipecat Cloud service. Since Pipecat Cloud is a real hosted product where such a policy could plausibly exist, this axis applies, but no evidence supports it. Missing for 10: any privacy policy, ToS clause, or documentation stating customer data is not used to train models, and no independent corroboration.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents~partialclaimed6/10

   Docs explicitly document configurable retention settings for conversation transcripts and audio recordings, giving users control over how long data is stored. However, evidence does not detail a deletion mechanism (e.g., data subject deletion requests, API/CLI commands to purge stored data) beyond retention duration settings. Missing for 10: explicit deletion/export controls, granular per-conversation deletion API, and independent verification of retention enforcement.

   • [claimed-docs] “Retention settings allow you to configure how long your conversational agent stores conversation transcripts and audio recordings.”
   • [claimed-docs] “Retention settings allow you to configure how long your conversational agent stores conversation transcripts and audio recordings”

   proof ↗⚑ flag

   Pipecat—none0/10

   Pipecat is a self-hostable open-source framework and also offers a hosted Pipecat Cloud option that processes voice/audio data, so data-retention/deletion controls are a fair question for this product category, but the evidence pack contains no documentation of retention policies, deletion APIs, or data lifecycle controls — only unrelated features like recording/transcription and metrics.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   ElevenLabs Agents—none0/10

   The evidence pack covers conversation transcript/audio retention settings (docs-26, docs-52) but nothing about opting out of product telemetry or usage tracking (e.g., CLI analytics, dashboard usage metrics) for the AI-native user. Retention configuration is about customer conversation data, not the vendor's own telemetry collection, so this axis has no supporting evidence.

   ⚑ flag

   Pipecat—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

Call control

1. developerEscalate a live call to a human with warm or blind transfer, passing context along

   weight 2 · round drawn
   ElevenLabs Agents—none0/10

   The evidence pack covers telephony features like SIP trunking, tools, webhooks, and MCP integrations, but nowhere documents call transfer (warm or blind) to a human agent or passing conversation context during handoff. Missing for 10: any mention of transfer_to_number/transfer_to_agent tool types, warm/blind transfer configuration, or context-passing during escalation.

   ⚑ flag

   Pipecat—none0/10

   The evidence pack covers Pipecat's telephony deployment (webhook/carrier integration for Twilio, Telnyx, etc.), conversation flow transitions via Pipecat Flows, and subagent hand-off for tasks, but nothing addresses call transfer to a human agent (warm or blind) or passing call context/metadata during such a transfer. This is a reasonable telephony axis for a voice AI framework, but no citation demonstrates or documents this capability.

   ⚑ flag

2. developerMy agent can send DTMF keypresses, navigate IVR menus, and detect or leave voicemail

   weight 1 · round drawn
   ElevenLabs Agents—none0/10

   The evidence pack confirms ElevenLabs Agents supports telephony integration via SIP trunking and phone numbers, so the telephony axis clearly applies, but nothing in the docs mentions DTMF tone sending, IVR menu navigation, or voicemail detection/leaving capabilities.

   • [claimed-docs] “This integration enables all customers to use their existing phone systems while leveraging ElevenLabs' advanced voice AI capabilities.”
   • [claimed-docs] “Route calls to AI agents without changing your existing phone infrastructure”

   proof ↗⚑ flag

   Pipecat—none0/10

   Pipecat clearly supports telephony transports (Twilio, Telnyx, Plivo, Exotel, SIP) for building call-handling bots, so this axis applies, but the evidence pack contains no mention of DTMF keypress sending, IVR menu navigation, or voicemail detection/leaving capabilities anywhere in the docs or community sources.

   • [claimed-docs] “Telephony bots have a different shape than WebRTC bots. The session doesn't start because a client sent your dispatcher an HTTP request — it…”
   • [claimed-docs] “the carrier (Twilio, Telnyx, Plivo, Exotel, or your SIP provider) is calling your webhook to tell you there's an inbound call”
   • [claimed-docs] “Any Transport: WebRTC, SIP, PSTN”
   • [claimed-docs] “python bot.py -t twilio -x your-name.ngrok.io”

   proof ↗⚑ flag

Campaigns

1. founderRun batch outbound call campaigns with scheduling and throughput controls

   weight 2 · round drawn
   ElevenLabs Agents—none0/10

   The evidence pack documents phone-number/SIP trunking integration for routing calls to agents, but contains no mention of batch outbound calling, campaign scheduling, or throughput/rate controls for outbound call volume. Since telephony campaign management is a fair capability to expect from a voice-agent telephony product, absence of evidence here is 'none' rather than 'na'.

   • [claimed-docs] “This integration enables all customers to use their existing phone systems while leveraging ElevenLabs' advanced voice AI capabilities.”
   • [claimed-docs] “Route calls to AI agents without changing your existing phone infrastructure”

   proof ↗⚑ flag

   Pipecat—none0/10

   Pipecat's telephony docs cover per-call webhook-driven session handling (inbound/outbound single calls via Twilio/Telnyx/etc.) and unlimited concurrency on Pipecat Cloud, but there is no evidence of batch campaign management, call scheduling, pacing/throughput controls, or dialer-list orchestration for outbound campaigns — this is a fair ask for a telephony-capable voice AI framework but no such capability is documented.

   • [claimed-docs] “Telephony bots have a different shape than WebRTC bots. The session doesn't start because a client sent your dispatcher an HTTP request — it…”
   • [claimed-docs] “the carrier (Twilio, Telnyx, Plivo, Exotel, or your SIP provider) is calling your webhook to tell you there's an inbound call”
   • [claimed-docs] “python bot.py -t twilio -x your-name.ngrok.io”
   • [claimed-docs] “Pipecat Cloud supports unlimited concurrency.”
   • [claimed-docs] “Any Transport: WebRTC, SIP, PSTN”

   proof ↗⚑ flag

Numbers

1. developerProvision phone numbers and run both inbound and outbound calls through the platform's API

   weight 3 · round to ElevenLabs Agents
   ElevenLabs Agents~partialclaimed6/10

   Docs confirm phone/telephony integration via SIP trunking, letting agents route inbound/outbound calls over existing phone infrastructure, and agents can be created/configured via the API. However, evidence does not explicitly show a documented API endpoint for provisioning new phone numbers or explicitly triggering outbound calls programmatically vs. via SIP trunk setup. Missing for 10: explicit phone-number provisioning API endpoint docs, explicit outbound-call-initiation API example, and independent/hands-on confirmation of inbound+outbound call flows.

   • [claimed-docs] “This integration enables all customers to use their existing phone systems while leveraging ElevenLabs' advanced voice AI capabilities.”
   • [claimed-docs] “Route calls to AI agents without changing your existing phone infrastructure”
   • [claimed-docs] “we'll start by creating an agent via the API or the web dashboard”

   proof ↗⚑ flag

   Pipecat~partialclaimed4/10

   Docs show Pipecat integrates with telephony carriers (Twilio, Telnyx, Plivo, Exotel, SIP) to receive inbound call webhooks and run bots over PSTN/SIP transports, and CLI examples show running a bot against Twilio (`python bot.py -t twilio -x ...`), covering the 'run calls' half of the story. However, there is no evidence Pipecat itself exposes an API to provision/purchase phone numbers — that is left to the carrier's own console/API, not documented as a Pipecat capability. Missing for 10: native phone-number provisioning API/CLI, outbound call initiation API details, and any first-party number-management workflow.

   • [claimed-docs] “Telephony bots have a different shape than WebRTC bots. The session doesn't start because a client sent your dispatcher an HTTP request — it…”
   • [claimed-docs] “the carrier (Twilio, Telnyx, Plivo, Exotel, or your SIP provider) is calling your webhook to tell you there's an inbound call”
   • [claimed-docs] “Any Transport: WebRTC, SIP, PSTN”
   • [claimed-docs] “python bot.py -t twilio -x your-name.ngrok.io”

   proof ↗⚑ flag

Sip

1. platform-engineerConnect my own carrier or PBX via SIP trunking (or import Twilio/Telnyx numbers) instead of being locked to bundled telephony

   weight 2 · round to Pipecat
   ElevenLabs Agents~partialclaimed7/10

   Docs explicitly describe a SIP trunking integration that lets customers connect their existing phone systems/carrier infrastructure to ElevenLabs agents without changing carriers, directly addressing the core ask of bringing your own PBX/carrier. However, the pack has no explicit mention of importing existing Twilio/Telnyx numbers or detailed carrier onboarding steps, and no independent/hands-on confirmation of a live SIP trunk setup working end-to-end. Missing for 10: explicit Twilio/Telnyx number import instructions, and third-party/hands-on validation of the SIP trunking flow.

   • [claimed-docs] “This integration enables all customers to use their existing phone systems while leveraging ElevenLabs' advanced voice AI capabilities.”
   • [claimed-docs] “Route calls to AI agents without changing your existing phone infrastructure”

   proof ↗⚑ flag

   Pipecat✓fullclaimed7/10

   Docs explicitly describe telephony bots being invoked by carrier webhooks including 'Twilio, Telnyx, Plivo, Exotel, or your SIP provider,' and highlight 'Any Transport: WebRTC, SIP, PSTN' plus a working CLI example (`python bot.py -t twilio -x ...ngrok.io`), showing genuine carrier/SIP flexibility rather than lock-in to one bundled provider. Missing for 10: concrete walkthrough of raw SIP trunk/PBX configuration (vs. just carrier webhooks), and independent/hands-on confirmation of a non-Twilio/Telnyx SIP trunk actually working in production.

   • [claimed-docs] “Telephony bots have a different shape than WebRTC bots. The session doesn't start because a client sent your dispatcher an HTTP request — it…”
   • [claimed-docs] “the carrier (Twilio, Telnyx, Plivo, Exotel, or your SIP provider) is calling your webhook to tell you there's an inbound call”
   • [claimed-docs] “Any Transport: WebRTC, SIP, PSTN”
   • [claimed-docs] “python bot.py -t twilio -x your-name.ngrok.io”

   proof ↗⚑ flag

Analytics

1. ai-native userThe platform's AI reviews my calls for me — scoring quality, flagging failures, and analyzing resolution automatically

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents✓fullclaimed7/10

   Docs explicitly describe automated post-call analysis: 'Success evaluation' scores conversation quality/goal achievement, 'Sentiment analysis' flags user sentiment, and 'Data collection' extracts structured resolution details — directly matching the story's scoring/flagging/resolution-analysis ask. Missing for 10: no independent/hands-on validation of analysis accuracy, and no explicit 'failure flagging' terminology beyond success-criteria evaluation.

   • [claimed-docs] “Success evaluation: Define custom criteria to assess conversation quality, goal achievement, and customer satisfaction”
   • [claimed-docs] “Data collection: Extract specific data points from conversations such as contact information, issue details, or any structured information”
   • [claimed-docs] “Sentiment analysis: Understand user sentiment across completed conversations”
   • [claimed-docs] “Define custom criteria to assess conversation quality, goal achievement, and customer satisfaction”
   • [claimed-docs] “Extract structured information from conversations such as contact details and business data.”

   proof ↗⚑ flag

   Pipecat—none0/10

   Pipecat ships developer-facing dev-time Evals (describe expected conversation, run against real agent) plus metrics/transcripts/audio-recording primitives, but there is no evidence of an AI system that automatically reviews production calls, scores quality, flags failures, or analyzes resolution after the fact — Evals is a pre-deployment testing tool, not post-call analytics.

   • [claimed-docs] “Pipecat Evals is the framework's built-in system for testing agent behavior. You describe a conversation and the behavior you expect, and Pi…”
   • [claimed-docs] “You describe a conversation and the behavior you expect, and Pipecat runs it against your real agent (the same pipeline, the same services, …”
   • [claimed-docs] “Pipecat offers built-in metrics for both, which can be enabled with straightforward configuration options.”
   • [claimed-docs] “Pipecat's turn events make it easy to collect both user and assistant messages as they occur.”

   proof ↗⚑ flag

2. founderSee call analytics — success rates, durations, outcomes, sentiment — in dashboards without building my own

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents~partialclaimed6/10

   Docs confirm built-in agent-analysis features—success evaluation, sentiment analysis, data extraction, and conversation search—managed via the ElevenAgents dashboard, which covers most of the story's asked-for metrics without custom-building analytics. However, there's no explicit evidence of a dashboard showing aggregated success-rate percentages or call-duration analytics/rollups across calls, only per-conversation criteria and settings. Missing for 10: dashboard screenshots/evidence of aggregate success-rate metrics, call-duration reporting/rollups, and independent corroboration that these appear in a unified analytics view.

   • [claimed-docs] “Success evaluation: Define custom criteria to assess conversation quality, goal achievement, and customer satisfaction”
   • [claimed-docs] “Data collection: Extract specific data points from conversations such as contact information, issue details, or any structured information”
   • [claimed-docs] “Sentiment analysis: Understand user sentiment across completed conversations”
   • [claimed-docs] “Searching conversations: Find messages by keyword or meaning across your conversation history.”
   • [claimed-docs] “ElevenLabs Agents are managed either through the ElevenAgents dashboard, the ElevenLabs API, the Agents CLI or the hosted MCP server.”
   • [claimed-docs] “Extract structured information from conversations such as contact details and business data.”

   proof ↗⚑ flag

   Pipecat—none0/10

   Pipecat exposes low-level building blocks (metrics, transcripts, audio recording, evals) that a developer could use to construct analytics, and a community debugging tool (Whisker) for pipeline tracing — but there is no evidence of a built-in dashboard surfacing success rates, call durations, outcomes, or sentiment without the founder building it themselves.

   • [claimed-docs] “Pipecat offers built-in metrics for both, which can be enabled with straightforward configuration options.”
   • [claimed-docs] “Pipecat's turn events make it easy to collect both user and assistant messages as they occur.”
   • [claimed-docs] “Pipecat's \`AudioBufferProcessor\` makes it easy to capture high-quality audio recordings of both the user and bot during interactions.”
   • [community] “With Whisker you can: View a live graph of your pipeline, watch frame processors flash in real time, select a processor to inspect its frame…”
   • [claimed-docs] “You describe a conversation and the behavior you expect, and Pipecat runs it against your real agent (the same pipeline, the same services, …”

   proof ↗⚑ flag

Monitoring

1. platform-engineerMonitor live calls in production and get alerts when agents misbehave or error rates spike

   weight 1 · round drawn
   ElevenLabs Agents~partialclaimed4/10

   Docs show post-call analytics (sentiment analysis, success evaluation, data extraction, conversation search) and a marketing claim to 'monitor' agents, but there is no evidence of live/real-time call monitoring dashboards, error-rate metrics, or alerting/notification mechanisms for misbehaving agents. Missing for 10: real-time monitoring dashboard, error-rate/latency metrics, alerting/notification system, on-call integration (e.g., PagerDuty/Slack alerts).

   • [claimed-docs] “Success evaluation: Define custom criteria to assess conversation quality, goal achievement, and customer satisfaction”
   • [claimed-docs] “Sentiment analysis: Understand user sentiment across completed conversations”
   • [claimed-docs] “Searching conversations: Find messages by keyword or meaning across your conversation history.”
   • [claimed-docs] “Define custom criteria to assess conversation quality, goal achievement, and customer satisfaction”
   • [claimed-docs] “Configure, deploy and monitor natural, human-sounding agents in 70+ languages with leading accuracy and ultra-low latency across voice or ch…”

   proof ↗⚑ flag

   Pipecat~partialcommunity4/10

   Pipecat provides building blocks for observability — built-in metrics (docs-5), turn-event transcripts (docs-4), audio recording (docs-3/21/29), and the community-highlighted Whisker tool that shows a live pipeline graph and frame-level debugger (pipecat-comm-3/4/5) — but these are development/debugging aids, not a production monitoring or alerting system. There is no evidence of dashboards for live production calls, error-rate-spike detection, or alert/notification integrations (e.g., PagerDuty, Slack alerts) for misbehaving agents. Missing for 10: production-grade dashboards for live call monitoring, automated error-rate/anomaly alerting, and integration with on-call/alerting systems.

   • [claimed-docs] “Pipecat offers built-in metrics for both, which can be enabled with straightforward configuration options.”
   • [claimed-docs] “Pipecat's turn events make it easy to collect both user and assistant messages as they occur.”
   • [community] “As someone who spends a lot of time looking at timestamped log lines to debug Pipecat pipelines, I'm a big fan of this work... For 1 and 3 […”
   • [community] “With Whisker you can: View a live graph of your pipeline, watch frame processors flash in real time, select a processor to inspect its frame…”
   • [community] “I had been thinking of working on something like this recently as a way to debug Pipecat pipelines. But the work Aleix has done goes far bey…”

   proof ↗⚑ flag

Testing

1. developerTest agents with simulated conversations or evals before putting them on real phone calls

   weight 2 · round drawn
   ElevenLabs Agents✓fullclaimed8/10

   ElevenLabs documents an explicit Agent Testing feature that verifies conversational responses, tool usage, and full multi-turn outcomes before deployment, including converting real conversations into simulated test cases, plus success-evaluation criteria for assessing conversation quality/goal achievement — directly matching the pre-deployment simulated testing story. Missing for 10: no independent/hands-on report validating test accuracy or CI-integration details, and no explicit mention of automated eval scoring pipelines beyond docs description.

   • [claimed-docs] “Agent testing lets you verify conversational responses, tool usage, and full multi-turn outcomes before you deploy.”
   • [claimed-docs] “Transform real conversations into test cases when you find an interaction where the agent underperformed.”
   • [claimed-docs] “Agent testing lets you verify conversational responses, tool usage, and full multi-turn outcomes before you deploy”
   • [claimed-docs] “Success evaluation: Define custom criteria to assess conversation quality, goal achievement, and customer satisfaction”
   • [claimed-docs] “Define custom criteria to assess conversation quality, goal achievement, and customer satisfaction”

   proof ↗⚑ flag

   Pipecat✓fullprobed8/10

   Pipecat Evals is a documented, built-in framework feature letting developers describe conversations/expected behavior and run them against the real agent pipeline before deployment, and the runtime probe confirms a working `pipecat eval` CLI subcommand exists alongside `init`/`cloud`. This directly addresses simulated-conversation testing before real phone calls, though evidence doesn't show telephony-specific eval scenarios or third-party validation of eval accuracy. Missing for 10: independent/community corroboration of Evals in practice, and explicit examples of testing telephony-specific call flows via evals.

   • [claimed-docs] “Pipecat Evals is the framework's built-in system for testing agent behavior. You describe a conversation and the behavior you expect, and Pi…”
   • [claimed-docs] “You describe a conversation and the behavior you expect, and Pipecat runs it against your real agent (the same pipeline, the same services, …”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Pipecat CLI (pypi pipecat-ai[cli]) ran keylessly via uvx — `pipecat --help` lists init (pr…”

   proof ↗⚑ flag

Post call

1. developerExtract structured data from every call — outcomes, entities, dispositions — delivered via API or webhook after the call

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents~partialclaimed6/10

   Docs show structured post-call outputs (data collection with contact/business data extraction, success-evaluation criteria, sentiment analysis) available via 'agent-analysis' features, and the platform is manageable via API. However, the pack never explicitly documents a webhook payload/callback delivering this structured analysis after a call, only webhook *tools* for in-call actions — missing for 10: explicit post-call webhook delivery mechanism/schema for outcomes and dispositions, and confirmation these analysis fields are retrievable via a documented API endpoint.

   • [claimed-docs] “Success evaluation: Define custom criteria to assess conversation quality, goal achievement, and customer satisfaction”
   • [claimed-docs] “Data collection: Extract specific data points from conversations such as contact information, issue details, or any structured information”
   • [claimed-docs] “Sentiment analysis: Understand user sentiment across completed conversations”
   • [claimed-docs] “Extract structured information from conversations such as contact details and business data”
   • [claimed-docs] “Extract structured information from conversations such as contact details and business data.”
   • [claimed-docs] “Tools give your assistant the ability to make external function calls to third party apps so you can get real-time information.”
   • [claimed-docs] “allow an assistant to trigger authenticated actions based on the conversation, like scheduling meetings or initiating order returns”

   proof ↗⚑ flag

   Pipecat~partialclaimed4/10

   Pipecat provides building blocks that a developer could use to build this — Flows functions can 'process data by interfacing with external systems and APIs' (function calling) and turn events/transcripts capture user and assistant messages, plus telephony webhooks for call events — but there is no documented turnkey feature for automatic post-call structured extraction (outcomes/entities/dispositions) delivered via a dedicated API or webhook. Missing for 10: a built-in post-call summary/disposition-extraction pipeline, a documented structured-output schema for call outcomes, and an out-of-box webhook delivery mechanism specifically for post-call analytics (only inbound-call webhooks and generic function-calling/transcript primitives are documented).

   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data ... 2. Progress the conversation by transitioning between nodes”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: Process data by interfacing with external systems and APIs...Progress the conversation by…”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data by interfacing with external systems and APIs... 2. Progress the conversa…”
   • [claimed-docs] “Flows auto-derives the function's metadata — name, description, parameter properties (with their descriptions), and which parameters are req…”
   • [claimed-docs] “Pipecat's turn events make it easy to collect both user and assistant messages as they occur.”
   • [claimed-docs] “Telephony bots have a different shape than WebRTC bots. The session doesn't start because a client sent your dispatcher an HTTP request — it…”
   • [claimed-docs] “the carrier (Twilio, Telnyx, Plivo, Exotel, or your SIP provider) is calling your webhook to tell you there's an inbound call”

   proof ↗⚑ flag

Tools

1. ai-native userMy voice agent can plug in MCP servers as tool sources so one integration grants it whole toolsets mid-call

   weight 2 · round to ElevenLabs Agents
   ElevenLabs Agents✓fullclaimed8/10

   Docs explicitly describe connecting conversational agents to external MCP servers as a tool source, enabling access to data/processing capabilities via that server mid-call, directly matching the story. Missing for 10: independent hands-on confirmation of mid-call multi-tool retrieval from a connected MCP server (only vendor docs, no third-party validation) and detail on how many tools/toolsets a single MCP connection exposes.

   • [claimed-docs] “ElevenLabs allows you to connect your conversational agents to external MCP servers. This enables your agents to: Access and process informa…”
   • [claimed-docs] “ElevenLabs allows you to connect your conversational agents to external MCP servers.”
   • [claimed-docs] “Access and process information from various data sources via the MCP server”

   proof ↗⚑ flag

   Pipecat—none0/10

   The evidence only shows Pipecat's context-hub *acting as* an MCP server for coding-agent tooling (pipecat-docs-11/38), which is the opposite direction from what the story asks — a voice agent consuming external MCP servers as tool sources mid-call. Function/tool calling is documented via Pipecat Flows (pipecat-docs-6/22/30/39), but no evidence shows the voice agent connecting to or importing toolsets from external MCP servers during a call.

   • [claimed-docs] “pipecat context-hub install registers the hub as an MCP server with your coding agent and builds the local index”
   • [claimed-docs] “pipecat context-hub install` registers the hub as an MCP server with your coding agent and builds the local index”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data ... 2. Progress the conversation by transitioning between nodes”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: Process data by interfacing with external systems and APIs...Progress the conversation by…”

   proof ↗⚑ flag

2. developerMy agent can call external APIs and custom functions mid-conversation and speak the result without awkward dead air

   weight 3 · round to Pipecat
   ElevenLabs Agents✓fullclaimed7/10

   Docs clearly describe multiple tool-calling mechanisms — webhook tools for external API calls, client-side tools, sandboxed custom JS, and MCP server integration — explicitly for making external function calls mid-conversation to fetch real-time info and trigger authenticated actions. Conversation-flow settings (turn-taking, interruption handling, ultra-low latency) address smooth conversational pacing, though not explicitly framed as avoiding dead air during tool execution. Missing for 10: explicit documentation of filler/hold-speech behavior while awaiting API results, and independent/hands-on verification that tool calls don't introduce audible pauses.

   • [claimed-docs] “Enable agents to call clients & APIs to perform actions”
   • [claimed-docs] “Tools executed directly on the client-side application (e.g., web browser, mobile app).”
   • [claimed-docs] “Tools give your assistant the ability to make external function calls to third party apps so you can get real-time information.”
   • [claimed-docs] “Custom JavaScript executed in a sandboxed environment on ElevenLabs' infrastructure.”
   • [claimed-docs] “ElevenLabs allows you to connect your conversational agents to external MCP servers. This enables your agents to: Access and process informa…”
   • [claimed-docs] “Tools allow ElevenLabs agents to perform actions beyond generating text responses.”
   • [claimed-docs] “allow an assistant to trigger authenticated actions based on the conversation, like scheduling meetings or initiating order returns”
   • [claimed-docs] “Conversation flow settings determine how your assistant handles periods of user silence, interruptions during speech, and turn-taking behavi…”
   • [claimed-docs] “Configure, deploy and monitor natural, human-sounding agents in 70+ languages with leading accuracy and ultra-low latency across voice or ch…”

   proof ↗⚑ flag

   Pipecat✓fullclaimed8/10

   Pipecat Flows functions explicitly support 'processing data by interfacing with external systems and APIs' mid-conversation and auto-derive function schemas from code, while interruption handling and turn detection docs describe how the pipeline manages timing/dead-air around such calls. Missing for 10: no concrete hands-on example/demo showing latency-masking (e.g., filler speech) during a live API call, and no independent benchmark of perceived dead-air during function execution.

   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data ... 2. Progress the conversation by transitioning between nodes”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: Process data by interfacing with external systems and APIs...Progress the conversation by…”
   • [claimed-docs] “Functions in Pipecat Flows serve two key purposes: 1. Process data by interfacing with external systems and APIs... 2. Progress the conversa…”
   • [claimed-docs] “Flows auto-derives the function's metadata — name, description, parameter properties (with their descriptions), and which parameters are req…”
   • [claimed-docs] “When the user starts speaking while the bot is talking, the bot stops immediately, in-flight work is cancelled, and the pipeline is ready fo…”
   • [claimed-docs] “Interruptions (also called barge-in) let the user talk over the bot. When the user starts speaking while the bot is talking, the bot stops i…”
   • [claimed-docs] “Smart Turn Detection uses an advanced machine learning model to determine when a user has finished speaking and your bot should respond.”

   proof ↗⚑ flag

Recording

1. platform-engineerRetrieve full call recordings and transcripts programmatically for every call

   weight 2 · round drawn
   ElevenLabs Agents~partialclaimed5/10

   Docs confirm that conversation transcripts and audio recordings are stored with configurable retention, and that conversations can be searched, analyzed, and have structured data extracted (docs-26/52, docs-21/23/56), implying underlying storage of call recordings/transcripts. However, there is no explicit documentation of a specific API endpoint or method for programmatically retrieving full call recordings and transcripts on a per-call basis. Missing for 10: an explicit 'get conversation' or 'get recording/transcript' API reference, example request/response showing programmatic export of audio + transcript per call.

   • [claimed-docs] “Retention settings allow you to configure how long your conversational agent stores conversation transcripts and audio recordings.”
   • [claimed-docs] “Retention settings allow you to configure how long your conversational agent stores conversation transcripts and audio recordings”
   • [claimed-docs] “Data collection: Extract specific data points from conversations such as contact information, issue details, or any structured information”
   • [claimed-docs] “Searching conversations: Find messages by keyword or meaning across your conversation history.”
   • [claimed-docs] “Extract structured information from conversations such as contact details and business data.”

   proof ↗⚑ flag

   Pipecat~partialclaimed5/10

   Pipecat provides building blocks — AudioBufferProcessor for capturing audio recordings and turn-event based transcript collection — that a platform engineer could wire into their own storage/retrieval system, but there's no documented built-in centralized recordings/transcripts store or query API for retrieving them programmatically across all calls. Missing for 10: a first-party recordings/transcripts storage service or retrieval API, evidence of per-call metadata/indexing, and any hands-on confirmation of retrieving historical call data at scale.

   • [claimed-docs] “Pipecat's \`AudioBufferProcessor\` makes it easy to capture high-quality audio recordings of both the user and bot during interactions.”
   • [claimed-docs] “Pipecat's AudioBufferProcessor makes it easy to capture high-quality audio recordings of both the user and bot during interactions.”
   • [claimed-docs] “Pipecat's `AudioBufferProcessor` makes it easy to capture high-quality audio recordings of both the user and bot during interactions.”
   • [claimed-docs] “Pipecat's turn events make it easy to collect both user and assistant messages as they occur.”

   proof ↗⚑ flag

Transcription

1. developerGet accurate real-time transcription with control over the STT provider, language models, or key terms

   weight 2 · round to Pipecat
   ElevenLabs Agents~partialclaimed4/10

   Docs confirm agents let developers choose or bring custom LLMs (docs-41) and mention real-time voice interaction (docs-37), but there is no evidence of selectable STT providers, transcription accuracy tuning, or 'key terms'/vocabulary boosting controls anywhere in the pack. Missing for 10: STT provider selection, transcription-specific accuracy settings, key-term/glossary support, and any independent verification of transcription accuracy claims.

   • [claimed-docs] “Choose from supported LLMs or bring your own custom model”
   • [claimed-docs] “Configure, deploy and monitor natural, human-sounding agents in 70+ languages with leading accuracy and ultra-low latency across voice or ch…”
   • [claimed-docs] “Conversation flow settings determine how your assistant handles periods of user silence, interruptions during speech, and turn-taking behavi…”

   proof ↗⚑ flag

   Pipecat~partialcommunity6/10

   Docs and community evidence confirm Pipecat lets developers swap STT providers and language models with one line of code (200+ services, local Whisper/Ollama), and it exposes STT-specific latency metrics (TTFS, P99) for tuning accuracy/speed tradeoffs. However, there is no evidence of support for passing custom vocabulary or 'key terms'/boost-word lists to STT providers, and no independent benchmark validating transcription accuracy claims. Missing for 10: explicit key-terms/vocabulary-boost configuration support, and independent/hands-on accuracy validation.

   • [claimed-docs] “Swap speech, language, and vision services from any of 200+ integrated providers and services — usually one line of code.”
   • [claimed-docs] “Pipecat ships with measured P99 latency values for each supported service so that turn detection can account for this delay automatically.”
   • [claimed-docs] “Time To Final Segment (TTFS) measures how long it takes from the moment a user stops speaking until the STT service delivers the final trans…”
   • [community] “Local inference is already supported via Pipecat, you can use ollama or any custom OpenAI endpoint. Local STT is also supported via whisper,…”
   • [community] “Yes, Pipecat already supports that natively, so this can be done easily with ollama... Also, check out any provider they support, and it can…”

   proof ↗⚑ flag

Voices

1. founderClone a custom brand voice and use it for my agents, with a documented consent process

   weight 2 · round drawn
   ElevenLabs Agents—none0/10

   The evidence pack describes selecting from 5k+ pre-built voices and voice customization (speed, multi-voice switching) but contains no mention of a voice-cloning feature for creating a custom brand voice, nor any documented consent process for cloning (e.g., voice verification, spoken consent statement) required by ElevenLabs' broader platform. Missing for 10: any docs on voice cloning/Instant or Professional Voice Cloning for Agents, and any consent-verification workflow.

   • [claimed-docs] “Select from 5k+ voices across 31 languages with customization options”
   • [claimed-docs] “Enable your agent to switch between different voices for multi-character conversations, storytelling, and language tutoring.”
   • [claimed-docs] “You can customize various aspects of your AI agent's voice to create a more natural and engaging conversation experience. This includes cont…”

   proof ↗⚑ flag

   Pipecat—none0/10

   Pipecat's evidence covers swapping among 200+ TTS/STT providers and general voice pipeline features, but nothing addresses custom voice cloning or any documented consent process for cloning a brand voice — this is a fair question since Pipecat is a voice-AI framework, but no evidence supports it being delivered.

   • [claimed-docs] “Swap speech, language, and vision services from any of 200+ integrated providers and services — usually one line of code.”
   • [claimed-docs] “Open source Python framework for building voice and multimodal AI pipelines. Orchestrate 150+ AI services with ultra-low latency.”

   proof ↗⚑ flag

2. developerChoose from a broad voice library or plug in multiple TTS providers to get the voice I want

   weight 2 · round to Pipecat
   ElevenLabs Agents~partialclaimed5/10

   Docs confirm a broad in-house voice library (5k+ voices across 31 languages, docs-6) with customization like speed, pronunciation, and multi-voice switching (docs-17/32/48), satisfying the first half of the story. However, there is no evidence of plugging in third-party/alternative TTS providers — the docs only mention bring-your-own LLM (docs-41), not bring-your-own TTS engine. Missing for 10: any documented support for external/third-party TTS provider integration, evidence of provider-agnostic voice pipeline, independent confirmation of using non-ElevenLabs voices.

   • [claimed-docs] “Select from 5k+ voices across 31 languages with customization options”
   • [claimed-docs] “Enable your agent to switch between different voices for multi-character conversations, storytelling, and language tutoring.”
   • [claimed-docs] “You can customize various aspects of your AI agent's voice to create a more natural and engaging conversation experience. This includes cont…”
   • [claimed-docs] “Enable your agent to switch between different voices for multi-character conversations, storytelling, and language tutoring”
   • [claimed-docs] “Choose from supported LLMs or bring your own custom model”

   proof ↗⚑ flag

   Pipecat✓fullcommunity8/10

   Docs and community evidence confirm Pipecat supports swapping speech services (TTS/STT) across 200+ integrated providers with typically one line of code, plus community confirmation of easy onboarding of any provider and local TTS/STT options (ollama, whisper). Missing for 10: an explicit enumerated 'voice library' list or curated voice catalog UI, and independent benchmark of ease/quality across providers beyond vendor/community anecdotes.

   • [claimed-docs] “Swap speech, language, and vision services from any of 200+ integrated providers and services — usually one line of code.”
   • [claimed-docs] “Open source Python framework for building voice and multimodal AI pipelines. Orchestrate 150+ AI services with ultra-low latency.”
   • [community] “Local inference is already supported via Pipecat, you can use ollama or any custom OpenAI endpoint. Local STT is also supported via whisper,…”
   • [community] “Yes, Pipecat already supports that natively, so this can be done easily with ollama... Also, check out any provider they support, and it can…”
   • [claimed-docs] “Pipecat ships with measured P99 latency values for each supported service so that turn detection can account for this delay automatically.”

   proof ↗⚑ flag