Vapi vs LiveKit Agents
LiveKit Agents wins · 14–18 (29 drawn)
Agent building — building agents — abstractions, tool wiring, control flowAgent building
Building agents — abstractions, tool wiring, control flow
Agent ops
ai-native userMy coding agent can provision a complete voice agent end to end — create the agent, attach a number, and place a call — through the API, CLI, or MCP without touching the dashboard
weight 3 · round to VapiVapi documents API/SDK-based assistant creation, phone number attachment, and call placement (vapi-docs-1, vapi-docs-18, vapi-docs-19), a CLI for managing assistants, phone numbers, and calls entirely from the terminal (vapi-docs-2, vapi-docs-31, vapi-probe-rt-1 confirming keyless install/version), and an official MCP server exposing these same operations to any MCP-compatible agent, verified live and auth-gated in a runtime probe (vapi-docs-33, vapi-probe-rt-2). Together these three surfaces (API, CLI, MCP) cover the full agent-provision-number-call workflow without dashboard use. missing for 10: no single end-to-end hands-on trace showing one agent chaining create→attach→call purely via CLI/MCP/API in one session, and no independent (non-vendor) confirmation of the full workflow succeeding.
- [claimed-docs] “const assistant = await vapi.assistant”
- [claimed-docs] “Manage assistants, phone numbers, and calls from your terminal”
- [claimed-docs] “Create a voice assistant, connect it to a phone number, and make your first calls.”
- [claimed-docs] “In under 5 minutes, you'll create a voice assistant and start talking to it over the phone.”
- [claimed-docs] “Build, test, and deploy voice AI applications without leaving your development environment.”
- [claimed-docs] “The Vapi MCP Server exposes Vapi APIs as tools via the Model Context Protocol (MCP), so you can manage assistants, phone numbers, and calls …”
- [probe] “official MCP server documented at https://docs.vapi.ai/sdk/mcp-server”
- [probe] “official CLI documented at https://docs.vapi.ai/cli”
- [probe] “PROBE runtime (recorded 2026-09-05): the official Vapi CLI installed via the vendor's one-liner (`curl -sSL https://vapi.ai/install.sh | bas…”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
LiveKit provides a real CLI (`lk`), a documented telephony/SIP stack for provisioning inbound/outbound trunks and placing calls, and framework docs explicitly note it's 'built for coding agents like Claude Code, Cursor, and Codex,' suggesting API/CLI-driven workflows are possible without the dashboard. However, the MCP support documented is for the agent's own tool-calling (consuming MCP servers as tools), not an MCP server exposing agent-provisioning/number-attachment/call-placing actions to a coding agent, and no single evidenced workflow shows an agent programmatically creating an agent, attaching a number, and placing a call end-to-end via CLI/API in one flow — the Agent Builder path shown is dashboard/browser-based (no-code), which contradicts the 'without touching dashboard' framing. Missing for 10: an MCP server (or CLI/API recipe) that lets a coding agent itself create an agent, provision/attach a phone number, and place a call in one documented end-to-end sequence.
- [github] “Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.”
- [claimed-docs] “LiveKit telephony lets you build AI-powered voice apps that handle inbound and outbound calls.”
- [claimed-docs] “Outbound trunks are used to place outgoing calls.”
- [probe] “official CLI documented at https://github.com/livekit/livekit-cli”
- [probe] “PROBE runtime (recorded 2026-09-05): `lk --version` printed `lk version 2.18.6` after a plain `brew install livekit-cli` — the official CLI …”
- [claimed-docs] “LiveKit is built for coding agents like Claude Code, Cursor, and Codex.”
- [github] “MCP support: Native support for MCP. Integrate tools provided by MCP servers with one line of code.”
- [claimed-docs] “LiveKit Agents has first-class support for Model Context Protocol (MCP) servers.”
ai-native userThe platform's own AI helps me author agents — generating or improving prompts, flows, and test cases from a description
weight 1 · round drawnVapinone0/10Evidence shows Vapi provides CLI, MCP server, tool/webhook infrastructure, and testing frameworks (Evals, Voice Test Suites), plus a Customer Support template, but nothing indicates the platform itself uses AI to generate or improve prompts, conversation flows, or test cases from a natural-language description — testing tools require manually defined mock conversations/scripts rather than AI-authored ones.
- [claimed-docs] “Voice Test Suites enable you to test your AI voice agents through simulated phone conversations.”
- [claimed-docs] “Evals is Vapi's AI agent testing framework that enables you to systematically test assistants and squads using mock conversations with autom…”
- [claimed-docs] “Our AI tester calls your voice agent and follows a script that simulates real customer behavior.”
- [claimed-docs] “you'll create mock conversations, define expected behaviors, and validate your agents work correctly before production”
- [claimed-docs] “Select the down arrow next to Create Assistant, then choose Customer Support.”
LiveKit Agentsnone0/10Evidence shows a no-code 'Agent Builder' for browser prototyping and a testing framework, but nothing indicates the platform itself uses AI to generate or improve prompts, flows, or test cases from a natural-language description of the desired agent.
- [claimed-docs] “LiveKit Agent Builder to prototype and deploy agents directly in your browser without writing code”
- [claimed-docs] “LiveKit Agent Builder: Prototype and deploy voice agents directly in your browser, without writing any code.”
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes. They integrate with your existing test suite using pytest (Python) or V…”
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes... Agent simulations run end-to-end conversations between your agent and…”
Build
developerBuild a working phone voice agent — prompt, voice, and phone number — and take my first live call within an hour
weight 3 · round to VapiDocs explicitly walk through creating a voice assistant, connecting a phone number, and making a first call in under 5 minutes (vapi-docs-18, vapi-docs-19), with voice selection via the Voice Library (vapi-docs-8, vapi-docs-24) and dashboard-driven assistant creation with templates (vapi-docs-16); community evidence corroborates a working live-call demo (vapi-comm-1). Missing for 10: independent third-party benchmarking of the full 'within an hour' timing claim beyond vendor docs and a single anecdotal community comment.
- [claimed-docs] “Create a voice assistant, connect it to a phone number, and make your first calls.”
- [claimed-docs] “In under 5 minutes, you'll create a voice assistant and start talking to it over the phone.”
- [claimed-docs] “Select the down arrow next to Create Assistant, then choose Customer Support.”
- [claimed-docs] “The Voice Library in the Vapi Dashboard lists every voice available to your organization. Browse and preview voices there”
- [claimed-docs] “Browse and preview voices there, then copy a voice's ID to use on an assistant.”
- [community] “Called the demo number, sounds smooth! Good luck.”
Docs show a <10‑minute voice-assistant quickstart (prompt+voice) plus a dedicated telephony/SIP stack for inbound/outbound calls (trunks, phone number), and runtime probes confirm the framework and self-hosted server actually install and boot without extra keys. Together these cover prompt, voice, and phone number needed for a first live call within an hour. Missing for 10: a single unified, hands-on-verified tutorial that walks through phone-number provisioning and first live call end-to-end (currently voice-quickstart and telephony docs are separate), and independent (non-vendor) confirmation of the 'within an hour' timeline.
- [claimed-docs] “Build and deploy a simple voice assistant in less than 10 minutes.”
- [claimed-docs] “Build and deploy a simple voice assistant with Python or Node.js in less than 10 minutes.”
- [claimed-docs] “LiveKit telephony lets you build AI-powered voice apps that handle inbound and outbound calls.”
- [claimed-docs] “Enable your voice agent to make or take phone calls.”
- [claimed-docs] “Outbound trunks are used to place outgoing calls.”
- [github] “Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.”
- [probe] “PROBE runtime (recorded 2026-09-05): `lk --version` printed `lk version 2.18.6` after a plain `brew install livekit-cli` — the official CLI …”
- [probe] “PROBE runtime (recorded 2026-09-05): `livekit-server --dev` (brew-installed, Apache-2.0 OSS) booted on this machine with NO account or key —…”
- [probe] “PROBE runtime (recorded 2026-09-05): pypi livekit-agents 1.8.0 installs and `import livekit.agents` succeeds with no key (npm @livekit/agent…”
developerRun conversations in multiple languages, including detecting and switching language mid-call
weight 2 · round drawnVapinone0/10The evidence pack covers CLI, MCP server, tools, voices, compliance, and testing, but contains no mention of multi-language support, language detection, or mid-call language switching capabilities. Absence of evidence for this applicable axis (multilingual voice agents are a fair capability to expect from a voice AI platform) means it must be judged 'none'. missing for 10: any mention of multi-language conversation support, language auto-detection, or mid-call language switching features.
LiveKit Agentsnone0/10No evidence in the pack addresses multilingual conversation support, language detection, or dynamic language switching mid-call; the docs cover turn detection, interruption handling, tool use, MCP, and telephony but never mention STT/TTS language selection or switching logic. This is a fair axis for a voice-agent framework (many STT/TTS providers support multi-language), so absence of evidence means 'none' rather than 'na'.
founderDesign multi-step conversation flows in a visual builder with branching, states, and handoffs without writing code
weight 2 · round to LiveKit AgentsVapinone0/10Evidence covers Vapi's assistant configuration, tools, CLI, MCP server, voices, and testing, but there is no mention of a visual flow builder with branching, states, or handoffs — Vapi appears configuration/code-driven (assistant config, function tools, webhooks) rather than a no-code visual conversation designer.
LiveKit mentions a browser-based 'Agent Builder' for no-code prototyping of voice agents, but the evidence never describes visual branching, explicit states, or handoff design — the core framework is fundamentally code-first (Python/Node.js) with tools, MCP, and turn-detection logic. missing for 10: documentation of branching/state UI, handoff modeling, or any screenshots/examples of the Agent Builder's flow-design capabilities beyond a vague no-code prototyping claim.
- [claimed-docs] “Prototype and deploy voice agents directly in your browser, without writing any code.”
- [claimed-docs] “LiveKit Agent Builder to prototype and deploy agents directly in your browser without writing code”
- [claimed-docs] “LiveKit Agent Builder: Prototype and deploy voice agents directly in your browser, without writing any code.”
Personalization
developerInject dynamic variables and per-caller context at call time so each conversation is personalized
weight 2 · round drawnVapinone0/10The evidence pack covers assistant creation, tools/webhooks, CLI, MCP server, voices, and testing, but contains no documentation of variable injection, assistantOverrides, or per-caller context personalization at call time. Missing for 10: docs on dynamic variable substitution (e.g., {{variableName}} templating), call-time overrides/metadata injection, and any example showing per-caller personalization.
LiveKit Agentsnone0/10The evidence pack covers tool use, MCP, telephony, testing, and turn detection, but nothing addresses injecting dynamic variables or per-caller context (e.g., participant/room metadata, job context) into an agent's prompt or session at call start. Missing for 10: any documentation of job/participant metadata APIs, per-call context injection, or dynamic prompt personalization.
developerGround the agent on my documents with a built-in knowledge base or RAG so it answers from my content
weight 2 · round to LiveKit AgentsVapinone0/10Vapi is a voice-agent platform focused on assistants, phone numbers, custom webhook tools, and telephony/testing; the evidence pack shows no built-in knowledge base or RAG feature for grounding assistants on uploaded documents. Custom tools/webhooks could be used to build a workaround, but no document ingestion or retrieval capability is documented.
- [claimed-docs] “Create your own webhook-based tools to extend assistant capabilities”
- [claimed-docs] “This guide shows you how to create custom tools, including Function Tools, for your Vapi assistants.”
- [claimed-docs] “Custom tools that you create... interact with your systems via webhooks”
LiveKit Agents supports calling external APIs/tools that a developer can use to implement RAG ("Call external APIs or lookup data for RAG"), and has full LLM tool-use and MCP integration for wiring in retrieval systems, but there is no built-in knowledge base, document ingestion, or vector-store/RAG pipeline shipped by the framework itself — developers must bring their own RAG implementation via the tools API. Missing for 10: built-in document indexing/vector store, out-of-the-box knowledge-base feature, first-party RAG pipeline or example showing document grounding end-to-end.
- [claimed-docs] “Call external APIs or lookup data for RAG.”
- [claimed-docs] “LiveKit Agents has full support for LLM tool use. This feature allows you to create a custom library of tools to extend your agent's context”
- [claimed-docs] “LiveKit Agents has full support for LLM tool use.”
- [github] “MCP support: Native support for MCP. Integrate tools provided by MCP servers with one line of code.”
- [claimed-docs] “Wrap an MCP server in an \`MCPToolset\` and pass it to the agent's \`tools\` parameter”
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round drawnVapi confirms an llms.txt file exists and is served at docs.vapi.ai/llms.txt (HTTP 200), explicitly instructing agents on how to fetch clean Markdown per page, which directly satisfies the story. missing for 10: no independent/community corroboration of agents actually consuming llms.txt in practice, only first-party probe evidence.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.vapi.ai/llms.txt # Vapi ## Instructions for AI Agents - For clean Markdown of any page, append `.…”
LiveKit publishes a working llms.txt (verified live at docs.livekit.io/llms.txt, HTTP 200) and agent-friendly markdown doc endpoints (e.g. /agents/.md), plus explicit docs noting the platform is 'built for coding agents like Claude Code, Cursor, and Codex,' confirming intentional support for AI-native doc consumption. missing for 10: no independent third-party confirmation that external agents successfully consume these endpoints in practice, and no dedicated documentation page explaining the llms.txt/agent-doc strategy itself.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.livekit.io/llms.txt # LiveKit docs > LiveKit is an open-source platform for building voice, video,…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.livekit.io/agents/.md LiveKit docs › Build Agents › Get Started › Introduction --- # Introduction …”
- [claimed-docs] “LiveKit is built for coding agents like Claude Code, Cursor, and Codex.”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to LiveKit AgentsVapi ships an official CLI (vapi-docs-2/31/32, confirmed working keylessly in vapi-probe-rt-1) that can manage assistants, calls, and forward webhooks, which supports scripted/CI-style usage, and SDKs/REST APIs imply headless programmatic calls. However there is no explicit CI/automation documentation, no examples of running in a pipeline, and the MCP server requires bearer-key auth (vapi-probe-rt-2) which is unaddressed for CI contexts. missing for 10: explicit CI/automation examples or docs, non-interactive auth/service-account flow for CI, evidence of headless voice-testing/evals running in a pipeline.
- [claimed-docs] “Manage assistants, phone numbers, and calls from your terminal”
- [claimed-docs] “Build, test, and deploy voice AI applications without leaving your development environment.”
- [claimed-docs] “The CLI auto-detects your tech stack and sets up everything you need.”
- [probe] “PROBE runtime (recorded 2026-09-05): the official Vapi CLI installed via the vendor's one-liner (`curl -sSL https://vapi.ai/install.sh | bas…”
- [claimed-docs] “Evals is Vapi's AI agent testing framework that enables you to systematically test assistants and squads using mock conversations with autom…”
- [claimed-docs] “you'll create mock conversations, define expected behaviors, and validate your agents work correctly before production”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
LiveKit Agents is a Python/Node.js SDK plus self-hostable server that runs as a headless programmatic participant with no UI dependency; it's pip/npm-installable, has a built-in test framework (pytest/Vitest, agent simulations) suitable for CI, and the server/CLI were verified runtime to work keylessly. missing for 10: no explicit documented CI pipeline example (e.g., GitHub Actions config) or headless-mode confirmation beyond inference from server/testing docs.
- [claimed-docs] “A programmatic participant is any code that joins a LiveKit room as a participant — this includes AI agents, media processors, or custom log…”
- [claimed-docs] “The Agents framework isn't limited to AI agents. You can use it to deploy any code that needs to process realtime media and data streams as …”
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes. They integrate with your existing test suite using pytest (Python) or V…”
- [claimed-docs] “Agent simulations run end-to-end conversations between your agent and an LLM-driven user, then evaluate the results across the full interact…”
- [github] “Builtin test framework: Write tests and use judges to ensure your agent is performing as expected.”
- [probe] “PROBE runtime (recorded 2026-09-05): `livekit-server --dev` (brew-installed, Apache-2.0 OSS) booted on this machine with NO account or key —…”
- [probe] “PROBE runtime (recorded 2026-09-05): pypi livekit-agents 1.8.0 installs and `import livekit.agents` succeeds with no key (npm @livekit/agent…”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round to LiveKit AgentsVapinone0/10The evidence shows Vapi exposes its own APIs as an MCP server (so external MCP clients like Claude Desktop can control Vapi), but this is the opposite direction from the story — there's no evidence that Vapi assistants can consume/plug-in external MCP servers to use their tools within a conversation. Vapi's tool docs describe webhook-based custom tools and OpenAI-style function calling only, with no mention of MCP-server ingestion.
- [claimed-docs] “so you can manage assistants, phone numbers, and calls from any MCP-compatible AI assistant (like Claude Desktop) or agent framework”
- [claimed-docs] “The Vapi MCP Server exposes Vapi APIs as tools via the Model Context Protocol (MCP), so you can manage assistants, phone numbers, and calls …”
- [claimed-docs] “Create your own webhook-based tools to extend assistant capabilities”
- [claimed-docs] “Custom tools that you create... interact with your systems via webhooks”
- [probe] “official MCP server documented at https://docs.vapi.ai/sdk/mcp-server”
Docs and GitHub explicitly state 'first-class support for Model Context Protocol (MCP) servers' with a documented pattern to wrap an MCP server in an MCPToolset and pass it to the agent's tools parameter, described as one line of code integration. This is first-party documentation without independent hands-on verification of MCP-specific usage. Missing for 10: independent/community confirmation of MCP tool integration working end-to-end, and more detail on multi-server or auth configurations.
- [claimed-docs] “Wrap an MCP server in an \`MCPToolset\` and pass it to the agent's \`tools\` parameter”
- [claimed-docs] “Wrap an MCP server in an `MCPToolset` and pass it to the agent's `tools` parameter”
- [claimed-docs] “LiveKit Agents has first-class support for Model Context Protocol (MCP) servers.”
- [github] “MCP support: Native support for MCP. Integrate tools provided by MCP servers with one line of code.”
- [github] “Native support for MCP. Integrate tools provided by MCP servers with one line of code.”
ai-native userConnect an agent via an official MCP server
weight 3 · round to VapiVapi ships an official hosted MCP server (mcp.vapi.ai) exposing its assistant/phone/call APIs as MCP tools for any MCP-compatible agent (Claude Desktop, agent frameworks), documented in first-party docs and confirmed live via a runtime probe (401 bearer-key gate exactly as documented). Missing for 10: independent/community hands-on testimonials specifically about using the MCP server (community evidence only covers general demo/founder trivia, not MCP usage).
- [claimed-docs] “so you can manage assistants, phone numbers, and calls from any MCP-compatible AI assistant (like Claude Desktop) or agent framework”
- [claimed-docs] “The Vapi MCP Server exposes Vapi APIs as tools via the Model Context Protocol (MCP), so you can manage assistants, phone numbers, and calls …”
- [probe] “official MCP server documented at https://docs.vapi.ai/sdk/mcp-server”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
ai-native userUse an official CLI
weight 2 · round drawnVapi ships a documented official CLI for managing assistants, phone numbers, and calls from the terminal, with webhook forwarding, org/environment switching, and tech-stack auto-detection, and this was independently verified in a runtime probe (successful install and `--version` output). missing for 10: no independent third-party review or community discussion of the CLI's day-to-day usage beyond the vendor docs and single install probe.
- [claimed-docs] “Manage assistants, phone numbers, and calls from your terminal”
- [claimed-docs] “vapi listen --forward-to localhost:3000/webhook”
- [claimed-docs] “Switch between organizations and environments seamlessly”
- [claimed-docs] “Forward webhooks to your local server for debugging”
- [claimed-docs] “Build, test, and deploy voice AI applications without leaving your development environment.”
- [claimed-docs] “The CLI auto-detects your tech stack and sets up everything you need.”
- [probe] “official CLI documented at https://docs.vapi.ai/cli”
- [probe] “PROBE runtime (recorded 2026-09-05): the official Vapi CLI installed via the vendor's one-liner (`curl -sSL https://vapi.ai/install.sh | bas…”
LiveKit ships an official CLI (livekit-cli), documented on GitHub and verified hands-on to install and run (`lk --version` works via brew install), confirming it's a real, functional official CLI supporting the agent workflow. missing for 10: no deep documentation of full CLI command surface for agent-specific workflows within the evidence pack.
ai-native userDrive the product through a documented public API
weight 3 · round to VapiVapi ships a documented public API/SDK (assistant management, calls, tools), a CLI for terminal-driven workflows, and a hosted MCP server exposing the API as tools, all confirmed by runtime probes (CLI installs and runs, MCP endpoint live and auth-gated as documented). This directly satisfies programmatic/AI-native control via a documented public interface. Missing for 10: no independent third-party developer report deeply exercising the API beyond docs/probes.
- [claimed-docs] “const assistant = await vapi.assistant”
- [claimed-docs] “Manage assistants, phone numbers, and calls from your terminal”
- [claimed-docs] “so you can manage assistants, phone numbers, and calls from any MCP-compatible AI assistant (like Claude Desktop) or agent framework”
- [claimed-docs] “The Vapi MCP Server exposes Vapi APIs as tools via the Model Context Protocol (MCP), so you can manage assistants, phone numbers, and calls …”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.vapi.ai/llms.txt # Vapi ## Instructions for AI Agents - For clean Markdown of any page, append `.…”
- [probe] “official MCP server documented at https://docs.vapi.ai/sdk/mcp-server”
- [probe] “official CLI documented at https://docs.vapi.ai/cli”
- [probe] “PROBE runtime (recorded 2026-09-05): the official Vapi CLI installed via the vendor's one-liner (`curl -sSL https://vapi.ai/install.sh | bas…”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
LiveKit Agents ships a well-documented, pip/npm-installable Python and Node.js SDK with extensive public API surface (tools, MCP, turn detection, telephony, testing), plus a llms.txt and hands-on verified installs/runtime confirming the API is real and usable by AI-native developers. Missing for 10: independent third-party API reference/versioning audit and deeper evidence of API stability guarantees beyond docs and probes.
- [claimed-docs] “The Agents framework lets you add any Python or Node.js program to LiveKit rooms as full realtime participants.”
- [claimed-docs] “A programmatic participant is any code that joins a LiveKit room as a participant — this includes AI agents, media processors, or custom log…”
- [claimed-docs] “The Agents framework isn't limited to AI agents. You can use it to deploy any code that needs to process realtime media and data streams as …”
- [github] “MCP support: Native support for MCP. Integrate tools provided by MCP servers with one line of code.”
- [claimed-docs] “LiveKit Agents has full support for LLM tool use. This feature allows you to create a custom library of tools to extend your agent's context”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.livekit.io/llms.txt # LiveKit docs > LiveKit is an open-source platform for building voice, video,…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.livekit.io/agents/.md LiveKit docs › Build Agents › Get Started › Introduction --- # Introduction …”
- [probe] “PROBE runtime (recorded 2026-09-05): pypi livekit-agents 1.8.0 installs and `import livekit.agents` succeeds with no key (npm @livekit/agent…”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to VapiVapi's enterprise docs mention Role Based Access Control (RBAC) and SSO, implying some access-scoping capability at the org level, and its MCP server/API require bearer API keys — but there is no documented mechanism for issuing per-agent, least-privilege scoped API keys or tokens tailored to a specific agent's permissions. missing for 10: explicit scoped API key creation/management UI or API, documentation of key-level permission granularity, and independent confirmation that RBAC restricts agent credentials rather than just human dashboard users.
- [claimed-docs] “Single Sign On (SSO) supported for Okta, Azure AD, SAML, and OIDC * Role Based Access Control (RBAC)”
- [claimed-docs] “Single Sign On (SSO) supported for Okta, Azure AD, SAML, and OIDC”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
LiveKit Agentsnone0/10The evidence pack covers voice/agent features, tool-calling, MCP, testing, and telephony, but contains no mention of scoped or least-privilege API key/token issuance for agents (e.g., LiveKit's grant-based access tokens or credential scoping). Since LiveKit is a platform with API keys and would plausibly support such scoping, this is an applicable but undocumented axis in the given evidence.
ai-native userBuild against official SDKs
weight 2 · round to LiveKit AgentsVapi documents an official JS/TS SDK usage pattern, official CLI (probe-confirmed installed and runnable), and official MCP server (probe-confirmed live and auth-gated), all clearly aimed at AI-native/agentic developer workflows including IDE assistant integration (Cursor/Windsurf/VSCode). missing for 10: independent third-party corroboration of SDK code quality and coverage across multiple languages beyond the docs snippets.
- [claimed-docs] “const assistant = await vapi.assistant”
- [claimed-docs] “Manage assistants, phone numbers, and calls from your terminal”
- [claimed-docs] “so you can manage assistants, phone numbers, and calls from any MCP-compatible AI assistant (like Claude Desktop) or agent framework”
- [claimed-docs] “The Vapi MCP Server exposes Vapi APIs as tools via the Model Context Protocol (MCP), so you can manage assistants, phone numbers, and calls …”
- [claimed-docs] “Your IDE's AI assistant (Cursor, Windsurf, VSCode) gains complete, accurate knowledge of Vapi's APIs and best practices. No more hallucinate…”
- [probe] “official MCP server documented at https://docs.vapi.ai/sdk/mcp-server”
- [probe] “official CLI documented at https://docs.vapi.ai/cli”
- [probe] “PROBE runtime (recorded 2026-09-05): the official Vapi CLI installed via the vendor's one-liner (`curl -sSL https://vapi.ai/install.sh | bas…”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
LiveKit provides official Python and Node.js SDKs (livekit-agents), verified hands-on to pip-install and import with no key, plus documented plugin ecosystem, tool use, MCP support, testing frameworks, and telephony integration—clearly an official SDK ecosystem for AI-native building. Missing for 10: independent third-party benchmarking of SDK API stability/versioning beyond community latency complaints unrelated to SDK build story.
- [claimed-docs] “The Agents framework lets you add any Python or Node.js program to LiveKit rooms as full realtime participants.”
- [claimed-docs] “LiveKit Agents includes a large ecosystem of open source plugins for a variety of AI providers.”
- [claimed-docs] “LiveKit Agents has full support for LLM tool use. This feature allows you to create a custom library of tools to extend your agent's context”
- [github] “MCP support: Native support for MCP. Integrate tools provided by MCP servers with one line of code.”
- [probe] “PROBE runtime (recorded 2026-09-05): pypi livekit-agents 1.8.0 installs and `import livekit.agents` succeeds with no key (npm @livekit/agent…”
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes. They integrate with your existing test suite using pytest (Python) or V…”
ai-native userSubscribe to events via webhooks
weight 2 · round to VapiVapi supports webhook-based events via Server URL (tool-calls messages), custom webhook tools, and CLI webhook forwarding for local debugging, giving AI-native users a documented event subscription mechanism. missing for 10: independent/hands-on confirmation of webhook delivery reliability, a full event-type catalog/schema, and signature/verification documentation.
- [claimed-docs] “Create your own webhook-based tools to extend assistant capabilities”
- [claimed-docs] “When tools are triggered, your Server URL receives a `tool-calls` message”
- [claimed-docs] “Vapi supports OpenAI-style tool/function calling. Assistants can ping your server to perform actions.”
- [claimed-docs] “Forward webhooks to your local server for debugging”
- [claimed-docs] “Custom tools that you create... interact with your systems via webhooks”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round drawnVapinone0/10Evidence covers building/testing assistants (Evals, Voice Test Suites), CLI/MCP management tools, and tool-calling, but nothing describes the product itself surfacing AI-generated insights or suggestions derived from a user's own call/usage data (e.g., analytics dashboards with AI-generated recommendations).
ai-native userSet up automations that run autonomously in the background
weight 2 · round to LiveKit AgentsVapi assistants operate autonomously once a call starts—handling conversation flow, invoking custom tools via webhooks, and triggering server-side automations without further human input (vapi-docs-6, vapi-docs-17, vapi-docs-23, vapi-docs-34). However, this autonomy is scoped to an active call session; there is no evidence of scheduled/cron-style background jobs or agent loops that run independently of a triggered call or user interaction. Missing for 10: evidence of scheduled/background triggers outside live calls, persistent autonomous task queues, or proactive (non-call-triggered) automation runs.
- [claimed-docs] “Create your own webhook-based tools to extend assistant capabilities”
- [claimed-docs] “When tools are triggered, your Server URL receives a `tool-calls` message”
- [claimed-docs] “Vapi supports OpenAI-style tool/function calling. Assistants can ping your server to perform actions.”
- [claimed-docs] “Custom tools that you create... interact with your systems via webhooks”
- [claimed-docs] “This guide shows you how to create custom tools, including Function Tools, for your Vapi assistants.”
LiveKit Agents workers run as persistent 'programmatic participants' deployed to LiveKit Cloud or self-hosted infrastructure, processing realtime streams and telephony calls autonomously in the background without human intervention (docs-8, docs-20, docs-30, docs-26, gh-2), and this is verified hands-on as a real pip-installable OSS framework with a working self-hosted backbone (probe-rt-2, probe-rt-3). Missing for 10: explicit scheduling/cron-trigger mechanisms, independent third-party reports of long-running unattended stability (the one community mention is about observability tooling gaps, not autonomy failure).
- [claimed-docs] “A programmatic participant is any code that joins a LiveKit room as a participant — this includes AI agents, media processors, or custom log…”
- [claimed-docs] “The Agents framework isn't limited to AI agents. You can use it to deploy any code that needs to process realtime media and data streams as …”
- [claimed-docs] “Run your agent on LiveKit Cloud's global infrastructure.”
- [claimed-docs] “LiveKit telephony lets you build AI-powered voice apps that handle inbound and outbound calls.”
- [github] “Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.”
- [probe] “PROBE runtime (recorded 2026-09-05): `livekit-server --dev` (brew-installed, Apache-2.0 OSS) booted on this machine with NO account or key —…”
- [probe] “PROBE runtime (recorded 2026-09-05): pypi livekit-agents 1.8.0 installs and `import livekit.agents` succeeds with no key (npm @livekit/agent…”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round drawnVapinone0/10Vapi's docs describe building and configuring voice AI assistants for end-users (phone/web calls, tools, testing) and even an MCP server/CLI for managing those assistants, but there is no evidence of a built-in AI assistant/copilot inside the Vapi product itself that the AI-native user can delegate platform tasks to (e.g., an in-dashboard copilot that configures assistants or writes tools for you). The CLI mentions external IDE assistants (Cursor/Windsurf) gaining API knowledge, but that's a third-party tool, not a built-in in-product assistant.
- [claimed-docs] “Your IDE's AI assistant (Cursor, Windsurf, VSCode) gains complete, accurate knowledge of Vapi's APIs and best practices. No more hallucinate…”
- [claimed-docs] “Manage assistants, phone numbers, and calls from your terminal”
- [claimed-docs] “so you can manage assistants, phone numbers, and calls from any MCP-compatible AI assistant (like Claude Desktop) or agent framework”
ai-native userOperate the product with natural-language commands
weight 2 · round to VapiVapi exposes an official MCP server that lets any MCP-compatible AI assistant (e.g. Claude Desktop) manage assistants, phone numbers, and calls via natural-language tool calls, and this endpoint is confirmed live and auth-gated in a runtime probe. It also ships a CLI and llms.txt docs optimized for AI agents to operate it. Missing for 10: no first-party evidence of a built-in chat/NL command console inside the Vapi product itself (beyond MCP/CLI proxies), and no independent hands-on report of an agent successfully performing multi-step tasks via MCP.
- [claimed-docs] “so you can manage assistants, phone numbers, and calls from any MCP-compatible AI assistant (like Claude Desktop) or agent framework”
- [claimed-docs] “The Vapi MCP Server exposes Vapi APIs as tools via the Model Context Protocol (MCP), so you can manage assistants, phone numbers, and calls …”
- [claimed-docs] “Your IDE's AI assistant (Cursor, Windsurf, VSCode) gains complete, accurate knowledge of Vapi's APIs and best practices. No more hallucinate…”
- [probe] “official MCP server documented at https://docs.vapi.ai/sdk/mcp-server”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
- [probe] “PROBE runtime (recorded 2026-09-05): the official Vapi CLI installed via the vendor's one-liner (`curl -sSL https://vapi.ai/install.sh | bas…”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.vapi.ai/llms.txt # Vapi ## Instructions for AI Agents - For clean Markdown of any page, append `.…”
LiveKit ships an 'Agent Builder' that lets users prototype/deploy voice agents in-browser without code, and docs explicitly note the framework is 'built for coding agents like Claude Code, Cursor, and Codex' plus agent-readable llms.txt docs — suggesting some AI-native/no-code operability. However there is no evidence of a native natural-language command interface for operating LiveKit itself (e.g., NL-driven CLI, chatops control plane); the MCP support described is for agents calling external tools, not for controlling LiveKit via NL. Missing for 10: a documented NL-driven interface/CLI for configuring or operating the platform itself, and independent confirmation of the Agent Builder's no-code UX actually being NL-based rather than GUI-based.
- [claimed-docs] “LiveKit is built for coding agents like Claude Code, Cursor, and Codex.”
- [claimed-docs] “Prototype and deploy voice agents directly in your browser, without writing any code.”
- [claimed-docs] “LiveKit Agent Builder to prototype and deploy agents directly in your browser without writing code”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.livekit.io/llms.txt # LiveKit docs > LiveKit is an open-source platform for building voice, video,…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.livekit.io/agents/.md LiveKit docs › Build Agents › Get Started › Introduction --- # Introduction …”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnVapinone0/10The evidence pack shows static markdown documentation, code snippets, a CLI, and an MCP server, but no interactive API reference or runnable/try-it-out examples are described anywhere in the docs pages cited.
LiveKit Agentsnone0/10Evidence shows extensive quickstart guides, markdown-based docs (llms.txt), and a no-code 'Agent Builder' prototyping tool, but nothing describes an interactive API reference (e.g., embedded code sandbox, live runnable examples, or Swagger-like explorer) for exploring the SDK/API itself.
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round drawnVapinone0/10The evidence pack covers Vapi's CLI, MCP server, docs, and SDKs, but no citation mentions an OpenAPI spec, API reference schema, or any machine-readable spec file available for download. This is a fair axis for an API-first product, but no supporting evidence exists in the pack.
LiveKit Agentsnone0/10The evidence pack documents LiveKit's SDKs, CLI, docs (llms.txt), and framework capabilities but contains no mention of a downloadable OpenAPI/Swagger spec or other machine-readable API description for LiveKit's server or Agents APIs. Since LiveKit exposes a real API surface (server API, cloud API), this axis applies but is unevidenced.
- [probe] “official CLI documented at https://github.com/livekit/livekit-cli”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.livekit.io/llms.txt # LiveKit docs > LiveKit is an open-source platform for building voice, video,…”
- [probe] “PROBE docs-md: HTTP 200 at https://docs.livekit.io/agents/.md LiveKit docs › Build Agents › Get Started › Introduction --- # Introduction …”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to VapiVapi offers dedicated Voice Test Suites (AI tester simulating conversations) and an Evals framework explicitly for creating mock conversations to 'validate your agents work correctly before production,' plus CLI webhook forwarding to localhost for local development/debugging — all separate from live production calls. Missing for 10: explicit vendor use of the term 'sandbox environment' and independent/hands-on verification that test runs are fully isolated from production data.
- [claimed-docs] “Voice Test Suites enable you to test your AI voice agents through simulated phone conversations.”
- [claimed-docs] “Evals is Vapi's AI agent testing framework that enables you to systematically test assistants and squads using mock conversations with autom…”
- [claimed-docs] “Our AI tester calls your voice agent and follows a script that simulates real customer behavior.”
- [claimed-docs] “you'll create mock conversations, define expected behaviors, and validate your agents work correctly before production”
- [claimed-docs] “Forward webhooks to your local server for debugging”
- [claimed-docs] “Build, test, and deploy voice AI applications without leaving your development environment.”
LiveKit provides a self-hostable dev server (`livekit-server --dev`) that boots locally with placeholder keys, separate from any production deployment, plus a built-in testing framework with behavioral tests and LLM-driven agent simulations that evaluate agent behavior without needing real production data. However, there is no explicitly branded 'sandbox environment' or staging/production data-isolation feature documented — the sandbox-like capability is inferred from dev-mode self-hosting and test simulations rather than a dedicated sandbox product feature. Missing for 10: an explicit sandbox/staging environment offering with documented separation from production data, and independent confirmation that test simulations never touch production data stores.
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes. They integrate with your existing test suite using pytest (Python) or V…”
- [claimed-docs] “Agent simulations run end-to-end conversations between your agent and an LLM-driven user, then evaluate the results across the full interact…”
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes... Agent simulations run end-to-end conversations between your agent and…”
- [github] “Builtin test framework: Write tests and use judges to ensure your agent is performing as expected.”
- [probe] “PROBE runtime (recorded 2026-09-05): `livekit-server --dev` (brew-installed, Apache-2.0 OSS) booted on this machine with NO account or key —…”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnVapinone0/10No evidence pack item mentions API versioning scheme, version numbers in endpoints, or any documented deprecation/sunset policy for Vapi's APIs; the docs cover features (CLI, MCP, tools, voices) but not API lifecycle governance.
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round drawnVapinone0/10The evidence pack documents CLI/SDK/MCP management of individual assistants, phone numbers, and calls, but nowhere shows bulk/batch endpoints or commands (e.g., batch-create, bulk-update, mass-delete) for operating across many items at once.
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round drawnVapi supports event-driven server-side tools/webhooks that fire on call events (tool-calls messages, server-url events) which act as a rule-trigger mechanism, but this is scoped to voice-call events only rather than a general-purpose automation/rules engine for arbitrary triggers. missing for 10: a documented general condition/trigger-action rules engine spanning non-call events, cross-system automation, or independent hands-on evidence of complex conditional automation chains.
- [claimed-docs] “When tools are triggered, your Server URL receives a `tool-calls` message”
- [claimed-docs] “Vapi supports OpenAI-style tool/function calling. Assistants can ping your server to perform actions.”
- [claimed-docs] “Custom tools that you create... interact with your systems via webhooks”
- [claimed-docs] “Create your own webhook-based tools to extend assistant capabilities”
LiveKit Agents supports event-driven automatic behaviors — tool-calling on LLM decisions, turn-detector triggering response timing, and adaptive interruption handling triggering barge-in logic — all of which are 'events auto-trigger actions' patterns built into the agent runtime. However, the evidence shows these as built-in framework behaviors and developer-coded event handlers rather than a user-facing declarative rule engine (e.g., no evidence of a 'when X happens do Y' config surface, webhook/rule subscription API, or no-code rule builder). Missing for 10: explicit rule-definition/webhook-trigger API exposed to non-developer users, documentation of a generalized event-subscription system beyond tool calls and turn logic, and independent confirmation of custom rule automation in production use.
- [claimed-docs] “Call external APIs or lookup data for RAG.”
- [claimed-docs] “LiveKit Agents has full support for LLM tool use. This feature allows you to create a custom library of tools to extend your agent's context”
- [claimed-docs] “LiveKit Agents has full support for LLM tool use.”
- [claimed-docs] “LiveKit's `TurnDetector` is an audio model that encodes user audio directly, capturing both _what_ is said and _how_ it's said.”
- [claimed-docs] “A turn detector model can predict that they have more to say and wait for them to finish before responding.”
- [claimed-docs] “Adaptive interruption handling allows an agent to respond naturally when users speak mid-response... to identify intentional interruptions (…”
- [claimed-docs] “the model analyzes the acoustic signals to identify intentional interruptions (barge-ins) from conversational backchanneling”
- [claimed-docs] “Adaptive interruption handling allows an agent to respond naturally when users speak mid-response.”
ai-native userSchedule recurring jobs or workflows
weight 2 · round drawnVapinone0/10No evidence in the pack shows Vapi supports scheduling recurring jobs, workflows, or automated recurring calls/tasks — the docs cover assistants, tools, CLI, MCP server, testing, and voice customization but nothing about cron-like scheduling or recurring automation triggers.
LiveKit Agentsnone0/10The evidence pack covers LiveKit Agents' realtime voice/video agent capabilities, tool use, MCP integration, telephony, and testing, but contains no mention of any scheduling, cron-like recurring job, or workflow-automation trigger mechanism. This is a fair capability to ask about for an automation-focused agent framework, but no evidence supports it.
ai-native userVersion, review, and roll back my automations
weight 1 · round to LiveKit AgentsVapinone0/10Vapi's docs cover assistant creation, tools, CLI, MCP server, and testing/evals, but there is no mention of versioning assistant configurations, review/approval workflows, or rollback to prior versions of automations. No evidence pack item addresses version history or rollback capability.
The pricing page explicitly claims 'Instant rollback to a previous agent deployment,' and the testing/simulation framework (behavioral tests, LLM-judge simulations) supports a review step before deployment, but there's no documented versioning system, changelog, or diff/history UI for automations. Missing for 10: explicit version history/diffing of agent configs, a documented review/approval workflow beyond test suites, and independent confirmation that rollback works in practice.
- [claimed-docs] “Instant rollback to a previous agent deployment”
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes. They integrate with your existing test suite using pytest (Python) or V…”
- [claimed-docs] “Agent simulations run end-to-end conversations between your agent and an LLM-driven user, then evaluate the results across the full interact…”
- [github] “Builtin test framework: Write tests and use judges to ensure your agent is performing as expected.”
Compliance trust — stories about compliance trust in this arenaCompliance trust
Stories about compliance trust in this arena
Compliance
founderMeet call-recording consent and disclosure obligations with per-call recording controls and configurable data retention
weight 2 · round drawnVapinone0/10The evidence pack covers HIPAA toggle, SSO/RBAC, testing/evals, and CLI/MCP tooling, but contains no mention of call-recording enable/disable controls, consent disclosure features, or configurable data retention policies for call recordings — a fair and applicable axis for a voice-calling platform.
LiveKit Agentsnone0/10No evidence in the pack mentions call recording, consent disclosure, or data retention controls anywhere in LiveKit Agents docs, GitHub, or community sources; the pack covers telephony, tool use, MCP, testing, and turn detection but nothing about recording/retention compliance features. missing for 10: recording consent/disclosure mechanisms, per-call recording toggles, configurable data retention policies, and any documentation or hands-on evidence of these compliance controls.
platform-engineerRun regulated workloads with HIPAA/BAA support, SOC 2, and data-residency options
weight 2 · round to VapiVapi documents HIPAA compliance toggle in the dashboard and enterprise SSO/RBAC, indicating some regulated-workload support, but evidence lacks explicit SOC 2 certification details, BAA execution process, or data-residency region options. missing for 10: SOC 2 report/certification evidence, explicit BAA process details, data-residency/region selection options, independent audit confirmation.
- [claimed-docs] “HIPAA compliance can be ensured by turning on HIPAA compliance in the Dashboard.”
- [claimed-docs] “Single Sign On (SSO) supported for Okta, Azure AD, SAML, and OIDC * Role Based Access Control (RBAC)”
- [claimed-docs] “Single Sign On (SSO) supported for Okta, Azure AD, SAML, and OIDC”
LiveKit Agentsnone0/10The evidence pack contains no mention of HIPAA/BAA agreements, SOC 2 certification, or data-residency controls anywhere in LiveKit's docs, GitHub repo, or probes; self-hosting and TURN/security items only cover TLS/SSL and self-hosted deployment, not compliance attestations. As a platform serving enterprise/regulated workloads, this axis clearly applies, so lack of evidence yields 'none'.
- [claimed-docs] “The good news is LiveKit includes an embedded TURN server. It's a secure TURN implementation that has integrated authentication with the res…”
- [claimed-docs] “In order to have a secure LiveKit deployment, you will need a domain as well as a SSL certificate for that domain.”
Deployment scale — stories about deployment scale in this arenaDeployment scale
Stories about deployment scale in this arena
Scale
platform-engineerSee documented concurrency limits and scale to many simultaneous calls without manual capacity begging
weight 2 · round to LiveKit AgentsVapinone0/10No evidence pack items document concurrency limits, per-org call caps, autoscaling behavior, or capacity-request processes; the docs cover CLI, MCP server, tools, voices, and enterprise SSO but nothing about simultaneous-call scaling limits or how to raise them.
Docs mention running agents on LiveKit Cloud's global infrastructure and self-hosting with an embedded TURN server, implying scale-out capability, but no evidence pack item cites concrete documented concurrency limits, per-instance call caps, or autoscaling guarantees; a community report even flags lack of visibility into per-call cost/latency at scale. missing for 10: published concurrency/capacity numbers, autoscaling documentation, load-testing benchmarks, and confirmation that scaling requires no manual quota requests.
- [claimed-docs] “Run your agent on LiveKit Cloud's global infrastructure.”
- [claimed-docs] “The good news is LiveKit includes an embedded TURN server. It's a secure TURN implementation that has integrated authentication with the res…”
- [community] “We've been running multiple voice AI agents on LiveKit and kept running into visibility issues — no way to measure TTFT, latency across STT …”
Self host
platform-engineerSelf-host the voice agent runtime from open-source code on my own infrastructure
weight 3 · round to LiveKit AgentsVapinone0/10Vapi is presented entirely as a hosted SaaS platform with a CLI, MCP server, and dashboard for managing cloud-hosted assistants; there is no evidence of open-source runtime code or any self-hosting option. One community note even references that a different product (pipecat by daily.co) is the open-source alternative, implicitly contrasting with Vapi's closed hosted model.
- [community] “Vapi is also built on media framework by daily.co. They have an open source version of voice ai called pipecat.”
- [claimed-docs] “Build, test, and deploy voice AI applications without leaving your development environment.”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
LiveKit Agents is open source, pip/npm installable and runs the agent runtime independent of any vendor account, and the underlying WebRTC/SIP server (livekit-server) is Apache-2.0 and self-hostable, confirmed via hands-on probes (booting `livekit-server --dev` locally with no keys, and installing livekit-agents/CLI keylessly) plus official self-hosting deployment docs covering TURN, TLS/domain setup. Missing for 10: no first-party production-scale self-hosting case study or independent report of large-scale self-hosted deployment beyond dev-mode probe.
- [probe] “PROBE runtime (recorded 2026-09-05): `lk --version` printed `lk version 2.18.6` after a plain `brew install livekit-cli` — the official CLI …”
- [probe] “PROBE runtime (recorded 2026-09-05): `livekit-server --dev` (brew-installed, Apache-2.0 OSS) booted on this machine with NO account or key —…”
- [probe] “PROBE runtime (recorded 2026-09-05): pypi livekit-agents 1.8.0 installs and `import livekit.agents` succeeds with no key (npm @livekit/agent…”
- [claimed-docs] “The good news is LiveKit includes an embedded TURN server. It's a secure TURN implementation that has integrated authentication with the res…”
- [claimed-docs] “In order to have a secure LiveKit deployment, you will need a domain as well as a SSL certificate for that domain.”
- [claimed-docs] “A programmatic participant is any code that joins a LiveKit room as a participant — this includes AI agents, media processors, or custom log…”
Latency turntaking — stories about latency turntaking in this arenaLatency turntaking
Stories about latency turntaking in this arena
Latency
platform-engineerSee documented end-to-end voice latency numbers or tuning guidance backing the platform's speed claims
weight 3 · round to VapiVapi's docs mention that 'Ultra Fast' vs 'High Intelligence' model settings trade off latency, giving conceptual tuning guidance, but there are no documented end-to-end latency numbers (e.g., ms benchmarks) or detailed tuning playbooks in the evidence pack. Missing for 10: concrete latency benchmark figures, component-level latency breakdowns (STT/LLM/TTS), and independent verification of speed claims.
- [claimed-docs] “Ultra Fast uses speed-optimized models for lower latency. High Intelligence accepts higher latency for greater capability.”
LiveKit Agentsnone0/10The evidence pack contains no documented end-to-end latency benchmarks or explicit tuning guidance for reducing turn-taking latency — only feature descriptions (turn detector, adaptive interruption handling) without numbers or configuration guidance. A community report explicitly states users could not measure TTFT or STT→LLM→TTS latency, reinforcing that this visibility/documentation is absent.
- [claimed-docs] “Adaptive interruption handling allows an agent to respond naturally when users speak mid-response... to identify intentional interruptions (…”
- [claimed-docs] “LiveKit's `TurnDetector` is an audio model that encodes user audio directly, capturing both _what_ is said and _how_ it's said.”
- [claimed-docs] “A turn detector model can predict that they have more to say and wait for them to finish before responding.”
- [claimed-docs] “the model analyzes the acoustic signals to identify intentional interruptions (barge-ins) from conversational backchanneling”
- [community] “We've been running multiple voice AI agents on LiveKit and kept running into visibility issues — no way to measure TTFT, latency across STT …”
Turn taking
developerRely on the agent to handle interruptions (barge-in) gracefully — stopping speech, updating context, and recovering the turn
weight 3 · round to LiveKit AgentsVapinone0/10The evidence pack covers CLI, MCP server, tools, voices, compliance, and testing, but contains no documentation or evidence about interruption handling, barge-in behavior, context updates during interruptions, or turn recovery mechanics. This is a core latency/turn-taking capability for voice agents, but nothing in the pack addresses it directly.
LiveKit Agents has well-documented, purpose-built features for barge-in: adaptive interruption handling distinguishes intentional interruptions from backchanneling, and the TurnDetector model analyzes acoustic and semantic signals to manage turn-taking recovery. These are first-party docs with specific technical detail on the mechanism (stopping speech, updating context, recovering turn), and the runtime probes confirm the framework is real and installable. Missing for 10: independent/hands-on evidence specifically testing barge-in behavior in production (only community evidence is about latency observability, not interruption handling itself) and no third-party benchmark of interruption accuracy.
- [claimed-docs] “Adaptive interruption handling allows an agent to respond naturally when users speak mid-response... to identify intentional interruptions (…”
- [claimed-docs] “the model analyzes the acoustic signals to identify intentional interruptions (barge-ins) from conversational backchanneling”
- [claimed-docs] “LiveKit's `TurnDetector` is an audio model that encodes user audio directly, capturing both _what_ is said and _how_ it's said.”
- [claimed-docs] “A turn detector model can predict that they have more to say and wait for them to finish before responding.”
- [claimed-docs] “Adaptive interruption handling allows an agent to respond naturally when users speak mid-response.”
- [claimed-docs] “LiveKit's TurnDetector is an audio model that encodes user audio directly, capturing both what is said and how it's said.”
- [probe] “PROBE runtime (recorded 2026-09-05): pypi livekit-agents 1.8.0 installs and `import livekit.agents` succeeds with no key (npm @livekit/agent…”
developerEnable noise suppression or audio filtering so the agent stays coherent on noisy real-world calls
weight 1 · round drawnVapinone0/10No evidence pack items mention noise suppression, background noise filtering, or audio denoising features for handling noisy real-world calls; the pack covers voices, tools, CLI, MCP, latency modes, and testing but nothing about audio filtering.
LiveKit Agentsnone0/10The evidence pack covers turn detection, interruption handling, telephony, MCP tools, and testing, but contains no mention of noise suppression or audio filtering plugins/features anywhere. Missing for 10: any docs or plugin reference to noise cancellation (e.g. Krisp/BVC), background noise filtering, or audio pre-processing configuration.
developerUse model-based end-of-turn detection beyond simple VAD silence timeouts so the agent doesn't talk over slow speakers
weight 2 · round to LiveKit AgentsVapinone0/10The evidence pack lacks any mention of model-based end-of-turn detection, turn-taking models, or semantic VAD alternatives; only latency tuning between 'Ultra Fast' and 'High Intelligence' model modes is documented, which does not address end-of-turn detection beyond silence-based VAD. Missing for 10: any documentation of a model-based or semantic end-of-turn detector, comparison to VAD-only timeout behavior, or configuration options for turn-taking sensitivity.
- [claimed-docs] “Ultra Fast uses speed-optimized models for lower latency. High Intelligence accepts higher latency for greater capability.”
LiveKit's TurnDetector is a dedicated audio+semantic model (not just VAD silence) that predicts whether a user has finished speaking or has more to say, waiting accordingly, and adaptive interruption handling further distinguishes real barge-ins from backchanneling — directly addressing not talking over slow speakers. This is documented in detail across multiple first-party doc pages plus GitHub feature lists. missing for 10: independent/hands-on benchmark or community validation of the turn-detector's real-world latency/accuracy impact.
- [claimed-docs] “LiveKit's `TurnDetector` is an audio model that encodes user audio directly, capturing both _what_ is said and _how_ it's said.”
- [claimed-docs] “A turn detector model can predict that they have more to say and wait for them to finish before responding.”
- [claimed-docs] “LiveKit's TurnDetector is an audio model that encodes user audio directly, capturing both what is said and how it's said.”
- [claimed-docs] “Adaptive interruption handling allows an agent to respond naturally when users speak mid-response... to identify intentional interruptions (…”
- [claimed-docs] “the model analyzes the acoustic signals to identify intentional interruptions (barge-ins) from conversational backchanneling”
- [claimed-docs] “Adaptive interruption handling allows an agent to respond naturally when users speak mid-response.”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to VapiVapi's API/SDK covers assistant, phone number, and call management, backed by a full CLI (vapi-docs-2 through 4, 27, 31-32, verified live in vapi-probe-rt-1) and an official MCP server exposing the same operations for external agents (vapi-docs-5,33, confirmed live and auth-gated in vapi-probe-rt-2). Dashboard-only conveniences like Voice Library browsing/preview remain UI-centric, though voice IDs and custom voices can be set via API (vapi-docs-9,24,26,35). Missing for 10: explicit confirmation that every dashboard-only feature (e.g., visual Voice Library browsing, HIPAA toggle) has a full API equivalent, and independent (non-vendor) verification of full UI/API parity.
- [claimed-docs] “Manage assistants, phone numbers, and calls from your terminal”
- [claimed-docs] “vapi listen --forward-to localhost:3000/webhook”
- [claimed-docs] “so you can manage assistants, phone numbers, and calls from any MCP-compatible AI assistant (like Claude Desktop) or agent framework”
- [claimed-docs] “The Vapi MCP Server exposes Vapi APIs as tools via the Model Context Protocol (MCP), so you can manage assistants, phone numbers, and calls …”
- [claimed-docs] “Build, test, and deploy voice AI applications without leaving your development environment.”
- [claimed-docs] “The CLI auto-detects your tech stack and sets up everything you need.”
- [probe] “PROBE runtime (recorded 2026-09-05): the official Vapi CLI installed via the vendor's one-liner (`curl -sSL https://vapi.ai/install.sh | bas…”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
- [claimed-docs] “You can use your own custom voice with any supported provider by setting the `voice` property in your assistant configuration”
- [claimed-docs] “Browse and preview voices there, then copy a voice's ID to use on an assistant.”
LiveKit Agents is API/SDK-first (Python/Node.js) with a full programmatic surface for building agents, tools, telephony, and testing, and there is also a no-code 'Agent Builder' UI for prototyping. However, the evidence doesn't confirm that everything achievable in that browser-based Agent Builder UI (or LiveKit Cloud dashboard features like deployment management, rollback) is equally exposed via API/CLI, so parity between UI and API is not fully demonstrated. missing for 10: explicit evidence that Agent Builder's no-code UI actions (and Cloud dashboard deployment/rollback controls) are all reachable via API/CLI, and confirmation of full CLI/API parity with dashboard features.
- [claimed-docs] “Prototype and deploy voice agents directly in your browser, without writing any code.”
- [claimed-docs] “LiveKit Agent Builder to prototype and deploy agents directly in your browser without writing code”
- [claimed-docs] “LiveKit Agent Builder: Prototype and deploy voice agents directly in your browser, without writing any code.”
- [claimed-docs] “Instant rollback to a previous agent deployment”
- [probe] “official CLI documented at https://github.com/livekit/livekit-cli”
- [probe] “PROBE runtime (recorded 2026-09-05): `lk --version` printed `lk version 2.18.6` after a plain `brew install livekit-cli` — the official CLI …”
- [claimed-docs] “A programmatic participant is any code that joins a LiveKit room as a participant — this includes AI agents, media processors, or custom log…”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnVapinone0/10No evidence of a data export feature, open-format data dumps, or account portability/deletion tooling; the docs cover CLI, MCP server, tools, and voice features but nothing about exporting assistants/call data/transcripts in open formats for user exit. Missing for 10: documented export functionality, open data formats supported, and any account/data portability guarantees.
LiveKit Agentsnone0/10The evidence pack documents LiveKit Agents' open-source, self-hostable nature (Apache-2.0 server, pip/npm installable framework) but contains no explicit mention of data export tooling, open data formats for conversation/session logs, or a documented exit/migration path for user data. Self-hosting mitigates lock-in in principle, but that is not the same as a documented 'export all data' capability.
- [probe] “PROBE runtime (recorded 2026-09-05): `livekit-server --dev` (brew-installed, Apache-2.0 OSS) booted on this machine with NO account or key —…”
- [probe] “PROBE runtime (recorded 2026-09-05): pypi livekit-agents 1.8.0 installs and `import livekit.agents` succeeds with no key (npm @livekit/agent…”
- [claimed-docs] “The good news is LiveKit includes an embedded TURN server. It's a secure TURN implementation that has integrated authentication with the res…”
ai-native userRead the product's source under an open license
weight 2 · round to LiveKit AgentsVapinone0/10Vapi is a closed, commercial SaaS platform; nothing in the evidence indicates its core source (server, dashboard, or model runtime) is published under an open license—only SDKs/CLI tooling and docs are mentioned, and the one open-source reference (pipecat) is a third-party project, not Vapi itself.
- [community] “Vapi is also built on media framework by daily.co. They have an open source version of voice ai called pipecat.”
LiveKit Agents is hosted at github.com/livekit/agents and documented as an open-source framework, with hands-on verification that the packages install and run without any account or key (pip/npm) and that the underlying server is Apache-2.0 OSS. The llms.txt docs also explicitly describe LiveKit as 'an open-source platform.' Missing for 10: an explicit license file/badge citation specifically for the livekit/agents repo (only inferred via probe evidence and sibling repo license).
- [github] “MCP support: Native support for MCP. Integrate tools provided by MCP servers with one line of code.”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.livekit.io/llms.txt # LiveKit docs > LiveKit is an open-source platform for building voice, video,…”
- [probe] “PROBE runtime (recorded 2026-09-05): `livekit-server --dev` (brew-installed, Apache-2.0 OSS) booted on this machine with NO account or key —…”
- [probe] “PROBE runtime (recorded 2026-09-05): pypi livekit-agents 1.8.0 installs and `import livekit.agents` succeeds with no key (npm @livekit/agent…”
- [probe] “official CLI documented at https://github.com/livekit/livekit-cli”
ai-native userSelf-host the core product
weight 3 · round to LiveKit AgentsVapinone0/10Vapi is presented exclusively as a hosted cloud platform (dashboard, hosted MCP server, hosted API/CLI against cloud endpoints); nothing in the evidence pack mentions a self-hostable core engine, open-source repo for the core product, or on-prem deployment option (the mentioned pipecat is a different open-source project, not Vapi itself). This is a fair axis for a voice-AI platform, but no evidence supports self-hosting.
- [community] “Vapi is also built on media framework by daily.co. They have an open source version of voice ai called pipecat.”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
- [claimed-docs] “The Vapi MCP Server exposes Vapi APIs as tools via the Model Context Protocol (MCP), so you can manage assistants, phone numbers, and calls …”
LiveKit provides a documented, Apache-2.0 open-source self-hosting deployment guide including embedded TURN server and SSL setup, and this was hands-on verified: `livekit-server --dev` boots with no account or key, and the open-source livekit-agents framework installs and imports keylessly via pip/npm. This directly demonstrates the core product (server + agents framework) can be self-hosted. Missing for 10: independent third-party production self-hosting case study beyond the probe verification.
- [claimed-docs] “The good news is LiveKit includes an embedded TURN server. It's a secure TURN implementation that has integrated authentication with the res…”
- [claimed-docs] “In order to have a secure LiveKit deployment, you will need a domain as well as a SSL certificate for that domain.”
- [probe] “PROBE runtime (recorded 2026-09-05): `livekit-server --dev` (brew-installed, Apache-2.0 OSS) booted on this machine with NO account or key —…”
- [probe] “PROBE runtime (recorded 2026-09-05): pypi livekit-agents 1.8.0 installs and `import livekit.agents` succeeds with no key (npm @livekit/agent…”
- [probe] “PROBE runtime (recorded 2026-09-05): `lk --version` printed `lk version 2.18.6` after a plain `brew install livekit-cli` — the official CLI …”
Pricing plans — plan structure and value — what each tier costs and what it unlocksPricing plans
Plan structure and value — what each tier costs and what it unlocks
Pricing
founderSee published per-minute or usage pricing and estimate cost per call before committing
weight 2 · round drawnVapinone0/10The evidence pack contains no mention of pricing pages, per-minute rates, usage-based cost breakdowns, or any pricing calculator/estimator; all evidence is about docs, CLI, MCP server, tools, and testing features. Absence of evidence for this applicable axis yields none.
LiveKit Agentsnone0/10The pack references a pricing URL (livekit.io/pricing) only for an unrelated 'rollback' feature and never quotes actual per-minute or usage rates; a community comment even highlights that operators struggled with 'cost per call' visibility, but no evidence shows published pricing figures a founder could use to estimate costs before committing.
- [claimed-docs] “Instant rollback to a previous agent deployment”
- [community] “We've been running multiple voice AI agents on LiveKit and kept running into visibility issues — no way to measure TTFT, latency across STT …”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round to LiveKit AgentsVapinone0/10The evidence pack contains no mention of data residency, regional storage options, or geographic data controls anywhere in Vapi's docs (only HIPAA toggle and SSO/RBAC are mentioned as compliance features). Missing for 10: any documentation of region selection, data residency guarantees, or storage location controls.
LiveKit Agents/server is open-source and self-hostable (embedded TURN server, deployment docs, hands-on verified self-host boot), which lets an operator choose their own hosting region/data location, but there is no explicit documented region-selection or data-residency feature for LiveKit Cloud itself. missing for 10: explicit LiveKit Cloud region-selection/data-residency controls, compliance certifications (e.g. GDPR/SOC2 region guarantees), and any customer-facing UI for choosing storage region.
- [claimed-docs] “The good news is LiveKit includes an embedded TURN server. It's a secure TURN implementation that has integrated authentication with the res…”
- [claimed-docs] “In order to have a secure LiveKit deployment, you will need a domain as well as a SSL certificate for that domain.”
- [probe] “PROBE runtime (recorded 2026-09-05): `livekit-server --dev` (brew-installed, Apache-2.0 OSS) booted on this machine with NO account or key —…”
ai-native userPrevent my data from being used to train AI models
weight 3 · round drawnVapinone0/10No evidence in the pack addresses data-training opt-out, model-training data policies, or privacy controls preventing AI training use; only HIPAA and SSO/RBAC enterprise features are mentioned, which are unrelated to this specific claim. missing for 10: explicit data-training opt-out policy, DPA/ToS language on training use, dashboard/API setting to exclude data from model training.
LiveKit Agentsnone0/10No evidence in the pack addresses data usage for AI model training, opt-out policies, or any privacy commitments regarding training data; the evidence pack is entirely about agent framework features (tools, telephony, testing, MCP support). This is a plausible axis for a platform handling user voice/media data, but absent any documentation on training-data usage or opt-out, it cannot be credited as delivered.
ai-native userControl data retention and deletion
weight 2 · round drawnVapinone0/10The evidence pack covers HIPAA, SSO/RBAC, tooling, CLI, and MCP server features but contains no documentation about data retention policies, call/recording deletion controls, or user-initiated data export/erasure mechanisms. Missing for 10: retention period settings, deletion/erasure APIs or dashboard controls, data export tools, and any policy documentation on how long call data/transcripts are stored.
LiveKit Agentsnone0/10The evidence pack contains no mention of data retention policies, recording/session deletion controls, or configurable retention windows for LiveKit Agents. While self-hosting is documented (implying infrastructure control), no explicit retention/deletion feature or API is evidenced.
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnVapinone0/10No evidence pack item addresses telemetry opt-out or usage tracking controls; Vapi's docs cover HIPAA/SSO/RBAC compliance but not a telemetry toggle.
Telephony — stories about telephony in this arenaTelephony
Stories about telephony in this arena
Call control
developerEscalate a live call to a human with warm or blind transfer, passing context along
weight 2 · round drawnVapinone0/10No evidence in the pack mentions call transfer, warm/blind transfer, or passing context to a human agent; the evidence covers assistants, CLI, MCP server, tools/webhooks, voices, and compliance, but nothing about live-call escalation/transfer capabilities.
LiveKit Agentsnone0/10LiveKit Agents documents inbound/outbound telephony (SIP trunks, phone calls) but the evidence pack contains no mention of call transfer (warm or blind) or passing conversational context to a human agent during a live call — this specific escalation capability is never described.
- [claimed-docs] “LiveKit telephony lets you build AI-powered voice apps that handle inbound and outbound calls.”
- [claimed-docs] “Enable your voice agent to make or take phone calls.”
- [claimed-docs] “Outbound trunks are used to place outgoing calls.”
- [github] “Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.”
developerMy agent can send DTMF keypresses, navigate IVR menus, and detect or leave voicemail
weight 1 · round drawnVapinone0/10The evidence pack contains no mention of DTMF keypress sending, IVR menu navigation, or voicemail detection/leaving functionality anywhere in the docs or community items — these telephony-specific capabilities are entirely unevidenced despite being a plausible axis for a voice AI telephony product.
LiveKit Agentsnone0/10Evidence confirms LiveKit's telephony/SIP stack supports inbound/outbound calls (docs-26, docs-31, docs-32, gh-2), but nothing in the pack mentions DTMF keypress sending, IVR menu navigation, or voicemail detection/leaving capabilities specifically. Missing for 10: any documentation or example of DTMF tone generation, automated IVR menu traversal, or voicemail detection/answering-machine-detection logic.
- [claimed-docs] “LiveKit telephony lets you build AI-powered voice apps that handle inbound and outbound calls.”
- [claimed-docs] “Enable your voice agent to make or take phone calls.”
- [claimed-docs] “Outbound trunks are used to place outgoing calls.”
- [github] “Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.”
Campaigns
founderRun batch outbound call campaigns with scheduling and throughput controls
weight 2 · round drawnVapinone0/10Evidence covers assistant creation, tools, CLI, MCP server, testing, and voice customization, but nothing addresses batch/outbound campaign management, call scheduling, or throughput/concurrency controls for bulk dialing. No mention of a campaigns API, CSV/list upload, dialer pacing, or rate-limiting controls for outbound calling at scale.
LiveKit Agentsnone0/10Evidence shows LiveKit Agents can place individual outbound calls via SIP outbound trunks (docs-26, docs-32, docs-31), but there is no evidence of any batch campaign management, call scheduling, or throughput/rate-limiting controls for running outbound calling campaigns at scale — these features are simply not documented anywhere in the pack.
- [claimed-docs] “LiveKit telephony lets you build AI-powered voice apps that handle inbound and outbound calls.”
- [claimed-docs] “Outbound trunks are used to place outgoing calls.”
- [claimed-docs] “Enable your voice agent to make or take phone calls.”
- [github] “Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.”
Numbers
developerProvision phone numbers and run both inbound and outbound calls through the platform's API
weight 3 · round to VapiDocs explicitly cover creating an assistant, connecting it to a phone number, and making inbound/outbound calls via the quickstart, plus CLI/MCP support for managing phone numbers and calls, and SIP for advanced telephony integration, with runtime probes confirming the CLI and MCP endpoints work as documented. Missing for 10: independent hands-on confirmation of actual outbound call placement via raw API (only demo/inbound anecdote in community evidence) and explicit multi-number provisioning workflow details.
- [claimed-docs] “const assistant = await vapi.assistant”
- [claimed-docs] “Create a voice assistant, connect it to a phone number, and make your first calls.”
- [claimed-docs] “In under 5 minutes, you'll create a voice assistant and start talking to it over the phone.”
- [claimed-docs] “Manage assistants, phone numbers, and calls from your terminal”
- [claimed-docs] “so you can manage assistants, phone numbers, and calls from any MCP-compatible AI assistant (like Claude Desktop) or agent framework”
- [claimed-docs] “The Vapi MCP Server exposes Vapi APIs as tools via the Model Context Protocol (MCP), so you can manage assistants, phone numbers, and calls …”
- [claimed-docs] “Use any SIP softphone (e.g., Zoiper, Linphone) to dial your SIP URI”
- [claimed-docs] “This guide shows you how to set up and test SIP calls to your Vapi assistant using any SIP client or softphone.”
- [probe] “PROBE runtime (recorded 2026-09-05): the official Vapi CLI installed via the vendor's one-liner (`curl -sSL https://vapi.ai/install.sh | bas…”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
- [community] “Called the demo number, sounds smooth! Good luck.”
LiveKit's SIP/telephony stack clearly supports inbound and outbound calling ('LiveKit telephony lets you build AI-powered voice apps that handle inbound and outbound calls', 'Outbound trunks are used to place outgoing calls', 'Enable your voice agent to make or take phone calls') and integrates with the agents runtime via SIP trunks. However, the evidence never shows LiveKit itself provisioning phone numbers through its own API — telephony typically requires configuring an external SIP trunk provider, and no docs here describe a native number-provisioning endpoint. Missing for 10: explicit documentation/evidence of a LiveKit API call that provisions/purchases phone numbers directly (rather than just configuring trunks against externally-acquired numbers), and independent/hands-on confirmation of the inbound+outbound call flow.
- [claimed-docs] “LiveKit telephony lets you build AI-powered voice apps that handle inbound and outbound calls.”
- [claimed-docs] “Enable your voice agent to make or take phone calls.”
- [claimed-docs] “Outbound trunks are used to place outgoing calls.”
- [github] “Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.”
Sip
platform-engineerConnect my own carrier or PBX via SIP trunking (or import Twilio/Telnyx numbers) instead of being locked to bundled telephony
weight 2 · round to LiveKit AgentsDocs confirm SIP connectivity: platform engineers can point any SIP client/softphone at a Vapi SIP URI to make/receive calls (vapi-docs-14, vapi-docs-22), which supports self-hosted/carrier SIP trunking use cases. However, the evidence never explicitly documents PBX/carrier trunk configuration options or a number-import workflow for Twilio/Telnyx accounts. missing for 10: explicit documentation of BYO-carrier/PBX trunk setup steps, and evidence of importing existing Twilio/Telnyx phone numbers into Vapi.
- [claimed-docs] “Use any SIP softphone (e.g., Zoiper, Linphone) to dial your SIP URI”
- [claimed-docs] “This guide shows you how to set up and test SIP calls to your Vapi assistant using any SIP client or softphone.”
LiveKit's SIP-based telephony stack lets agents make/receive calls via inbound and outbound SIP trunks, which is the standard mechanism for bringing your own carrier or PBX (docs-26, docs-32, gh-2, docs-31). This is architecture-agnostic SIP, not a bundled/proprietary telephony lock-in, satisfying the platform-engineer's need to connect external trunks. Missing for 10: explicit documented walkthroughs or examples of importing Twilio/Telnyx numbers specifically, and no independent/hands-on confirmation of a real carrier trunk connection succeeding.
- [claimed-docs] “LiveKit telephony lets you build AI-powered voice apps that handle inbound and outbound calls.”
- [claimed-docs] “Outbound trunks are used to place outgoing calls.”
- [github] “Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.”
- [claimed-docs] “Enable your voice agent to make or take phone calls.”
Testing analytics — stories about testing analytics in this arenaTesting analytics
Stories about testing analytics in this arena
Analytics
ai-native userThe platform's AI reviews my calls for me — scoring quality, flagging failures, and analyzing resolution automatically
weight 2 · round to VapiVapi's Evals framework provides automated validation of assistant behavior via mock conversations and its Voice Test Suites use an AI tester to simulate calls and check outcomes, which is adjacent to automated call review, but neither is documented as automatically scoring, flagging failures in, or analyzing resolution of real production calls after the fact. Missing for 10: evidence of automatic post-call quality scoring on live/production transcripts, explicit failure-flagging analytics, and resolution-outcome analysis dashboards distinct from pre-production test suites.
- [claimed-docs] “Evals is Vapi's AI agent testing framework that enables you to systematically test assistants and squads using mock conversations with autom…”
- [claimed-docs] “you'll create mock conversations, define expected behaviors, and validate your agents work correctly before production”
- [claimed-docs] “Voice Test Suites enable you to test your AI voice agents through simulated phone conversations.”
- [claimed-docs] “Our AI tester calls your voice agent and follows a script that simulates real customer behavior.”
LiveKit Agentsdisputedcontradicted3/10LiveKit ships a dev-time testing framework with 'judges' and agent simulations to evaluate scripted interactions (docs-9/10/25, gh-4), but this is pre-deployment test tooling, not automatic scoring/flagging/resolution-analysis of live production calls. A hands-on community report explicitly says operators running real agents on LiveKit have no built-in way to measure latency or cost per call, let alone automated quality/resolution review, contradicting the idea that the platform reviews calls for you. Missing for 10: production call-level QA scoring, automatic failure flagging on real traffic, resolution/outcome analysis dashboards.
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes. They integrate with your existing test suite using pytest (Python) or V…”
- [claimed-docs] “Agent simulations run end-to-end conversations between your agent and an LLM-driven user, then evaluate the results across the full interact…”
- [github] “Builtin test framework: Write tests and use judges to ensure your agent is performing as expected.”
- [community] “We've been running multiple voice AI agents on LiveKit and kept running into visibility issues — no way to measure TTFT, latency across STT …”
founderSee call analytics — success rates, durations, outcomes, sentiment — in dashboards without building my own
weight 2 · round drawnVapinone0/10The evidence pack documents assistant creation, tools, testing (Evals, Voice Test Suites), CLI/MCP management, and voice library features, but contains no mention of a dashboard showing call success rates, durations, outcomes, or sentiment analytics — the core of this story.
LiveKit Agentsnone0/10The evidence pack shows testing/simulation tools (pytest/Vitest, LLM-driven simulations) but no built-in production analytics dashboard for success rates, durations, outcomes, or sentiment. A community report explicitly confirms this gap — users running LiveKit voice agents said they had 'no way to measure TTFT, latency across STT→LLM→TTS, or even cost per call' and found debugging painful, indicating no such dashboard exists out of the box.
- [community] “We've been running multiple voice AI agents on LiveKit and kept running into visibility issues — no way to measure TTFT, latency across STT …”
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes. They integrate with your existing test suite using pytest (Python) or V…”
- [claimed-docs] “Agent simulations run end-to-end conversations between your agent and an LLM-driven user, then evaluate the results across the full interact…”
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes... Agent simulations run end-to-end conversations between your agent and…”
Monitoring
platform-engineerMonitor live calls in production and get alerts when agents misbehave or error rates spike
weight 1 · round drawnVapinone0/10The evidence pack covers pre-production testing (Voice Test Suites, Evals) and webhook debugging via the CLI, but contains no mention of live call monitoring dashboards, real-time alerting, or error-rate-spike detection for production traffic. This is a fair axis for a voice AI platform, but nothing in the pack demonstrates it.
LiveKit Agentsnone0/10No evidence pack items describe production monitoring dashboards, error-rate alerting, or call-quality observability tooling for LiveKit Agents; the only related evidence is a community report explicitly describing the lack of visibility into TTFT, per-call latency, and cost — a gap, not a delivered capability.
- [community] “We've been running multiple voice AI agents on LiveKit and kept running into visibility issues — no way to measure TTFT, latency across STT …”
Testing
developerTest agents with simulated conversations or evals before putting them on real phone calls
weight 2 · round to VapiVapi documents two dedicated testing features directly matching the story: Voice Test Suites for simulated phone conversations via an AI tester following scripted customer behavior, and Evals, a testing framework for mock conversations with automated validation before production. Missing for 10: independent/hands-on developer corroboration of these specific testing features beyond vendor docs.
- [claimed-docs] “Voice Test Suites enable you to test your AI voice agents through simulated phone conversations.”
- [claimed-docs] “Our AI tester calls your voice agent and follows a script that simulates real customer behavior.”
- [claimed-docs] “Evals is Vapi's AI agent testing framework that enables you to systematically test assistants and squads using mock conversations with autom…”
- [claimed-docs] “you'll create mock conversations, define expected behaviors, and validate your agents work correctly before production”
Docs explicitly describe both behavioral tests (pytest/Vitest) and agent simulations that run end-to-end conversations with an LLM-driven user, evaluating results before deployment, and GitHub notes a 'builtin test framework' with judges — directly matching the testing-before-real-calls story, complemented by separate telephony/SIP support for real calls. Missing for 10: independent/hands-on validation of the eval/simulation framework and explicit documentation tying test simulations to pre-phone-call validation workflows.
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes. They integrate with your existing test suite using pytest (Python) or V…”
- [claimed-docs] “Agent simulations run end-to-end conversations between your agent and an LLM-driven user, then evaluate the results across the full interact…”
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes... Agent simulations run end-to-end conversations between your agent and…”
- [claimed-docs] “Behavioral tests verify specific interactions and expected outcomes... using pytest (Python) or Vitest (Node.js)”
- [github] “Builtin test framework: Write tests and use judges to ensure your agent is performing as expected.”
- [github] “Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.”
- [claimed-docs] “LiveKit telephony lets you build AI-powered voice apps that handle inbound and outbound calls.”
Tools function calling — stories about tools function calling in this arenaTools function calling
Stories about tools function calling in this arena
Post call
developerExtract structured data from every call — outcomes, entities, dispositions — delivered via API or webhook after the call
weight 2 · round drawnVapinone0/10The evidence pack shows Vapi's webhook/tool-calling system for live in-call actions (tool-calls messages, custom webhook tools, function calling) but contains no mention of a post-call structured-data/analysis feature (outcomes, entities, dispositions) delivered via API or webhook after the call ends. Missing for 10: any docs on end-of-call reports, structured data extraction schemas, call analysis/summary webhooks, or an API endpoint returning call outcome/entity data.
- [claimed-docs] “When tools are triggered, your Server URL receives a `tool-calls` message”
- [claimed-docs] “Vapi supports OpenAI-style tool/function calling. Assistants can ping your server to perform actions.”
- [claimed-docs] “Custom tools that you create... interact with your systems via webhooks”
- [claimed-docs] “Create your own webhook-based tools to extend assistant capabilities”
LiveKit Agentsnone0/10LiveKit Agents provides tool-calling, MCP integration, and telephony hooks, but nothing in the evidence describes a built-in mechanism for automatically extracting structured call outcomes/entities/dispositions and delivering them via API or webhook after a call ends — that would have to be custom-built by the developer using the tool-calling primitives. Since no such capability is documented, this applicable axis is unmet.
- [claimed-docs] “Call external APIs or lookup data for RAG.”
- [claimed-docs] “LiveKit Agents has full support for LLM tool use. This feature allows you to create a custom library of tools to extend your agent's context”
- [claimed-docs] “LiveKit telephony lets you build AI-powered voice apps that handle inbound and outbound calls.”
- [claimed-docs] “Outbound trunks are used to place outgoing calls.”
Tools
ai-native userMy voice agent can plug in MCP servers as tool sources so one integration grants it whole toolsets mid-call
weight 2 · round to LiveKit AgentsVapinone0/10Evidence only shows Vapi exposing its own APIs as an MCP *server* for external AI assistants (e.g., Claude Desktop) to manage calls/assistants — the reverse direction of the story. There is no documentation or probe showing a Vapi voice assistant can itself act as an MCP *client*, plugging in external MCP servers as tool sources mid-call; tool integration is instead described only via webhook-based custom tools and OpenAI-style function calling.
- [claimed-docs] “so you can manage assistants, phone numbers, and calls from any MCP-compatible AI assistant (like Claude Desktop) or agent framework”
- [claimed-docs] “The Vapi MCP Server exposes Vapi APIs as tools via the Model Context Protocol (MCP), so you can manage assistants, phone numbers, and calls …”
- [claimed-docs] “Create your own webhook-based tools to extend assistant capabilities”
- [claimed-docs] “Vapi supports OpenAI-style tool/function calling. Assistants can ping your server to perform actions.”
- [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://mcp.vapi.ai/mcp returned HTTP 401 — t…”
LiveKit Agents has first-class, documented MCP support ("first-class support for Model Context Protocol (MCP) servers", "Integrate tools provided by MCP servers with one line of code") via `MCPToolset`, which wraps an MCP server and passes it directly to the agent's tools parameter — enabling mid-call toolset access for voice agents. This is corroborated across both docs and GitHub README. Missing for 10: independent hands-on validation of actual mid-call MCP toolset usage beyond vendor docs.
- [claimed-docs] “Wrap an MCP server in an \`MCPToolset\` and pass it to the agent's \`tools\` parameter”
- [claimed-docs] “Wrap an MCP server in an `MCPToolset` and pass it to the agent's `tools` parameter”
- [claimed-docs] “LiveKit Agents has first-class support for Model Context Protocol (MCP) servers.”
- [github] “MCP support: Native support for MCP. Integrate tools provided by MCP servers with one line of code.”
- [github] “Native support for MCP. Integrate tools provided by MCP servers with one line of code.”
developerMy agent can call external APIs and custom functions mid-conversation and speak the result without awkward dead air
weight 3 · round to LiveKit AgentsVapi's docs clearly show mid-call function/tool calling via webhook Server URLs and OpenAI-style tool-calls messages, letting the assistant fetch external API results and use them in conversation (vapi-docs-6, vapi-docs-7, vapi-docs-17, vapi-docs-20, vapi-docs-23, vapi-docs-34). However, the pack lacks explicit evidence about mechanisms for avoiding 'dead air' during the API call latency (e.g., async tool config, filler phrases, or interim speech) — only general latency-tuning docs for model selection are present (vapi-docs-15). Missing for 10: explicit documentation/demo of filler/interim speech or async tool handling during function execution, and independent hands-on confirmation that the conversation flow feels seamless during a live tool call.
- [claimed-docs] “Create your own webhook-based tools to extend assistant capabilities”
- [claimed-docs] “Server URL: The endpoint where your function is hosted”
- [claimed-docs] “When tools are triggered, your Server URL receives a `tool-calls` message”
- [claimed-docs] “This guide shows you how to create custom tools, including Function Tools, for your Vapi assistants.”
- [claimed-docs] “Vapi supports OpenAI-style tool/function calling. Assistants can ping your server to perform actions.”
- [claimed-docs] “Custom tools that you create... interact with your systems via webhooks”
- [claimed-docs] “Ultra Fast uses speed-optimized models for lower latency. High Intelligence accepts higher latency for greater capability.”
LiveKit Agents has full documented support for LLM tool/function calling (custom Python/Node functions calling external APIs mid-conversation) plus MCP server tool integration, combined with adaptive interruption handling and turn-detection to avoid awkward dead air while tools execute. Runtime probes confirm the framework is real and installable, corroborating the docs claims. missing for 10: independent/hands-on evidence specifically showing tool-call latency handled gracefully in a live conversation (only vendor docs cover this exact combination).
- [claimed-docs] “Call external APIs or lookup data for RAG.”
- [claimed-docs] “LiveKit Agents has full support for LLM tool use. This feature allows you to create a custom library of tools to extend your agent's context”
- [claimed-docs] “LiveKit Agents has full support for LLM tool use.”
- [claimed-docs] “LiveKit Agents has first-class support for Model Context Protocol (MCP) servers.”
- [claimed-docs] “Wrap an MCP server in an \`MCPToolset\` and pass it to the agent's \`tools\` parameter”
- [claimed-docs] “Adaptive interruption handling allows an agent to respond naturally when users speak mid-response... to identify intentional interruptions (…”
- [claimed-docs] “the model analyzes the acoustic signals to identify intentional interruptions (barge-ins) from conversational backchanneling”
- [probe] “PROBE runtime (recorded 2026-09-05): pypi livekit-agents 1.8.0 installs and `import livekit.agents` succeeds with no key (npm @livekit/agent…”
Transcription recording — stories about transcription recording in this arenaTranscription recording
Stories about transcription recording in this arena
Recording
platform-engineerRetrieve full call recordings and transcripts programmatically for every call
weight 2 · round drawnVapinone0/10The evidence pack contains no documentation, API reference, or example demonstrating retrieval of call recordings or transcripts, despite this being a natural, expected capability for a voice AI platform; only tangential tooling (CLI, MCP server, custom tools, testing/evals) is covered.
Transcription
developerGet accurate real-time transcription with control over the STT provider, language models, or key terms
weight 2 · round to LiveKit AgentsVapinone0/10The evidence pack contains no documentation about selecting/configuring an STT provider, choosing language models for transcription, or defining custom key terms/vocabulary for real-time transcription — topics like 'transcriber', 'Deepgram', or keyword boosting are absent. While Vapi is clearly a voice AI platform where such controls are a fair axis, none of the provided docs, community items, or probes address it.
LiveKit Agents supports swapping STT/LLM providers via its open-source plugin ecosystem and lets you override API key/base URL for any provider (docs-12, docs-24, docs-35), giving real control over transcription and language-model choice. However there is no evidence of support for custom vocabulary/key-term or hotword boosting in STT, and no independent benchmark of transcription accuracy. Missing for 10: key-term/hotword/phrase-hint configuration support, accuracy benchmarks or independent corroboration of transcription quality.
- [claimed-docs] “LiveKit Agents includes a large ecosystem of open source plugins for a variety of AI providers.”
- [claimed-docs] “You can use LiveKit Inference to access many of these models directly through LiveKit Cloud, or you can use the open source plugins to conne…”
- [claimed-docs] “For any provider not included, you can override the API key and base URL at initialization for the LLM, STT, and TTS interfaces in the plugi…”
Voices tts — stories about voices tts in this arenaVoices tts
Stories about voices tts in this arena
Voices
founderClone a custom brand voice and use it for my agents, with a documented consent process
weight 2 · round drawnVapinone0/10Vapi docs show you can plug in a 'custom voice' by setting the voice property with a provider ID (vapi-docs-9, vapi-docs-26) and browse a Voice Library (vapi-docs-8, vapi-docs-35), but nothing describes an actual voice-cloning workflow or any documented consent/verification process required before cloning a brand voice. Missing for 10: a described voice-cloning feature/flow, a documented consent or identity-verification process, and any policy language governing voice cloning.
- [claimed-docs] “You can use your own custom voice with any supported provider by setting the `voice` property in your assistant configuration”
- [claimed-docs] “You can use your own custom voice with any supported provider by setting the voice property in your assistant configuration”
- [claimed-docs] “The Voice Library in the Vapi Dashboard lists every voice available to your organization. Browse and preview voices there”
- [claimed-docs] “The **Voice Library** in the [Vapi Dashboard] lists every voice available to your organization. Browse and preview voices there, then copy a…”
developerChoose from a broad voice library or plug in multiple TTS providers to get the voice I want
weight 2 · round drawnDocs confirm a Voice Library with many previewable voices and support for custom voices with any supported provider, indicating multi-provider TTS flexibility, but the evidence never names or lists specific TTS providers (e.g., ElevenLabs, PlayHT, Azure) or details plugging in third-party/custom TTS engines beyond voice ID selection. missing for 10: explicit list of supported TTS providers, documentation of custom/BYO TTS provider integration mechanics, independent hands-on confirmation of voice quality/variety.
- [claimed-docs] “The Voice Library in the Vapi Dashboard lists every voice available to your organization. Browse and preview voices there”
- [claimed-docs] “You can use your own custom voice with any supported provider by setting the `voice` property in your assistant configuration”
- [claimed-docs] “Browse and preview voices there, then copy a voice's ID to use on an assistant.”
- [claimed-docs] “You can use your own custom voice with any supported provider by setting the voice property in your assistant configuration”
- [claimed-docs] “The **Voice Library** in the [Vapi Dashboard] lists every voice available to your organization. Browse and preview voices there, then copy a…”
Docs confirm a large open-source plugin ecosystem for TTS/STT/LLM providers, plus LiveKit Inference for provider access, and an override mechanism for API key/base URL for providers not natively included, supporting multi-provider TTS flexibility. However, there's no explicit mention of a 'voice library' (e.g., curated voice catalog/selection UI) or enumeration of specific TTS providers/voices, so the 'broad voice library' half of the story is unevidenced. Missing for 10: explicit voice catalog/list of supported TTS providers and voices, evidence of ease of switching between TTS voices, independent confirmation of provider breadth.
- [claimed-docs] “LiveKit Agents includes a large ecosystem of open source plugins for a variety of AI providers.”
- [claimed-docs] “You can use LiveKit Inference to access many of these models directly through LiveKit Cloud, or you can use the open source plugins to conne…”
- [claimed-docs] “For any provider not included, you can override the API key and base URL at initialization for the LLM, STT, and TTS interfaces in the plugi…”