Amplitude vs Mixpanel
Amplitude wins · 19–12 (21 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round drawnDirect probe confirms a live llms.txt at amplitude.com/llms.txt describing itself as an AI-native platform, plus a docs.md machine-readable docs mirror and an OpenAPI spec, giving agents multiple structured entry points. missing for 10: independent third-party confirmation that agents actually consume these files effectively in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://amplitude.com/llms.txt # Amplitude Amplitude is the AI-native digital analytics platform built for AI b…”
- [probe] “PROBE docs-md: HTTP 200 at https://amplitude.com/docs.md # Amplitude Documentation Official documentation for Amplitude analytics, SDKs, an…”
- [probe] “PROBE openapi: HTTP 200 at https://amplitude.com/openapi.json — contains "openapi" key”
A direct probe confirms Mixpanel serves a working llms.txt at docs.mixpanel.com/llms.txt (HTTP 200) with a curated index of docs, and Mixpanel also documents an MCP server for agent-native data access. Missing for 10: no independent/community corroboration of an agent actually consuming llms.txt successfully.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.mixpanel.com/llms.txt # Mixpanel - [What is Mixpanel?](https://docs.mixpanel.com/docs/what-is-mixp…”
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [probe] “official MCP server documented at https://docs.mixpanel.com/docs/mcp”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to AmplitudeAmplitude offers headless automation surfaces suitable for CI: HTTP V2 event ingestion API, Dashboard REST API, Experiment APIs for managing flags/experiments 'from CI, scripts, or your own tools', a CLI-based wizard, and an OpenAPI spec probe. These support scripted/automated use without a GUI, but there's no evidence of a full CLI for ongoing operations (only a setup wizard), no CI-specific recipes/examples, and no independent/hands-on confirmation that headless CI workflows work reliably. missing for 10: dedicated CLI beyond setup wizard, documented CI/CD pipeline examples, independent verification of headless automation in practice.
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
- [claimed-docs] “Manage flags, experiments, and deployments from CI, scripts, or your own tools with the Experiment APIs.”
- [claimed-docs] “Run the Wizard CLI to instrument your codebase and get a first dashboard from one terminal command.”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
- [probe] “PROBE openapi: HTTP 200 at https://amplitude.com/openapi.json — contains "openapi" key”
Mixpanel exposes REST-based APIs (Raw Event Export, Query API, Engage API, JQL) that can be scripted and run headlessly/in CI for data export and querying, and warehouse connectors support automated syncs. However, there's no evidence of an official CLI, SDK for automation pipelines, or documented CI/CD integration pattern — the product is primarily a UI-driven analytics platform with API access as a secondary capability. missing for 10: dedicated CLI tool, CI/CD pipeline examples or GitHub Actions integration, headless automation SDK, and any hands-on/community evidence of running Mixpanel in automated pipelines.
- [claimed-docs] “To export raw event data, use the Raw Event Export API. The events payload is returned as JSONL”
- [claimed-docs] “The raw export API allows you to download your event data as it is received and stored within Mixpanel, complete with all event properties”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The JQL JavaScript code must define a main() function. This JavaScript will be compiled, and the main() function will be executed by the que…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
- [claimed-docs] “you can sync data from data warehouses like Snowflake, BigQuery, Databricks, Redshift, and Postgres to Mixpanel”
- [claimed-docs] “Use the Feature Flags API to assign your users to variants for experiments, rollout, and releases.”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round drawnAmplitudenone0/10All evidence describes Amplitude publishing its own MCP server so external AI tools (Claude, Cursor, etc.) can pull Amplitude data — this is Amplitude acting as an MCP *server*, not as a client that plugs in and consumes external MCP servers' tools. There is no evidence Amplitude itself connects to or invokes third-party MCP servers to extend its own AI features.
- [claimed-docs] “Bring Amplitude’s behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Bring Amplitude's behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Start a conversation about your users or product and get expert-level answers in seconds that you can share with your whole org. No SQL, con…”
- [probe] “official MCP server documented at https://amplitude.com/mcp-server”
Mixpanelnone0/10Evidence only shows Mixpanel exposing its OWN hosted MCP server so external AI assistants can query Mixpanel data (mixpanel-docs-10, mixpanel-docs-19, mixpanel-probe-3) — this is the reverse direction of the story, which asks whether Mixpanel can consume/plug in external MCP servers to use their tools. No evidence exists of Mixpanel acting as an MCP client or supporting third-party MCP tool integration.
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
- [probe] “official MCP server documented at https://docs.mixpanel.com/docs/mcp”
ai-native userConnect an agent via an official MCP server
weight 3 · round drawnAmplitude is not itself an agent but a SaaS analytics platform, so the axis applies; it documents an official MCP server at amplitude.com/mcp-server that brings Amplitude's behavioral data into AI tools like Claude, Cursor, OpenAI, and Lovable, corroborated by a probe confirming the page exists. Missing for 10: independent/hands-on third-party verification of the MCP server actually working in practice.
- [claimed-docs] “Bring Amplitude’s behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Bring Amplitude's behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Start a conversation about your users or product and get expert-level answers in seconds that you can share with your whole org. No SQL, con…”
- [probe] “official MCP server documented at https://amplitude.com/mcp-server”
Mixpanel documents a hosted, official MCP server that gives AI assistants direct access to Mixpanel data, exposing tools like natural-language querying of events/funnels/retention and dashboard creation, confirmed by both docs and a live probe of the docs page. missing for 10: independent/hands-on verification of the MCP server working in practice, and details on authentication/scopes beyond the docs.
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [claimed-docs] “Create-Dashboard | Create a new dashboard with text cards and reports”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
- [claimed-docs] “Once connected, you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation …”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required”
- [probe] “official MCP server documented at https://docs.mixpanel.com/docs/mcp”
ai-native userUse an official CLI
weight 2 · round to AmplitudeAmplitude ships a Wizard CLI that auto-instruments a codebase and gets a first dashboard from one terminal command (amplitude-docs-27, amplitude-docs-1), which counts as an official CLI, but this is scoped to onboarding/setup rather than a general-purpose AI-native CLI for ongoing agentic workflows. Missing for 10: documentation of a broader CLI with commands beyond initial instrumentation, evidence of AI-native/agentic use cases via the CLI, and independent hands-on corroboration of the CLI's capabilities.
- [claimed-docs] “Run the Wizard CLI to instrument your codebase and get a first dashboard from one terminal command.”
- [claimed-docs] “The wizard detects your framework, installs the right SDK, adds your API key, and fires a test event.”
Mixpanelnone0/10The evidence pack documents Mixpanel's APIs, MCP server, JQL query language, and export methods, but contains no mention of an official command-line interface (CLI) tool for Mixpanel. A SaaS analytics platform could plausibly ship a CLI for automation/scripting, so the axis applies, but no evidence supports it.
ai-native userDrive the product through a documented public API
weight 3 · round drawnAmplitude documents a broad public HTTP API surface (event ingestion, dashboard REST API, cohort management, event/property CRUD, experiment/feature-flag APIs) plus a confirmed OpenAPI spec and llms.txt/docs.md endpoints for machine consumption, indicating a well-documented, programmatically drivable API. missing for 10: independent third-party corroboration of API completeness/reliability and no evidence of official client libraries per language beyond SDK mentions.
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
- [claimed-docs] “Export your project's event data.”
- [claimed-docs] “Manage flags, experiments, and deployments from CI, scripts, or your own tools with the Experiment APIs.”
- [claimed-docs] “List all your cohorts in Amplitude, export a cohort in Amplitude, or upload a cohort.”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
- [claimed-docs] “Create, get, update, and delete categories, event types, event properties, and user properties.”
- [probe] “PROBE openapi: HTTP 200 at https://amplitude.com/openapi.json — contains "openapi" key”
- [probe] “PROBE llms.txt: HTTP 200 at https://amplitude.com/llms.txt # Amplitude Amplitude is the AI-native digital analytics platform built for AI b…”
- [probe] “PROBE docs-md: HTTP 200 at https://amplitude.com/docs.md # Amplitude Documentation Official documentation for Amplitude analytics, SDKs, an…”
Mixpanel documents a comprehensive public REST-style API surface (Query API, Raw Event Export API, Engage API, Feature Flags API, JQL) plus a hosted MCP server that lets AI assistants query events, funnels, retention, and replays via natural language, giving AI-native users both programmatic and agentic access. missing for 10: no discoverable OpenAPI/Swagger spec (probe found only 404s) and no independent/hands-on corroboration of the MCP server or APIs actually working reliably.
- [claimed-docs] “To export raw event data, use the Raw Event Export API. The events payload is returned as JSONL”
- [claimed-docs] “The raw export API allows you to download your event data as it is received and stored within Mixpanel, complete with all event properties”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
- [claimed-docs] “Use the Feature Flags API to assign your users to variants for experiments, rollout, and releases.”
- [claimed-docs] “Create custom queries using Javascript code”
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.mixpanel.com/openapi.json, https://docs.mixpanel.com/swagger.json, https://docs.mixpane…”
- [probe] “official MCP server documented at https://docs.mixpanel.com/docs/mcp”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round drawnAmplitudenone0/10Evidence shows API keys, an MCP server, and OpenAPI spec, but nothing about scoped/least-privilege credential issuance (e.g., role-based API tokens, granular permission scopes) for agent use is documented.
Mixpanelnone0/10Mixpanel offers an MCP server and various APIs, but the evidence pack contains no mention of scoped/least-privilege API keys, service accounts, or permission-limited tokens for agent use — only generic API/export/query docs and generic security incident commentary unrelated to credential scoping.
ai-native userBuild against official SDKs
weight 2 · round to AmplitudeAmplitude documents a full suite of official SDKs (web, mobile, server, game engines) sharing a unified identity/event model and API surface, backed by an installer wizard, HTTP V2 ingestion endpoint, and a published OpenAPI spec, all discoverable via llms.txt/docs.md for AI-native consumption. Missing for 10: independent/hands-on developer corroboration of SDK integration ease and no explicit AI-agent-specific SDK usage examples beyond the MCP server.
- [claimed-docs] “Evaluate flags locally for low-latency rollout decisions.”
- [claimed-docs] “Auto-instrument page views, clicks, and sessions out of the box.”
- [claimed-docs] “Every SDK shares the same identity, event, and consent model, so a user instrumented once is understood everywhere.”
- [claimed-docs] “Run the Wizard CLI to instrument your codebase and get a first dashboard from one terminal command.”
- [claimed-docs] “Use Experiment SDKs to flag and test features across web, mobile, and backend.”
- [claimed-docs] “Use the same API surface across web, mobile, server, and game engines.”
- [claimed-docs] “Unified SDKs provide one integration point for multiple Amplitude products.”
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
- [probe] “PROBE llms.txt: HTTP 200 at https://amplitude.com/llms.txt # Amplitude Amplitude is the AI-native digital analytics platform built for AI b…”
- [probe] “PROBE docs-md: HTTP 200 at https://amplitude.com/docs.md # Amplitude Documentation Official documentation for Amplitude analytics, SDKs, an…”
- [probe] “PROBE openapi: HTTP 200 at https://amplitude.com/openapi.json — contains "openapi" key”
Evidence shows official APIs (Query API, Raw/Engage Export API, Feature Flags API) and a JQL scripting interface for building custom analytics, which support programmatic integration, but there is no explicit mention of official client SDKs (e.g., JS/Python/mobile libraries), language coverage, or code samples that an AI-native builder would need. Missing for 10: explicit SDK documentation/list of languages, code snippets, GitHub repos, and independent corroboration of SDK reliability.
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
- [claimed-docs] “Use the Feature Flags API to assign your users to variants for experiments, rollout, and releases.”
- [claimed-docs] “Create custom queries using Javascript code”
- [claimed-docs] “The JQL JavaScript code must define a main() function. This JavaScript will be compiled, and the main() function will be executed by the que…”
ai-native userSubscribe to events via webhooks
weight 2 · round drawnAmplitudenone0/10No evidence of any webhook subscription mechanism for events; Amplitude's evidence covers SDKs, APIs for ingest/export, integrations, and an MCP server, but nothing describes outbound webhooks for event subscriptions. missing for 10: any webhook API/documentation, event subscription mechanism, or third-party confirmation of webhook support.
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round to AmplitudeAmplitude AI (docs-10, docs-34) explicitly lets users ask plain-language questions and get expert-level answers/insights across their data, and the MCP server (docs-11/17, probe-4) surfaces this behavioral context directly inside AI tools. This directly matches the story of getting AI-generated insights inside the product. Missing for 10: independent/hands-on validation of the AI insight quality (only vendor docs present), and no detail on proactive suggestion generation vs. query-response only.
- [claimed-docs] “Ask questions in plain language — Amplitude AI surfaces answers across all your data.”
- [claimed-docs] “Start a conversation about your users or product and get expert-level answers in seconds that you can share with your whole org. No SQL, con…”
- [claimed-docs] “Bring Amplitude’s behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Bring Amplitude's behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [probe] “official MCP server documented at https://amplitude.com/mcp-server”
Mixpanel's own docs/marketing describe AI-driven insight surfacing ('Agents monitor your product continuously, surface insights before you think to ask' and example outputs like 'Signups dipped 19% last week'), AI trend-spotting across replays/heatmaps, AI summaries of replay playlists, and natural-language querying via its MCP server. These directly match the story of AI-generated insights/suggestions inside the product. Missing for 10: independent/hands-on validation of these AI insight features' accuracy or usefulness, and more concrete product detail beyond marketing copy.
- [claimed-docs] “Agents monitor your product continuously, surface insights before you think to ask, and tell your team exactly what to do next.”
- [claimed-docs] “Diagnose what changed, and why, in seconds, delivered as a dashboard you and your team can keep working from.”
- [claimed-docs] “Signups dipped 19% last week — a brief pullback against otherwise steady growth.”
- [claimed-docs] “Use AI to spot trends across hundreds of replays and heatmaps, then zoom in yourself to get to the “why” behind every change.”
- [claimed-docs] “Save playlists of replays based on events or properties, and quickly review with AI summaries, making it easy to revisit groups of replays t…”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
- [claimed-docs] “Once connected, you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation …”
ai-native userSet up automations that run autonomously in the background
weight 2 · round to AmplitudeAmplitude offers pieces that resemble autonomous background automation—flag rollout/rollback rules that react to data (amplitude-docs-19), automated in-product guides/surveys triggers (amplitude-docs-14), CI/script-driven experiment management (amplitude-docs-13), and an embeddable agent that 'completes tasks for users' (amplitude-docs-26)—but there is no dedicated, documented automation/workflow builder explicitly for AI-native users to configure and run persistent background automations. missing for 10: a first-party 'automations' or workflow engine with triggers/schedules, independent evidence of autonomous background execution, and confirmation the embedded agent runs unattended rather than on-demand.
- [claimed-docs] “Release behind a flag, ramp to the audience you choose, and turn it off the moment something looks wrong.”
- [claimed-docs] “Launch in-product guides and surveys to onboard, announce, and learn from users.”
- [claimed-docs] “Manage flags, experiments, and deployments from CI, scripts, or your own tools with the Experiment APIs.”
- [claimed-docs] “Embed an agent grounded in your help content and user behavior. Answer questions, guide flows, and complete tasks for users in the moments t…”
Mixpanel markets background 'Agents' that continuously monitor data and surface insights/recommendations autonomously (mixpanel-docs-44), suggesting some agentic automation capability, but there is no documentation on how a user actually configures or triggers these automations, what actions they can take, or independent confirmation they run unattended. missing for 10: setup/configuration docs for these agents, description of trigger conditions and autonomous actions, independent/hands-on evidence of them running in the background.
- [claimed-docs] “Agents monitor your product continuously, surface insights before you think to ask, and tell your team exactly what to do next.”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · round to AmplitudeAmplitude AI lets users ask plain-language questions and get answers across their data, and an embeddable agent can 'answer questions, guide flows, and complete tasks for users,' which shows some built-in delegatable AI assistant capability. However, evidence is limited to marketing docs snippets with no hands-on demonstration of task delegation scope, autonomy limits, or independent corroboration. missing for 10: independent/hands-on verification of the AI assistant actually completing delegated tasks, detail on task scope/autonomy, and confirmation this is available to the analytics-user persona (not just embedded end-user product).
- [claimed-docs] “Ask questions in plain language — Amplitude AI surfaces answers across all your data.”
- [claimed-docs] “Embed an agent grounded in your help content and user behavior. Answer questions, guide flows, and complete tasks for users in the moments t…”
- [claimed-docs] “Start a conversation about your users or product and get expert-level answers in seconds that you can share with your whole org. No SQL, con…”
Mixpanel's marketing copy references built-in AI features such as 'Agents' that 'monitor your product continuously... and tell your team exactly what to do next,' AI-generated replay summaries, and AI trend-spotting across replays/heatmaps, suggesting some in-product AI assistant capability. However, these are thin marketing snippets rather than detailed product docs describing how a user actually delegates multi-step tasks to an assistant, and the only well-documented AI integration (MCP server) is for external AI assistants to query Mixpanel data, not a built-in assistant for users. Missing for 10: detailed first-party documentation of the 'Agents' feature and how tasks are delegated/executed, concrete UI/workflow examples, and independent hands-on corroboration.
- [claimed-docs] “Agents monitor your product continuously, surface insights before you think to ask, and tell your team exactly what to do next.”
- [claimed-docs] “Use AI to spot trends across hundreds of replays and heatmaps, then zoom in yourself to get to the “why” behind every change.”
- [claimed-docs] “Save playlists of replays based on events or properties, and quickly review with AI summaries, making it easy to revisit groups of replays t…”
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
ai-native userOperate the product with natural-language commands
weight 2 · round drawnAmplitude AI lets users 'ask questions in plain language' and get answers across data, and the MCP server brings this natural-language querying into external AI tools (Claude, Cursor, etc.), directly enabling natural-language operation of the product's analytics capabilities. This is corroborated by dedicated docs pages and a probe confirming the MCP server exists. Missing for 10: independent/hands-on verification of natural-language command accuracy and breadth (e.g., can it also configure flags/dashboards via NL, not just query), and no community evidence testing this specific feature.
- [claimed-docs] “Ask questions in plain language — Amplitude AI surfaces answers across all your data.”
- [claimed-docs] “Bring Amplitude’s behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Bring Amplitude's behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Start a conversation about your users or product and get expert-level answers in seconds that you can share with your whole org. No SQL, con…”
- [probe] “official MCP server documented at https://amplitude.com/mcp-server”
Mixpanel documents a hosted MCP server that lets AI assistants query events, funnels, flows, retention, and session replays using natural language without dashboard navigation, directly matching the story of operating the product via NL commands. This is first-party documentation without independent hands-on corroboration of the NL experience itself. Missing for 10: independent/community validation of the MCP natural-language workflow in practice, and detail on the breadth/reliability of commands beyond the listed examples.
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
- [claimed-docs] “Once connected, you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation …”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required”
- [claimed-docs] “Create-Dashboard | Create a new dashboard with text cards and reports”
- [probe] “official MCP server documented at https://docs.mixpanel.com/docs/mcp”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnAmplitudenone0/10Evidence confirms Amplitude has API documentation (HTTP V2 endpoint, Dashboard REST API, various CRUD endpoints) and even an openapi.json spec file, but nothing in the pack shows an interactive reference UI with runnable/try-it examples — no Swagger-style console, no code sandbox, no mention of 'try it' functionality.
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
- [claimed-docs] “Create, get, update, and delete categories, event types, event properties, and user properties.”
- [probe] “PROBE openapi: HTTP 200 at https://amplitude.com/openapi.json — contains "openapi" key”
Mixpanelnone0/10While Mixpanel documents various APIs (Query API, Raw Export API, JQL), there is no evidence of an interactive API reference with runnable/try-it examples; a probe for a standard OpenAPI/Swagger spec returned 404s across all candidate paths, suggesting no interactive API console exists.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.mixpanel.com/openapi.json, https://docs.mixpanel.com/swagger.json, https://docs.mixpane…”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round to AmplitudeA direct probe confirms a machine-readable OpenAPI spec is available at https://amplitude.com/openapi.json (contains an 'openapi' key), and Amplitude also exposes llms.txt/docs.md machine-readable docs. Missing for 10: no independent/third-party corroboration of the spec's completeness or usage examples beyond the probe.
- [probe] “PROBE openapi: HTTP 200 at https://amplitude.com/openapi.json — contains "openapi" key”
- [probe] “PROBE llms.txt: HTTP 200 at https://amplitude.com/llms.txt # Amplitude Amplitude is the AI-native digital analytics platform built for AI b…”
- [probe] “PROBE docs-md: HTTP 200 at https://amplitude.com/docs.md # Amplitude Documentation Official documentation for Amplitude analytics, SDKs, an…”
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
Mixpanelnone0/10Mixpanel has extensive REST/Query/Export APIs, but direct probes for a machine-readable OpenAPI/Swagger spec (openapi.json, swagger.json, etc.) all returned 404, and no documentation evidence points to a downloadable spec file—only human-readable reference docs are cited.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.mixpanel.com/openapi.json, https://docs.mixpanel.com/swagger.json, https://docs.mixpane…”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round drawnAmplitudenone0/10No evidence pack mentions a sandbox, test project, or staging environment separate from production data; docs reference API keys, SDKs, and CI/deployment but nothing about isolating test data from production. Missing for 10: sandbox/test environment documentation, guidance on separating test vs production data, any staging API keys or environment flags.
Mixpanelnone0/10No evidence pack item mentions a sandbox, test/dev project, or any mechanism to let an AI assistant query or write data without touching the production Mixpanel dataset — the MCP server docs (mixpanel-docs-10, mixpanel-docs-19) only describe querying live production data. Missing for 10: any documented sandbox/test project mode, data isolation for AI testing, or non-production environment support.
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
- [probe] “official MCP server documented at https://docs.mixpanel.com/docs/mcp”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnAmplitudenone0/10Evidence shows Amplitude has REST/HTTP APIs (e.g., HTTP V2 endpoint, Dashboard REST API, Experiment APIs) and an OpenAPI spec, confirming APIs exist, but nothing in the pack documents API versioning conventions or a deprecation/sunset policy for these APIs. The axis applies to any API-driven platform, but no evidence supports the specific claim of a documented deprecation policy.
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
- [claimed-docs] “Manage flags, experiments, and deployments from CI, scripts, or your own tools with the Experiment APIs.”
- [probe] “PROBE openapi: HTTP 200 at https://amplitude.com/openapi.json — contains "openapi" key”
Mixpanelnone0/10No evidence pack item mentions API versioning, version numbers, or a documented deprecation/sunset policy for Mixpanel's APIs; the openapi.json probe returned 404s and no versioning docs were found. missing for 10: API version scheme documentation, deprecation/sunset policy, changelog or migration guides for breaking changes.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.mixpanel.com/openapi.json, https://docs.mixpanel.com/swagger.json, https://docs.mixpane…”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round drawnAmplitude's APIs support programmatic mass operations—batch event ingestion via HTTP V2 (docs-2), CRUD across event types/properties (docs-38), cohort list/export/upload (docs-29), and CI/script-driven flag/experiment management (docs-13)—which enable bulk-style automation for AI-native users. However there's no explicit documentation of a dedicated bulk-operations UI or API endpoint for acting on 'many items at once' as a single call. Missing for 10: explicit bulk-update/bulk-delete endpoints, documented batch size limits, and independent evidence of bulk workflows being used successfully.
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
- [claimed-docs] “Create, get, update, and delete categories, event types, event properties, and user properties.”
- [claimed-docs] “List all your cohorts in Amplitude, export a cohort in Amplitude, or upload a cohort.”
- [claimed-docs] “Manage flags, experiments, and deployments from CI, scripts, or your own tools with the Experiment APIs.”
- [claimed-docs] “Export your project's event data.”
Mixpanel offers APIs (Raw Event Export, JQL, Query API) that allow programmatic, high-volume data operations (bulk export, custom scripted queries, bulk filtering/cohort retrieval), and its MCP server lets AI agents query events, funnels, retention, etc. via natural language. However, the MCP tool list only shows single-item actions like 'Create-Dashboard', with no documented bulk-operation tools (e.g., batch create/update/delete across many items) for an AI-native agent to invoke. missing for 10: explicit AI-agent-invocable bulk operation tools (batch create/update/delete), independent confirmation of bulk actions succeeding at scale via MCP or API in practice.
- [claimed-docs] “To export raw event data, use the Raw Event Export API. The events payload is returned as JSONL”
- [claimed-docs] “To export raw event data, use the Raw Event Export API... This method is recommended for high volume exports.”
- [claimed-docs] “Create custom queries using Javascript code”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The JQL JavaScript code must define a main() function. This JavaScript will be compiled, and the main() function will be executed by the que…”
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [claimed-docs] “Create-Dashboard | Create a new dashboard with text cards and reports”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round drawnAmplitudenone0/10The evidence pack covers analytics, experimentation, session replay, and AI-query features, but contains no documentation of a rules/automation engine that triggers actions automatically based on events (e.g., event-triggered webhooks, workflows, or conditional messaging). Feature flags/experiments and in-product guides are mentioned but only as manually configured campaigns, not as an automatic event-triggered rules system.
- [claimed-docs] “Launch in-product guides and surveys to onboard, announce, and learn from users.”
- [claimed-docs] “Release behind a flag, ramp to the audience you choose, and turn it off the moment something looks wrong.”
- [claimed-docs] “Roll out behind a flag and run A/B tests so the data, not a hunch, drives the release.”
Mixpanelnone0/10Mixpanel's evidence covers analytics reporting, exports, session replay, and an MCP server for querying data with natural language, but there is no documented feature for defining rule-based triggers that automatically fire actions on events (e.g., alerts, webhooks, workflow automation). The closest mention, 'Agents monitor your product continuously... tell your team what to do next' (mixpanel-docs-44), describes passive AI insight surfacing, not user-defined event-triggered automation.
- [claimed-docs] “Agents monitor your product continuously, surface insights before you think to ask, and tell your team exactly what to do next.”
- [claimed-docs] “Use the Feature Flags API to assign your users to variants for experiments, rollout, and releases.”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
ai-native userSchedule recurring jobs or workflows
weight 2 · round drawnAmplitudenone0/10Amplitude is an analytics/experimentation platform; nothing in the evidence pack references scheduling recurring jobs, workflows, or automated cron-like tasks. There's no mention of scheduled reports, recurring data exports, or workflow automation triggers.
Mixpanelnone0/10Mixpanel is an analytics platform with export APIs, MCP server for queries, and dashboards, but there is no evidence of any capability to schedule recurring jobs or workflows (e.g., scheduled reports, cron-like automation, recurring data syncs beyond warehouse mirroring). No scheduling/automation feature is documented anywhere in the evidence pack.
Dashboards reporting — stories about dashboards reporting in this arenaDashboards reporting
Stories about dashboards reporting in this arena
Ai analytics ops
ai-native userHave an agent build a dashboard of my key metrics end-to-end via the API or MCP server
weight 3 · round to MixpanelAmplitude has documented APIs (Dashboard REST API for reading chart data, event/property CRUD, cohort APIs) and an official MCP server that lets AI tools like Claude/Cursor query Amplitude data in plain language, which covers agent-driven analytics querying. However, evidence shows the Dashboard API is read-only (fetching JSON of existing dashboard graphs) and 'Create a dashboard' appears to be a UI action, not an API/MCP-exposed operation, so there's no documented end-to-end agent workflow that programmatically builds a new dashboard from scratch via API or MCP. Missing for 10: an API/MCP endpoint or example specifically for creating/configuring dashboards (not just querying data), and any hands-on/community proof of an agent successfully building a dashboard end-to-end.
- [claimed-docs] “Bring Amplitude’s behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Bring Amplitude's behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Start a conversation about your users or product and get expert-level answers in seconds that you can share with your whole org. No SQL, con…”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
- [claimed-docs] “Create a dashboard”
- [claimed-docs] “Create, get, update, and delete categories, event types, event properties, and user properties.”
- [probe] “official MCP server documented at https://amplitude.com/mcp-server”
Mixpanel provides a documented hosted MCP server explicitly with a Create-Dashboard action ('Create a new dashboard with text cards and reports') plus natural-language querying of events, funnels, flows, retention, session replays, complementing the Query/Export APIs an agent could also use end-to-end. This directly matches the story of an agent building a dashboard via MCP or API, though there's no independent hands-on report confirming a real agent successfully completed this end-to-end. missing for 10: independent/hands-on verification of an agent actually building a dashboard via MCP or API, and more detail on API-only dashboard creation outside MCP.
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [claimed-docs] “Create-Dashboard | Create a new dashboard with text cards and reports”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
- [claimed-docs] “Once connected, you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation …”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required”
- [probe] “official MCP server documented at https://docs.mixpanel.com/docs/mcp”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
Dashboards
developerCreate and manage dashboards and saved insights programmatically via the API
weight 2 · round drawnThe docs mention a Dashboard REST API that returns JSON data for existing dashboard graphs and a general 'Create a dashboard' doc, plus API endpoints for cohorts and event/property metadata, but none of these clearly describe full CRUD (create/update/delete) for dashboards or 'saved insights' via API—only read/export access is explicitly documented. Missing for 10: explicit API endpoints for creating/updating/deleting dashboards, a documented 'saved insights' API object, and independent confirmation that these operations work programmatically.
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
- [claimed-docs] “Create a dashboard”
- [claimed-docs] “List all your cohorts in Amplitude, export a cohort in Amplitude, or upload a cohort.”
- [claimed-docs] “Create, get, update, and delete categories, event types, event properties, and user properties.”
Mixpanel's hosted MCP server exposes a 'Create-Dashboard' tool that can create dashboards with text cards and reports, and lets you query events/funnels/retention/etc. programmatically, but this is MCP-tool access for AI assistants, not a documented general-purpose REST API for full CRUD (create/update/delete) of dashboards and saved insights. The Query API (docs-39, docs-43) only supports reading/filtering data, not creating or managing dashboard objects. Missing for 10: explicit REST API endpoints for creating, updating, and deleting dashboards/saved insights outside the MCP context, and any evidence of programmatic dashboard sharing/permissions management.
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [claimed-docs] “Create-Dashboard | Create a new dashboard with text cards and reports”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
- [probe] “official MCP server documented at https://docs.mixpanel.com/docs/mcp”
product-managerCompose saved insights into shareable dashboards for my team
weight 2 · round to AmplitudeDocs confirm core dashboard-building capabilities: creating dashboards, charts/insights, funnels/segments, and organizing dashboards/notebooks into team-owned spaces for collaboration, which directly supports composing insights into shareable team dashboards. Missing for 10: explicit documentation of sharing/permissions mechanics (e.g., link sharing, access controls) and independent/hands-on confirmation of the dashboard-building workflow beyond first-party docs.
- [claimed-docs] “Charts, dashboards, and insights. Build funnels, segment users, answer "why".”
- [claimed-docs] “Create a dashboard”
- [claimed-docs] “Use spaces to organize charts, dashboards, and notebooks for the team that owns the work.”
- [claimed-docs] “Build funnels, segment users, answer "why".”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
Mixpanel's marketing/docs reference dashboards as an output (e.g. MCP tool 'Create-Dashboard' with text cards and reports, and marketing copy about 'a dashboard you and your team can keep working from'), implying saved insights can be composed into shareable team dashboards. However there is no first-party documentation detailing the dashboard-building UI, sharing/permissions workflow, or team collaboration mechanics beyond these brief mentions. Missing for 10: dedicated dashboard-creation docs, sharing/permission controls, and independent/hands-on evidence of team usage.
- [claimed-docs] “Create-Dashboard | Create a new dashboard with text cards and reports”
- [claimed-docs] “Diagnose what changed, and why, in seconds, delivered as a dashboard you and your team can keep working from.”
- [claimed-docs] “Connect your product data to your company strategy in one interactive view, so teams can align faster, act with confidence, and stay account…”
product-managerSubscribe to dashboards and get alerted when a key metric moves abnormally
weight 1 · round to MixpanelAmplitudenone0/10The evidence pack covers dashboards, charts, cohorts, and a Dashboard REST API, but contains no mention of subscribing to dashboards or receiving alerts when a metric moves abnormally (e.g., anomaly detection, alerting rules, or subscription/notification features). missing for 10: evidence of dashboard subscription feature, evidence of anomaly/alert detection on metrics, evidence of notification delivery (email/Slack) tied to metric changes.
- [claimed-docs] “Create a dashboard”
- [claimed-docs] “Use spaces to organize charts, dashboards, and notebooks for the team that owns the work.”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
Evidence shows Mixpanel has AI 'Agents' that monitor products continuously and surface anomalies (e.g., 'Signups dipped 19% last week') and can 'tell your team exactly what to do next,' suggesting some proactive alerting capability, but there is no explicit documentation of a dashboard subscription mechanism or configurable alert thresholds for PMs. missing for 10: explicit dashboard subscription feature, alert configuration/thresholds, delivery channels (email/Slack) for alerts.
- [claimed-docs] “Agents monitor your product continuously, surface insights before you think to ask, and tell your team exactly what to do next.”
- [claimed-docs] “Signups dipped 19% last week — a brief pullback against otherwise steady growth.”
- [claimed-docs] “Diagnose what changed, and why, in seconds, delivered as a dashboard you and your team can keep working from.”
Event ingestion — stories about event ingestion in this arenaEvent ingestion
Stories about event ingestion in this arena
Identity
developerBulk-import historical events from another analytics tool or a data export into the platform
weight 1 · round to MixpanelAmplitude's HTTP V2 API for sending events from a server (amplitude-docs-2) and third-party ETL/integration connectors like Fivetran and RudderStack (amplitude-intdir-5, amplitude-intdir-12) suggest a path for importing external or historical event data, but there is no explicit documentation of a dedicated bulk/batch historical import tool or backfill process. Missing for 10: explicit bulk-import/backfill API documentation, guidance on importing from competitor tools (e.g., Mixpanel/Segment exports), and any hands-on confirmation of successful historical data migration.
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
- [claimed-docs] “Fivetran is the leader in automated data integration, delivering ready-to-use connectors that adapt to change.”
- [claimed-docs] “RudderStack provides Event streaming, ETL Pipelines, and Reverse ETL Pipelines that allow customers to have a 360 degree actionable view of …”
- [claimed-docs] “Export your project's event data.”
Mixpanel supports ingesting historical/bulk data via Warehouse Connectors (syncing from Snowflake, BigQuery, Databricks, Redshift, Postgres) and has a general import/ingestion API implied by its event JSON model, but the evidence pack lacks explicit documentation of a dedicated bulk 'Import API' for historical event backfill or migration from another analytics tool. missing for 10: explicit historical/bulk import API docs, guidance on importing from other analytics platforms (e.g., Amplitude/Segment migration), and confirmation of backdated event timestamp support.
- [claimed-docs] “you can sync data from data warehouses like Snowflake, BigQuery, Databricks, Redshift, and Postgres to Mixpanel”
- [claimed-docs] “With Warehouse Connectors, you can sync data from data warehouses like Snowflake, BigQuery, Databricks, Redshift, and Postgres to Mixpanel.”
- [claimed-docs] “you can sync data from data warehouses like Snowflake, BigQuery, Databricks, Redshift, and Postgres to Mixpanel.”
- [claimed-docs] “SET CHANGE\TRACKING = TRUE; # or ALTER VIEW SET CHANGE\TRACKING = TRUE; \\\ Mixpanel will create and manage the necessary STREAM objects in …”
- [claimed-docs] “SET TBLPROPERTIES (delta.enableChangeDataFeed = true); \\\ \\Considerations when using Mirror with Databricks:\\ \ Mirror is not supported o…”
- [claimed-docs] “Every data point sent to Mixpanel is stored as JSON in our data store.”
developerIdentify users across devices and merge anonymous pre-signup activity into their identified profile
weight 2 · round to AmplitudeThe only relevant evidence is a generic identity-model claim ('Every SDK shares the same identity, event, and consent model, so a user instrumented once is understood everywhere'), which gestures at cross-context identity resolution but never explicitly describes cross-device merging or converting anonymous pre-signup activity into an identified user profile. missing for 10: explicit documentation of user/device ID merge behavior, identify API or alias mechanism for anonymous-to-known user transition, and any independent confirmation of cross-device identity stitching.
- [claimed-docs] “Every SDK shares the same identity, event, and consent model, so a user instrumented once is understood everywhere.”
Mixpanelnone0/10The evidence pack contains no mention of Mixpanel's identity resolution features (e.g., identify(), $merge, alias, distinct_id merging) for tying anonymous pre-signup activity to an identified user profile. While this is a well-known Mixpanel capability generally, none of the provided docs, community posts, or probes reference it, so per the evidence-only rule this must be judged as unsupported.
Instrumentation
product-managerCapture clicks, pageviews, and form interactions automatically without writing tracking code for each event
weight 2 · round to AmplitudeAmplitude's docs claim built-in autocapture ('Auto-instrument page views, clicks, and sessions out of the box' and a Wizard CLI that 'instruments your codebase' in one command), directly matching the story of code-free automatic event capture. However, all support is first-party marketing/docs with no independent hands-on confirmation of the autocapture specifically, and community feedback (amplitude-comm-3) reports that real-world integration is 'very hard' and data is 'often wrong,' introducing real-world doubt about the ease claimed. Missing for 10: independent/hands-on verification that autocapture works as advertised without extra config, and resolution of the community-reported integration difficulty.
- [claimed-docs] “Auto-instrument page views, clicks, and sessions out of the box.”
- [claimed-docs] “Run the Wizard CLI to instrument your codebase and get a first dashboard from one terminal command.”
- [claimed-docs] “The wizard detects your framework, installs the right SDK, adds your API key, and fires a test event.”
- [community] “Amplitude is NOT a great product when you actually try to implement it. I have seen at least 3 companies pay for them... only to realize tha…”
Mixpanelnone0/10The evidence pack covers Mixpanel's event model, funnels, retention, exports, warehouse connectors, and session replay, but nowhere documents an autocapture/autotrack feature that automatically records clicks, pageviews, and form interactions without custom instrumentation. Session Replay (mixpanel-docs-4/23/28/33) records video sessions but does not equate to automatic event-level tracking for analytics events.
developerSend events from web, mobile, and backend apps through official SDKs for the major languages and platforms
weight 3 · round to AmplitudeDocs explicitly describe a unified SDK model spanting web, mobile, server, and game engines with a shared identity/event model (amplitude-docs-36, amplitude-docs-16, amplitude-docs-37), plus a direct HTTP V2 endpoint for backend/server-side event sending (amplitude-docs-2) and auto-instrumentation for web/mobile (amplitude-docs-4). This covers the core of the story across major platforms via official SDKs. Missing for 10: an explicit enumerated list of supported languages/platforms and independent (non-vendor) confirmation of SDK reliability/breadth beyond docs claims.
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
- [claimed-docs] “Auto-instrument page views, clicks, and sessions out of the box.”
- [claimed-docs] “Every SDK shares the same identity, event, and consent model, so a user instrumented once is understood everywhere.”
- [claimed-docs] “Use Experiment SDKs to flag and test features across web, mobile, and backend.”
- [claimed-docs] “Use the same API surface across web, mobile, server, and game engines.”
- [claimed-docs] “Unified SDKs provide one integration point for multiple Amplitude products.”
Mixpanelnone0/10The evidence pack never mentions any specific SDKs (JavaScript, iOS, Android, Python, Java, etc.) for sending events from web, mobile, or backend apps — it only discusses tracking methods generically (warehouse connectors, event ingestion server rewrite, export APIs) without documenting an SDK catalog. Missing for 10: explicit mention of official web SDK, mobile SDKs (iOS/Android), backend/server SDKs, and any platform coverage list.
- [community] “Mixpanel rewrote their event-ingestion server from Erlang to Python after two years, citing that 'the code has become difficult to maintain.…”
- [claimed-docs] “Every data point sent to Mixpanel is stored as JSON in our data store.”
- [claimed-docs] “SET CHANGE\TRACKING = TRUE; # or ALTER VIEW SET CHANGE\TRACKING = TRUE; \\\ Mixpanel will create and manage the necessary STREAM objects in …”
Flags experiments — stories about flags experiments in this arenaFlags experiments
Stories about flags experiments in this arena
Ai analytics ops
ai-native userHave an agent create, target, and toggle feature flags through the API or MCP server
weight 2 · round to AmplitudeAmplitude's Experiment APIs let scripts/CI/tools create, target, and toggle flags (amplitude-docs-13, amplitude-docs-19, amplitude-docs-28, amplitude-docs-42), so an agent could drive flag lifecycle via API. However, the documented MCP server (amplitude-docs-11/17/34) is scoped to bringing analytics/behavioral context into AI tools for Q&A, not to flag creation/toggling, so the 'MCP server' half of the story is unsupported. Missing for 10: MCP-server tools for flag CRUD/targeting/toggling, explicit agent-oriented examples of calling the Experiment API, and any independent verification of this workflow.
- [claimed-docs] “Manage flags, experiments, and deployments from CI, scripts, or your own tools with the Experiment APIs.”
- [claimed-docs] “Release behind a flag, ramp to the audience you choose, and turn it off the moment something looks wrong.”
- [claimed-docs] “Roll out behind a flag and run A/B tests so the data, not a hunch, drives the release.”
- [claimed-docs] “Define the audience to target users by properties, cohorts, or segments.”
- [claimed-docs] “Bring Amplitude’s behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Start a conversation about your users or product and get expert-level answers in seconds that you can share with your whole org. No SQL, con…”
Mixpanel documents a Feature Flags API for assigning users to variants for experiments, rollout, and releases (mixpanel-docs-36), and a separate hosted MCP server for natural-language querying and dashboard creation (mixpanel-docs-10/11/19). However, there is no evidence that flag creation or toggling is exposed via the MCP server, nor documentation confirming full CRUD (create/toggle) operations through the API beyond targeting/assignment. missing for 10: MCP tool list including feature-flag creation/toggle, explicit API endpoints for creating and toggling flags, and any hands-on confirmation of agent-driven flag management.
- [claimed-docs] “Use the Feature Flags API to assign your users to variants for experiments, rollout, and releases.”
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [claimed-docs] “Create-Dashboard | Create a new dashboard with text cards and reports”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
Experimentation
product-managerRun A/B experiments with goal metrics and statistical significance reported on the results
weight 3 · round to AmplitudeAmplitude's Feature Experiment product explicitly supports A/B, multivariate, and bandit testing with audience targeting, causal lift measurement, and visual editing to ship variants, plus Experiment APIs for programmatic control (docs-25, docs-41, docs-42, docs-13, docs-15, docs-28). Statistical significance/causal lift is referenced ('measure causal lift') but no deep documentation of the specific stats methodology or independent hands-on validation of experiment result reporting is present. Missing for 10: detailed documentation of significance testing methodology/confidence intervals, and independent/hands-on corroboration of the experiment results UI.
- [claimed-docs] “Ship with confidence. Test variants, target segments, measure causal lift.”
- [claimed-docs] “Run A/B, multivariate, and bandit tests so the data, not the loudest opinion, makes the call.”
- [claimed-docs] “Define the audience to target users by properties, cohorts, or segments.”
- [claimed-docs] “Manage flags, experiments, and deployments from CI, scripts, or your own tools with the Experiment APIs.”
- [claimed-docs] “Edit pages with a visual editor and ship the winning variant without a code deploy.”
- [claimed-docs] “Roll out behind a flag and run A/B tests so the data, not a hunch, drives the release.”
- [claimed-docs] “Use Experiment SDKs to flag and test features across web, mobile, and backend.”
- [claimed-docs] “Release behind a flag, ramp to the audience you choose, and turn it off the moment something looks wrong.”
Mixpanel documents a Feature Flags API for assigning variants to experiments/rollouts and marketing copy claims users can 'design, run, and analyze tests and rollouts' alongside analytics, indicating some experimentation capability. However, there is no explicit documentation of goal-metric configuration or statistical-significance calculation/reporting for A/B tests. missing for 10: explicit stat-sig reporting docs, goal-metric setup workflow, independent corroboration of experiment results UI.
- [claimed-docs] “Close the gap between shipping fast and shipping right by designing, running, and analyzing tests and rollouts in the same place you do all …”
- [claimed-docs] “Use the Feature Flags API to assign your users to variants for experiments, rollout, and releases.”
Feature flags
developerRoll out features behind flags targeted by user properties, cohorts, and percentage rollouts
weight 3 · round to AmplitudeAmplitude's Experiment/Feature Flags docs explicitly cover targeting by user properties, cohorts, and segments (docs-42), percentage-based ramping/rollouts (docs-19, docs-28), local evaluation for low latency (docs-3), and SDKs across web/mobile/server (docs-35), directly matching the story. missing for 10: independent/hands-on verification of flag targeting behavior beyond vendor docs, and no community evidence specifically confirming the flagging/rollout feature works as described.
- [claimed-docs] “Evaluate flags locally for low-latency rollout decisions.”
- [claimed-docs] “Release behind a flag, ramp to the audience you choose, and turn it off the moment something looks wrong.”
- [claimed-docs] “Roll out behind a flag and run A/B tests so the data, not a hunch, drives the release.”
- [claimed-docs] “Use Experiment SDKs to flag and test features across web, mobile, and backend.”
- [claimed-docs] “Define the audience to target users by properties, cohorts, or segments.”
- [claimed-docs] “Manage flags, experiments, and deployments from CI, scripts, or your own tools with the Experiment APIs.”
Mixpanel documents a Feature Flags API for assigning users to variants for experiments, rollouts, and releases, and separately supports cohort creation via user properties, plus a marketing page claiming tests/rollouts are done in the same place as analytics. However, there's no documentation showing explicit percentage-rollout controls or cohort/property-based targeting rules specifically applied to feature flags. missing for 10: detailed docs on percentage-based rollout configuration, explicit flag targeting rules by user property/cohort, and any independent/hands-on confirmation of the flagging workflow.
- [claimed-docs] “Use the Feature Flags API to assign your users to variants for experiments, rollout, and releases.”
- [claimed-docs] “Properties allow you to create groups of users (aka cohorts) and also enable you to filter for certain events or users.”
- [claimed-docs] “Close the gap between shipping fast and shipping right by designing, running, and analyzing tests and rollouts in the same place you do all …”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
Funnels retention — stories about funnels retention in this arenaFunnels retention
Stories about funnels retention in this arena
Behavioral analysis
product-managerBuild multi-step funnels with filters and breakdowns to find where users drop off
weight 3 · round to MixpanelDocs explicitly reference building funnels, segmenting users, and answering 'why' via charts/dashboards, which is core to this story, but there is no direct documentation detail on breakdown/filter mechanics within a funnel, nor independent hands-on validation of funnel accuracy — one community comment even flags data accuracy concerns during implementation. missing for 10: explicit funnel filter/breakdown feature documentation, independent hands-on corroboration of funnel drop-off analysis working as described.
- [claimed-docs] “Build funnels, segment users, answer "why".”
- [claimed-docs] “Charts, dashboards, and insights. Build funnels, segment users, answer "why".”
- [claimed-docs] “Create user cohorts”
- [community] “Amplitude is NOT a great product when you actually try to implement it. I have seen at least 3 companies pay for them... only to realize tha…”
Mixpanel's funnels report explicitly measures conversion through a series of events, properties enable filtering/cohorting, and the Query API supports breakdowns and filtering of events; marketing copy also touts discovering 'critical drop-off points.' Missing for 10: a hands-on community example specifically confirming breakdown usage inside funnels (only adjacent Flow feature has direct user testimonials) and independent verification of the breakdown UI itself.
- [claimed-docs] “Measure conversions through a series of events”
- [claimed-docs] “Properties allow you to create groups of users (aka cohorts) and also enable you to filter for certain events or users.”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “Discover critical drop-off points and calibrate user journeys to optimize every touchpoint for maximum engagement, retention, and growth.”
- [community] “On Mixpanel's Flow feature: 'This is the coolest thing I have seen in Analytics for a long time. This really gets to the heart of most of th…”
product-managerAnalyze behavior at the account or company level, not just per user, for B2B products
weight 2 · round to MixpanelAmplitudenone0/10The evidence pack covers user-level analytics, cohorts, funnels, session replay, and experimentation, but contains no mention of account-level or company/group-based analytics (e.g., grouping events by an 'account' or 'company' entity) that B2B PMs need. Since account-based analysis is a plausible and common analytics-platform capability, its absence here is 'none' rather than 'na'.
- [claimed-docs] “Build funnels, segment users, answer "why".”
- [claimed-docs] “Create user cohorts”
- [claimed-docs] “Charts, dashboards, and insights. Build funnels, segment users, answer "why".”
- [claimed-docs] “Define the audience to target users by properties, cohorts, or segments.”
Mixpanel does support account/company-level (group) analytics for B2B use cases, but the evidence pack lacks first-party docs describing this feature directly — the only confirmation comes from a community report noting that group-level/account-level analytics 'useful for B2B SaaS' exists but is 'tucked away on one of Mixpanel's higher price plans,' limiting access for many customers. Missing for 10: first-party documentation of the Group Analytics feature, details on setup/limits, and independent hands-on validation beyond the pricing-gate complaint.
- [community] “Group-level/account-level analytics useful for B2B SaaS is 'tucked away on one of Mixpanel's higher price plans,' limiting access for smalle…”
product-managerMeasure retention over time and slice it by behavioral cohorts
weight 3 · round drawnAmplitude is core analytics product with explicit retention/behavioral-cohort features: funnels/segmentation ('Build funnels, segment users, answer why'), user cohort creation, and cohort management APIs (list/export/upload). This directly supports retention-over-time analysis sliced by behavioral cohorts, which is a first-class Amplitude use case. Missing for 10: no explicit named 'Retention chart' documentation excerpt and no independent/hands-on report specifically confirming retention-by-cohort workflow quality.
- [claimed-docs] “Build funnels, segment users, answer "why".”
- [claimed-docs] “Create user cohorts”
- [claimed-docs] “Charts, dashboards, and insights. Build funnels, segment users, answer "why".”
- [claimed-docs] “List all your cohorts in Amplitude, export a cohort in Amplitude, or upload a cohort.”
- [claimed-docs] “Create a dashboard”
Mixpanel's Retention report explicitly assesses user engagement over time, and properties/cohorts allow filtering and grouping of users for behavioral segmentation, directly matching the story. Documentation confirms retention analysis, cohort creation via properties, and query/export APIs that support slicing by cohort. Missing for 10: independent hands-on evidence of retention-by-cohort slicing in practice and no community corroboration specifically on retention feature usability.
- [claimed-docs] “The Retention report in Mixpanel is designed to assess user engagement over a specified period of time.”
- [claimed-docs] “Properties allow you to create groups of users (aka cohorts) and also enable you to filter for certain events or users.”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
product-managerExplore the paths users actually take before or after a key event
weight 2 · round to MixpanelAmplitude's docs confirm funnel building and segmentation ('Build funnels, segment users, answer why') and session replay lets PMs see exact user sessions tied to funnel drop-off points, which supports exploring behavior around key events. However, no explicit mention of a dedicated Path/Journey analysis feature (e.g., Amplitude's Pathfinder/User Paths chart type) appears in the evidence, so the story's specific 'paths before/after an event' capability is only indirectly covered via funnels + session replay. Missing for 10: explicit path/journey analysis chart documentation, independent user validation of path exploration accuracy.
- [claimed-docs] “Build funnels, segment users, answer "why".”
- [claimed-docs] “Charts, dashboards, and insights. Build funnels, segment users, answer "why".”
- [claimed-docs] “Go from a chart, funnel, or support ticket to the exact session.”
- [claimed-docs] “Watch real user sessions to understand the why behind your metrics. Spot frustration, tie errors to UX, and measure engagement across every …”
- [claimed-docs] “See exactly what users experience. Replay sessions with full event and network context.”
Mixpanel's 'Flows' report is explicitly designed to visualize the paths users take before/after a given event, and this is corroborated by independent hands-on community praise ('gets to the heart of most of the information you need on user flows') as well as the docs' framing around discovering drop-off points and calibrating user journeys. However, community feedback also flags real friction (hover-based UI hard to use, requests to unify page-view vs custom events) and no probe confirms path exploration works flawlessly today. missing for 10: recent independent corroboration of UI usability, explicit documentation of 'before event' path exploration (most docs emphasize funnels/retention/after-event flows).
- [claimed-docs] “Discover critical drop-off points and calibrate user journeys to optimize every touchpoint for maximum engagement, retention, and growth.”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
- [community] “On Mixpanel's Flow feature: 'This is the coolest thing I have seen in Analytics for a long time. This really gets to the heart of most of th…”
- [community] “Paying Mixpanel customer feedback on Flow: 'My biggest request is the same as the biggest request for the rest of Mixpanel: remove the wall …”
- [community] “Usability complaints about Flow's hover-based UI: 'Requires too much effort (rolling over hover states) to dig out information' and is 'Very…”
Self serve insights
founderAnswer "which activation step loses users?" myself with funnels and drop-off analysis, without an analyst
weight 3 · round drawnAmplitude's core product is explicitly built around funnels, segmentation, and self-serve answers ('Build funnels, segment users, answer why') plus an AI assistant that answers plain-language questions without SQL or a data team, directly matching the founder self-serve need. Community evidence corroborates ease of adoption across non-analyst roles, though one review notes implementation/data-quality difficulties. Missing for 10: independent hands-on validation specifically of funnel/drop-off analysis accuracy and ease for a non-analyst founder persona.
- [claimed-docs] “Build funnels, segment users, answer "why".”
- [claimed-docs] “Charts, dashboards, and insights. Build funnels, segment users, answer "why".”
- [claimed-docs] “Ask questions in plain language — Amplitude AI surfaces answers across all your data.”
- [claimed-docs] “Start a conversation about your users or product and get expert-level answers in seconds that you can share with your whole org. No SQL, con…”
- [community] “We got on for the generous free plan, stayed for the simple UX. Amplitude has been the only analytics product that we've gotten the whole te…”
- [community] “Amplitude is NOT a great product when you actually try to implement it. I have seen at least 3 companies pay for them... only to realize tha…”
Mixpanel's core product is self-serve funnels ('measure conversions through a series of events') and drop-off analysis, with docs explicitly framing this as a no-code, click-based, self-serve capability ('interactive reports that let you query and visualize the results with just a few clicks', 'Discover critical drop-off points'), plus Session Replay to explore 'why customers drop off' — all designed for non-analyst self-service. Community evidence corroborates the self-serve UI power ('PMs spend hours in mixpanel quickly generating custom report after custom report') though also notes UI friction in an older Flow feature. Missing for 10: no first-party founder/non-technical-user case study specifically validating a founder (not a PM/analyst) independently diagnosing activation drop-off, and one community note flags usability complaints on a related flow visualization.
- [claimed-docs] “Measure conversions through a series of events”
- [claimed-docs] “Discover critical drop-off points and calibrate user journeys to optimize every touchpoint for maximum engagement, retention, and growth.”
- [claimed-docs] “It enables you to track how users engage with your product and analyze this data with interactive reports that let you query and visualize t…”
- [claimed-docs] “Now, with Mixpanel Session Replay, you can quickly follow up by exploring **why** customers drop off.”
- [community] “'Getting a front end anywhere near as slick and powerful as Mixpanel's is an even bigger challenge than the data pipeline... I've seen PMs s…”
- [community] “On Mixpanel's Flow feature: 'This is the coolest thing I have seen in Analytics for a long time. This really gets to the heart of most of th…”
- [community] “Usability complaints about Flow's hover-based UI: 'Requires too much effort (rolling over hover states) to dig out information' and is 'Very…”
Integrations — connecting to other tools — breadth and depth of built-in integrationsIntegrations
Connecting to other tools — breadth and depth of built-in integrations
Destinations
developerStream events onward to third-party destinations (CRM, ad platforms, webhooks) from the analytics platform
weight 2 · round to AmplitudeAmplitudedisputedcontradicted6/10Amplitude's integrations directory documents connections to CRM (Salesforce, HubSpot), reverse-ETL/streaming tools (Hightouch, RudderStack, Segment), and states it can integrate with any system, supporting onward event streaming to third-party destinations. However, a hands-on community report explicitly states 'they dont do a good job in sending data back to things like SFDC,' directly contradicting the claimed CRM-sync capability for at least one major destination. Missing for 10: independent corroboration of reliable webhook/ad-platform streaming, and resolution of the documented SFDC sync complaint.
- [claimed-docs] “Connect the leading data and marketing platforms to Amplitude. Don’t see what you need? We can integrate with any system.”
- [claimed-docs] “Hightouch syncs data from data warehouses into tools that teams rely on to run the business (CRM, email, ERP, etc.”
- [claimed-docs] “Salesforce provides customer relationship management software and applications focused on sales, customer service, marketing automation, ana…”
- [claimed-docs] “RudderStack provides Event streaming, ETL Pipelines, and Reverse ETL Pipelines that allow customers to have a 360 degree actionable view of …”
- [community] “Amplitude is NOT a great product when you actually try to implement it. I have seen at least 3 companies pay for them... only to realize tha…”
Mixpanelnone0/10Evidence shows only inbound warehouse syncing (Snowflake/BigQuery/etc. into Mixpanel) and pull-based export APIs (Raw Event Export, Query API, CSV), none of which describe pushing/streaming events out to CRM systems, ad platforms, or webhooks. The partner integrations directory lists third-party platforms (mParticle, Census, Vendo) but their descriptions frame them as bringing data INTO Mixpanel, not Mixpanel streaming events out to them.
- [claimed-docs] “you can sync data from data warehouses like Snowflake, BigQuery, Databricks, Redshift, and Postgres to Mixpanel”
- [claimed-docs] “To export raw event data, use the Raw Event Export API. The events payload is returned as JSONL”
- [claimed-docs] “With Warehouse Connectors, you can sync data from data warehouses like Snowflake, BigQuery, Databricks, Redshift, and Postgres to Mixpanel.”
- [claimed-docs] “You can export data from your project using a variety of methods, including lightweight CSVs generated from the UI, large volume raw data ex…”
- [claimed-docs] “mParticle is an AI Customer Data Platform that powers the entire marketing stack with real-time customer data. Companies like NBCUniversal, …”
- [claimed-docs] “Census is the #1 Data Activation platform that unites revenue teams around a single source of truth in the cloud data warehouse. Our no-code…”
- [claimed-docs] “Vendo is a marketing data platform for growth teams. It connects the tools you already use — Shopify, Google Ads, Meta Ads, TikTok, Klaviyo,…”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round drawnAmplitude exposes broad APIs (HTTP V2 event ingestion, Dashboard REST API, Experiment APIs for flags/experiments, Cohorts API, taxonomy CRUD, data export) that mirror many core UI workflows, and an official MCP server lets AI tools query behavioral data. However, several flagship UI capabilities—session replay viewing/analysis, the visual A/B test editor, in-product guides/surveys creation, dashboard/chart creation, and the AI 'ask in plain language' analyst—are documented only as UI features with no corresponding API endpoints in the evidence pack. Missing for 10: API parity for session replay, visual experiment editor, dashboard/notebook creation, and guides/surveys authoring; independent confirmation that API coverage matches UI feature-for-feature.
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
- [claimed-docs] “Export your project's event data.”
- [claimed-docs] “Manage flags, experiments, and deployments from CI, scripts, or your own tools with the Experiment APIs.”
- [claimed-docs] “List all your cohorts in Amplitude, export a cohort in Amplitude, or upload a cohort.”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
- [claimed-docs] “Create, get, update, and delete categories, event types, event properties, and user properties.”
- [claimed-docs] “Watch real user sessions to understand the why behind your metrics.”
- [claimed-docs] “Edit pages with a visual editor and ship the winning variant without a code deploy.”
- [claimed-docs] “Create a dashboard”
- [probe] “PROBE openapi: HTTP 200 at https://amplitude.com/openapi.json — contains "openapi" key”
- [probe] “official MCP server documented at https://amplitude.com/mcp-server”
Mixpanel exposes a broad API surface—Query API (mirrors report results), Raw Export/Engage APIs, Feature Flags API, JQL for custom queries, and an MCP server that can create dashboards and query via natural language—giving strong programmatic coverage of most analytics workflows (mixpanel-docs-39, mixpanel-docs-43, mixpanel-docs-36, mixpanel-docs-37, mixpanel-docs-10, mixpanel-docs-11). However, some UI-only capabilities are explicitly documented, e.g. Session Replay sampling is managed 'directly from the Mixpanel UI — no code changes or redeployment required' (mixpanel-docs-5, mixpanel-docs-29), indicating a gap rather than full UI/API parity. Missing for 10: explicit confirmation that dashboard configuration, session-replay settings, and all admin/UI-only controls are fully API-accessible, and independent verification of complete parity beyond vendor docs.
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
- [claimed-docs] “Use the Feature Flags API to assign your users to variants for experiments, rollout, and releases.”
- [claimed-docs] “Create custom queries using Javascript code”
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [claimed-docs] “Create-Dashboard | Create a new dashboard with text cards and reports”
- [claimed-docs] “teams can manage Session Replay sampling directly from the Mixpanel UI — no code changes or redeployment required.”
- [claimed-docs] “teams can manage Session Replay sampling directly from the Mixpanel UI — no code changes or redeployment required”
ai-native userExport all of my data in open formats and leave
weight 3 · round to MixpanelAmplitude provides data export mechanisms (event data export, Dashboard REST API returning JSON, cohort export/upload) via documented APIs, which support portability of raw data. However, there's no evidence of a comprehensive full-account export (e.g., all configuration, dashboards, experiments, session replays) in a single open-format bundle, nor any documented account-deletion/data-portability workflow for 'leaving' the platform. missing for 10: full-account export tooling covering dashboards/configs/experiments/session-replay data, an explicit data-portability/account-closure workflow, and independent verification that exported formats are truly open and complete.
- [claimed-docs] “Export your project's event data.”
- [claimed-docs] “List all your cohorts in Amplitude, export a cohort in Amplitude, or upload a cohort.”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
Mixpanel documents multiple open-format export paths — Raw Event Export API returning JSONL (mixpanel-docs-7,9,18,31), CSV export from the UI (mixpanel-docs-25,30), Engage API for user profiles (mixpanel-docs-8), and Query API for aggregated data (mixpanel-docs-39,43) — enabling a user to pull out event, profile, and query data in standard formats. Community commentary even praises Mixpanel's export policy as 'great' when discussing data ownership (mixpanel-comm-13), corroborating that full data export works in practice. Missing for 10: no explicit statement of exporting *all* config/dashboard/schema metadata, and no independent bulk-export walkthrough beyond forum anecdote.
- [claimed-docs] “To export raw event data, use the Raw Event Export API. The events payload is returned as JSONL”
- [claimed-docs] “The raw export API allows you to download your event data as it is received and stored within Mixpanel, complete with all event properties”
- [claimed-docs] “Every data point sent to Mixpanel is stored as JSON in our data store.”
- [claimed-docs] “You can export data from your project using a variety of methods, including lightweight CSVs generated from the UI, large volume raw data ex…”
- [claimed-docs] “You may export small batches of events as a CSV for use in quick analysis by navigating to the Events tab”
- [claimed-docs] “To export User Profile data, use the Engage API.”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [community] “Complaint about vendor lock-in: 'The tool is lovely, their export policy is great, but there's something about actually owning your data' — …”
ai-native userRead the product's source under an open license
weight 2 · round drawnAmplitudenone0/10Amplitude is a closed-source, proprietary SaaS analytics platform; the evidence pack shows only some SDKs (e.g., Flutter) are open-source but no evidence the core product's source is open-licensed or readable. This axis applies to any product (a buyer could ask about source availability), but there's no evidence of an open license covering Amplitude's core codebase.
- [claimed-docs] “The Flutter SDK is an open-source UI for developing applications for Android, iOS, Windows, Mac, Linux, Google Fuchsia and the Web.”
ai-native userSelf-host the core product
weight 3 · round drawnAmplitudenone0/10Amplitude is offered exclusively as a hosted SaaS analytics platform; no evidence of a self-hostable core product, open-source release, or on-prem deployment option appears anywhere in the docs, integrations, or community threads.
Privacy cookieless — stories about privacy cookieless in this arenaPrivacy cookieless
Stories about privacy cookieless in this arena
Privacy first analytics
founderTrack product usage without third-party cookies so I can minimize or avoid cookie consent banners
weight 2 · round to AmplitudeAmplitude's docs mention server-side event sending (HTTP V2 endpoint) and a shared consent model with options to mask sensitive content and honor consent, which implies some capacity for privacy-conscious, non-cookie-dependent tracking, but there is no explicit claim about avoiding third-party cookies or reducing cookie-consent banners. missing for 10: explicit documentation of cookieless/first-party tracking, cookie policy details, or guidance on reducing consent-banner requirements.
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
- [claimed-docs] “Every SDK shares the same identity, event, and consent model, so a user instrumented once is understood everywhere.”
- [claimed-docs] “Mask sensitive content, honor consent, and control what gets captured.”
Mixpanelnone0/10The evidence pack contains no documentation on cookieless tracking, first-party identity resolution, or reducing/avoiding cookie consent banners — Mixpanel's docs focus on events, funnels, retention, session replay, exports, and MCP, none of which address cookie usage or consent-banner avoidance. Missing for 10: any statement on cookieless tracking methods, first-party data collection design, or consent-banner impact.
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round to MixpanelAmplitudenone0/10No evidence in the pack mentions data residency, region selection, EU/US hosting options, or any data storage location controls for Amplitude. This is a fair axis for an analytics SaaS platform handling user data, but absence of evidence means it cannot be credited as delivered. missing for 10: any mention of regional data hosting options, EU/US residency controls, or data location settings.
Mixpanel documents an EU Data Residency Program allowing customers to process and store personal data in Europe, directly addressing regional data residency choice [mixpanel-docs-38], backed by GDPR-oriented deletion/retrieval APIs [mixpanel-docs-12]. Missing for 10: no detail on which regions beyond EU are supported, no independent/hands-on confirmation of the residency mechanics or migration process, and no mention of per-project region selection UI.
- [claimed-docs] “Mixpanel provides you with the option to process and store your customers' personal data in Europe via our EU Data Residency Program.”
- [claimed-docs] “Mixpanel deletion and retrieval APIs are in place to help Mixpanel implementations meet the requirements outlined by the General Data Protec…”
ai-native userPrevent my data from being used to train AI models
weight 3 · round drawnAmplitudenone0/10No evidence in the pack addresses AI-training opt-out or data-usage-for-AI-training controls; the pack covers analytics, session replay, feature flags, and an MCP server but nothing about preventing data from being used to train AI models.
Mixpanelnone0/10No evidence in the pack addresses AI-training opt-out or any policy preventing customer data from being used to train AI models; privacy docs only mention GDPR deletion/retrieval APIs and EU data residency, not AI-training exclusion.
- [claimed-docs] “Mixpanel deletion and retrieval APIs are in place to help Mixpanel implementations meet the requirements outlined by the General Data Protec…”
- [claimed-docs] “Mixpanel provides you with the option to process and store your customers' personal data in Europe via our EU Data Residency Program.”
ai-native userControl data retention and deletion
weight 2 · round to MixpanelDocs mention consent controls and content masking for session replay (amplitude-docs-33) and CRUD including delete for event/user properties via API (amplitude-docs-38), which touch on privacy controls, but there's no explicit documentation of a user-level data retention policy or GDPR-style deletion/export mechanism for end-user data. missing for 10: explicit data retention policy settings, a documented user data deletion/erasure API or workflow, and any independent confirmation that deletion requests are honored.
- [claimed-docs] “Mask sensitive content, honor consent, and control what gets captured.”
- [claimed-docs] “Create, get, update, and delete categories, event types, event properties, and user properties.”
Mixpanel documents explicit deletion and retrieval APIs built to meet GDPR requirements, plus an EU Data Residency Program for regional data storage/processing, giving users concrete levers to control retention and deletion of their data ([mixpanel-docs-12], [mixpanel-docs-38]). Missing for 10: independent/hands-on verification of the deletion APIs actually working, and documentation of configurable retention windows or automated data-expiry settings beyond GDPR compliance language.
- [claimed-docs] “Mixpanel deletion and retrieval APIs are in place to help Mixpanel implementations meet the requirements outlined by the General Data Protec…”
- [claimed-docs] “Mixpanel provides you with the option to process and store your customers' personal data in Europe via our EU Data Residency Program.”
ai-native userOpt out of telemetry and usage tracking
weight 2 · round to AmplitudeAmplitude documents a shared 'consent model' across SDKs and features to 'honor consent, and control what gets captured' for session replay, implying some opt-out/consent tooling exists in its data-collection stack, but there is no dedicated documentation about an AI-native user opting out of Amplitude's own telemetry or usage tracking of the product itself. missing for 10: explicit opt-out/DNT mechanism documentation, privacy policy specifics for AI-native/developer users, independent confirmation that consent controls actually block tracking.
- [claimed-docs] “Every SDK shares the same identity, event, and consent model, so a user instrumented once is understood everywhere.”
- [claimed-docs] “Mask sensitive content, honor consent, and control what gets captured.”
Mixpanelnone0/10Evidence pack shows Mixpanel's GDPR deletion/retrieval APIs and EU data residency options for customer data stored in Mixpanel, but there is no mention of any mechanism for an AI-native user (e.g., of the MCP server or AI features) to opt out of telemetry/usage tracking collected about their own interactions with Mixpanel's tools.
- [claimed-docs] “Mixpanel deletion and retrieval APIs are in place to help Mixpanel implementations meet the requirements outlined by the General Data Protec…”
- [claimed-docs] “Mixpanel provides you with the option to process and store your customers' personal data in Europe via our EU Data Residency Program.”
Session replay — stories about session replay in this arenaSession replay
Stories about session replay in this arena
Replay
product-managerJump from a funnel drop-off or an error event straight to session replays of the affected users
weight 2 · round to AmplitudeAmplitude's session replay docs explicitly describe going from a chart, funnel, or support ticket to the exact session, catching errors and rage clicks across sessions, and tying errors to UX — directly matching the funnel-drop-off/error-to-replay jump story. This is well documented across multiple docs pages with clear first-party detail on the workflow. missing for 10: independent/hands-on third-party corroboration that the funnel-to-replay jump works smoothly in practice.
- [claimed-docs] “Go from a chart, funnel, or support ticket to the exact session.”
- [claimed-docs] “Catch rage clicks, dead clicks, and errors across sessions.”
- [claimed-docs] “Watch real user sessions to understand the why behind your metrics. Spot frustration, tie errors to UX, and measure engagement across every …”
- [claimed-docs] “See exactly what users experience. Replay sessions with full event and network context.”
- [claimed-docs] “Focus replays on specific users, pages, and flows.”
Mixpanel explicitly documents Session Replay as a way to 'follow up by exploring why customers drop off' from funnels, and playlists can be saved 'based on events or properties' for review, supporting drop-off-to-replay workflows. However, there is no explicit documentation of jumping from an error event specifically into replays, nor independent/hands-on confirmation of this workflow. Missing for 10: explicit error-event-to-replay linkage, independent user corroboration of the workflow.
- [claimed-docs] “Now, with Mixpanel Session Replay, you can quickly follow up by exploring why customers drop off.”
- [claimed-docs] “Now, with Mixpanel Session Replay, you can quickly follow up by exploring **why** customers drop off.”
- [claimed-docs] “Save playlists of replays based on events or properties, and quickly review with AI summaries, making it easy to revisit groups of replays t…”
- [claimed-docs] “Discover critical drop-off points and calibrate user journeys to optimize every touchpoint for maximum engagement, retention, and growth.”
- [claimed-docs] “Use AI to spot trends across hundreds of replays and heatmaps, then zoom in yourself to get to the “why” behind every change.”
product-managerWatch recordings of real user sessions with sensitive input masked by default
weight 3 · round to AmplitudeAmplitude's session replay docs explicitly state you can watch real user sessions ([amplitude-docs-8],[amplitude-docs-21],[amplitude-docs-24]) and that sensitive content is masked by default with consent controls ([amplitude-docs-33]), directly matching the story. Missing for 10: independent/hands-on verification of default masking behavior and more detail on configurability/exceptions.
- [claimed-docs] “Watch real user sessions to understand the why behind your metrics.”
- [claimed-docs] “Watch real user sessions to understand the why behind your metrics. Spot frustration, tie errors to UX, and measure engagement across every …”
- [claimed-docs] “See exactly what users experience. Replay sessions with full event and network context.”
- [claimed-docs] “Mask sensitive content, honor consent, and control what gets captured.”
- [claimed-docs] “Go from a chart, funnel, or support ticket to the exact session.”
- [claimed-docs] “Focus replays on specific users, pages, and flows.”
Mixpanel's Session Replay lets PMs watch real user session recordings and even use AI to summarize/playlist replays (mixpanel-docs-4, mixpanel-docs-27, mixpanel-docs-42), but the evidence pack contains no documentation confirming that sensitive input fields (e.g., passwords, form data) are masked by default in these recordings. Privacy docs cover GDPR/EU data residency (mixpanel-docs-12, mixpanel-docs-38) but not replay-specific input masking. Missing for 10: explicit documentation of default input/text masking or redaction controls within Session Replay, and any independent confirmation that masking works as claimed.
- [claimed-docs] “Now, with Mixpanel Session Replay, you can quickly follow up by exploring why customers drop off.”
- [claimed-docs] “Save playlists of replays based on events or properties, and quickly review with AI summaries, making it easy to revisit groups of replays t…”
- [claimed-docs] “Mixpanel Session Replay provides the fastest way to gain a comprehensive understanding of your customers and make informed product decisions”
- [claimed-docs] “Mixpanel deletion and retrieval APIs are in place to help Mixpanel implementations meet the requirements outlined by the General Data Protec…”
- [claimed-docs] “Mixpanel provides you with the option to process and store your customers' personal data in Europe via our EU Data Residency Program.”
Warehouse sql — stories about warehouse sql in this arenaWarehouse sql
Stories about warehouse sql in this arena
Ai analytics ops
ai-native userHave an agent answer growth questions from live product data by running queries through the API or MCP server
weight 3 · round drawnAmplitude ships an official MCP server that lets AI tools (Claude, Cursor, OpenAI, etc.) query behavioral data conversationally without SQL or dashboards, plus REST/Dashboard/Export APIs for programmatic access to live product data. Together these enable an agent to answer growth questions from live data via API or MCP. Missing for 10: independent hands-on verification of the MCP server's query accuracy/reliability and no community corroboration of agent-driven analysis in practice.
- [claimed-docs] “Bring Amplitude’s behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Bring Amplitude's behavioral context directly into your AI tools—Claude, Cursor, OpenAI, Lovable, Figma, and more—to get answers where you w…”
- [claimed-docs] “Start a conversation about your users or product and get expert-level answers in seconds that you can share with your whole org. No SQL, con…”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
- [claimed-docs] “Create, get, update, and delete categories, event types, event properties, and user properties.”
- [probe] “official MCP server documented at https://amplitude.com/mcp-server”
- [probe] “PROBE openapi: HTTP 200 at https://amplitude.com/openapi.json — contains "openapi" key”
Mixpanel documents a hosted MCP server that lets AI assistants query events, funnels, flows, retention, and session replays via natural language (no dashboard navigation), plus a Query API for filtering/breaking down events, cohorts, and profiles, and a raw export/JQL API for deeper queries—together covering agent-driven growth Q&A over live product data. Missing for 10: independent/hands-on verification of MCP server reliability or accuracy, and no evidence of complex ad-hoc SQL-like warehouse querying via MCP beyond the documented report types.
- [claimed-docs] “Mixpanel provides a hosted Model Context Protocol (MCP) server that gives AI assistants direct access to your Mixpanel data.”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required.”
- [claimed-docs] “Once connected, you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation …”
- [claimed-docs] “you can query events, funnels, flows, retention, session replays, and more using natural language — no dashboard navigation required”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
- [probe] “official MCP server documented at https://docs.mixpanel.com/docs/mcp”
Raw access
developerRun analytical queries and pull raw event data back through a documented query/export API
weight 3 · round to MixpanelAmplitude documents a Dashboard REST API returning JSON query results, an Export API for raw event data, and other CRUD APIs (events, cohorts, taxonomy), with an OpenAPI spec confirmed reachable — covering both analytical queries and raw event export as the story requires. Missing for 10: independent/hands-on developer corroboration of the export/query API's reliability and rate limits beyond first-party docs.
- [claimed-docs] “Send data directly from your server to the HTTP V2 endpoint.”
- [claimed-docs] “Export your project's event data.”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
- [claimed-docs] “Create, get, update, and delete categories, event types, event properties, and user properties.”
- [probe] “PROBE openapi: HTTP 200 at https://amplitude.com/openapi.json — contains "openapi" key”
Mixpanel documents a Query API for analytical queries (filter/breakdown events, cohorts, activity streams) and a separate Raw Event Export API returning full JSONL event data, plus JQL for custom JS-based queries — directly matching the query/export API story. Missing for 10: no independent/hands-on developer corroboration of API reliability or rate limits beyond first-party docs.
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
- [claimed-docs] “To export raw event data, use the Raw Event Export API. The events payload is returned as JSONL”
- [claimed-docs] “The raw export API allows you to download your event data as it is received and stored within Mixpanel, complete with all event properties”
- [claimed-docs] “To export raw event data, use the Raw Event Export API... This method is recommended for high volume exports.”
- [claimed-docs] “Create custom queries using Javascript code”
- [claimed-docs] “The JQL JavaScript code must define a main() function. This JavaScript will be compiled, and the main() function will be executed by the que…”
developerQuery my raw events with SQL (or an SQL-equivalent query language) inside the platform
weight 3 · round to MixpanelAmplitudenone0/10Evidence shows Amplitude offers dashboards, charts, cohorts, and an AI natural-language query interface (which explicitly markets 'No SQL' as a benefit), plus REST/export APIs and warehouse integrations like BigQuery/Snowflake for data export — but nothing indicates an in-platform SQL or SQL-equivalent query editor over raw events. missing for 10: any documented SQL/notebook query interface inside Amplitude, evidence of a query language for ad-hoc raw event access, or hands-on confirmation of SQL support.
- [claimed-docs] “Ask questions in plain language — Amplitude AI surfaces answers across all your data.”
- [claimed-docs] “Start a conversation about your users or product and get expert-level answers in seconds that you can share with your whole org. No SQL, con…”
- [claimed-docs] “Export your project's event data.”
- [claimed-docs] “Get data that's displayed on the dashboard graphs in JSON format through the Dashboard REST API.”
- [claimed-docs] “BigQuery is Google's fully managed, petabyte scale, low cost analytics data warehouse.”
- [claimed-docs] “Snowflake makes it easy to amass all your data, enable rapid analytics, and derive data-driven insights for all your business users.”
Mixpanel doesn't offer SQL, but its JQL (JavaScript Query Language) lets developers write custom code queries against raw event data directly in the platform, and the Query API can filter/break down events and return activity streams — a functional but not syntactic SQL-equivalent. missing for 10: no actual SQL support, no evidence JQL/Query API is positioned as a full SQL-equivalent, and no independent/hands-on corroboration of its expressiveness for raw-event querying.
- [claimed-docs] “Create custom queries using Javascript code”
- [claimed-docs] “The JQL JavaScript code must define a main() function. This JavaScript will be compiled, and the main() function will be executed by the que…”
- [claimed-docs] “The Mixpanel Query API endpoints can: Filter or break down Events * Return user profiles. * Return an activity/event stream for a user. * Re…”
- [claimed-docs] “The Query API allows you to obtain the same calculated and formatted results that you see from a report in our web app.”
Warehouse
developerSync events between the platform and my own data warehouse (query warehouse tables in-product or export events continuously to it)
weight 2 · round to MixpanelAmplitude documents an Export API for event data and lists first-party integrations with warehouses (BigQuery, Snowflake, Databricks) plus reverse-ETL tools (Hightouch, Fivetran, RudderStack) that support syncing events to/from a warehouse, satisfying the 'continuous export' half of the story. However, there's no evidence of an in-product warehouse query feature (e.g., querying warehouse tables directly within Amplitude's UI). Missing for 10: in-product warehouse querying capability, and independent/hands-on confirmation of the export-to-warehouse pipelines actually working reliably.
- [claimed-docs] “Export your project's event data.”
- [claimed-docs] “BigQuery is Google's fully managed, petabyte scale, low cost analytics data warehouse.”
- [claimed-docs] “Databricks is a unified, open analytics platform that enables building, deploying, sharing, and maintaining enterprise-grade data, analytics…”
- [claimed-docs] “Snowflake makes it easy to amass all your data, enable rapid analytics, and derive data-driven insights for all your business users.”
- [claimed-docs] “Hightouch syncs data from data warehouses into tools that teams rely on to run the business (CRM, email, ERP, etc.”
- [claimed-docs] “Fivetran is the leader in automated data integration, delivering ready-to-use connectors that adapt to change.”
Mixpanel's Warehouse Connectors let you sync data from Snowflake, BigQuery, Databricks, Redshift, and Postgres into Mixpanel, and the Raw Event Export API (JSONL, recommended for high-volume) lets you pull events out for warehouse loading, satisfying both directions of the story at a basic level. However, continuous/automated export to a warehouse is not natively built-in as a managed reverse-sync pipeline—Mixpanel's own partner directory lists third-party Reverse ETL tools like Census for pushing Mixpanel data to warehouses, implying continuous outbound sync relies on partner tooling rather than a first-party feature. Missing for 10: evidence of a native continuous/scheduled export-to-warehouse pipeline (not just an on-demand API), and independent/hands-on confirmation that Warehouse Connectors and export API work reliably at scale.
- [claimed-docs] “you can sync data from data warehouses like Snowflake, BigQuery, Databricks, Redshift, and Postgres to Mixpanel”
- [claimed-docs] “With Warehouse Connectors, you can sync data from data warehouses like Snowflake, BigQuery, Databricks, Redshift, and Postgres to Mixpanel.”
- [claimed-docs] “you can sync data from data warehouses like Snowflake, BigQuery, Databricks, Redshift, and Postgres to Mixpanel.”
- [claimed-docs] “To export raw event data, use the Raw Event Export API. The events payload is returned as JSONL”
- [claimed-docs] “The raw export API allows you to download your event data as it is received and stored within Mixpanel, complete with all event properties”
- [claimed-docs] “To export raw event data, use the Raw Event Export API... This method is recommended for high volume exports.”
- [claimed-docs] “You can export data from your project using a variety of methods, including lightweight CSVs generated from the UI, large volume raw data ex…”
- [claimed-docs] “Census is the #1 Data Activation platform that unites revenue teams around a single source of truth in the cloud data warehouse. Our no-code…”
- [claimed-docs] “SET CHANGE\TRACKING = TRUE; # or ALTER VIEW SET CHANGE\TRACKING = TRUE; \\\ Mixpanel will create and manage the necessary STREAM objects in …”
- [claimed-docs] “SET TBLPROPERTIES (delta.enableChangeDataFeed = true); \\\ \\Considerations when using Mirror with Databricks:\\ \ Mirror is not supported o…”
Not comparable on these axes
ai-native userVersion, review, and roll back my automations
weight 1 · not comparableAmplitude's feature-flag/experiment system supports rollback-like control ('turn it off the moment something looks wrong') and can be managed via CI/scripts, but there is no evidence of a version history, change review, or audit/rollback log for automations. missing for 10: explicit versioning/change-history for flags or automations, a review/approval workflow, and documented rollback-to-prior-version mechanics.
- [claimed-docs] “Release behind a flag, ramp to the audience you choose, and turn it off the moment something looks wrong.”
- [claimed-docs] “Manage flags, experiments, and deployments from CI, scripts, or your own tools with the Experiment APIs.”
- [claimed-docs] “Roll out behind a flag and run A/B tests so the data, not a hunch, drives the release.”
Mixpaneln/aMixpanel is a product analytics platform (events, funnels, retention, session replay, dashboards, MCP queries) rather than an automation/workflow tool with versionable automations; there is no concept of 'automations' to version, review, or roll back in the evidence pack, making this axis a category mismatch rather than a missing feature.