Product Analytics arenaBuyer checklist
Every requirement we judge product analytics products against, as a ready-to-send RFP checklist — with each item's priority, why it matters, and how the top-ranked products score on it today.
53 requirements · 12 themes · verdicts for 4 products · updated 2026-09-16 · priorities mirror the story weights our scoring uses (methodology)
Show the markdown export
# Product Analytics — buyer checklist (RFP) Derived from ProductArena's evidence-graded user-story taxonomy for Product Analytics: 53 judged requirements. Priorities mirror story weights (3 = must-have, 2 = should-have, 1 = nice-to-have). ## Agenticness - [ ] **[must-have]** Plug MCP servers into this product so it can use their tools - [ ] **[must-have]** Connect an agent via an official MCP server - [ ] **[must-have]** Drive the product through a documented public API - [ ] **[must-have]** Delegate tasks to a built-in AI assistant inside the product - [ ] **[should-have]** Point an agent at llms.txt or agent-oriented docs - [ ] **[should-have]** Run the product headlessly / in CI for automation - [ ] **[should-have]** Use an official CLI - [ ] **[should-have]** Issue scoped/least-privilege API credentials for an agent - [ ] **[should-have]** Build against official SDKs - [ ] **[should-have]** Subscribe to events via webhooks - [ ] **[should-have]** Get AI-generated insights and suggestions from my data inside the product - [ ] **[should-have]** Set up automations that run autonomously in the background - [ ] **[should-have]** Operate the product with natural-language commands - [ ] **[should-have]** Explore an interactive API reference with runnable examples - [ ] **[should-have]** Download a machine-readable API spec (OpenAPI or equivalent) - [ ] **[should-have]** Rely on versioned APIs with a documented deprecation policy - [ ] **[nice-to-have]** Test against a sandbox environment without touching production data ## Automation depth - [ ] **[must-have]** Define rules that trigger actions automatically on events - [ ] **[should-have]** Perform bulk operations across many items at once - [ ] **[should-have]** Schedule recurring jobs or workflows - [ ] **[nice-to-have]** Version, review, and roll back my automations ## Dashboards reporting - [ ] **[must-have]** Have an agent build a dashboard of my key metrics end-to-end via the API or MCP server - [ ] **[should-have]** Create and manage dashboards and saved insights programmatically via the API - [ ] **[should-have]** Compose saved insights into shareable dashboards for my team - [ ] **[nice-to-have]** Subscribe to dashboards and get alerted when a key metric moves abnormally ## Event ingestion - [ ] **[must-have]** Send events from web, mobile, and backend apps through official SDKs for the major languages and platforms - [ ] **[should-have]** Identify users across devices and merge anonymous pre-signup activity into their identified profile - [ ] **[should-have]** Capture clicks, pageviews, and form interactions automatically without writing tracking code for each event - [ ] **[nice-to-have]** Bulk-import historical events from another analytics tool or a data export into the platform ## Flags experiments - [ ] **[must-have]** Run A/B experiments with goal metrics and statistical significance reported on the results - [ ] **[must-have]** Roll out features behind flags targeted by user properties, cohorts, and percentage rollouts - [ ] **[should-have]** Have an agent create, target, and toggle feature flags through the API or MCP server ## Funnels retention - [ ] **[must-have]** Build multi-step funnels with filters and breakdowns to find where users drop off - [ ] **[must-have]** Measure retention over time and slice it by behavioral cohorts - [ ] **[must-have]** Answer "which activation step loses users?" myself with funnels and drop-off analysis, without an analyst - [ ] **[should-have]** Analyze behavior at the account or company level, not just per user, for B2B products - [ ] **[should-have]** Explore the paths users actually take before or after a key event ## Integrations - [ ] **[should-have]** Stream events onward to third-party destinations (CRM, ad platforms, webhooks) from the analytics platform ## Openness - [ ] **[must-have]** Export all of my data in open formats and leave - [ ] **[must-have]** Self-host the core product - [ ] **[should-have]** Do everything through the API that I can do in the UI - [ ] **[should-have]** Read the product's source under an open license ## Privacy cookieless - [ ] **[should-have]** Track product usage without third-party cookies so I can minimize or avoid cookie consent banners ## Privacy posture - [ ] **[must-have]** Prevent my data from being used to train AI models - [ ] **[should-have]** Choose where my data is stored (region/residency) - [ ] **[should-have]** Control data retention and deletion - [ ] **[should-have]** Opt out of telemetry and usage tracking ## Session replay - [ ] **[must-have]** Watch recordings of real user sessions with sensitive input masked by default - [ ] **[should-have]** Jump from a funnel drop-off or an error event straight to session replays of the affected users ## Warehouse sql - [ ] **[must-have]** Have an agent answer growth questions from live product data by running queries through the API or MCP server - [ ] **[must-have]** Run analytical queries and pull raw event data back through a documented query/export API - [ ] **[must-have]** Query my raw events with SQL (or an SQL-equivalent query language) inside the platform - [ ] **[should-have]** Sync events between the platform and my own data warehouse (query warehouse tables in-product or export events continuously to it) --- Source: https://ultrametric.ai/productarena/arena/product-analytics (evidence-graded verdicts for 4 products) · methodology: https://ultrametric.ai/productarena/methodology
Chips show the top 4 ranked products' current verdict on each requirement — ✓ full · ~ partial · ! disputed · — none · n/a not applicable.
Agenticness — how well agents can access and operate the productAgenticness· 17 items
How well agents can access and operate the product
- must-have
ai-native userPlug MCP servers into this product so it can use their tools
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- must-have
ai-native userConnect an agent via an official MCP server
Core requirement — weighs 3× in arena scoring · 3 of 4 products fully deliver this today
- must-have
ai-native userDrive the product through a documented public API
Core requirement — weighs 3× in arena scoring · all 4 products fully deliver this today
- must-have
ai-native userDelegate tasks to a built-in AI assistant inside the product
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- should-have
ai-native userPoint an agent at llms.txt or agent-oriented docs
Important, not disqualifying — weighs 2× in arena scoring · 3 of 4 products fully deliver this today
- should-have
ai-native userRun the product headlessly / in CI for automation
Important, not disqualifying — weighs 2× in arena scoring · 1 of 4 products fully deliver this today
- should-have
ai-native userUse an official CLI
Important, not disqualifying — weighs 2× in arena scoring · 1 of 4 products fully deliver this today
- should-have
ai-native userIssue scoped/least-privilege API credentials for an agent
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userBuild against official SDKs
Important, not disqualifying — weighs 2× in arena scoring · 2 of 4 products fully deliver this today
- should-have
ai-native userSubscribe to events via webhooks
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userGet AI-generated insights and suggestions from my data inside the product
Important, not disqualifying — weighs 2× in arena scoring · 2 of 4 products fully deliver this today
- should-have
ai-native userSet up automations that run autonomously in the background
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userOperate the product with natural-language commands
Important, not disqualifying — weighs 2× in arena scoring · 2 of 4 products fully deliver this today
- should-have
ai-native userExplore an interactive API reference with runnable examples
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
Important, not disqualifying — weighs 2× in arena scoring · 2 of 4 products fully deliver this today
- should-have
ai-native userRely on versioned APIs with a documented deprecation policy
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- nice-to-have
ai-native userTest against a sandbox environment without touching production data
Differentiator, not a dealbreaker — weighs 1× in arena scoring · no product fully delivers this yet
Automation depth — how much of the product can run unattendedAutomation depth· 4 items
How much of the product can run unattended
- must-have
ai-native userDefine rules that trigger actions automatically on events
Core requirement — weighs 3× in arena scoring · 1 of 4 products fully deliver this today
- should-have
ai-native userPerform bulk operations across many items at once
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userSchedule recurring jobs or workflows
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- nice-to-have
ai-native userVersion, review, and roll back my automations
Differentiator, not a dealbreaker — weighs 1× in arena scoring · no product fully delivers this yet
Dashboards reporting — stories about dashboards reporting in this arenaDashboards reporting· 4 items
Stories about dashboards reporting in this arena
- must-have
ai-native userHave an agent build a dashboard of my key metrics end-to-end via the API or MCP server
Core requirement — weighs 3× in arena scoring · 1 of 4 products fully deliver this today
- should-have
developerCreate and manage dashboards and saved insights programmatically via the API
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
product-managerCompose saved insights into shareable dashboards for my team
Important, not disqualifying — weighs 2× in arena scoring · 2 of 4 products fully deliver this today
- nice-to-have
product-managerSubscribe to dashboards and get alerted when a key metric moves abnormally
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 1 of 4 products fully deliver this today
Event ingestion — stories about event ingestion in this arenaEvent ingestion· 4 items
Stories about event ingestion in this arena
- must-have
developerSend events from web, mobile, and backend apps through official SDKs for the major languages and platforms
Core requirement — weighs 3× in arena scoring · 2 of 4 products fully deliver this today
- should-have
developerIdentify users across devices and merge anonymous pre-signup activity into their identified profile
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
product-managerCapture clicks, pageviews, and form interactions automatically without writing tracking code for each event
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- nice-to-have
developerBulk-import historical events from another analytics tool or a data export into the platform
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 1 of 4 products fully deliver this today
Flags experiments — stories about flags experiments in this arenaFlags experiments· 3 items
Stories about flags experiments in this arena
- must-have
product-managerRun A/B experiments with goal metrics and statistical significance reported on the results
Core requirement — weighs 3× in arena scoring · 2 of 4 products fully deliver this today
- must-have
developerRoll out features behind flags targeted by user properties, cohorts, and percentage rollouts
Core requirement — weighs 3× in arena scoring · 2 of 3 products fully deliver this today
- should-have
ai-native userHave an agent create, target, and toggle feature flags through the API or MCP server
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
Funnels retention — stories about funnels retention in this arenaFunnels retention· 5 items
Stories about funnels retention in this arena
- must-have
product-managerBuild multi-step funnels with filters and breakdowns to find where users drop off
Core requirement — weighs 3× in arena scoring · 3 of 4 products fully deliver this today
- must-have
product-managerMeasure retention over time and slice it by behavioral cohorts
Core requirement — weighs 3× in arena scoring · 3 of 4 products fully deliver this today
- must-have
founderAnswer "which activation step loses users?" myself with funnels and drop-off analysis, without an analyst
Core requirement — weighs 3× in arena scoring · 3 of 4 products fully deliver this today
- should-have
product-managerAnalyze behavior at the account or company level, not just per user, for B2B products
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
product-managerExplore the paths users actually take before or after a key event
Important, not disqualifying — weighs 2× in arena scoring · 2 of 4 products fully deliver this today
Integrations — connecting to other tools — breadth and depth of built-in integrationsIntegrations· 1 item
Connecting to other tools — breadth and depth of built-in integrations
- should-have
developerStream events onward to third-party destinations (CRM, ad platforms, webhooks) from the analytics platform
Important, not disqualifying — weighs 2× in arena scoring · 1 of 4 products fully deliver this today
Openness — open source, data portability, and self-hosting storiesOpenness· 4 items
Open source, data portability, and self-hosting stories
- must-have
ai-native userExport all of my data in open formats and leave
Core requirement — weighs 3× in arena scoring · 2 of 4 products fully deliver this today
- must-have
ai-native userSelf-host the core product
Core requirement — weighs 3× in arena scoring · 1 of 4 products fully deliver this today
- should-have
ai-native userDo everything through the API that I can do in the UI
Important, not disqualifying — weighs 2× in arena scoring · 1 of 4 products fully deliver this today
- should-have
ai-native userRead the product's source under an open license
Important, not disqualifying — weighs 2× in arena scoring · 2 of 4 products fully deliver this today
Privacy cookieless — stories about privacy cookieless in this arenaPrivacy cookieless· 1 item
Stories about privacy cookieless in this arena
- should-have
founderTrack product usage without third-party cookies so I can minimize or avoid cookie consent banners
Important, not disqualifying — weighs 2× in arena scoring · 1 of 4 products fully deliver this today
Privacy posture — data-handling and privacy storiesPrivacy posture· 4 items
Data-handling and privacy stories
- must-have
ai-native userPrevent my data from being used to train AI models
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- should-have
ai-native userChoose where my data is stored (region/residency)
Important, not disqualifying — weighs 2× in arena scoring · 1 of 4 products fully deliver this today
- should-have
ai-native userControl data retention and deletion
Important, not disqualifying — weighs 2× in arena scoring · 1 of 4 products fully deliver this today
- should-have
ai-native userOpt out of telemetry and usage tracking
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
Session replay — stories about session replay in this arenaSession replay· 2 items
Stories about session replay in this arena
- must-have
product-managerWatch recordings of real user sessions with sensitive input masked by default
Core requirement — weighs 3× in arena scoring · 1 of 3 products fully deliver this today
- should-have
product-managerJump from a funnel drop-off or an error event straight to session replays of the affected users
Important, not disqualifying — weighs 2× in arena scoring · 1 of 3 products fully deliver this today
Warehouse sql — stories about warehouse sql in this arenaWarehouse sql· 4 items
Stories about warehouse sql in this arena
- must-have
ai-native userHave an agent answer growth questions from live product data by running queries through the API or MCP server
Core requirement — weighs 3× in arena scoring · 2 of 4 products fully deliver this today
- must-have
developerRun analytical queries and pull raw event data back through a documented query/export API
Core requirement — weighs 3× in arena scoring · 3 of 4 products fully deliver this today
- must-have
developerQuery my raw events with SQL (or an SQL-equivalent query language) inside the platform
Core requirement — weighs 3× in arena scoring · 1 of 4 products fully deliver this today
- should-have
developerSync events between the platform and my own data warehouse (query warehouse tables in-product or export events continuously to it)
Important, not disqualifying — weighs 2× in arena scoring · 1 of 4 products fully deliver this today
Full evidence behind every verdict lives on the arena page and each product page — chips above deep-link straight to the judged story.