Feature Flags & Experimentation Arena
Feature Flags & Experimentation arenaBuyer checklist
Every requirement we judge feature flags & experimentation products against, as a ready-to-send RFP checklist — with each item's priority, why it matters, and how the top-ranked products score on it today.
54 requirements · 11 themes · verdicts for 5 products · updated 2026-09-05 · priorities mirror the story weights our scoring uses (methodology)
Show the markdown export
# Feature Flags & Experimentation — buyer checklist (RFP) Derived from ProductArena's evidence-graded user-story taxonomy for Feature Flags & Experimentation: 54 judged requirements. Priorities mirror story weights (3 = must-have, 2 = should-have, 1 = nice-to-have). ## Agenticness - [ ] **[must-have]** Plug MCP servers into this product so it can use their tools - [ ] **[must-have]** Connect an agent via an official MCP server - [ ] **[must-have]** Drive the product through a documented public API - [ ] **[must-have]** Delegate tasks to a built-in AI assistant inside the product - [ ] **[should-have]** Point an agent at llms.txt or agent-oriented docs - [ ] **[should-have]** Run the product headlessly / in CI for automation - [ ] **[should-have]** Use an official CLI - [ ] **[should-have]** Issue scoped/least-privilege API credentials for an agent - [ ] **[should-have]** Build against official SDKs - [ ] **[should-have]** Subscribe to events via webhooks - [ ] **[should-have]** Get AI-generated insights and suggestions from my data inside the product - [ ] **[should-have]** Set up automations that run autonomously in the background - [ ] **[should-have]** Operate the product with natural-language commands - [ ] **[should-have]** Explore an interactive API reference with runnable examples - [ ] **[should-have]** Download a machine-readable API spec (OpenAPI or equivalent) - [ ] **[should-have]** Rely on versioned APIs with a documented deprecation policy - [ ] **[nice-to-have]** Test against a sandbox environment without touching production data ## Automation depth - [ ] **[must-have]** Define rules that trigger actions automatically on events - [ ] **[should-have]** Perform bulk operations across many items at once - [ ] **[should-have]** Schedule recurring jobs or workflows - [ ] **[nice-to-have]** Version, review, and roll back my automations ## Deployment self host - [ ] **[must-have]** Self-host the full flag platform from an open-source distribution, keeping evaluation data on my infrastructure - [ ] **[nice-to-have]** Run a relay/edge proxy so flags stay served when the vendor is unreachable and SDK traffic stays inside my network ## Experimentation - [ ] **[must-have]** Run A/B and multivariate experiments on flags and see which variation wins on my metrics - [ ] **[should-have]** Trust a documented statistics engine (Bayesian or frequentist, with variance-reduction options) behind experiment results - [ ] **[should-have]** Define experiment metrics from my own data — warehouse tables or ingested events — instead of a black-box metric store - [ ] **[nice-to-have]** Read experiment configurations and results programmatically to summarize outcomes and recommend ship/rollback decisions ## Flag management - [ ] **[must-have]** Create a feature flag and toggle it live in production within minutes of signing up - [ ] **[must-have]** Target flags with attribute-based rules and reusable segments so the right users see the right variation - [ ] **[should-have]** Manage separate environments (dev/staging/prod) with independent flag states and scoped SDK keys - [ ] **[should-have]** Serve multivariate flags and dynamic configuration values (strings, numbers, JSON), not just booleans - [ ] **[should-have]** Find stale flags and code references so temporary flags actually get removed from the codebase - [ ] **[nice-to-have]** Target or exclude specific individual users for a flag (allowlists, beta testers, internal accounts) ## Governance audit - [ ] **[must-have]** Create and toggle flags through documented APIs, CLIs, or MCP — and the platform can force my changes through approval workflows instead of letting me write to production unreviewed - [ ] **[must-have]** Require approvals or change requests before production flag changes go live - [ ] **[must-have]** Every flag change is recorded in an audit log — who changed what, when, and to which value - [ ] **[should-have]** Restrict who can change which flags with roles, permissions, and scoped API tokens ## Openness - [ ] **[must-have]** Export all of my data in open formats and leave - [ ] **[must-have]** Self-host the core product - [ ] **[should-have]** Do everything through the API that I can do in the UI - [ ] **[should-have]** Read the product's source under an open license ## Pricing plans - [ ] **[should-have]** See published pricing and understand what drives cost (seats, MAUs, events, requests) before committing ## Privacy posture - [ ] **[must-have]** Prevent my data from being used to train AI models - [ ] **[should-have]** Choose where my data is stored (region/residency) - [ ] **[should-have]** Control data retention and deletion - [ ] **[should-have]** Opt out of telemetry and usage tracking ## Rollouts releases - [ ] **[must-have]** Roll a flag out progressively by percentage with consistent bucketing, ramping from 1% to 100% without redeploying - [ ] **[should-have]** Guard a rollout with metrics so a regression is detected and the release is rolled back automatically - [ ] **[nice-to-have]** Schedule flag changes and releases to happen at a specific future time ## Sdk delivery - [ ] **[must-have]** My server SDKs evaluate flags locally from a cached ruleset — microsecond decisions with no network call per flag check - [ ] **[should-have]** Evaluate flags at the edge (CDN workers or an edge/relay layer) close to users - [ ] **[should-have]** Use official SDKs across my whole stack — backend, web, and mobile — with consistent flag behavior - [ ] **[should-have]** Flag changes propagate to connected SDKs in seconds via streaming or fast polling — a kill switch actually kills - [ ] **[nice-to-have]** Use the vendor through OpenFeature providers so my flag code isn't locked to one vendor's SDK API --- Source: https://ultrametric.ai/productarena/arena/feature-flags (evidence-graded verdicts for 5 products) · methodology: https://ultrametric.ai/productarena/methodology
Chips show the top 5 ranked products' current verdict on each requirement — ✓ full · ~ partial · ! disputed · — none · n/a not applicable.
Agenticness — how well agents can access and operate the productAgenticness· 17 items
How well agents can access and operate the product
- must-have
ai-native userPlug MCP servers into this product so it can use their tools
Core requirement — weighs 3× in arena scoring
- must-have
ai-native userConnect an agent via an official MCP server
Core requirement — weighs 3× in arena scoring · all 5 products fully deliver this today
- must-have
ai-native userDrive the product through a documented public API
Core requirement — weighs 3× in arena scoring · all 5 products fully deliver this today
- must-have
ai-native userDelegate tasks to a built-in AI assistant inside the product
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- should-have
ai-native userPoint an agent at llms.txt or agent-oriented docs
Important, not disqualifying — weighs 2× in arena scoring · 3 of 5 products fully deliver this today
- should-have
ai-native userRun the product headlessly / in CI for automation
Important, not disqualifying — weighs 2× in arena scoring · all 5 products fully deliver this today
- should-have
ai-native userUse an official CLI
Important, not disqualifying — weighs 2× in arena scoring · 3 of 5 products fully deliver this today
- should-have
ai-native userIssue scoped/least-privilege API credentials for an agent
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userBuild against official SDKs
Important, not disqualifying — weighs 2× in arena scoring · 4 of 5 products fully deliver this today
- should-have
ai-native userSubscribe to events via webhooks
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
- should-have
ai-native userGet AI-generated insights and suggestions from my data inside the product
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userSet up automations that run autonomously in the background
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userOperate the product with natural-language commands
Important, not disqualifying — weighs 2× in arena scoring · 3 of 5 products fully deliver this today
- should-have
ai-native userExplore an interactive API reference with runnable examples
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userRely on versioned APIs with a documented deprecation policy
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- nice-to-have
ai-native userTest against a sandbox environment without touching production data
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 3 of 5 products fully deliver this today
Automation depth — how much of the product can run unattendedAutomation depth· 4 items
How much of the product can run unattended
- must-have
ai-native userDefine rules that trigger actions automatically on events
Core requirement — weighs 3× in arena scoring · 1 of 5 products fully deliver this today
- should-have
ai-native userPerform bulk operations across many items at once
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userSchedule recurring jobs or workflows
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- nice-to-have
ai-native userVersion, review, and roll back my automations
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 1 of 5 products fully deliver this today
Deployment self host — stories about deployment self host in this arenaDeployment self host· 2 items
Stories about deployment self host in this arena
- must-have
platform engineerSelf-host the full flag platform from an open-source distribution, keeping evaluation data on my infrastructure
Core requirement — weighs 3× in arena scoring · 3 of 5 products fully deliver this today
- nice-to-have
platform engineerRun a relay/edge proxy so flags stay served when the vendor is unreachable and SDK traffic stays inside my network
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 4 of 5 products fully deliver this today
Experimentation — stories about experimentation in this arenaExperimentation· 4 items
Stories about experimentation in this arena
- must-have
product managerRun A/B and multivariate experiments on flags and see which variation wins on my metrics
Core requirement — weighs 3× in arena scoring · 4 of 5 products fully deliver this today
- should-have
product managerTrust a documented statistics engine (Bayesian or frequentist, with variance-reduction options) behind experiment results
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
- should-have
product managerDefine experiment metrics from my own data — warehouse tables or ingested events — instead of a black-box metric store
Important, not disqualifying — weighs 2× in arena scoring · 2 of 5 products fully deliver this today
- nice-to-have
ai agentRead experiment configurations and results programmatically to summarize outcomes and recommend ship/rollback decisions
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 2 of 5 products fully deliver this today
Flag management — stories about flag management in this arenaFlag management· 6 items
Stories about flag management in this arena
- must-have
developerCreate a feature flag and toggle it live in production within minutes of signing up
Core requirement — weighs 3× in arena scoring · all 5 products fully deliver this today
- must-have
developerTarget flags with attribute-based rules and reusable segments so the right users see the right variation
Core requirement — weighs 3× in arena scoring · 3 of 5 products fully deliver this today
- should-have
developerManage separate environments (dev/staging/prod) with independent flag states and scoped SDK keys
Important, not disqualifying — weighs 2× in arena scoring · 4 of 5 products fully deliver this today
- should-have
developerServe multivariate flags and dynamic configuration values (strings, numbers, JSON), not just booleans
Important, not disqualifying — weighs 2× in arena scoring · 2 of 5 products fully deliver this today
- should-have
platform engineerFind stale flags and code references so temporary flags actually get removed from the codebase
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
- nice-to-have
developerTarget or exclude specific individual users for a flag (allowlists, beta testers, internal accounts)
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 4 of 5 products fully deliver this today
Governance audit — stories about governance audit in this arenaGovernance audit· 4 items
Stories about governance audit in this arena
- must-have
ai agentCreate and toggle flags through documented APIs, CLIs, or MCP — and the platform can force my changes through approval workflows instead of letting me write to production unreviewed
Core requirement — weighs 3× in arena scoring · 2 of 5 products fully deliver this today
- must-have
platform engineerRequire approvals or change requests before production flag changes go live
Core requirement — weighs 3× in arena scoring · 4 of 5 products fully deliver this today
- must-have
platform engineerEvery flag change is recorded in an audit log — who changed what, when, and to which value
Core requirement — weighs 3× in arena scoring · 1 of 5 products fully deliver this today
- should-have
platform engineerRestrict who can change which flags with roles, permissions, and scoped API tokens
Important, not disqualifying — weighs 2× in arena scoring · 3 of 5 products fully deliver this today
Openness — open source, data portability, and self-hosting storiesOpenness· 4 items
Open source, data portability, and self-hosting stories
- must-have
ai-native userExport all of my data in open formats and leave
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- must-have
ai-native userSelf-host the core product
Core requirement — weighs 3× in arena scoring · 3 of 5 products fully deliver this today
- should-have
ai-native userDo everything through the API that I can do in the UI
Important, not disqualifying — weighs 2× in arena scoring · 3 of 5 products fully deliver this today
- should-have
ai-native userRead the product's source under an open license
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
Pricing plans — plan structure and value — what each tier costs and what it unlocksPricing plans· 1 item
Plan structure and value — what each tier costs and what it unlocks
- should-have
product managerSee published pricing and understand what drives cost (seats, MAUs, events, requests) before committing
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
Privacy posture — data-handling and privacy storiesPrivacy posture· 4 items
Data-handling and privacy stories
- must-have
ai-native userPrevent my data from being used to train AI models
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- should-have
ai-native userChoose where my data is stored (region/residency)
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userControl data retention and deletion
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userOpt out of telemetry and usage tracking
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
Rollouts releases — stories about rollouts releases in this arenaRollouts releases· 3 items
Stories about rollouts releases in this arena
- must-have
developerRoll a flag out progressively by percentage with consistent bucketing, ramping from 1% to 100% without redeploying
Core requirement — weighs 3× in arena scoring · all 5 products fully deliver this today
- should-have
platform engineerGuard a rollout with metrics so a regression is detected and the release is rolled back automatically
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
- nice-to-have
developerSchedule flag changes and releases to happen at a specific future time
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 3 of 5 products fully deliver this today
Sdk delivery — stories about sdk delivery in this arenaSdk delivery· 5 items
Stories about sdk delivery in this arena
- must-have
platform engineerMy server SDKs evaluate flags locally from a cached ruleset — microsecond decisions with no network call per flag check
Core requirement — weighs 3× in arena scoring · 1 of 5 products fully deliver this today
- should-have
platform engineerEvaluate flags at the edge (CDN workers or an edge/relay layer) close to users
Important, not disqualifying — weighs 2× in arena scoring · 3 of 5 products fully deliver this today
- should-have
developerUse official SDKs across my whole stack — backend, web, and mobile — with consistent flag behavior
Important, not disqualifying — weighs 2× in arena scoring · 2 of 5 products fully deliver this today
- should-have
developerFlag changes propagate to connected SDKs in seconds via streaming or fast polling — a kill switch actually kills
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
- nice-to-have
platform engineerUse the vendor through OpenFeature providers so my flag code isn't locked to one vendor's SDK API
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 1 of 5 products fully deliver this today
Full evidence behind every verdict lives on the arena page and each product page — chips above deep-link straight to the judged story.