Statsig vs Flagsmith

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Statsig
   Statsig✓fullprobed9/10

   Statsig publishes a working llms.txt (probed live, HTTP 200) and offers a dedicated Docs MCP server that lets AI clients read documentation directly, confirmed via a full keyless initialize handshake at /api/mcp. This directly satisfies pointing an agent at agent-oriented docs. Missing for 10: independent third-party confirmation of agents actually using llms.txt in practice beyond the probe.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.statsig.com/llms.txt # Statsig Documentation > Official documentation for Statsig feature flags, e…”
   • [claimed-docs] “Use the Statsig Docs MCP server to let supported AI clients read public Statsig documentation directly from /api/mcp.”
   • [claimed-docs] “Use the Statsig Docs MCP server to let supported AI clients read public Statsig documentation directly from /api/mcp”
   • [probe] “PROBE runtime (recorded 2026-09-05): the documented no-auth docs MCP server at https://docs.statsig.com/api/mcp completed a FULL keyless ini…”

   proof ↗⚑ flag

   Flagsmith~partialprobed5/10

   Flagsmith has no llms.txt file (confirmed 404 via direct probe), but it does publish agent-oriented documentation — a dedicated docs page describing the MCP server 'for AI assistants and agents' to programmatically access the Admin API, plus a documented CLI. This is genuine agent-facing documentation, just not in the llms.txt convention. Missing for 10: an actual llms.txt or equivalent machine-readable agent doc index, and independent confirmation an agent can successfully consume the MCP docs page as a pointer target.

   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.flagsmith.com/llms.txt”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [probe] “official MCP server documented at https://docs.flagsmith.com/integrating-with-flagsmith/mcp-server”
   • [claimed-docs] “The Flagsmith CLI lets you manage flags, segments, features, projects and environments from your terminal, and evaluate flags the way an SDK…”
   • [probe] “official CLI documented at https://docs.flagsmith.com/integrating-with-flagsmith/CLI”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round drawn
   Statsig✓fullprobed8/10

   Statsig ships an official CLI (@statsig/siggy) that runs keylessly and is explicitly documented and probe-verified for CI/CD automation of Feature Gates and Experiments, plus a Console API/HTTP API for headless programmatic control. Runtime probes confirm the CLI executes via npx in a scriptable, non-interactive way suitable for automation pipelines. Missing for 10: no independent third-party CI pipeline case study or deeper docs on CLI command coverage beyond gates/experiments.

   • [claimed-docs] “The Statsig CLI is a command-line interface that helps manage Feature Gates, Experiments, and Dynamic Configs within the Statsig platform.”
   • [claimed-docs] “You can integrate the CLI into CI/CD pipelines to automate the management of Feature Gates and Experiments.”
   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [claimed-docs] “Overview of the Statsig HTTP API for retrieving feature gate, experiment, and dynamic config values and logging events directly without an S…”
   • [probe] “official CLI documented at https://docs.statsig.com/statsigcli/introduction”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Statsig CLI (npm @statsig/siggy) ran keylessly via `npx -y @statsig/siggy --version`, prin…”

   proof ↗⚑ flag

   Flagsmith✓fullprobed8/10

   Flagsmith ships an official CLI and Admin API that can create/evaluate flags entirely headlessly, and a probe confirms a full keyless self-hosted Docker create-and-evaluate loop with no UI involved, plus server-side SDKs for automated environments. missing for 10: explicit CI/CD pipeline examples (e.g., GitHub Actions) and independent third-party CI usage reports.

   • [claimed-docs] “The Flagsmith CLI lets you manage flags, segments, features, projects and environments from your terminal, and evaluate flags the way an SDK…”
   • [claimed-docs] “Anything you can do in the Flagsmith dashboard, you can also do via the Admin API.”
   • [claimed-docs] “This is the API you use to programmatically manage your Flagsmith projects. Anything you can do in the Flagsmith dashboard, you can also do …”
   • [claimed-docs] “You can use Docker to set up an entire Flagsmith Feature Flag environment locally”
   • [claimed-docs] “You can use Docker to set up an entire [Flagsmith Feature Flag](https://www.flagsmith.com) environment locally”
   • [probe] “official CLI documented at https://docs.flagsmith.com/integrating-with-flagsmith/CLI”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”

   proof ↗⚑ flag

3. ai-native userConnect an agent via an official MCP server

   weight 3 · round drawn
   Statsig✓fullprobed9/10

   Statsig ships an official MCP server integrating with agent tools like Codex, Cursor, and Claude Code for querying Statsig data, plus a separate docs MCP server; runtime probes confirm both endpoints are live, with the main MCP server enforcing OAuth and the docs server completing a full handshake. missing for 10: independent third-party (non-vendor) hands-on report of using the MCP server in a real agent workflow beyond docs/probes.

   • [claimed-docs] “The Statsig MCP (Model Context Protocol) server brings the power of Statsig into tools like Codex, Cursor, and Claude Code. With this setup,…”
   • [claimed-docs] “The Statsig MCP (Model Context Protocol) server brings the power of Statsig into tools like Codex, Cursor, and Claude Code.”
   • [claimed-docs] “Use the Statsig Docs MCP server to let supported AI clients read public Statsig documentation directly from /api/mcp.”
   • [probe] “official MCP server documented at https://docs.statsig.com/integrations/mcp/overview”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.statsig.com/v1/mcp returned HTTP …”
   • [probe] “PROBE runtime (recorded 2026-09-05): the documented no-auth docs MCP server at https://docs.statsig.com/api/mcp completed a FULL keyless ini…”

   proof ↗⚑ flag

   Flagsmith✓fullprobed9/10

   Flagsmith documents an official MCP Server that gives AI assistants/agents programmatic access to the Admin API via the Model Context Protocol, and a runtime probe confirms a live hosted MCP endpoint (mcp.flagsmith.com) responding to JSON-RPC initialize calls (key-gated). This is first-party, dedicated, and independently verified as functioning. Missing for 10: no independent/community hands-on report of actually using an agent through the MCP server in a real workflow.

   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API”
   • [probe] “official MCP server documented at https://docs.flagsmith.com/integrating-with-flagsmith/mcp-server”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the documented hosted MCP server https://mcp.flagsmith.com returned…”

   proof ↗⚑ flag

4. ai-native userUse an official CLI

   weight 2 · round to Statsig
   Statsig✓fullprobed9/10

   Statsig ships an official CLI (Statsig CLI / @statsig/siggy) for managing Feature Gates, Experiments, and Dynamic Configs, documented for CI/CD automation, and a runtime probe confirms it executes keylessly via npx and is scriptable. missing for 10: no independent (non-Statsig) hands-on review of the CLI's full feature set beyond the version-check probe.

   • [claimed-docs] “The Statsig CLI is a command-line interface that helps manage Feature Gates, Experiments, and Dynamic Configs within the Statsig platform.”
   • [claimed-docs] “You can integrate the CLI into CI/CD pipelines to automate the management of Feature Gates and Experiments.”
   • [probe] “official CLI documented at https://docs.statsig.com/statsigcli/introduction”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Statsig CLI (npm @statsig/siggy) ran keylessly via `npx -y @statsig/siggy --version`, prin…”

   proof ↗⚑ flag

   Flagsmith✓fullprobed8/10

   Flagsmith documents an official CLI that manages flags, segments, features, projects and environments and can evaluate flags like an SDK, confirmed live at the documented URL. missing for 10: independent/hands-on community verification of the CLI's functionality and details on installation/versioning maturity.

   • [claimed-docs] “The Flagsmith CLI lets you manage flags, segments, features, projects and environments from your terminal, and evaluate flags the way an SDK…”
   • [probe] “official CLI documented at https://docs.flagsmith.com/integrating-with-flagsmith/CLI”

   proof ↗⚑ flag

5. ai-native userDrive the product through a documented public API

   weight 3 · round to Statsig
   Statsig✓fullprobed9/10

   Statsig documents multiple public API surfaces (HTTP API for gate/experiment retrieval and event logging, full CRUD Console API, and CLI) that let developers/agents drive the product programmatically, and runtime probes confirm the CLI works keylessly and the MCP servers respond live. missing for 10: a discoverable OpenAPI/Swagger spec (probe found 404s on standard OpenAPI paths), which would give full machine-readable API discoverability.

   • [claimed-docs] “Overview of the Statsig HTTP API for retrieving feature gate, experiment, and dynamic config values and logging events directly without an S…”
   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [claimed-docs] “The Statsig CLI is a command-line interface that helps manage Feature Gates, Experiments, and Dynamic Configs within the Statsig platform.”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Statsig CLI (npm @statsig/siggy) ran keylessly via `npx -y @statsig/siggy --version`, prin…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.statsig.com/openapi.json, https://docs.statsig.com/swagger.json, https://docs.statsig.c…”

   proof ↗⚑ flag

   Flagsmith✓fullprobed8/10

   Flagsmith exposes a well-documented Admin API ('Anything you can do in the Flagsmith dashboard, you can also do via the Admin API'), a public CLI, and an official MCP server explicitly built for AI assistants/agents, with a runtime probe confirming the hosted MCP endpoint is live and key-gated and a self-hosted API create/evaluate roundtrip working keylessly. This directly satisfies driving the product via documented public API for an AI-native user. Missing for 10: a discoverable OpenAPI/swagger spec or llms.txt (both probed and returned 404), and independent/third-party corroboration of the MCP server's real-world agent usage beyond first-party docs.

   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API”
   • [claimed-docs] “Anything you can do in the Flagsmith dashboard, you can also do via the Admin API.”
   • [claimed-docs] “This is the API you use to programmatically manage your Flagsmith projects. Anything you can do in the Flagsmith dashboard, you can also do …”
   • [claimed-docs] “The Flagsmith CLI lets you manage flags, segments, features, projects and environments from your terminal, and evaluate flags the way an SDK…”
   • [probe] “official MCP server documented at https://docs.flagsmith.com/integrating-with-flagsmith/mcp-server”
   • [probe] “official CLI documented at https://docs.flagsmith.com/integrating-with-flagsmith/CLI”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the documented hosted MCP server https://mcp.flagsmith.com returned…”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.flagsmith.com/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.flagsmith.com/openapi.json, https://docs.flagsmith.com/swagger.json, https://docs.flags…”

   proof ↗⚑ flag

6. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to Flagsmith
   Statsig~partialprobed3/10

   Statsig documents general access-management/SSO/SCIM controls and shows its hosted MCP server requires an OAuth bearer token rather than being fully open (statsig-probe-rt-2), implying some authenticated, permissioned access surface an agent could use. However there is no explicit documentation of issuing scoped or least-privilege API keys/tokens specifically for agent use (e.g., read-only or project-scoped Console API keys). Missing for 10: explicit docs on creating scoped/limited-permission API keys, role-based key scoping for agents, and any guidance on least-privilege credential issuance for AI agents.

   • [claimed-docs] “In many enterprise environments, organizations use SSO and SCIM together to improve security and usability.”
   • [claimed-docs] “Basic settings include automatically adding new users with the same email domain to your project. Statsig also supports SSO to simplify invi…”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.statsig.com/v1/mcp returned HTTP …”

   proof ↗⚑ flag

   Flagsmith~partialprobed5/10

   Flagsmith exposes an Admin API with RBAC for fine-grained access control, and its documented MCP server requires an API key to authenticate agent access (confirmed via a live 401 probe), showing credential-gated agent access. However, there is no explicit documentation of issuing scoped, least-privilege tokens specifically for agents (e.g., read-only or resource-limited keys) beyond general user-role RBAC. Missing for 10: explicit agent-specific scoped/short-lived API key issuance, documented token scopes (read/write/resource-limited), and independent confirmation of least-privilege enforcement for MCP/agent credentials.

   • [claimed-docs] “Role-based access control (RBAC) provides fine-grained access management of Flagsmith resources.”
   • [claimed-docs] “Only allow certain users to modify your production environments.”
   • [claimed-docs] “Purpose: Creating, updating, and deleting projects, environments, flags, segments, and users.”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the documented hosted MCP server https://mcp.flagsmith.com returned…”
   • [probe] “official MCP server documented at https://docs.flagsmith.com/integrating-with-flagsmith/mcp-server”

   proof ↗⚑ flag

7. ai-native userBuild against official SDKs

   weight 2 · round to Statsig
   Statsig✓fullclaimed7/10

   Statsig's docs explicitly describe SDK architecture (client vs. server evaluation, on-device eval SDKs, deterministic evaluation across platforms) and even offer an HTTP API for cases without an SDK, confirming a first-party SDK ecosystem that AI-native developers can build against. Missing for 10: a concrete list/matrix of supported languages/platforms and independent hands-on confirmation of SDK usage beyond docs.

   • [claimed-docs] “Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always retu…”
   • [claimed-docs] “Statsig's client-side On-Device Eval SDKs provide an alternate client-side architecture. The device keeps the definition of each experiment …”
   • [claimed-docs] “Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always retu…”
   • [claimed-docs] “Statsig's client-side On-Device Eval SDKs provide an alternate client-side architecture. The device keeps the definition of each experiment …”
   • [claimed-docs] “Overview of the Statsig HTTP API for retrieving feature gate, experiment, and dynamic config values and logging events directly without an S…”

   proof ↗⚑ flag

   Flagsmith~partialclaimed6/10

   Flagsmith advertises official SDKs across 15+ languages and frameworks (React, Next.js, .NET, Java, etc.) and positions itself for web, mobile, and server-side integration, giving AI-native developers concrete SDKs to build against. However the evidence pack lacks SDK-specific documentation depth (API references, code samples, changelogs) beyond a marketing blurb. Missing for 10: dedicated per-SDK docs/examples, versioning/maintenance evidence, and independent corroboration of SDK quality.

   • [github] “Choose from 15+ popular languages like Typescript, .NET, Java, and more. Integrate with any framework, including React, Next.js, and more”
   • [claimed-docs] “Manage feature flags and remote config across web, mobile, and server-side apps.”

   proof ↗⚑ flag

8. ai-native userSubscribe to events via webhooks

   weight 2 · round to Flagsmith
   Statsig—none0/10

   The evidence pack contains no mention of webhooks or event subscription mechanisms anywhere in Statsig's docs, community posts, or probes — only feature gates, experiments, CLI, MCP, and access management are covered. Webhooks are a plausible axis for a flagging/analytics platform, so absence of evidence yields none rather than na.

   ⚑ flag

   Flagsmith~partialclaimed5/10

   Flagsmith documents Audit Log Webhooks that let you stream events (flag/segment/identity changes) into your own infrastructure, which is a real webhook subscription mechanism. However, evidence only covers audit-log-triggered webhooks, not a broader event/webhook API for arbitrary flag or environment change subscriptions, and there's no independent corroboration of reliability. Missing for 10: documentation of a general-purpose webhook subscription API beyond audit logs, and any hands-on/independent verification of webhook delivery in production use.

   • [claimed-docs] “You can stream your Audit Logs into your own infrastructure using Audit Log Webhooks.”
   • [claimed-docs] “Every action taken within the Flagsmith administration application is tracked and logged.”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Statsig
   Statsig~partialprobed5/10

   Statsig's MCP server explicitly lets AI clients (Claude Code, Cursor, Codex) 'ask questions, explore experiments, and access your Statsig data using AI,' which is a form of AI-generated insight retrieval, and this is confirmed live via a working OAuth-gated MCP endpoint. However this is an external-agent integration rather than a native in-product AI insights/suggestions feature (e.g., no evidence of built-in AI-generated experiment summaries, anomaly explanations, or suggestion widgets inside the Statsig console itself). Missing for 10: native in-app AI-generated insights/copilot UI, evidence of proactive AI suggestions surfaced directly in the console rather than via external MCP client.

   • [claimed-docs] “The Statsig MCP (Model Context Protocol) server brings the power of Statsig into tools like Codex, Cursor, and Claude Code. With this setup,…”
   • [claimed-docs] “The Statsig MCP (Model Context Protocol) server brings the power of Statsig into tools like Codex, Cursor, and Claude Code.”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.statsig.com/v1/mcp returned HTTP …”
   • [probe] “official MCP server documented at https://docs.statsig.com/integrations/mcp/overview”

   proof ↗⚑ flag

   Flagsmith—none0/10

   Flagsmith provides an MCP server for external AI agents to call its Admin API (docs-17/29) and a 'release pipeline' described as an 'intelligent release assistant' (docs-37), but this is rule-based automation and API access, not evidence of the product itself generating AI insights or suggestions from data (e.g., anomaly detection, usage analysis, recommended flags/segments). No documented AI/ML-driven insight or suggestion feature is present in the evidence pack.

   • [claimed-docs] “Release Pipelines act as your intelligent release assistant, automating your entire feature flag deployment process.”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Flagsmith
   Statsig~partialprobed6/10

   Statsig supports background-running automations like Scheduled Rollouts (gradual gate rollout without manual intervention) and Autotune (autonomously shifts traffic toward winning variants), plus CLI integration into CI/CD pipelines for automated gate/experiment management. These are real autonomous background mechanisms, but they are experimentation/rollout automations rather than general-purpose AI-agent automations, and there's no evidence of scheduling/triggering arbitrary agentic workflows. Missing for 10: evidence of general-purpose scheduled/triggered agent workflows beyond rollout/autotune, and independent verification that these run reliably unattended in production.

   • [claimed-docs] “Gradually deploy a feature over time by setting up a Feature Gate as a Scheduled Rollout.”
   • [claimed-docs] “Unlike a standard A/B test, Autotune shifts traffic toward the winning variant to maximize a target metric.”
   • [claimed-docs] “Autotune shifts traffic toward the winning variant to maximize a target metric.”
   • [claimed-docs] “You can integrate the CLI into CI/CD pipelines to automate the management of Feature Gates and Experiments.”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Statsig CLI (npm @statsig/siggy) ran keylessly via `npx -y @statsig/siggy --version`, prin…”

   proof ↗⚑ flag

   Flagsmith~partialprobed7/10

   Flagsmith documents concrete autonomous background automation: Scheduled Flags apply changes at a future time without manual intervention, and Release Pipelines define stages with triggers/actions that automatically progress rollouts to environments/audiences, generating change requests on promotion. The MCP server also lets AI agents programmatically drive these flows, and a runtime probe confirms the hosted MCP endpoint is live (though key-gated, un-exercised end-to-end). Missing for 10: independent/hands-on verification that Release Pipelines actually execute staged autonomous rollouts in practice, and a verified authenticated MCP-driven automation run rather than just a 401 probe.

   • [claimed-docs] “Scheduled flags allow you to queue and automatically apply changes to feature flags at a specified future time, eliminating the need for man…”
   • [claimed-docs] “Scheduled flags allow you to queue and automatically apply changes to feature flags at a specified future time”
   • [claimed-docs] “Define stages that your flags progress through automatically, with triggers and actions that control rollouts to selected environments and a…”
   • [claimed-docs] “Release Pipelines can automatically generate change requests when promoting flags to production, eliminating manual effort and streamlining …”
   • [claimed-docs] “Release Pipelines act as your intelligent release assistant, automating your entire feature flag deployment process.”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API”
   • [probe] “official MCP server documented at https://docs.flagsmith.com/integrating-with-flagsmith/mcp-server”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the documented hosted MCP server https://mcp.flagsmith.com returned…”

   proof ↗⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round drawn
   Statsig—none0/10

   Evidence covers Statsig's MCP server, which lets external AI tools (Cursor, Claude Code, Codex) query Statsig data — the opposite direction of an in-product assistant that users delegate tasks to. No evidence of a built-in AI assistant embedded in the Statsig console itself.

   ⚑ flag

   Flagsmith—none0/10

   Flagsmith exposes an MCP server so external AI agents can call its Admin API, and calls its automated release pipeline an 'intelligent release assistant,' but neither is a built-in AI assistant inside the Flagsmith product that a user can converse with or delegate tasks to — the MCP server is a server-side integration point for external agents, not a first-party in-app assistant.

   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API”
   • [claimed-docs] “Release Pipelines act as your intelligent release assistant, automating your entire feature flag deployment process.”
   • [probe] “official MCP server documented at https://docs.flagsmith.com/integrating-with-flagsmith/mcp-server”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the documented hosted MCP server https://mcp.flagsmith.com returned…”

   proof ↗⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round to Flagsmith
   Statsig~partialprobed6/10

   Statsig ships a first-party MCP server (verified live via runtime probe) that lets AI clients like Claude Code and Cursor query experiments, gates, and data using natural language, which is the core mechanism for natural-language operation. However, this is an agentic/AI-integration layer bolted onto a console/CLI/API-driven product rather than a native NL command interface across the whole product surface. missing for 10: evidence of natural-language command support within the core Statsig console UI itself (not just via external MCP-connected agents), independent hands-on validation of NL command accuracy/reliability, and coverage of write-actions (creating/modifying gates or experiments) via natural language rather than read-only querying.

   • [claimed-docs] “The Statsig MCP (Model Context Protocol) server brings the power of Statsig into tools like Codex, Cursor, and Claude Code. With this setup,…”
   • [claimed-docs] “The Statsig MCP (Model Context Protocol) server brings the power of Statsig into tools like Codex, Cursor, and Claude Code.”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.statsig.com/v1/mcp returned HTTP …”
   • [probe] “official MCP server documented at https://docs.statsig.com/integrations/mcp/overview”

   proof ↗⚑ flag

   Flagsmith~partialprobed7/10

   Flagsmith ships an official hosted MCP server (mcp.flagsmith.com) exposing the Admin API to AI assistants/agents, confirmed live via a runtime probe (401 requiring auth, not a failure), which enables natural-language operation through an MCP-compatible AI client, plus a CLI for terminal-based flag management. However, this is agent-mediated (via MCP client) rather than native NLP in the product itself, and there's no hands-on evidence of a full natural-language command actually being executed end-to-end through an AI assistant. Missing for 10: a demonstrated end-to-end natural-language interaction transcript, and independent (non-vendor) corroboration of the MCP server working with a real AI agent.

   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API”
   • [probe] “official MCP server documented at https://docs.flagsmith.com/integrating-with-flagsmith/mcp-server”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the documented hosted MCP server https://mcp.flagsmith.com returned…”
   • [claimed-docs] “The Flagsmith CLI lets you manage flags, segments, features, projects and environments from your terminal, and evaluate flags the way an SDK…”
   • [probe] “official CLI documented at https://docs.flagsmith.com/integrating-with-flagsmith/CLI”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Statsig—none0/10

   Statsig documents an HTTP API and Console API but there is no evidence of an interactive API reference with runnable/try-it examples; explicit probes for OpenAPI/Swagger endpoints returned 404s, indicating no such interactive explorer exists.

   • [claimed-docs] “Overview of the Statsig HTTP API for retrieving feature gate, experiment, and dynamic config values and logging events directly without an S…”
   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.statsig.com/openapi.json, https://docs.statsig.com/swagger.json, https://docs.statsig.c…”

   proof ↗⚑ flag

   Flagsmith—none0/10

   The evidence pack explicitly shows probes for an OpenAPI/Swagger spec and llms.txt returning 404s, and no citation describes an interactive, runnable API reference (e.g., Swagger UI, Postman collection, or live code playground). Only static API overview docs and an Admin API description are mentioned, none of which are runnable/interactive.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.flagsmith.com/openapi.json, https://docs.flagsmith.com/swagger.json, https://docs.flags…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://docs.flagsmith.com/llms.txt”
   • [claimed-docs] “Authentication: Uses a public, non-secret Environment Key. Security: Open by design. The Environment Key can be exposed in client-side code.”
   • [claimed-docs] “This is the API you use to programmatically manage your Flagsmith projects. Anything you can do in the Flagsmith dashboard, you can also do …”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   Statsig—none0/10

   Statsig documents a Console API and HTTP API but a direct probe for OpenAPI/swagger specs at common paths (openapi.json, swagger.json, api/openapi.json, .well-known/openapi.json) all returned 404, and no docs reference a downloadable machine-readable spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.statsig.com/openapi.json, https://docs.statsig.com/swagger.json, https://docs.statsig.c…”
   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [claimed-docs] “Overview of the Statsig HTTP API for retrieving feature gate, experiment, and dynamic config values and logging events directly without an S…”

   proof ↗⚑ flag

   Flagsmith—none0/10

   Flagsmith documents a full Admin API and CLI/MCP integrations, but explicit probes for a machine-readable spec (openapi.json, swagger.json, .well-known/openapi.json) all returned 404, and no evidence pack item points to a downloadable OpenAPI/Swagger spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.flagsmith.com/openapi.json, https://docs.flagsmith.com/swagger.json, https://docs.flags…”
   • [claimed-docs] “This is the API you use to programmatically manage your Flagsmith projects. Anything you can do in the Flagsmith dashboard, you can also do …”
   • [claimed-docs] “Purpose: Creating, updating, and deleting projects, environments, flags, segments, and users.”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Flagsmith
   Statsig—none0/10

   The evidence pack documents feature gates, experiments, CLI, and MCP integrations, but nothing describes a distinct sandbox/staging environment or project isolated from production data for testing purposes — only production-oriented tools like emergency disable switches, gate overrides, and scheduled rollouts are mentioned. This is a fair question for a feature-flag/experimentation platform, but no evidence confirms a sandbox capability exists.

   • [claimed-docs] “Feature Gates, commonly known as feature flags, allow you to toggle the behavior of your product in real time without deploying new code.”
   • [claimed-docs] “Have an emergency disable switch that lets you immediately turn off a particular code branch for users in production”
   • [claimed-docs] “After your Feature Gate is live, view Feature Gate exposures in the Statsig console to monitor who encounters your gate.”
   • [claimed-docs] “set up Feature Gate overrides to allow specific users to bypass the gate.”
   • [claimed-docs] “Basic settings include automatically adding new users with the same email domain to your project. Statsig also supports SSO to simplify invi…”

   proof ↗⚑ flag

   Flagsmith~partialprobed7/10

   Flagsmith supports multiple environments (dev/staging/prod) and fully self-hostable Docker setups that let users create, toggle, and evaluate flags without touching production, and a probe confirms a full local create-and-evaluate flag loop working offline; RBAC also lets teams restrict who can modify production. However, none of this is framed specifically as an 'AI-native sandbox' testing workflow, and the MCP/agent-facing docs don't call out sandbox vs. production isolation explicitly. Missing for 10: an explicit sandbox/staging environment guide oriented at AI agents, and confirmation that the MCP server itself supports pointing at a non-prod environment safely.

   • [github] “Just wrap a section of code with a flag, and then use Flagsmith to toggle that feature on or off for different environments, users or user s…”
   • [claimed-docs] “Self-hosting Flagsmith allows you to run the complete Flagsmith platform within your own infrastructure, giving you full control over your d…”
   • [claimed-docs] “You can use Docker to set up an entire Flagsmith Feature Flag environment locally”
   • [claimed-docs] “You can use Docker to set up an entire [Flagsmith Feature Flag](https://www.flagsmith.com) environment locally”
   • [claimed-docs] “Only allow certain users to modify your production environments.”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Statsig—none0/10

   Statsig documents an HTTP API and Console API but the evidence pack contains no mention of API versioning scheme or a documented deprecation policy; a probe for OpenAPI spec files (which would typically reveal version info) returned 404s across all candidate paths.

   • [claimed-docs] “Overview of the Statsig HTTP API for retrieving feature gate, experiment, and dynamic config values and logging events directly without an S…”
   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.statsig.com/openapi.json, https://docs.statsig.com/swagger.json, https://docs.statsig.c…”

   proof ↗⚑ flag

   Flagsmith—none0/10

   The evidence pack documents Flagsmith's Admin API, SDKs, CLI, and MCP server, but nowhere mentions API version numbers, a versioning scheme, or a documented deprecation policy; probes even show openapi.json/spec endpoints returning 404, suggesting no discoverable API contract documentation. Missing for 10: any statement of API versioning strategy, deprecation timelines, or migration guides for breaking changes.

   • [claimed-docs] “Authentication: Uses a public, non-secret Environment Key. Security: Open by design. The Environment Key can be exposed in client-side code.”
   • [claimed-docs] “Purpose: Creating, updating, and deleting projects, environments, flags, segments, and users.”
   • [claimed-docs] “Anything you can do in the Flagsmith dashboard, you can also do via the Admin API.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.flagsmith.com/openapi.json, https://docs.flagsmith.com/swagger.json, https://docs.flags…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round drawn
   Statsig~partialprobed6/10

   Statsig's Console API provides a CRUD API for managing gates, experiments, and configs programmatically, and the CLI (confirmed runnable via npx) can be scripted/integrated into CI/CD, enabling automation across many items rather than one-by-one UI clicks. However, no evidence explicitly documents a bulk/batch endpoint or bulk-edit feature, so bulk operations must be assembled by looping API/CLI calls rather than a native bulk primitive; missing for 10: explicit bulk/batch API endpoints, documented multi-item transactional updates, and independent confirmation of large-scale automation use.

   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [claimed-docs] “The Statsig CLI is a command-line interface that helps manage Feature Gates, Experiments, and Dynamic Configs within the Statsig platform.”
   • [claimed-docs] “You can integrate the CLI into CI/CD pipelines to automate the management of Feature Gates and Experiments.”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Statsig CLI (npm @statsig/siggy) ran keylessly via `npx -y @statsig/siggy --version`, prin…”

   proof ↗⚑ flag

   Flagsmith~partialprobed6/10

   Flagsmith's Admin API, CLI, and MCP server expose full CRUD over flags, segments, projects, environments and users programmatically, which lets an AI-native user script operations across many items in a loop (docs-16, docs-18, docs-17/29, probe-3/4). However, there is no documented explicit bulk/batch endpoint, bulk import/export tool, or multi-item transactional operation — only single-resource CRUD calls that must be iterated. Missing for 10: dedicated bulk/batch API endpoints, bulk import/export documentation, evidence of atomic multi-item operations.

   • [claimed-docs] “Purpose: Creating, updating, and deleting projects, environments, flags, segments, and users.”
   • [claimed-docs] “The Flagsmith CLI lets you manage flags, segments, features, projects and environments from your terminal, and evaluate flags the way an SDK…”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API”
   • [probe] “official MCP server documented at https://docs.flagsmith.com/integrating-with-flagsmith/mcp-server”
   • [probe] “official CLI documented at https://docs.flagsmith.com/integrating-with-flagsmith/CLI”
   • [claimed-docs] “This is the API you use to programmatically manage your Flagsmith projects. Anything you can do in the Flagsmith dashboard, you can also do …”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Flagsmith
   Statsig~partialclaimed4/10

   Statsig supports rule-based automation such as targeting rules on Feature Gates, Scheduled Rollouts, and Autotune (which automatically shifts traffic toward a winning variant based on live metrics) — these are forms of automatic rule-triggered behavior, but they are evaluation/targeting rules and traffic-shift algorithms rather than a general 'on event X, trigger action Y' automation engine. Missing for 10: documented event-triggered workflow/webhook automation (e.g., alert-to-action pipelines), explicit event-based trigger definitions distinct from targeting/eval rules, and independent hands-on evidence of such automation working end-to-end.

   • [claimed-docs] “Gradually deploy a feature over time by setting up a Feature Gate as a Scheduled Rollout.”
   • [claimed-docs] “Unlike a standard A/B test, Autotune shifts traffic toward the winning variant to maximize a target metric.”
   • [claimed-docs] “Schedule gradual feature rollouts to safely deploy new code”
   • [claimed-docs] “Autotune shifts traffic toward the winning variant to maximize a target metric.”
   • [claimed-docs] “set up Feature Gate overrides to allow specific users to bypass the gate.”
   • [claimed-docs] “Infra Analytics: Monitor and debug service health alongside product outcomes. Ingest metrics and traces with OpenTelemetry, search and group…”

   proof ↗⚑ flag

   Flagsmith~partialclaimed6/10

   Release Pipelines let users define stages with 'triggers and actions' that automatically progress flags/rollouts to environments and audiences, and Scheduled Flags apply changes automatically at a specified time — both are rule-based automated actions. Audit Log Webhooks can stream internal events for downstream automation. However, evidence covers only time/stage-based and rollout triggers rather than generic arbitrary-event-triggered actions (e.g., inbound webhook conditions tied to external signals), and there's no independent/hands-on corroboration of the pipeline automation working as described. Missing for 10: broader event-source flexibility (e.g., external event ingestion triggering flag changes), and independent verification of release-pipeline automation in practice.

   • [claimed-docs] “Define stages that your flags progress through automatically, with triggers and actions that control rollouts to selected environments and a…”
   • [claimed-docs] “Release Pipelines can automatically generate change requests when promoting flags to production, eliminating manual effort and streamlining …”
   • [claimed-docs] “Release Pipelines act as your intelligent release assistant, automating your entire feature flag deployment process.”
   • [claimed-docs] “Scheduled flags allow you to queue and automatically apply changes to feature flags at a specified future time, eliminating the need for man…”
   • [claimed-docs] “Scheduled flags allow you to queue and automatically apply changes to feature flags at a specified future time”
   • [claimed-docs] “You can stream your Audit Logs into your own infrastructure using Audit Log Webhooks.”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round to Flagsmith
   Statsig—none0/10

   Statsig only documents one-time 'Scheduled Rollouts' for feature gates and CLI/CI-CD scripting of flag management (statsig-docs-3, statsig-docs-25, statsig-docs-31); there is no evidence of a recurring job/workflow scheduler (e.g., cron-like recurring automation) that an AI-native user could set up. missing for 10: any documented recurring/cron-style job scheduling, workflow orchestration, or automated recurring task execution beyond a single scheduled feature rollout.

   • [claimed-docs] “Gradually deploy a feature over time by setting up a Feature Gate as a Scheduled Rollout.”
   • [claimed-docs] “Schedule gradual feature rollouts to safely deploy new code”
   • [claimed-docs] “You can integrate the CLI into CI/CD pipelines to automate the management of Feature Gates and Experiments.”

   proof ↗⚑ flag

   Flagsmith~partialclaimed4/10

   Flagsmith supports 'Scheduled flags' to queue a one-time future flag change, and 'Release Pipelines' that automatically progress flags through stages with triggers/actions — both provide some automation depth, but neither is documented as a recurring/cron-style job scheduler that repeats on an interval. Missing for 10: explicit recurring/cron scheduling, repeatable automated workflow triggers beyond a single future timestamp, and any AI-agent-driven orchestration of repeating tasks.

   • [claimed-docs] “Scheduled flags allow you to queue and automatically apply changes to feature flags at a specified future time, eliminating the need for man…”
   • [claimed-docs] “Scheduled flags allow you to queue and automatically apply changes to feature flags at a specified future time”
   • [claimed-docs] “Define stages that your flags progress through automatically, with triggers and actions that control rollouts to selected environments and a…”
   • [claimed-docs] “Release Pipelines can automatically generate change requests when promoting flags to production, eliminating manual effort and streamlining …”
   • [claimed-docs] “Release Pipelines act as your intelligent release assistant, automating your entire feature flag deployment process.”

   proof ↗⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round to Flagsmith
   Statsig~partialclaimed4/10

   Statsig's feature gates support rollback-like controls (emergency disable switch, scheduled rollouts) and CLI/Console API can be scripted into CI/CD for programmatic management, implying some versioned control over flag state, but the evidence never documents an explicit audit/version history, diffing, or approval-review workflow for these 'automations'. missing for 10: explicit version history/audit log for gate or experiment changes, a review/approval workflow before changes go live, and a documented one-click rollback to a prior version (only an on/off disable switch is shown).

   • [claimed-docs] “Have an emergency disable switch that lets you immediately turn off a particular code branch for users in production”
   • [claimed-docs] “Gradually deploy a feature over time by setting up a Feature Gate as a Scheduled Rollout.”
   • [claimed-docs] “The Statsig CLI is a command-line interface that helps manage Feature Gates, Experiments, and Dynamic Configs within the Statsig platform.”
   • [claimed-docs] “You can integrate the CLI into CI/CD pipelines to automate the management of Feature Gates and Experiments.”
   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [claimed-docs] “After your Feature Gate is live, view Feature Gate exposures in the Statsig console to monitor who encounters your gate.”

   proof ↗⚑ flag

   Flagsmith✓fullclaimed7/10

   Flagsmith provides a four-eyes Change Request workflow analogous to PR review (docs-10), full audit logs tracking every flag/segment/identity change over time (docs-8, docs-39), and Release Pipelines with automated stage-gated rollouts and auto-generated change requests (docs-6, docs-34, docs-37), plus instant toggle-off to revert behavior without redeploying (flagsmith-gh-1). Together these cover versioning, review, and effective rollback of flag-driven automations. missing for 10: an explicit named 'rollback to previous flag version' feature (evidence only shows audit trail + manual re-toggle, not one-click revert), and independent/hands-on confirmation of the change-request/pipeline workflow in practice.

   • [claimed-docs] “Every action taken within the Flagsmith administration application is tracked and logged. This allows you to easily retrace the events and v…”
   • [claimed-docs] “Feature Change Requests help creating a four-eyes workflow (create, approve, publish) to updating feature flags, similarly to Pull Requests …”
   • [claimed-docs] “Define stages that your flags progress through automatically, with triggers and actions that control rollouts to selected environments and a…”
   • [claimed-docs] “Release Pipelines can automatically generate change requests when promoting flags to production, eliminating manual effort and streamlining …”
   • [claimed-docs] “Release Pipelines act as your intelligent release assistant, automating your entire feature flag deployment process.”
   • [github] “Make changes remotely: Easily toggle individual features on and off, and make changes without deploying new code”
   • [claimed-docs] “Every action taken within the Flagsmith administration application is tracked and logged.”

   proof ↗⚑ flag

Proxy

1. platform engineerRun a relay/edge proxy so flags stay served when the vendor is unreachable and SDK traffic stays inside my network

   weight 1 · round to Flagsmith
   Statsig—none0/10

   No evidence in the pack describes a self-hostable relay/edge proxy for offline or in-network flag serving; documentation covers SDKs, HTTP API, CLI, and MCP but nothing about a proxy component that caches flags locally when Statsig's servers are unreachable.

   ⚑ flag

   Flagsmith✓fullclaimed8/10

   Flagsmith documents a self-hosted Edge Proxy service that runs the Flagsmith Engine within your own infrastructure and is connected to directly by server-side SDKs, keeping evaluation traffic local and resilient to vendor outages. This directly matches the platform-engineer story of running a relay/edge proxy for network-local flag serving. Missing for 10: independent hands-on validation of failover behavior when the vendor API is unreachable, and more detail on sync/refresh mechanics between proxy and main API.

   • [claimed-docs] “The Flagsmith Edge Proxy is a service that you host yourself, that allows you to run an instance of the Flagsmith Engine close to your serve…”
   • [claimed-docs] “Run the Edge Proxy within in your own infrastructure and connect to it from your server-side SDKs”
   • [claimed-docs] “Self-hosting Flagsmith allows you to run the complete Flagsmith platform within your own infrastructure, giving you full control over your d…”

   proof ↗⚑ flag

Self host

1. platform engineerSelf-host the full flag platform from an open-source distribution, keeping evaluation data on my infrastructure

   weight 3 · round to Flagsmith
   Statsig—none0/10

   No evidence in the pack mentions an open-source self-hosted distribution of the full Statsig platform; all material describes the hosted SaaS console, SDKs, CLI, and MCP integrations that connect to Statsig's cloud API. Absence of evidence for this applicable axis means it is not delivered.

   ⚑ flag

   Flagsmith✓fullprobed9/10

   Flagsmith's docs explicitly describe self-hosting the complete platform for full data/security control, with a Docker-based setup guide, and this was independently verified via a real keyless self-host probe that created an org/project/environment/flag and successfully evaluated it against the local instance. missing for 10: no independent third-party audit of production-scale self-hosted deployments or HA/scaling guidance beyond the basic Docker quickstart.

   • [claimed-docs] “Self-hosting Flagsmith allows you to run the complete Flagsmith platform within your own infrastructure, giving you full control over your d…”
   • [claimed-docs] “You can use Docker to set up an entire Flagsmith Feature Flag environment locally”
   • [claimed-docs] “You can use Docker to set up an entire [Flagsmith Feature Flag](https://www.flagsmith.com) environment locally”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”

   proof ↗⚑ flag

Agent ops

1. ai agentRead experiment configurations and results programmatically to summarize outcomes and recommend ship/rollback decisions

   weight 1 · round to Statsig
   Statsig✓fullprobed8/10

   Statsig provides a Console API (CRUD for experiments/gates), an HTTP API for retrieving experiment/gate values, a CLI scriptable in CI, and an official MCP server explicitly documented to let AI tools like Claude Code/Cursor/Codex 'ask questions, explore experiments, and access your Statsig data' — directly enabling an AI agent to programmatically read experiment configs/results. Runtime probes confirm the MCP server and CLI are live and reachable. Missing for 10: explicit documented example of an agent using these APIs to produce a ship/rollback recommendation (only general data-access capability is evidenced, not the recommendation workflow itself).

   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [claimed-docs] “Overview of the Statsig HTTP API for retrieving feature gate, experiment, and dynamic config values and logging events directly without an S…”
   • [claimed-docs] “The Statsig MCP (Model Context Protocol) server brings the power of Statsig into tools like Codex, Cursor, and Claude Code. With this setup,…”
   • [claimed-docs] “The Statsig CLI is a command-line interface that helps manage Feature Gates, Experiments, and Dynamic Configs within the Statsig platform.”
   • [claimed-docs] “You can integrate the CLI into CI/CD pipelines to automate the management of Feature Gates and Experiments.”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Statsig CLI (npm @statsig/siggy) ran keylessly via `npx -y @statsig/siggy --version`, prin…”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.statsig.com/v1/mcp returned HTTP …”

   proof ↗⚑ flag

   Flagsmith~partialprobed5/10

   Flagsmith documents an Experimentation feature with a Bayesian statistics engine for reading A/B test results (flagsmith-docs-7/25/35/38), and separately an official MCP server plus Admin API giving agents programmatic access to flags/segments/environments (flagsmith-docs-17/29, probe-3, probe-rt-1). However, no evidence confirms the MCP/Admin API surface exposes the experiment analytics/results data (vs. just flag/segment CRUD), so an agent's ability to programmatically pull experiment outcomes (not just configs) for ship/rollback recommendations is unconfirmed. Missing for 10: documented API/MCP endpoint for reading experiment statistics results, and any example of an agent consuming that data to produce a recommendation.

   • [claimed-docs] “serve variations of a feature with a multivariate flag, collect events from your application into a managed data warehouse, and read the res…”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API”
   • [probe] “official MCP server documented at https://docs.flagsmith.com/integrating-with-flagsmith/mcp-server”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the documented hosted MCP server https://mcp.flagsmith.com returned…”
   • [claimed-docs] “Flagsmith Experimentation lets you run A/B tests end to end on the platform: serve variations of a feature with a multivariate flag, collect…”
   • [claimed-docs] “Purpose: Creating, updating, and deleting projects, environments, flags, segments, and users.”

   proof ↗⚑ flag

Analysis

1. product managerTrust a documented statistics engine (Bayesian or frequentist, with variance-reduction options) behind experiment results

   weight 2 · round drawn
   Statsig~partialcommunity6/10

   Statsig documents a full experimentation suite with variance-reduction techniques (CUPED, stratified sampling, switchback tests) and hypothesis/metric scorecards, and community sentiment praises its 'stat rigor,' but the evidence never names or details whether the underlying engine is Bayesian, frequentist, or a documented hybrid methodology. missing for 10: explicit documentation of the statistical model (Bayesian vs frequentist), confidence-interval/p-value methodology details, and independent validation of statistical correctness beyond anecdotal praise.

   • [claimed-docs] “Statsig offers a full suite of experiment measurement tools, including CUPED, Stratified Sampling, Switchback Tests, and more.”
   • [claimed-docs] “The Scorecard feature makes this easy by letting you enter your hypothesis and select both primary and secondary metrics.”
   • [claimed-docs] “Statsig Warehouse Native is an experimentation platform that runs analysis directly in your data warehouse.”
   • [community] “Hats off to Statsig. They built a stellar product. Superior to many of their industry competitors like Optimizely. Back when I was on an int…”

   proof ↗⚑ flag

   Flagsmith~partialclaimed6/10

   Flagsmith explicitly documents an experimentation feature with a 'built-in Bayesian statistics engine' reading results from multivariate flags and warehoused events, giving product managers a named statistical method behind results. However, the docs give no detail on the underlying model, confidence intervals, sample-ratio checks, or frequentist alternative, and there is no independent/hands-on corroboration of the engine's correctness or variance-reduction techniques. Missing for 10: documentation of specific statistical methodology/assumptions, variance-reduction (e.g., CUPED) support, and third-party validation of the engine's results.

   • [claimed-docs] “serve variations of a feature with a multivariate flag, collect events from your application into a managed data warehouse, and read the res…”
   • [claimed-docs] “read the results with a built-in Bayesian statistics engine”
   • [claimed-docs] “collect events from your application into a managed data warehouse, and read the results with a built-in Bayesian statistics engine.”
   • [claimed-docs] “Flagsmith Experimentation lets you run A/B tests end to end on the platform: serve variations of a feature with a multivariate flag, collect…”

   proof ↗⚑ flag

Experiments

1. product managerRun A/B and multivariate experiments on flags and see which variation wins on my metrics

   weight 3 · round to Statsig
   Statsig✓fullcommunity9/10

   Statsig's docs directly describe running A/B/n and multivariate experiments with primary/secondary metrics via the Scorecard, plus Layers for managing multiple experiments and Autotune for identifying/shifting toward winning variants, and community feedback corroborates its statistical rigor versus competitors. Missing for 10: independent hands-on validation of the results/analysis UI itself (e.g., screenshots or third-party review of the 'winning variation' scorecard output) rather than only docs.

   • [claimed-docs] “Statsig experimentation runs randomized controlled trials (A/B or A/B/n tests) that measure how product changes affect your key metrics.”
   • [claimed-docs] “The Scorecard feature makes this easy by letting you enter your hypothesis and select both primary and secondary metrics.”
   • [claimed-docs] “Unlike a standard A/B test, Autotune shifts traffic toward the winning variant to maximize a target metric.”
   • [claimed-docs] “Autotune shifts traffic toward the winning variant to maximize a target metric.”
   • [claimed-docs] “A Layer allows you to manage multiple experiments and feature flags together.”
   • [community] “Hats off to Statsig. They built a stellar product. Superior to many of their industry competitors like Optimizely. Back when I was on an int…”
   • [community] “Have implemented Statsig in two companies as alternatives to LaunchDarkly and loved it both times. Going to be interesting how their other b…”

   proof ↗⚑ flag

   Flagsmith✓fullclaimed7/10

   Flagsmith's Experimentation feature explicitly supports serving multivariate (A/B/n) flag variations, collecting events into a managed data warehouse, and reading results via a built-in Bayesian statistics engine, directly matching the PM story of running A/B/MVT tests and seeing which variation wins on metrics. Missing for 10: independent/hands-on evidence of the experimentation UI in action (e.g., screenshots or third-party reports of actual win/loss determination), and detail on which metrics/goals can be tracked or how statistical significance is surfaced to a non-technical PM.

   • [claimed-docs] “serve variations of a feature with a multivariate flag, collect events from your application into a managed data warehouse, and read the res…”
   • [claimed-docs] “Flagsmith Experimentation lets you run A/B tests end to end on the platform: serve variations of a feature with a multivariate flag, collect…”
   • [claimed-docs] “Multivariate flags let you define multiple variants with percentage weightings (A/B/n).”
   • [claimed-docs] “read the results with a built-in Bayesian statistics engine”
   • [claimed-docs] “collect events from your application into a managed data warehouse, and read the results with a built-in Bayesian statistics engine.”

   proof ↗⚑ flag

Metrics

1. product managerDefine experiment metrics from my own data — warehouse tables or ingested events — instead of a black-box metric store

   weight 2 · round to Statsig
   Statsig✓fullclaimed7/10

   Statsig Warehouse Native explicitly runs experiment analysis directly on the customer's own data warehouse rather than a black-box metric store, and Statsig's core event-ingestion pipeline lets PMs define metrics from ingested events with a full analysis toolkit (CUPED, stratified sampling, switchback tests). missing for 10: independent/hands-on corroboration of Warehouse Native in production and detail on the custom-metric-definition UI/workflow itself.

   • [claimed-docs] “Statsig Warehouse Native is an experimentation platform that runs analysis directly in your data warehouse.”
   • [claimed-docs] “Statsig offers a full suite of experiment measurement tools, including CUPED, Stratified Sampling, Switchback Tests, and more.”
   • [claimed-docs] “Statsig experimentation runs randomized controlled trials (A/B or A/B/n tests) that measure how product changes affect your key metrics.”
   • [claimed-docs] “Experiments enable you to run randomized controlled trials (A/B or A/B/n tests) to measure the impact of product changes on key metrics.”

   proof ↗⚑ flag

   Flagsmith—none0/10

   Flagsmith's experimentation feature explicitly funnels events into its own 'managed data warehouse' and evaluates them via a built-in Bayesian engine, the opposite of letting a PM define metrics from their own warehouse tables or ingested events — there is no evidence of BYO-warehouse or custom event-source metric definition.

   • [claimed-docs] “serve variations of a feature with a multivariate flag, collect events from your application into a managed data warehouse, and read the res…”
   • [claimed-docs] “collect events from your application into a managed data warehouse, and read the results with a built-in Bayesian statistics engine.”
   • [claimed-docs] “Flagsmith Experimentation lets you run A/B tests end to end on the platform: serve variations of a feature with a multivariate flag, collect…”
   • [claimed-docs] “read the results with a built-in Bayesian statistics engine”

   proof ↗⚑ flag

Environments

1. developerManage separate environments (dev/staging/prod) with independent flag states and scoped SDK keys

   weight 2 · round to Flagsmith
   Statsig—none0/10

   The evidence pack covers feature gates, experiments, CLI, console API, and access management, but nowhere mentions environment tags (dev/staging/prod) or environment-scoped SDK keys — a core Statsig platform capability that isn't documented here. Missing for 10: explicit environment/project configuration docs, environment-scoped SDK key generation, and any mention of promoting flag state across environments.

   ⚑ flag

   Flagsmith✓fullprobed8/10

   Flagsmith's core model is organized around projects with multiple environments (dev/staging/prod), each with its own independent flag states and its own scoped Environment/API key that's safe to expose client-side, confirmed by docs and a live probe showing environment-scoped key evaluation working end to end. RBAC further lets teams restrict who can modify production environments specifically. missing for 10: no explicit doc snippet walking through creating multiple named environments side-by-side or promoting flag state between them, and no independent/community corroboration beyond vendor docs and the probe.

   • [claimed-docs] “Authentication: Uses a public, non-secret Environment Key. Security: Open by design. The Environment Key can be exposed in client-side code.”
   • [claimed-docs] “Purpose: Creating, updating, and deleting projects, environments, flags, segments, and users.”
   • [claimed-docs] “Only allow certain users to modify your production environments.”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”
   • [claimed-docs] “Manage feature flags and remote config across web, mobile, and server-side apps.”

   proof ↗⚑ flag

Flags

1. developerCreate a feature flag and toggle it live in production within minutes of signing up

   weight 3 · round to Flagsmith
   Statsig✓fullcommunity8/10

   Docs provide a first-feature tutorial creating a gate, targeting users, and checking it live via SDK within a browser console (statsig-docs-24), plus a free-tier signup with feature flags included (statsig-docs-17), console-based exposure monitoring and overrides (statsig-docs-19, statsig-docs-29), and community praise confirming Statsig ships fast, developer-friendly tooling (statsig-comm-1, statsig-comm-2). Missing for 10: independent hands-on account of actual time-to-first-flag from signup and no third-party benchmark of onboarding speed.

   • [claimed-docs] “This tutorial creates a feature gate, targets mobile platforms and internal testers, then checks it live with the JavaScript SDK in your bro…”
   • [claimed-docs] “You get feature flags and 2 million metered events for free.”
   • [claimed-docs] “After your Feature Gate is live, view Feature Gate exposures in the Statsig console to monitor who encounters your gate.”
   • [claimed-docs] “set up Feature Gate overrides to allow specific users to bypass the gate.”
   • [community] “Hats off to Statsig. They built a stellar product. Superior to many of their industry competitors like Optimizely. Back when I was on an int…”
   • [community] “Have implemented Statsig in two companies as alternatives to LaunchDarkly and loved it both times. Going to be interesting how their other b…”

   proof ↗⚑ flag

   Flagsmith✓fullprobed9/10

   Docs and a hands-on runtime probe confirm the core flow: create a feature via the dashboard/API, name it, toggle it on/off per environment without deploying code, and evaluate it live via SDK/API within minutes — the probe even demonstrates a full keyless create-and-evaluate loop from scratch. Docker quickstart and clear UI steps (Create Feature, toggle) further support fast time-to-value. missing for 10: no independent third-party timing benchmark or new-user onboarding video confirming the 'within minutes' claim beyond docs/probe.

   • [claimed-docs] “Go to the **Features** section in your dashboard. Click **Create Feature**.”
   • [claimed-docs] “Enter a descriptive name for your flag (e.g., new_ui_enabled). Fill in the available fields according to the specifications of your feature.”
   • [github] “Make changes remotely: Easily toggle individual features on and off, and make changes without deploying new code”
   • [github] “Easily toggle individual features on and off, and make changes without deploying new code”
   • [claimed-docs] “You can use Docker to set up an entire Flagsmith Feature Flag environment locally”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”

   proof ↗⚑ flag

2. developerServe multivariate flags and dynamic configuration values (strings, numbers, JSON), not just booleans

   weight 2 · round to Flagsmith
   Statsig~partialclaimed6/10

   Statsig docs reference Dynamic Configs alongside Feature Gates and Experiments (e.g., statsig-docs-9, statsig-docs-10, statsig-docs-11, statsig-docs-31 mention 'Dynamic Configs' as a managed entity distinct from booleans), and experiments/layers imply multivariate parameter delivery (statsig-docs-4, statsig-docs-26, statsig-docs-30). However, no evidence explicitly confirms support for strings/numbers/JSON value types or shows a dynamic config JSON payload example. missing for 10: explicit documentation or example showing dynamic config value types (string/number/JSON), SDK code sample retrieving a non-boolean config value.

   • [claimed-docs] “The Statsig CLI is a command-line interface that helps manage Feature Gates, Experiments, and Dynamic Configs within the Statsig platform.”
   • [claimed-docs] “Overview of the Statsig HTTP API for retrieving feature gate, experiment, and dynamic config values and logging events directly without an S…”
   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [claimed-docs] “You can integrate the CLI into CI/CD pipelines to automate the management of Feature Gates and Experiments.”
   • [claimed-docs] “A Layer allows you to manage multiple experiments and feature flags together.”
   • [claimed-docs] “Statsig experimentation runs randomized controlled trials (A/B or A/B/n tests) that measure how product changes affect your key metrics.”

   proof ↗⚑ flag

   Flagsmith✓fullprobed9/10

   Flagsmith docs explicitly describe multivariate flags with percentage-weighted variants and remote config values, and core management docs show configurable flag fields (not just boolean toggles) plus experimentation using multivariate flags. This is corroborated by a self-hosted probe showing a real flag create/evaluate roundtrip. missing for 10: explicit doc confirmation of string/number/JSON config value types beyond variant weighting, and independent hands-on confirmation of non-boolean config value retrieval via SDK.

   • [claimed-docs] “Multivariate flags let you define multiple variants with percentage weightings (A/B/n).”
   • [claimed-docs] “serve variations of a feature with a multivariate flag, collect events from your application into a managed data warehouse, and read the res…”
   • [claimed-docs] “Enter a descriptive name for your flag (e.g., new_ui_enabled). Fill in the available fields according to the specifications of your feature.”
   • [claimed-docs] “Go to the **Features** section in your dashboard. Click **Create Feature**.”
   • [claimed-docs] “Flagsmith Experimentation lets you run A/B tests end to end on the platform: serve variations of a feature with a multivariate flag, collect…”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”

   proof ↗⚑ flag

Lifecycle

1. platform engineerFind stale flags and code references so temporary flags actually get removed from the codebase

   weight 2 · round drawn
   Statsig—none0/10

   The evidence pack covers feature gates, experiments, rollout scheduling, overrides, CLI, and MCP integrations, but nothing documents stale-flag detection, flag usage/code-reference scanning, or cleanup workflows. This is a reasonable axis for a feature-flag platform, but no capability is evidenced.

   ⚑ flag

   Flagsmith—none0/10

   The evidence pack covers targeting, rollouts, experimentation, audit logs, RBAC, and change requests, but contains no mention of stale-flag detection, code reference scanning, or flag lifecycle/cleanup tooling that would help identify and remove unused flags from a codebase. This is a fair capability to expect from a mature flag-management platform (competitors ship code-reference scanners), so absence of evidence means 'none' rather than 'na'.

   ⚑ flag

Targeting

1. developerTarget or exclude specific individual users for a flag (allowlists, beta testers, internal accounts)

   weight 1 · round to Flagsmith
   Statsig✓fullclaimed8/10

   Docs explicitly cover overrides for targeting/excluding specific users and a guided tutorial that targets internal testers/mobile platforms, directly matching the allowlist/beta-tester/internal-account use case. missing for 10: no independent hands-on confirmation of override UI behavior beyond first-party docs.

   • [claimed-docs] “set up Feature Gate overrides to allow specific users to bypass the gate.”
   • [claimed-docs] “This tutorial creates a feature gate, targets mobile platforms and internal testers, then checks it live with the JavaScript SDK in your bro…”
   • [claimed-docs] “After your Feature Gate is live, view Feature Gate exposures in the Statsig console to monitor who encounters your gate.”

   proof ↗⚑ flag

   Flagsmith✓fullclaimed9/10

   Flagsmith explicitly supports overriding feature flags for individual users via 'identities', documented for internal testing, QA, and personalized experiences, plus segment-based targeting for group-based allowlists (e.g., beta testers). This is a well-documented core feature directly matching the story. Missing for 10: independent third-party hands-on confirmation of identity-level overrides specifically (probes verify flag creation/eval but not the identity-override workflow itself).

   • [claimed-docs] “Override feature flags for individual users ("identities"). Enables internal testing, QA, customer support, or personalised experiences at t…”
   • [claimed-docs] “Override feature flags for individual users ("identities").”
   • [claimed-docs] “Define **segments**—groups of users matching rules based on traits (e.g., location, plan, app version, usage).”
   • [claimed-docs] “Define segments—groups of users matching rules based on traits (e.g., location, plan, app version, usage). Override feature flags for all us…”

   proof ↗⚑ flag

2. developerTarget flags with attribute-based rules and reusable segments so the right users see the right variation

   weight 3 · round to Flagsmith
   Statsig~partialclaimed5/10

   Docs confirm Feature Gates support attribute-based targeting (e.g., targeting mobile platforms and internal testers) and per-user overrides, indicating rule-based flag targeting is supported. However, the evidence pack never mentions a dedicated 'segments' or reusable audience-list feature, so reusable, cross-flag targeting groups are unevidenced. missing for 10: explicit documentation of reusable segment/audience definitions, evidence of segment reuse across multiple flags/experiments, and independent/hands-on confirmation of complex rule builders.

   • [claimed-docs] “This tutorial creates a feature gate, targets mobile platforms and internal testers, then checks it live with the JavaScript SDK in your bro…”
   • [claimed-docs] “set up Feature Gate overrides to allow specific users to bypass the gate.”
   • [claimed-docs] “Feature Gates, commonly known as feature flags, allow you to toggle the behavior of your product in real time without deploying new code.”

   proof ↗⚑ flag

   Flagsmith✓fullprobed9/10

   Docs clearly describe trait-based targeting (geography, subscription, device, version), reusable segments defined by trait rules, per-identity overrides, percentage-split rollouts within segments, and multivariate flags — all core to attribute-based targeting and segment-driven variation delivery, corroborated by a runtime probe confirming a real flag create/evaluate loop. Missing for 10: independent (non-vendor) third-party validation of segment/attribute targeting behavior at scale beyond the single self-host probe.

   • [claimed-docs] “Target features to users with specific traits (such as geography, subscription level, device type, or application version).”
   • [claimed-docs] “Define **segments**—groups of users matching rules based on traits (e.g., location, plan, app version, usage).”
   • [claimed-docs] “Define segments—groups of users matching rules based on traits (e.g., location, plan, app version, usage). Override feature flags for all us…”
   • [claimed-docs] “Override feature flags for individual users ("identities").”
   • [claimed-docs] “Add a rule defined with the % Split condition. Specify a value between 1 and 100 to define what percentage of your user base is included wit…”
   • [claimed-docs] “Multivariate flags let you define multiple variants with percentage weightings (A/B/n).”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”

   proof ↗⚑ flag

Access

1. platform engineerRestrict who can change which flags with roles, permissions, and scoped API tokens

   weight 2 · round to Flagsmith
   Statsig~partialclaimed4/10

   Statsig docs reference enterprise access management with SSO/SCIM and a Console API for programmatic control, implying some access-control layer exists, but no evidence details role definitions, granular permissions, or scoped/limited API tokens for restricting who can change specific flags. Missing for 10: explicit RBAC documentation, permission granularity for flag edits, and scoped/read-vs-write API token support.

   • [claimed-docs] “In many enterprise environments, organizations use SSO and SCIM together to improve security and usability.”
   • [claimed-docs] “Basic settings include automatically adding new users with the same email domain to your project. Statsig also supports SSO to simplify invi…”
   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”

   proof ↗⚑ flag

   Flagsmith✓fullclaimed9/10

   Flagsmith documents fine-grained RBAC to restrict who can modify resources (e.g., 'only allow certain users to modify your production environments'), plus a four-eyes change-request workflow for approvals, and clearly distinguishes scoped API tokens (public non-secret client-side environment keys vs. privileged Admin API keys for CRUD operations). SSO/SAML and 2FA further support access governance. Missing for 10: no explicit documentation of scoped/granular API token permissions (e.g., per-project or read-only server tokens) beyond the client/server key distinction, and no independent/community validation of RBAC granularity in practice.

   • [claimed-docs] “Role-based access control (RBAC) provides fine-grained access management of Flagsmith resources.”
   • [claimed-docs] “Only allow certain users to modify your production environments.”
   • [claimed-docs] “Feature Change Requests help creating a four-eyes workflow (create, approve, publish) to updating feature flags, similarly to Pull Requests …”
   • [claimed-docs] “Purpose: Creating, updating, and deleting projects, environments, flags, segments, and users.”
   • [claimed-docs] “Authentication: Uses a public, non-secret Environment Key. Security: Open by design. The Environment Key can be exposed in client-side code.”
   • [claimed-docs] “SAML/SSO Authentication”
   • [claimed-docs] “Two-Factor Authentication (2FA)”

   proof ↗⚑ flag

Agent ops

1. ai agentCreate and toggle flags through documented APIs, CLIs, or MCP — and the platform can force my changes through approval workflows instead of letting me write to production unreviewed

   weight 3 · round to Flagsmith
   Statsig~partialprobed4/10

   Statsig documents multiple agent-friendly interfaces for creating/toggling flags — a CLI (also usable in CI/CD), a Console CRUD API, and an official MCP server reachable via OAuth — giving strong coverage of the 'documented APIs, CLIs, or MCP' half of the story. However, there is no evidence of an approval/review workflow that intercepts or blocks agent-initiated changes before they hit production; access-management docs mention SSO/SCIM for identity but not change-approval gating. Missing for 10: documented approval-workflow/change-review gating for flag mutations, evidence that CLI/API/MCP writes can be routed through a review step rather than applied directly.

   • [claimed-docs] “The Statsig CLI is a command-line interface that helps manage Feature Gates, Experiments, and Dynamic Configs within the Statsig platform.”
   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [claimed-docs] “The Statsig MCP (Model Context Protocol) server brings the power of Statsig into tools like Codex, Cursor, and Claude Code. With this setup,…”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Statsig CLI (npm @statsig/siggy) ran keylessly via `npx -y @statsig/siggy --version`, prin…”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the hosted MCP server https://api.statsig.com/v1/mcp returned HTTP …”
   • [claimed-docs] “In many enterprise environments, organizations use SSO and SCIM together to improve security and usability.”

   proof ↗⚑ flag

   Flagsmith✓fullprobed7/10

   Flagsmith documents an official MCP server, CLI, and Admin API that let an AI agent create/read/toggle flags (flagsmith-docs-17/29, probe-3, probe-rt-1, docs-18/40), and separately documents a four-eyes Change Request workflow plus RBAC that can restrict who may modify production flags, similar to PR review (flagsmith-docs-10, flagsmith-docs-26). This gives both halves of the story: programmatic write access and an approval-gate mechanism. missing for 10: explicit documentation/hands-on proof that API/MCP/CLI-originated writes are automatically routed through Change Requests (rather than change requests being a dashboard-only workflow), and no independent verification that RBAC blocks an agent's direct API write to production.

   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [probe] “official MCP server documented at https://docs.flagsmith.com/integrating-with-flagsmith/mcp-server”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the documented hosted MCP server https://mcp.flagsmith.com returned…”
   • [claimed-docs] “The Flagsmith CLI lets you manage flags, segments, features, projects and environments from your terminal, and evaluate flags the way an SDK…”
   • [claimed-docs] “This is the API you use to programmatically manage your Flagsmith projects. Anything you can do in the Flagsmith dashboard, you can also do …”
   • [claimed-docs] “Feature Change Requests help creating a four-eyes workflow (create, approve, publish) to updating feature flags, similarly to Pull Requests …”
   • [claimed-docs] “Only allow certain users to modify your production environments.”
   • [claimed-docs] “Every action taken within the Flagsmith administration application is tracked and logged. This allows you to easily retrace the events and v…”

   proof ↗⚑ flag

Approvals

1. platform engineerRequire approvals or change requests before production flag changes go live

   weight 3 · round to Flagsmith
   Statsig—none0/10

   No evidence describes an approval workflow, change request, or review gate before flag changes go live in production; docs mention scheduled rollouts, overrides, and audit-adjacent features like exposure logging but not a governance approval mechanism.

   ⚑ flag

   Flagsmith✓fullclaimed9/10

   Flagsmith documents a native four-eyes Feature Change Request workflow (create, approve, publish) explicitly analogous to GitHub PRs, plus Release Pipelines that can auto-generate change requests when promoting flags to production, and RBAC to restrict who can modify production environments. This directly satisfies the governance requirement for approvals before production flag changes go live. Missing for 10: independent/hands-on verification of the approval gating in practice (only first-party docs cited).

   • [claimed-docs] “Feature Change Requests help creating a four-eyes workflow (create, approve, publish) to updating feature flags, similarly to Pull Requests …”
   • [claimed-docs] “Release Pipelines can automatically generate change requests when promoting flags to production, eliminating manual effort and streamlining …”
   • [claimed-docs] “Release Pipelines act as your intelligent release assistant, automating your entire feature flag deployment process.”
   • [claimed-docs] “Only allow certain users to modify your production environments.”
   • [claimed-docs] “Role-based access control (RBAC) provides fine-grained access management of Flagsmith resources.”

   proof ↗⚑ flag

Audit

1. platform engineerEvery flag change is recorded in an audit log — who changed what, when, and to which value

   weight 3 · round to Flagsmith
   Statsig—none0/10

   The evidence pack covers feature gates, experiments, SSO/SCIM, CLI, Console API, and MCP integrations, but nowhere mentions an audit log or change history capturing who changed a flag, when, or to what value. This is a standard governance capability for feature-flag platforms, so its complete absence from the docs pack means the axis applies but is unsupported.

   ⚑ flag

   Flagsmith✓fullclaimed9/10

   Flagsmith documents that every action in the admin application (flag, identity, segment changes) is tracked and logged, directly matching the who/what/when/value audit requirement, and offers webhook streaming to external systems for governance workflows. missing for 10: independent/hands-on confirmation of the audit log's exact field-level detail (e.g., explicit before/after value display) beyond vendor docs.

   • [claimed-docs] “Every action taken within the Flagsmith administration application is tracked and logged. This allows you to easily retrace the events and v…”
   • [claimed-docs] “Every action taken within the Flagsmith administration application is tracked and logged.”
   • [claimed-docs] “You can stream your Audit Logs into your own infrastructure using Audit Log Webhooks.”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Flagsmith
   Statsig✓fullprobed8/10

   Statsig documents a full Console API that is explicitly the CRUD API for 'performing the actions available on console.statsig.com without using the web UI,' plus a separate HTTP API for gate/experiment evaluation and event logging, and an official CLI wrapping these for CI/CD scripting — together covering feature gates, experiments, dynamic configs, and access management via API/CLI rather than only the UI. MCP servers and CLI were confirmed live/reachable in runtime probes, reinforcing that programmatic parity is real and operational, not just documented aspiration. Missing for 10: no independent third-party audit confirming 1:1 UI/API feature parity (e.g., some newer UI features like Autotune or Warehouse Native scorecards aren't explicitly confirmed as API-exposed), and no public OpenAPI spec was found (404s on standard paths).

   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [claimed-docs] “Overview of the Statsig HTTP API for retrieving feature gate, experiment, and dynamic config values and logging events directly without an S…”
   • [claimed-docs] “The Statsig CLI is a command-line interface that helps manage Feature Gates, Experiments, and Dynamic Configs within the Statsig platform.”
   • [claimed-docs] “You can integrate the CLI into CI/CD pipelines to automate the management of Feature Gates and Experiments.”
   • [probe] “PROBE runtime (recorded 2026-09-05): the official Statsig CLI (npm @statsig/siggy) ran keylessly via `npx -y @statsig/siggy --version`, prin…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.statsig.com/openapi.json, https://docs.statsig.com/swagger.json, https://docs.statsig.c…”

   proof ↗⚑ flag

   Flagsmith✓fullprobed9/10

   Flagsmith explicitly documents that 'Anything you can do in the Flagsmith dashboard, you can also do via the Admin API' (flagsmith-docs-30/40), and this is corroborated by a real probe showing a full create-project/environment/flag/evaluate roundtrip via the API (flagsmith-probe-rt-2). It further exposes this API to AI agents via an official MCP server and CLI (flagsmith-docs-17/18, flagsmith-probe-3/4, flagsmith-probe-rt-1), reinforcing API-UI parity for AI-native use. Missing for 10: a published OpenAPI/swagger spec was probed and returned 404 (flagsmith-probe-2), so machine-readable API completeness is not independently verifiable beyond the vendor's own claim.

   • [claimed-docs] “Anything you can do in the Flagsmith dashboard, you can also do via the Admin API.”
   • [claimed-docs] “This is the API you use to programmatically manage your Flagsmith projects. Anything you can do in the Flagsmith dashboard, you can also do …”
   • [claimed-docs] “The Flagsmith MCP Server gives AI assistants and agents programmatic access to the Flagsmith Admin API through the Model Context Protocol”
   • [claimed-docs] “The Flagsmith CLI lets you manage flags, segments, features, projects and environments from your terminal, and evaluate flags the way an SDK…”
   • [probe] “official MCP server documented at https://docs.flagsmith.com/integrating-with-flagsmith/mcp-server”
   • [probe] “official CLI documented at https://docs.flagsmith.com/integrating-with-flagsmith/CLI”
   • [probe] “PROBE runtime (recorded 2026-09-05): keyless JSON-RPC initialize POST to the documented hosted MCP server https://mcp.flagsmith.com returned…”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.flagsmith.com/openapi.json, https://docs.flagsmith.com/swagger.json, https://docs.flags…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to Flagsmith
   Statsig~partialclaimed4/10

   Statsig provides a Console API and CLI for programmatic access to configuration data (feature gates, experiments, dynamic configs), which supports exporting configuration/setup data in a scriptable way, but there is no evidence of a bulk data export feature for user-level event/analytics data in open/standard formats, nor documentation of data portability or account deletion/export workflows for users wanting to leave the platform. missing for 10: documented bulk export of analytics/event data in open formats (CSV/JSON/Parquet), explicit data portability or 'leave the platform' export tooling, evidence of exporting experiment results/metrics data, and any community confirmation of successful full data export.

   • [claimed-docs] “The Console API is the CRUD API for performing the actions available on console.statsig.com without using the web UI.”
   • [claimed-docs] “The Statsig CLI is a command-line interface that helps manage Feature Gates, Experiments, and Dynamic Configs within the Statsig platform.”
   • [claimed-docs] “You can integrate the CLI into CI/CD pipelines to automate the management of Feature Gates and Experiments.”

   proof ↗⚑ flag

   Flagsmith~partialprobed6/10

   Flagsmith's Admin API exposes full CRUD access to all projects, environments, flags, segments, and users ("anything you can do in the dashboard, you can also do via the Admin API"), and self-hosting via Docker gives full ownership of the underlying Postgres data store, together enabling a user to extract and own all their data outside the SaaS. However there is no documented one-click 'export all data' feature, no bulk export-to-JSON/CSV tool, and probes show no llms.txt or public OpenAPI spec is served, so a turnkey open-format export path is not explicitly evidenced. missing for 10: explicit bulk data-export feature/format, documented data-portability guarantee, and a working OpenAPI/machine-readable spec (docs.flagsmith.com/openapi.json returned 404).

   • [claimed-docs] “Anything you can do in the Flagsmith dashboard, you can also do via the Admin API.”
   • [claimed-docs] “This is the API you use to programmatically manage your Flagsmith projects. Anything you can do in the Flagsmith dashboard, you can also do …”
   • [claimed-docs] “Self-hosting Flagsmith allows you to run the complete Flagsmith platform within your own infrastructure, giving you full control over your d…”
   • [claimed-docs] “You can use Docker to set up an entire Flagsmith Feature Flag environment locally”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.flagsmith.com/openapi.json, https://docs.flagsmith.com/swagger.json, https://docs.flags…”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round to Flagsmith
   Statsig—none0/10

   No evidence in the pack indicates Statsig's core platform or source code is released under an open license; only docs, CLI, MCP servers, and SaaS features are mentioned, none of which speak to open-source licensing of the product's source.

   ⚑ flag

   Flagsmith✓fullclaimed7/10

   The Flagsmith source code is publicly hosted on GitHub (repeatedly cited) and the project provides self-hosting docs, Docker images, and full platform control, all consistent with an openly licensed codebase. However, the evidence pack never explicitly names or links the license file/type, so the claim of a specific open license is inferred rather than directly documented. Missing for 10: explicit license identifier/file citation (e.g., BSD-3-Clause LICENSE), independent confirmation of license terms.

   • [github] “Make changes remotely: Easily toggle individual features on and off, and make changes without deploying new code”
   • [github] “Just wrap a section of code with a flag, and then use Flagsmith to toggle that feature on or off for different environments, users or user s…”
   • [github] “Easily toggle individual features on and off, and make changes without deploying new code”
   • [claimed-docs] “Self-hosting Flagsmith allows you to run the complete Flagsmith platform within your own infrastructure, giving you full control over your d…”
   • [claimed-docs] “You can use Docker to set up an entire Flagsmith Feature Flag environment locally”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round to Flagsmith
   Statsig—none0/10

   No evidence anywhere in the pack mentions self-hosting or an on-prem/open-source deployment of the core Statsig platform; all documentation describes a hosted SaaS console, hosted MCP/API endpoints, and cloud-based warehouse-native analysis. Self-hosting is a fair ask for this category (competitors offer it), so absence of evidence means 'none' rather than 'na'.

   ⚑ flag

   Flagsmith✓fullprobed10/10

   Flagsmith documents comprehensive self-hosting via Docker (full platform, own infrastructure, full control over data/security), and a hands-on probe confirms a complete keyless self-host roundtrip — running the official Docker image, creating a project/environment/flag, and successfully evaluating it via the API with no account or license required.

   • [claimed-docs] “Self-hosting Flagsmith allows you to run the complete Flagsmith platform within your own infrastructure, giving you full control over your d…”
   • [claimed-docs] “You can use Docker to set up an entire Flagsmith Feature Flag environment locally”
   • [claimed-docs] “You can use Docker to set up an entire [Flagsmith Feature Flag](https://www.flagsmith.com) environment locally”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”

   proof ↗⚑ flag

Pricing

1. product managerSee published pricing and understand what drives cost (seats, MAUs, events, requests) before committing

   weight 2 · round to Statsig
   Statsig~partialclaimed3/10

   Docs mention a free tier tied to 'events' as a metered unit, giving a partial glimpse of cost drivers, but there is no evidence of a full published pricing page detailing seats, MAUs, or request-based costs across paid tiers. missing for 10: published pricing tiers/page, seat-based pricing, MAU-based pricing, request-based pricing, enterprise pricing details.

   • [claimed-docs] “You get feature flags and 2 million metered events for free.”

   proof ↗⚑ flag

   Flagsmith—none0/10

   The evidence pack only references the pricing page URL for isolated feature call-outs (SAML/SSO, 2FA) but contains no evidence about actual pricing tiers, seat costs, MAU/event/request-based pricing, or how usage drives cost. missing for 10: published pricing tiers, cost drivers (seats/MAUs/events/requests), any breakdown of what determines plan cost.

   • [claimed-docs] “SAML/SSO Authentication”
   • [claimed-docs] “Two-Factor Authentication (2FA)”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round to Flagsmith
   Statsig—none0/10

   No evidence in the pack mentions data residency, regional storage options, or geographic data controls for Statsig's platform; the documentation excerpts cover feature flags, experiments, MCP, CLI, and access management but never address where data is stored or user choice of region.

   ⚑ flag

   Flagsmith~partialprobed6/10

   Flagsmith supports data residency indirectly via self-hosting (full control over data location, deployable in any infrastructure/region) and an Edge Proxy that can be run within a customer's own infrastructure, which lets an AI-native user choose where flag data lives. However, there's no documented SaaS-side region picker (e.g., dedicated EU/US managed cloud regions) — residency control is achieved only by fully self-hosting rather than a first-class 'choose region' setting. Missing for 10: explicit managed-cloud region selection options, documentation of data residency guarantees for the hosted SaaS offering.

   • [claimed-docs] “Self-hosting Flagsmith allows you to run the complete Flagsmith platform within your own infrastructure, giving you full control over your d…”
   • [claimed-docs] “You can use Docker to set up an entire Flagsmith Feature Flag environment locally”
   • [claimed-docs] “The Flagsmith Edge Proxy is a service that you host yourself, that allows you to run an instance of the Flagsmith Engine close to your serve…”
   • [claimed-docs] “You can use Docker to set up an entire [Flagsmith Feature Flag](https://www.flagsmith.com) environment locally”
   • [claimed-docs] “Run the Edge Proxy within in your own infrastructure and connect to it from your server-side SDKs”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”

   proof ↗⚑ flag

2. ai-native userControl data retention and deletion

   weight 2 · round to Flagsmith
   Statsig—none0/10

   The evidence pack covers feature flags, experimentation, MCP integrations, and access management (SSO/SCIM), but contains no mention of data retention policies, deletion controls, or GDPR/CCPA-style data lifecycle management. This is a fair axis for a SaaS analytics/experimentation platform handling user event data, so absence of evidence yields 'none' rather than 'na'.

   ⚑ flag

   Flagsmith~partialclaimed5/10

   Flagsmith's Admin API and dashboard explicitly support creating, updating, and deleting projects, environments, flags, segments, and users, with a 'protected' flag option to guard against accidental deletion, and self-hosting gives full control over where data lives. However there is no explicit documentation of a data-retention policy, automated data expiry, or a GDPR-style 'delete my data' workflow beyond generic CRUD deletion of resources. Missing for 10: documented retention windows/expiry settings, formal data-deletion/right-to-erasure workflow, and audit-log retention controls.

   • [claimed-docs] “Purpose: Creating, updating, and deleting projects, environments, flags, segments, and users.”
   • [claimed-docs] “add `protected` to prevent them from accidentally being deleted”
   • [claimed-docs] “Self-hosting Flagsmith allows you to run the complete Flagsmith platform within your own infrastructure, giving you full control over your d…”
   • [claimed-docs] “Every action taken within the Flagsmith administration application is tracked and logged. This allows you to easily retrace the events and v…”

   proof ↗⚑ flag

Rollouts

1. platform engineerGuard a rollout with metrics so a regression is detected and the release is rolled back automatically

   weight 2 · round to Statsig
   Statsig~partialclaimed5/10

   Statsig supports gradual/scheduled rollouts, feature-gate emergency kill switches, and metric-based experimentation with alerting for regressions (Infra Analytics 'set alerts to catch regressions'), giving the pieces needed to guard a rollout with metrics. However, none of the evidence shows an automated rollback action triggered by a detected metric regression — alerts and kill switches appear to require manual intervention rather than an automatic release rollback loop. Missing for 10: documented automatic rollback/kill-switch triggered by metric threshold breach, closed-loop guardrail-to-rollback automation, and independent confirmation this works in production.

   • [claimed-docs] “Have an emergency disable switch that lets you immediately turn off a particular code branch for users in production”
   • [claimed-docs] “Gradually deploy a feature over time by setting up a Feature Gate as a Scheduled Rollout.”
   • [claimed-docs] “Infra Analytics: Monitor and debug service health alongside product outcomes. Ingest metrics and traces with OpenTelemetry, search and group…”
   • [claimed-docs] “Monitor and debug service health alongside product outcomes. Ingest metrics and traces with OpenTelemetry, search and group logs, and set al…”
   • [claimed-docs] “Schedule gradual feature rollouts to safely deploy new code”
   • [claimed-docs] “The Scorecard feature makes this easy by letting you enter your hypothesis and select both primary and secondary metrics.”

   proof ↗⚑ flag

   Flagsmith—none0/10

   Flagsmith documents percentage rollouts, scheduled flags, and Release Pipelines with 'triggers and actions,' plus an Experimentation module with Bayesian stats — but none of the evidence describes automatic metric-based regression detection that triggers an automatic rollback of a release. The pipeline features found are about staged promotion and change-request automation, not closed-loop guarded rollouts tied to live health/error metrics.

   • [claimed-docs] “Define stages that your flags progress through automatically, with triggers and actions that control rollouts to selected environments and a…”
   • [claimed-docs] “Release Pipelines can automatically generate change requests when promoting flags to production, eliminating manual effort and streamlining …”
   • [claimed-docs] “Release Pipelines act as your intelligent release assistant, automating your entire feature flag deployment process.”
   • [claimed-docs] “Flagsmith Experimentation lets you run A/B tests end to end on the platform: serve variations of a feature with a multivariate flag, collect…”
   • [claimed-docs] “Add a rule defined with the % Split condition. Specify a value between 1 and 100 to define what percentage of your user base is included wit…”

   proof ↗⚑ flag

2. developerRoll a flag out progressively by percentage with consistent bucketing, ramping from 1% to 100% without redeploying

   weight 3 · round to Flagsmith
   Statsig✓fullclaimed8/10

   Statsig docs explicitly describe scheduled/gradual rollouts to safely deploy new code without redeploying, and deterministic evaluation ensures consistent bucketing across evaluations for the same user given the same gate state. Percentage-based ramping (1%→100%) is a core documented Feature Gate/Scheduled Rollout capability, backed by consistent hashing via deterministic evaluation. missing for 10: explicit mention of percentage granularity/UI slider mechanics and independent hands-on confirmation of ramp behavior.

   • [claimed-docs] “Gradually deploy a feature over time by setting up a Feature Gate as a Scheduled Rollout.”
   • [claimed-docs] “Schedule gradual feature rollouts to safely deploy new code”
   • [claimed-docs] “Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always retu…”
   • [claimed-docs] “Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always retu…”
   • [claimed-docs] “Feature Gates, commonly known as feature flags, allow you to toggle the behavior of your product in real time without deploying new code.”

   proof ↗⚑ flag

   Flagsmith✓fullprobed9/10

   Flagsmith docs explicitly document percentage-based rollouts via a % Split segment rule (1-100%) that can be adjusted without redeploying, and this is corroborated by a runtime probe showing a live flag create/evaluate loop. Consistent bucketing is implied by the segment-based percentage split mechanism used across SDKs/environments, though the docs don't explicitly describe the hashing/bucketing algorithm in detail. missing for 10: explicit documentation of the deterministic bucketing/hashing algorithm ensuring consistent user assignment across ramp changes.

   • [claimed-docs] “Add a rule defined with the % Split condition. Specify a value between 1 and 100 to define what percentage of your user base is included wit…”
   • [claimed-docs] “Add a rule defined with the **% Split** condition. Specify a value between 1 and 100 to define what percentage of your user base is included…”
   • [claimed-docs] “Rollouts by percentage (also known as staged feature rollouts) allow you to test a new feature with a small subset of your user base.”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”

   proof ↗⚑ flag

Scheduling

1. developerSchedule flag changes and releases to happen at a specific future time

   weight 1 · round to Flagsmith
   Statsig~partialclaimed6/10

   Docs confirm Statsig supports 'Scheduled Rollout' for Feature Gates, letting you gradually deploy a feature over time rather than instantly ('Gradually deploy a feature over time by setting up a Feature Gate as a Scheduled Rollout', 'Schedule gradual feature rollouts to safely deploy new code'), which implies future-time scheduling of flag state changes. However, the pack lacks detail on setting an exact future timestamp for a release event, calendar/date-picker UI specifics, or independent confirmation of this exact workflow. Missing for 10: explicit documentation of specifying an exact date/time for a rollout step, screenshots or hands-on walkthrough, and third-party confirmation that scheduled changes fire reliably.

   • [claimed-docs] “Gradually deploy a feature over time by setting up a Feature Gate as a Scheduled Rollout.”
   • [claimed-docs] “Schedule gradual feature rollouts to safely deploy new code”

   proof ↗⚑ flag

   Flagsmith✓fullclaimed9/10

   Flagsmith's docs explicitly describe a Scheduled Flags feature that lets you queue changes to feature flags and have them automatically applied at a specified future time, removing the need for manual intervention at that moment — directly matching the story. This is further complemented by Release Pipelines, which can automatically stage/progress rollouts with triggers, reinforcing scheduled/automated release workflows. missing for 10: independent hands-on confirmation of the scheduling UI/behavior beyond first-party docs.

   • [claimed-docs] “Scheduled flags allow you to queue and automatically apply changes to feature flags at a specified future time, eliminating the need for man…”
   • [claimed-docs] “Scheduled flags allow you to queue and automatically apply changes to feature flags at a specified future time”
   • [claimed-docs] “Define stages that your flags progress through automatically, with triggers and actions that control rollouts to selected environments and a…”
   • [claimed-docs] “Release Pipelines act as your intelligent release assistant, automating your entire feature flag deployment process.”

   proof ↗⚑ flag

Edge

1. platform engineerEvaluate flags at the edge (CDN workers or an edge/relay layer) close to users

   weight 2 · round to Flagsmith
   Statsig—none0/10

   The evidence pack covers Statsig's feature gates, on-device eval SDKs for client-side, server SDKs, and CLI/API surfaces, but nothing addresses evaluating flags specifically at CDN/edge workers or an edge/relay layer close to users. Missing for 10: any mention of Cloudflare Workers, edge runtime SDKs, or a proxy/relay layer for edge evaluation.

   • [claimed-docs] “Statsig's client-side On-Device Eval SDKs provide an alternate client-side architecture. The device keeps the definition of each experiment …”
   • [claimed-docs] “Statsig's client-side On-Device Eval SDKs provide an alternate client-side architecture. The device keeps the definition of each experiment …”
   • [claimed-docs] “Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always retu…”

   proof ↗⚑ flag

   Flagsmith~partialcommunity5/10

   Flagsmith ships an official Edge Proxy that can be self-hosted 'close to your servers' and queried by server-side SDKs (flagsmith-docs-14, flagsmith-docs-28), which addresses the general 'evaluate close to users' need, but this is a self-run relay service rather than a native CDN/edge-worker (e.g., Cloudflare Workers, Fastly Compute) integration. A community comment also notes Flagsmith's evaluation is 'primarily server-side' rather than fully local/in-memory, unlike some flagd-style hybrid approaches (flagsmith-comm-1), tempering claims of true edge-local evaluation. Missing for 10: a documented CDN/edge-worker runtime or SDK (e.g., Cloudflare/Fastly integration), multi-region edge network evidence, and independent latency benchmarks confirming edge-proximity gains.

   • [claimed-docs] “The Flagsmith Edge Proxy is a service that you host yourself, that allows you to run an instance of the Flagsmith Engine close to your serve…”
   • [claimed-docs] “Run the Edge Proxy within in your own infrastructure and connect to it from your server-side SDKs”
   • [community] “I would have thought so [flags evaluated in-memory locally]. But flagsmith apparently does primarily server-side eval. And even OpenFeature …”

   proof ↗⚑ flag

Evaluation

1. platform engineerMy server SDKs evaluate flags locally from a cached ruleset — microsecond decisions with no network call per flag check

   weight 3 · round to Statsig
   Statsig~partialclaimed5/10

   Docs confirm deterministic evaluation 'even across different platforms (client or server)' implying server SDKs replicate rule logic locally, and explicitly describe on-device eval SDKs keeping definitions in memory for fast evaluation — but this on-device architecture is documented only for client SDKs, not explicitly server SDKs. Missing for 10: explicit server-SDK docs on background polling/streaming to a local cache, no-network-call guarantee per check, and microsecond-latency benchmarks specific to server SDKs.

   • [claimed-docs] “Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always retu…”
   • [claimed-docs] “Statsig's client-side On-Device Eval SDKs provide an alternate client-side architecture. The device keeps the definition of each experiment …”
   • [claimed-docs] “Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always retu…”
   • [claimed-docs] “Statsig's client-side On-Device Eval SDKs provide an alternate client-side architecture. The device keeps the definition of each experiment …”

   proof ↗⚑ flag

   Flagsmith!disputedcontradicted3/10

   Flagsmith docs describe an Edge Proxy you can self-host to reduce latency (flagsmith-docs-14/28), but this is still a network hop to a proxy, not a local in-process cached rule evaluation. A knowledgeable community source explicitly states Flagsmith 'does primarily server-side eval' rather than in-memory local evaluation (flagsmith-comm-1), and the probed runtime evidence shows flag reads going through a live API call (flagsmith-probe-rt-2), directly contradicting the 'no network call per flag check' claim. missing for 10: first-party SDK docs describing local ruleset caching/polling with microsecond in-process evaluation, and hands-on confirmation that server SDKs avoid per-check network calls.

   • [claimed-docs] “The Flagsmith Edge Proxy is a service that you host yourself, that allows you to run an instance of the Flagsmith Engine close to your serve…”
   • [claimed-docs] “Run the Edge Proxy within in your own infrastructure and connect to it from your server-side SDKs”
   • [community] “I would have thought so [flags evaluated in-memory locally]. But flagsmith apparently does primarily server-side eval. And even OpenFeature …”
   • [probe] “PROBE runtime (recorded 2026-09-05): full KEYLESS self-host roundtrip on this machine — the official flagsmith/flagsmith unified docker imag…”

   proof ↗⚑ flag

Sdks

1. developerUse official SDKs across my whole stack — backend, web, and mobile — with consistent flag behavior

   weight 2 · round to Flagsmith
   Statsig~partialclaimed5/10

   Statsig documents deterministic evaluation guaranteeing identical results 'across different platforms (client or server)' and offers On-Device Eval client SDKs plus a JS SDK tutorial that also targets mobile platforms, implying a consistent cross-stack SDK model. However, the pack never enumerates specific backend/web/mobile SDK names or shows hands-on parity testing across a real multi-platform stack. Missing for 10: an explicit list of officially supported language/platform SDKs, and independent/hands-on evidence confirming identical flag behavior when used together across backend, web, and mobile in a real project.

   • [claimed-docs] “Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always retu…”
   • [claimed-docs] “Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always retu…”
   • [claimed-docs] “Statsig's client-side On-Device Eval SDKs provide an alternate client-side architecture. The device keeps the definition of each experiment …”
   • [claimed-docs] “Statsig's client-side On-Device Eval SDKs provide an alternate client-side architecture. The device keeps the definition of each experiment …”
   • [claimed-docs] “This tutorial creates a feature gate, targets mobile platforms and internal testers, then checks it live with the JavaScript SDK in your bro…”

   proof ↗⚑ flag

   Flagsmith~partialcommunity6/10

   Flagsmith explicitly markets support for 15+ languages/frameworks across backend, web, and mobile, and its architecture (environment keys, consistent flag/segment/targeting model) implies consistent behavior across SDKs. However, the evidence pack lacks any per-SDK documentation detail, SDK-to-SDK consistency testing, or independent confirmation of parity across specific SDKs, and community commentary raises questions about server-side vs local evaluation consistency (flagd vs local evaluation) that isn't directly addressed. missing for 10: per-platform SDK documentation snippets, evidence of consistent flag evaluation logic/tests across SDKs, independent/hands-on confirmation of behavior parity across backend/web/mobile.

   • [github] “Choose from 15+ popular languages like Typescript, .NET, Java, and more. Integrate with any framework, including React, Next.js, and more”
   • [claimed-docs] “Manage feature flags and remote config across web, mobile, and server-side apps.”
   • [community] “I would have thought so [flags evaluated in-memory locally]. But flagsmith apparently does primarily server-side eval. And even OpenFeature …”

   proof ↗⚑ flag

Standards

1. platform engineerUse the vendor through OpenFeature providers so my flag code isn't locked to one vendor's SDK API

   weight 1 · round drawn
   Statsig—none0/10

   No evidence pack item mentions OpenFeature or a provider implementation for it; all SDK references are to Statsig's own proprietary SDKs, CLI, HTTP API, and Console API. This is a fair axis for a feature-flag/experimentation platform, so absence of evidence yields 'none'.

   ⚑ flag

   Flagsmith—none0/10

   No evidence in the pack mentions OpenFeature support, an OpenFeature provider implementation, or any compatibility layer for the OpenFeature spec — Flagsmith is only shown with its own proprietary SDKs, API, and CLI. Missing for 10: any mention of an OpenFeature provider, spec compliance, or documented integration with the OpenFeature ecosystem.

   ⚑ flag

Streaming

1. developerFlag changes propagate to connected SDKs in seconds via streaming or fast polling — a kill switch actually kills

   weight 2 · round to Statsig
   Statsig~partialclaimed7/10

   Statsig's docs explicitly describe real-time toggling and an 'emergency disable switch' to immediately turn off a code branch in production, plus deterministic evaluation and on-device SDKs for fast client updates — directly supporting the kill-switch and fast-propagation story. However, there is no explicit documentation of streaming vs polling mechanics, propagation latency in seconds, or independent/hands-on verification of SDK update speed. Missing for 10: documented streaming/polling interval specifics, third-party benchmark or hands-on confirmation of sub-second/seconds-level propagation.

   • [claimed-docs] “Feature Gates, commonly known as feature flags, allow you to toggle the behavior of your product in real time without deploying new code.”
   • [claimed-docs] “Have an emergency disable switch that lets you immediately turn off a particular code branch for users in production”
   • [claimed-docs] “Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always retu…”
   • [claimed-docs] “Statsig's client-side On-Device Eval SDKs provide an alternate client-side architecture. The device keeps the definition of each experiment …”

   proof ↗⚑ flag

   Flagsmith—none0/10

   The evidence pack describes flag targeting, rollouts, and toggling but contains no documentation of streaming updates, real-time push, or specific polling intervals that would substantiate seconds-level propagation to SDKs; a community comment even notes Flagsmith does 'primarily server-side eval' rather than confirming fast client propagation. Missing for 10: documented real-time/streaming SDK update mechanism, polling interval specs, and evidence of kill-switch propagation latency.

   • [community] “I would have thought so [flags evaluated in-memory locally]. But flagsmith apparently does primarily server-side eval. And even OpenFeature …”

   proof ↗⚑ flag