Composio vs Gram (Speakeasy)

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Composio
   Composio✓fullprobed9/10

   Composio hosts a live llms.txt at docs.composio.dev/llms.txt (verified 200 with agent-oriented summary content), directly satisfying the story of pointing an agent at agent-oriented docs; this is corroborated by a probe rather than just vendor claim. Missing for 10: no independent/community report of an agent actually consuming llms.txt successfully in the wild.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.composio.dev/llms.txt # Composio Documentation > Composio powers 1,400+ toolkits, tool search, con…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed8/10

   Speakeasy directly hosts a working llms.txt (HTTP 200, confirmed by probe) plus a docs.md markdown-native page, meaning an agent can be pointed at agent-oriented docs formats; the docs also emphasize AI-native/agent workflows throughout (SKILL.md, platform MCP server, CLI). missing for 10: no evidence of llms.txt existing for Gram product docs specifically (only the root Speakeasy domain), and no independent/hands-on confirmation an agent actually consumed and acted on the llms.txt content.

   • [probe] “PROBE llms.txt: HTTP 200 at https://www.speakeasy.com/llms.txt # Speakeasy > Speakeasy is the AI control plane for the AI-native enterprise…”
   • [probe] “PROBE docs-md: HTTP 200 at https://www.speakeasy.com/docs/ai-control-plane.md # AI Control Plane Documentation {/* * Renders through the …”
   • [claimed-docs] “A skill is a versioned `SKILL.md` manifest: reusable instructions that agents load on demand.”
   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Composio
   Composio✓fullprobed7/10

   Composio ships an SDK (Python/TS packages verified to install and import), a REST API (openapi.json), and a CLI, all of which can be scripted/invoked headlessly without a UI — sessions are created programmatically (composio.sessions.create) and tools executed via API/CLI, which is exactly the pattern used in CI automation. Missing for 10: explicit CI/CD pipeline docs or examples (e.g., GitHub Actions integration) and independent confirmation of a fully non-interactive auth flow for headless CI runs.

   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”
   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [probe] “official CLI documented at https://docs.composio.dev/docs/cli”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/composio/): `npm install @composio/core` and `uv pip install composio…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialprobed6/10

   Gram ships an official CLI (verified runnable via runtime probe) that can build/publish MCP servers and toolsets, which supports CI/headless automation, and docs confirm a working command-line reference. However, there is no explicit CI/CD pipeline documentation, no GitHub Actions example, and no headless auth/service-account guidance for unattended runs. Missing for 10: documented CI examples (e.g., GitHub Actions/pipeline configs), non-interactive/service-account authentication flow for the CLI, and independent hands-on confirmation of full automation beyond version-check.

   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to Gram (Speakeasy)
   Composio—none0/10

   All evidence describes Composio exposing its own hosted MCP server/toolkits for agents to consume (composio-docs-4, composio-probe-3, composio-probe-rt-1) — the reverse of this story, which asks whether a user can plug external MCP servers INTO Composio so it can use their tools. No evidence shows Composio importing or connecting to third-party MCP servers as a client.

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullclaimed7/10

   Gram documents fronting/governing third-party MCP servers ('put a governed endpoint in front of an MCP server someone else runs') and an in-dashboard harness to connect, authenticate, and chat with a model that calls tools from a connected server, plus a real-world case (MoonPay, 200+ MCP servers aggregated under the gateway). This directly satisfies plugging external MCP servers in so their tools become usable. Missing for 10: independent/hands-on verification of third-party server integration in production and more detail on tool-call fidelity across diverse upstream servers.

   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”
   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”
   • [claimed-docs] “MoonPay brought 200+ MCP servers under the MCP Gateway, going from proof of concept to a company-wide rollout in 30 days.”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round to Gram (Speakeasy)
   Composio✓fullprobed8/10

   Composio documents and exposes an official hosted MCP server (session `mcp: true`, and the shared 'Composio Connect' MCP URL) that any MCP-compatible agent/client can point at, and a live runtime probe confirms the endpoint (connect.composio.dev/mcp) is real and gated by managed OAuth as documented. This is a fair and clearly-applicable capability for an integration platform, backed by first-party docs plus independent endpoint verification. Missing for 10: an independent third-party report of a successful end-to-end agent session over this MCP endpoint (only auth-challenge probe, not a full tool-call transcript).

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Combine `mcp: true` with the direct-tools preset to get one MCP URL that serves exactly the tools you list”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed9/10

   Gram documents and exposes an official 'Platform MCP' server that lets MCP clients (Claude Code, Cursor, Codex, etc.) connect directly, plus generates/hosts MCP servers as its core product function, and a CLI/runtime probe confirms the tooling actually installs and runs. missing for 10: independent (non-vendor) hands-on confirmation that an external agent successfully connects to and operates the Platform MCP server end-to-end.

   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [probe] “official MCP server documented at https://www.speakeasy.com/docs/ai-control-plane/reference/platform-mcp”
   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to Composio
   Composio✓fullprobed9/10

   Composio ships an official CLI with documented install/setup for Codex, Claude Code, and terminal use, enabling agents or developers to connect apps, execute tools, inspect schemas, and debug projects locally (composio-docs-10, composio-docs-11, composio-probe-4). missing for 10: no independent/hands-on user report specifically validating the CLI (community evidence relates to unrelated connection issues, not the CLI itself).

   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [claimed-docs] “Install the native plugin for Codex or Claude Code: composio setup --target auto”
   • [probe] “official CLI documented at https://docs.composio.dev/docs/cli”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed8/10

   Gram ships an official CLI documented at reference/command-line, used to install toolsets as MCP servers and keep itself updated, and a runtime probe confirms the binary actually downloads and runs (`gram --version`). missing for 10: independent/third-party review of CLI usage beyond the vendor docs and a single runtime probe, and more detail on the full breadth of CLI commands beyond install/update.

   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to Composio
   Composio✓fullprobed9/10

   Composio publishes a documented, structured API surface: an OpenAPI spec (verified live via probe), SDK session creation calls, meta-tools, CLI, and MCP endpoints, all documented on docs.composio.dev and confirmed reachable/installable via runtime probes (openapi.json 200, SDK installs succeed, MCP endpoint live behind auth). This gives AI-native users multiple concrete, documented ways (REST API, Python/TS SDK, CLI, MCP) to drive the product programmatically. Missing for 10: independent third-party review validating API completeness/stability beyond vendor docs and probes.

   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”
   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/composio/): `npm install @composio/core` and `uv pip install composio…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialprobed7/10

   Gram exposes a documented Platform MCP server and CLI that let an AI-native user/agent drive the product programmatically (find/add MCP servers, author skills, manage policies, read logs) without the dashboard, and the CLI was independently verified to install and run. However, a direct probe for a conventional public REST/OpenAPI API (openapi.json, swagger.json, etc.) returned 404s, so there is no evidence of a general-purpose documented HTTP API beyond the MCP/CLI surfaces. Missing for 10: a documented REST/OpenAPI public API endpoint, independent confirmation of the Platform MCP server's functional breadth beyond docs.

   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [probe] “official MCP server documented at https://www.speakeasy.com/docs/ai-control-plane/reference/platform-mcp”
   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.speakeasy.com/openapi.json, https://www.speakeasy.com/swagger.json, https://www.speakeas…”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to Gram (Speakeasy)
   Composio!disputedcontradicted5/10

   Composio's docs show real scoping controls—restricting a session to specific toolkits, generating an MCP URL for an exact tool subset, and meta-tools that authorize apps on demand ([composio-docs-3],[composio-docs-5],[composio-docs-14])—but independent security research found a leaked Composio key returned live Gmail, GitHub, and CircleCI tokens, indicating credentials are not tightly least-privilege in practice ([composio-comm-2]). This is a concrete hands-on/security counter-example directly on the scoped-credential axis, so the claim is disputed rather than cleanly fulfilled. Missing for 10: no documentation of granular per-scope/permission-level credential issuance (only toolkit/tool-list restriction), and no rebuttal or fix evidence for the leaked-key finding.

   • [claimed-docs] “To restrict the set, pass `toolkits` when you create the session.”
   • [claimed-docs] “Combine `mcp: true` with the direct-tools preset to get one MCP URL that serves exactly the tools you list”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [community] “A leaked Composio key returned live Gmail, GitHub and CircleCI tokens (security research on exposed credentials in Composio's agent integrat…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed6/10

   Gram supports OAuth 2.0/2.1 integration with external providers, RBAC enforced at team/server/tool level synced to enterprise IDPs, and governs upstream credentials for third-party MCP servers with shadow-server blocking — collectively enabling scoped access control for agent tool use. However, there's no explicit documentation of per-agent least-privilege token minting/vending or fine-grained scope selection at credential-issuance time; the emphasis is on gateway-level RBAC and OAuth pass-through rather than dynamically scoped credential issuance. Missing for 10: explicit per-agent credential scoping/token minting mechanism, documented least-privilege scope configuration UI, and independent verification of enforcement.

   • [claimed-docs] “you can use any OAuth 2.0 provider you have set up (such as Auth0, Okta, Keycloak, or your own OAuth server)”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Composio
   Composio✓fullprobed8/10

   Composio ships official SDKs for both TypeScript (@composio/core) and Python (composio), verified via real installs that imported cleanly, alongside provider adapters that map tools into native framework formats and OpenAPI-documented endpoints. missing for 10: independent third-party tutorials/showcases building production apps with the SDKs, and no evidence of SDKs beyond Python/TypeScript (e.g. Go, Java).

   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/composio/): `npm install @composio/core` and `uv pip install composio…”
   • [claimed-docs] “A provider is the adapter that turns Composio tools into the native tool format your framework expects, so you don't write glue code.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence pack documents Gram's CLI, MCP server distribution, OAuth, skills, and observability features, but nowhere mentions official client SDKs (e.g., Python/TypeScript/Go libraries) for building against Gram's platform itself — this is a fair question for a platform product but is simply unaddressed.

   ⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round to Composio
   Composio✓fullclaimed7/10

   Composio documents a triggers/webhooks system where users register a webhook URL per project and Composio POSTs signed event payloads for every trigger, directly matching the subscribe-to-events story. missing for 10: independent/hands-on confirmation of webhook delivery reliability, event catalog/filtering details, and retry/latency behavior are not evidenced beyond first-party docs.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's evidence covers MCP server generation, tool logs, CLI, and RBAC, but nothing describes webhook subscriptions or event-driven push notifications for AI agents; missing for 10: webhook subscription mechanism, event schema/documentation, or any mention of push-based event delivery.

   ⚑ flag

Agentic features

1. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Composio
   Composio~partialcommunity5/10

   Composio's triggers feature lets developers register a webhook so Composio pushes event notifications (e.g., new email, PR) that can kick off agent logic without polling, which is a building block for background automation [composio-docs-9]. However, Composio itself doesn't host or run the autonomous agent loop — execution still happens in the developer's own agent/runtime that reacts to the webhook, and there's no evidence of a built-in scheduler or persistent background-running agent. Community evidence also shows integration reliability issues (Instagram connect failing) that would undercut unattended automation [composio-comm-1]. Missing for 10: evidence of Composio actually orchestrating/running an agent autonomously (not just delivering trigger events), a scheduling/cron-like automation primitive, and independent confirmation that background-triggered automations run reliably end-to-end.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

2. ai-native userOperate the product with natural-language commands

   weight 2 · round to Gram (Speakeasy)
   Composio~partialclaimed5/10

   Composio's CLI plugin integrates with Claude Code/Codex so a coding agent (or its user) can invoke Composio's tools via natural-language prompts inside those agents, and meta-tools let an agent discover/execute tools autonomously. However, Composio itself is an integration/tool-serving layer rather than a natural-language interface — there's no first-party chat/NL command surface of its own, only its embedding inside other agentic tools. Missing for 10: a dedicated natural-language command interface owned by Composio itself, and independent evidence of end-users issuing free-form NL commands successfully.

   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [claimed-docs] “Install the native plugin for Codex or Claude Code: composio setup --target auto”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed7/10

   Gram documents a platform MCP server that lets an org admin operate the product (find/add servers, manage risk policies, author skills, read observability) entirely from within an MCP client like Claude Code or Cursor using natural language, without opening the dashboard — this is direct agentic/NL control of the product itself. Missing for 10: independent/hands-on verification that these natural-language admin operations actually execute correctly (only the CLI binary itself was runtime-probed, not the platform-mcp NL workflow).

   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [probe] “official MCP server documented at https://www.speakeasy.com/docs/ai-control-plane/reference/platform-mcp”
   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Composio—none0/10

   The pack shows an OpenAPI spec exists (composio-probe-2) and standard docs pages, but there is no evidence of an interactive API reference (e.g., Swagger/Redoc-style try-it console) with runnable, in-browser examples — missing for 10: dedicated interactive API explorer, runnable code snippets, evidence of a try-it-now console.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram is positioned as an AI control plane/MCP gateway focused on server generation, governance, and observability; there is no evidence of an interactive API reference UI with runnable examples for exploring endpoints, nor documentation of such a feature.

   ⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round to Composio
   Composio✓fullprobed9/10

   A live probe confirms Composio serves a valid machine-readable OpenAPI spec at https://docs.composio.dev/openapi.json (HTTP 200, contains 'openapi' key), directly satisfying the story. Missing for 10: no independent third-party confirmation of spec completeness/versioning beyond the probe check.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence shows Gram consumes OpenAPI documents as input to build MCP servers, but there is no evidence Gram itself publishes a downloadable machine-readable spec of its own API — the direct probe for openapi.json/swagger.json/well-known paths all returned 404.

   • [probe] “PROBE openapi: all candidate paths 404 (https://www.speakeasy.com/openapi.json, https://www.speakeasy.com/swagger.json, https://www.speakeas…”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Gram (Speakeasy)
   Composio—none0/10

   Evidence shows a 'remote sandbox' feature (COMPOSIO_REMOTE_WORKBENCH) for running Python code on bulk/large tool responses, and a docs teaser mentioning 'sand[box]', but nothing describes an isolated test environment that lets a user exercise Composio's actual toolkits/connected accounts without touching live production data or APIs — connections like Instagram/Gmail appear to hit real production endpoints (per composio-comm-1/2).

   • [claimed-docs] “COMPOSIO_REMOTE_WORKBENCH** — Run Python in a remote sandbox for bulk operations or processing large tool responses.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.composio.dev/llms.txt # Composio Documentation > Composio powers 1,400+ toolkits, tool search, con…”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed4/10

   Gram provides an in-dashboard 'Connect' harness to test MCP servers with a model before wiring up a real client, which offers some sandbox-like isolation from production agent traffic, but there is no explicit mention of separate sandbox vs. production data/environments, test fixtures, or guarantees that tool calls in this harness avoid hitting live production data sources. missing for 10: explicit sandbox/staging environment concept, isolation guarantees from production data, mock or test data support during agent testing.

   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Composio—none0/10

   The evidence pack shows API/SDK docs, CLI, MCP endpoints, and an OpenAPI spec, but nowhere mentions API versioning conventions or a documented deprecation policy for breaking changes. Missing for 10: any changelog, version-numbering scheme (e.g. v1/v2 endpoints), or explicit deprecation/sunset policy documentation.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence pack covers OpenAPI-generated MCP servers, OAuth, governance, CLI, observability, and skills, but contains no documentation of API versioning practices or a deprecation policy for Gram's own APIs/servers.

   ⚑ flag

Multi tenant

1. platform-engineerMaintain separate connected accounts and scoped tokens per end user of my product, so each user's agent acts only on that user's credentials

   weight 3 · round to Composio
   Composio✓fullclaimed7/10

   Composio's session model is built around per-user identity (`user_id`) and each user must complete their own Connect Link OAuth flow, so an agent only executes tools using the connected end-user's own credentials, with callback_url and white-labeling to fit into a platform's UX. Missing for 10: independent/hands-on verification that token scoping is enforced per user in production, and explicit detail on scope-limiting (vs. just per-user connection) mechanics.

   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”
   • [claimed-docs] “The agent posts the Connect Link straight into the conversation. The user clicks it, signs in, and the agent retries the tool and continues.…”
   • [claimed-docs] “To send users back to your app after they connect, pass a `callback_url`”
   • [claimed-docs] “You can swap in your logo and name, then restyle the whole page to match your product.”
   • [claimed-docs] “To restrict the set, pass `toolkits` when you create the session.”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's evidence describes OAuth 2.0 provider integration and team/server/tool-level RBAC synced to an enterprise IDP (gram-docs-2, gram-gh-2), but nothing documents per-end-user connected accounts or scoped tokens so that each of a platform-engineer's own product's end users gets isolated credentials — the RBAC described is org/team-level, not end-user-level multi-tenant credential isolation. Missing for 10: explicit documentation of per-end-user connected-account/token scoping, any multi-tenant credential isolation guide, and evidence of enforcement that one end user's agent cannot use another's credentials.

   • [claimed-docs] “you can use any OAuth 2.0 provider you have set up (such as Auth0, Okta, Keycloak, or your own OAuth server)”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”

   proof ↗⚑ flag

Oauth

1. developerLet the platform run OAuth flows for downstream tools and vault the resulting credentials encrypted, with automatic token refresh

   weight 3 · round to Gram (Speakeasy)
   Composio!disputedcontradicted4/10

   Composio's docs describe hosted OAuth via Connect Link with no auth UI to build, callback_url redirects, and white-labeled auth pages (composio-docs-6,7,8), implying it manages and stores credentials for downstream tool auth. However, independent security research found a leaked Composio API key exposed live Gmail, GitHub, and CircleCI tokens, undermining the 'encrypted vault' claim in practice, and a user reported OAuth connect flows failing outright (400 errors) for at least one integration. Missing for 10: explicit documentation of encryption-at-rest mechanics, automatic token-refresh behavior, and any vendor response/fix to the leaked-token incident.

   • [claimed-docs] “The agent posts the Connect Link straight into the conversation. The user clicks it, signs in, and the agent retries the tool and continues.…”
   • [claimed-docs] “To send users back to your app after they connect, pass a `callback_url`”
   • [claimed-docs] “You can swap in your logo and name, then restyle the whole page to match your product.”
   • [community] “A leaked Composio key returned live Gmail, GitHub and CircleCI tokens (security research on exposed credentials in Composio's agent integrat…”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed5/10

   Docs confirm Gram lets you configure external OAuth 2.0 providers (Auth0, Okta, Keycloak, custom) for downstream tool auth and enforces OAuth2.1/RBAC for platform access, implying it manages OAuth flows for connected servers, but there is no explicit documentation of encrypted credential vaulting or automatic token refresh mechanics. Missing for 10: explicit description of credential encryption at rest, automatic refresh-token handling, and any hands-on/independent verification of these security claims.

   • [claimed-docs] “you can use any OAuth 2.0 provider you have set up (such as Auth0, Okta, Keycloak, or your own OAuth server)”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”

   proof ↗⚑ flag

White label

1. platform-engineerWhite-label the authorization flow with my own branding and my own OAuth app credentials

   weight 1 · round drawn
   Composio~partialclaimed6/10

   Composio's docs explicitly describe white-labeling the auth flow (swap logo/name, restyle the page) and building custom callback flows, directly matching the branding half of the story. However, there is no evidence in the pack that platform-engineers can supply their own OAuth app client ID/secret (bring-your-own OAuth credentials) rather than using Composio-managed auth apps. Missing for 10: documentation of custom/BYO OAuth app credential support, independent confirmation that white-labeling works end-to-end in production.

   • [claimed-docs] “You can swap in your logo and name, then restyle the whole page to match your product.”
   • [claimed-docs] “The agent posts the Connect Link straight into the conversation. The user clicks it, signs in, and the agent retries the tool and continues.…”
   • [claimed-docs] “To send users back to your app after they connect, pass a `callback_url`”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed6/10

   Gram supports bringing your own OAuth 2.0 provider (Auth0, Okta, Keycloak, or a custom server) so the underlying credentials are fully controlled by the platform-engineer, and RBAC/OAuth2.1 syncs to enterprise IDPs. However, there is no documented evidence of UI-level 'white-labeling' — custom branding/logo/domain on the consent or login screen itself — only backend provider configuration. Missing for 10: evidence of custom branding/theming of the OAuth consent screen or hosted login page, and confirmation that the authorization UI itself (not just the provider) can be rebranded.

   • [claimed-docs] “you can use any OAuth 2.0 provider you have set up (such as Auth0, Okta, Keycloak, or your own OAuth server)”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Composio
   Composio~partialcommunity5/10

   Composio documents parallel tool execution via meta-tools and a remote sandbox workbench explicitly for 'bulk operations or processing large tool responses,' supporting some bulk automation across many items. However, there's no dedicated bulk-operation API, batch endpoint, or hands-on evidence demonstrating bulk operations at scale, and one community report shows a basic connection failure (Instagram 400 error), raising reliability doubts. missing for 10: a documented batch/bulk API or endpoint, concrete examples of bulk operations across many items, independent hands-on verification of bulk workflows succeeding at scale.

   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [claimed-docs] “COMPOSIO_REMOTE_WORKBENCH** — Run Python in a remote sandbox for bulk operations or processing large tool responses.”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence pack covers MCP server hosting, governance, observability, RBAC, and CLI/skills features, but nowhere describes any bulk or batch operation capability (e.g., bulk approving tools, bulk managing servers, or batch tool invocation across many items). This is a plausible axis for a control-plane product managing many servers/toolsets, but no evidence shows it delivers.

   ⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Composio
   Composio~partialclaimed5/10

   Composio's Triggers system delivers event notifications to a registered webhook (signed, verifiable), which lets a user's own system react to events — but Composio itself doesn't appear to expose a first-party rule-definition or condition/action automation builder; the 'trigger action automatically' logic must be implemented by the user's external service consuming the webhook. missing for 10: a documented rules/automation engine (conditions, filters, chained actions) inside Composio itself, and independent evidence of end-to-end automatic action execution rather than just event delivery.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's evidence covers governance, observability, tool logging, RBAC, and MCP server management, but there is no mention of user-definable rule/trigger systems that automatically fire actions in response to events (e.g., event-driven automation, if-this-then-that rules, webhooks-to-action). Risk policies and access controls are governance constructs, not event-triggered automation rules.

   ⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round drawn
   Composio—none0/10

   Composio documents event-driven triggers delivered via webhooks (composio-docs-9), but there is no evidence of a scheduling primitive for recurring jobs/workflows (e.g., cron-style automation) — triggers fire on external events, not on a time-based recurring schedule. Missing for 10: any documented scheduler, cron syntax, or recurring-job API, and independent confirmation that scheduled workflows actually run.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram is an MCP control plane focused on generating, governing, observing, and distributing MCP servers/toolsets/skills, but nothing in the evidence describes scheduling recurring jobs or workflows (e.g., cron-like triggers or automated recurring execution) — it only covers request-time tool calls, logging, and access control.

   ⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round to Gram (Speakeasy)
   Composio—none0/10

   The evidence pack covers Composio's tool/session/MCP/auth infrastructure but contains no mention of versioning automations, review workflows, or rollback of workflows/agent configurations. No changelog, version history, diff/review, or revert capability is documented anywhere in the pack.

   ⚑ flag

   Gram (Speakeasy)~partialclaimed4/10

   Gram documents that skills are versioned (`SKILL.md` manifest) and provides audit/observability logs of tool calls, giving some traceability, but there is no evidence of a review workflow or an explicit rollback mechanism for automations/toolsets. missing for 10: explicit review process, rollback/revert functionality, version history UI for toolsets or MCP server configs.

   • [claimed-docs] “A skill is a versioned `SKILL.md` manifest: reusable instructions that agents load on demand.”
   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”

   proof ↗⚑ flag

Bridge

1. developerBridge a locally running MCP server into the platform as a managed remote connection without deploying it

   weight 1 · round to Gram (Speakeasy)
   Composio—none0/10

   Composio's docs describe the platform hosting its own MCP server (sessions with mcp:true, Composio Connect shared URL) so agents can consume Composio's tools via MCP, but there is no evidence of the reverse flow — registering or tunneling a developer's own locally-running MCP server into Composio to be exposed as a managed remote connection.

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed5/10

   Docs mention 'tunneled MCP servers' alongside hosted and shadow servers in the observability/tool-logs reference, implying a mechanism to bridge a locally running server into the platform without redeploying it, and the third-party governance guide covers fronting externally-run servers. However, no dedicated guide, CLI command, or hands-on walkthrough for setting up a local-to-remote tunnel is included in the evidence pack. Missing for 10: a dedicated tunnel setup guide/CLI reference, concrete steps for exposing a local dev MCP server, and independent confirmation the tunnel actually works.

   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”
   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”

   proof ↗⚑ flag

Access control

1. platform-engineerRestrict which servers and individual tools an agent may call through allowlists or per-tool access control

   weight 2 · round to Gram (Speakeasy)
   Composio✓fullclaimed7/10

   Composio's docs describe restricting a session to specific toolkits (composio-docs-3) and combining `mcp:true` with a direct-tools preset to serve an MCP URL with exactly the listed tools (composio-docs-5), directly enabling allowlist-style, per-tool/server access control for an agent. Missing for 10: independent/hands-on verification of enforcement, granular per-user or role-based policy controls, and audit/logging of denied calls.

   • [claimed-docs] “To restrict the set, pass `toolkits` when you create the session.”
   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Combine `mcp: true` with the direct-tools preset to get one MCP URL that serves exactly the tools you list”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullclaimed8/10

   Gram documents team/server/tool-level RBAC enforced with OAuth2.1 synced to enterprise IDPs, and explicitly blocks unsanctioned shadow MCP servers by default, directly matching allowlist/per-tool access control for a platform-engineer persona. This is corroborated by third-party rollout evidence (MoonPay) and governance guides for wrapping third-party MCP servers with access control. Missing for 10: independent hands-on verification of the RBAC/permission enforcement mechanics beyond vendor docs and GitHub README claims.

   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”
   • [claimed-docs] “MoonPay brought 200+ MCP servers under the MCP Gateway, going from proof of concept to a company-wide rollout in 30 days.”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”

   proof ↗⚑ flag

Audit

1. platform-engineerSee a log of every tool call that passed through the platform, with enough detail to audit or replay what an agent did

   weight 2 · round to Gram (Speakeasy)
   Composio—none0/10

   The evidence pack describes sessions, meta-tools, MCP hosting, auth flows, and a webhook mechanism for triggers, but nowhere documents a queryable audit log of tool calls with replay detail for platform engineers. Webhook triggers (composio-docs-9) cover external events, not a comprehensive tool-call audit trail.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullclaimed8/10

   Gram documents a raw execution log capturing every tool call across hosted, tunneled, shadow MCP servers, skills, and local tools ([gram-docs-5]), plus usage/cost tracking and session deep-dives ([gram-gh-1]) that support audit and replay-style analysis. RBAC/OAuth-scoped access adds context for auditability of who called what ([gram-gh-2]). Missing for 10: independent/hands-on verification of log completeness or replay functionality specifically, and detail on log retention/export format.

   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”
   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”

   proof ↗⚑ flag

Catalog

1. developerReach a deep catalog of third-party apps and APIs (hundreds to thousands of integrations) through the platform

   weight 3 · round to Composio
   Composio✓fullprobed9/10

   Composio's own docs and llms.txt claim 1,400+ toolkits reachable via meta-tools, sessions, MCP, and CLI, and probes confirm the docs/OpenAPI/MCP endpoint are live and functioning (though gated by auth), plus SDK packages install cleanly. One community report shows an isolated integration failure (Instagram OAuth) which doesn't contradict the breadth of the catalog itself. Missing for 10: independent third-party verification of the actual '1,400+' count beyond vendor claims.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.composio.dev/llms.txt # Composio Documentation > Composio powers 1,400+ toolkits, tool search, con…”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/composio/): `npm install @composio/core` and `uv pip install composio…”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's documented model is generating MCP servers from your own OpenAPI/TypeScript source or governing a third-party MCP server you already run — there is no evidence of a pre-built catalog of hundreds/thousands of ready-made third-party app/API integrations comparable to a connector marketplace. The MoonPay case study mentions bringing 200+ existing internally-run MCP servers under governance, not a Gram-provided catalog to browse and connect.

   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”
   • [claimed-docs] “MoonPay brought 200+ MCP servers under the MCP Gateway, going from proof of concept to a company-wide rollout in 30 days.”

   proof ↗⚑ flag

Connect

1. developerPoint my MCP-compatible client at a hosted remote server URL and reach downstream tools without deploying any infrastructure

   weight 3 · round drawn
   Composio✓fullprobed8/10

   Composio documents a hosted MCP server reachable via a shared URL (Composio Connect) or per-session mcp:true flag, requiring no self-hosted infrastructure, and a live probe confirms the endpoint at connect.composio.dev/mcp is active and auth-gated as documented. Missing for 10: independent hands-on confirmation of a successful authenticated tool call through the hosted endpoint beyond the 401 challenge probe.

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Combine `mcp: true` with the direct-tools preset to get one MCP URL that serves exactly the tools you list”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed8/10

   Gram's Control Plane auto-generates hosted MCP servers from an OpenAPI doc or TypeScript project, so developers get a remote server URL to point MCP clients at without deploying infra themselves, and this is corroborated by the CLI runtime probe and a real customer rollout (MoonPay's 200+ servers under the MCP Gateway). missing for 10: independent hands-on confirmation of an end-to-end client connecting to a hosted URL (only the CLI tool itself was runtime-verified, not the hosted server reachability).

   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [claimed-docs] “MoonPay brought 200+ MCP servers under the MCP Gateway, going from proof of concept to a company-wide rollout in 30 days.”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”

   proof ↗⚑ flag

Generate

1. developerTurn an existing API definition (OpenAPI) or custom code into a hosted MCP server without writing protocol plumbing

   weight 2 · round to Gram (Speakeasy)
   Composio—none0/10

   Evidence shows Composio can expose its own pre-built toolkits as hosted MCP servers (via `mcp:true` sessions or Composio Connect) and ships an OpenAPI spec for its own API, but there is no evidence of a feature letting developers upload a custom OpenAPI definition or their own code and have Composio auto-generate a hosted MCP server from it — the toolkits appear to be Composio-curated integrations, not user-supplied API/code onboarding.

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed8/10

   Gram's Control Plane generates a hosted MCP server directly from an OpenAPI document or a TypeScript Functions project, with no protocol plumbing required from the developer, and a CLI/runtime probe confirms the tooling actually installs and runs. missing for 10: independent (non-vendor) hands-on account of the OpenAPI-to-MCP generation flow itself, and more detail on custom-code/TypeScript function support beyond the docs mention.

   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”

   proof ↗⚑ flag

Publish

1. developerPublish my own MCP server on the platform for distribution, discovery, and installation by others

   weight 2 · round to Gram (Speakeasy)
   Composio—none0/10

   Composio's docs describe hosting MCP servers built from its own toolkits/sessions and exposing a shared Composio Connect MCP endpoint for consuming tools, plus a CLI for local tool use — all about consuming Composio-provided tools, not about a developer publishing a custom MCP server for others to discover and install. No evidence of a publish/marketplace/registry flow for third-party-authored MCP servers.

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed8/10

   Gram lets developers generate/host MCP servers from OpenAPI or TypeScript sources on its Control Plane, with governance, RBAC, tool logs, and a CLI/dashboard to publish, discover, and install servers into MCP clients, and the CLI was independently verified to install and run. missing for 10: no independent (non-vendor) evidence of a public discovery/marketplace listing showing other developers actually finding and installing a third party's published server, and no detail on versioning/publishing workflow specifics beyond docs.

   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [claimed-docs] “MoonPay brought 200+ MCP servers under the MCP Gateway, going from proof of concept to a company-wide rollout in 30 days.”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [probe] “official MCP server documented at https://www.speakeasy.com/docs/ai-control-plane/reference/platform-mcp”

   proof ↗⚑ flag

Triggers

1. developerSubscribe my agent to events from connected downstream apps (triggers/webhooks) instead of polling them

   weight 2 · round to Composio
   Composio✓fullclaimed7/10

   Composio documents a triggers/webhooks system where developers register one webhook URL per project and Composio POSTs signed trigger events from connected apps, directly enabling event subscription instead of polling. However, missing for 10: independent/hands-on confirmation of trigger reliability, details on per-app trigger configuration granularity, and evidence of delivery guarantees or retry semantics.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's documented capabilities center on generating/govern/observe MCP servers from OpenAPI specs and exposing tools for agents to call (pull-based); there is no mention anywhere in the evidence of webhook subscriptions, event triggers, or push notifications from downstream apps to agents. This is a plausible axis for an MCP gateway/control-plane product, but no evidence shows it delivers it.

   ⚑ flag

Analytics

1. platform-engineerSee usage analytics for the servers and tools my org consumes or publishes — call volumes, users, errors, and cost

   weight 2 · round to Gram (Speakeasy)
   Composio—none0/10

   Evidence covers sessions, MCP hosting, auth, triggers, CLI, and SDK installation, but nothing in the pack describes any analytics/usage dashboard, call volume metrics, per-user breakdowns, error rates, or cost reporting for consumed or published tools/servers.

   ⚑ flag

   Gram (Speakeasy)✓fullclaimed7/10

   Gram documents tool-call logs across hosted/tunneled/shadow servers and local tools, and the GitHub repo description explicitly claims usage tracking across teams by tokens/cost, budgets, and tool effectiveness measurement, directly matching call volumes/cost/usage. Errors are implied via raw execution logs but not explicitly broken out as an error-rate metric, and there's no independent/hands-on corroboration of the analytics dashboard itself. Missing for 10: explicit error-rate reporting UI evidence, independent third-party validation of the analytics dashboard beyond vendor docs/GitHub description.

   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”
   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round drawn
   Composio~partialprobed6/10

   Composio ships a full OpenAPI-documented API, SDK, CLI, and hosted MCP server covering sessions, auth, triggers, and tool execution (composio-docs-1,9,10; composio-probe-2,3,4), suggesting an API-first architecture where most UI-configurable actions (auth, webhooks, toolkits, white-labeling) have API/CLI equivalents. However there's no explicit documentation asserting full 1:1 parity between the dashboard UI and the API, and a community report shows at least one integration (Instagram connect) failing via the API path with a 400 error, suggesting some workflows may not work as smoothly as advertised. Missing for 10: an explicit parity statement or audit showing every UI feature (e.g. dashboard analytics, project settings) is API-accessible, and independent confirmation that failures like the Instagram case are edge cases rather than systemic gaps.

   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”
   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”
   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “official CLI documented at https://docs.composio.dev/docs/cli”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialprobed6/10

   Gram exposes a Platform MCP server and CLI that let an org admin add/configure MCP servers, author skills, manage risk policies, and read observability data without opening the dashboard (gram-docs-4, gram-docs-7, gram-probe-5), and the CLI is confirmed to actually install and run (gram-probe-rt-1). However, no public REST/OpenAPI surface was found (gram-probe-3: openapi.json paths all 404), so full UI-parity via a general API is unconfirmed. Missing for 10: a documented general-purpose REST/OpenAPI API covering all dashboard actions, and independent confirmation that every UI feature (not just admin/config tasks) is reachable programmatically.

   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.speakeasy.com/openapi.json, https://www.speakeasy.com/swagger.json, https://www.speakeas…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   Composio—none0/10

   No evidence anywhere in the pack of a data export feature, open-format export, or account portability tooling; documentation covers sessions, MCP, auth, CLI, and triggers but nothing about exporting user data or configurations. The axis is fair for a SaaS integration platform holding connection/auth/trigger data, but no evidence supports it.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence describes Gram's inputs (OpenAPI specs, SKILL.md manifests) and platform features like RBAC, logging, and CLI, but there is no documented data-export feature letting a user pull out their configured toolsets, logs, policies, or org data in open formats to migrate away. Axis applies to a platform like this but no evidence supports it.

   ⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Composio—none0/10

   No evidence of an open-license public source repository; evidence only covers docs, CLI, SDK packages (which are published but not shown to be open-licensed source), and hosted MCP/cloud infrastructure. Composio appears to be primarily a closed hosted SaaS platform, with no mention of a license or public repo for its core source.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence shows a public GitHub repository (github.com/speakeasy-api/gram) exists, but nothing in the pack states or documents an open-source license or that the full source is freely readable/licensed for reuse; docs focus on hosted product features (CLI, MCP servers, RBAC, observability) rather than source availability or licensing terms.

   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   Composio—none0/10

   All evidence describes Composio as a hosted/cloud SaaS (sessions, hosted MCP endpoint, cloud webhook delivery) with no documentation or mention of a self-hosted deployment option; community evidence even references 'your cloud solution.' Self-hosting is a fair axis for this type of platform product, but no evidence shows it is supported.

   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”
   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   While Gram's code is hosted on GitHub (gram-gh-1, gram-gh-2), the evidence pack contains no documentation, guide, or deployment instructions describing a supported self-hosted deployment of the core Control Plane — all docs describe the hosted SaaS platform, dashboard, and CLI/skill features rather than self-hosting.

   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”

   proof ↗⚑ flag

Pricing

1. developerPrototype my agent's tool connections on a meaningful free tier before paying anything

   weight 1 · round drawn
   Composio—none0/10

   The evidence pack contains no mention of pricing tiers, free tier limits, or cost structure for Composio; it only covers technical docs (sessions, MCP, CLI, auth) and unrelated community/security notes. Pricing-plans is a fair axis for a SaaS developer platform, but there is no evidence it delivers a meaningful free tier.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence pack contains no mention of pricing tiers, a free tier, or cost thresholds for Gram — nothing describes what is available without payment. This is a fair pricing-plans axis for any commercial dev tool, but there's simply no documentation or citation about free-tier limits or trial terms.

   ⚑ flag

2. developerSee transparent pricing per tool call or connection plus documented rate limits before committing to the platform

   weight 2 · round drawn
   Composio—none0/10

   The evidence pack contains no pricing pages, per-tool-call cost tables, or documented rate limits anywhere in the docs, probes, or community items — only technical usage docs and unrelated auth/security anecdotes. Missing for 10: any pricing page or per-call cost breakdown, any documented rate-limit thresholds, any plan comparison before commitment.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   No evidence pack item mentions pricing, cost per tool call/connection, or documented rate limits; all citations concern architecture, RBAC, skills, and CLI features. Absence of evidence for an applicable and standard pricing-transparency axis means this must be scored none.

   ⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   Composio—none0/10

   No evidence in the pack addresses data residency, region selection, or storage location controls for Composio's hosted services; nothing mentions where data is stored or any regional configuration options.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   No evidence in the pack mentions data residency, region selection, or geographic storage controls for Gram; the documentation focuses on governance, RBAC, tool logs, and MCP server hosting but never addresses where data is physically stored.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Composio—none0/10

   No evidence in the pack addresses data usage for AI model training, opt-out settings, or privacy controls related to training data; documentation covers sessions, MCP, auth, and CLI but nothing about training-data exclusion policies.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's evidence pack covers MCP server governance, RBAC, observability, and tool logging, but nothing addresses whether user/tool-call data is used to train AI models or an opt-out policy for such training — missing for 10: any data-training policy statement, opt-out controls, or model-training disclosure.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Composio—none0/10

   The evidence pack contains no documentation or mention of data retention policies, data deletion APIs, or user-controlled data lifecycle management for Composio — only session/auth/tool-execution mechanics are covered.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence pack covers governance, RBAC, audit logs, OAuth, and shadow-server blocking, but no citation addresses data retention policies, deletion controls, or user data lifecycle management for the AI-native platform.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Composio—none0/10

   No evidence pack item addresses a telemetry/usage-tracking opt-out mechanism, privacy settings, or data collection policy for Composio; the docs cover sessions, MCP, auth, CLI, and triggers but nothing about telemetry controls.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   No evidence in the pack addresses whether Gram's CLI, dashboard, or platform collects vendor-side telemetry/usage data or offers an opt-out; the only related items describe customer-facing usage analytics (tracking AI agent activity), not the product's own telemetry practices. Missing for 10: any mention of a telemetry setting, opt-out flag, or privacy policy statement about data collection from the CLI/platform itself.

   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”

   proof ↗⚑ flag

Api

1. ai agentQuery the platform's registry or catalog through a documented API to discover servers and tools programmatically

   weight 3 · round to Composio
   Composio✓fullprobed8/10

   Composio exposes a documented OpenAPI spec (probe confirms HTTP 200 with openapi key), meta-tools for programmatic discovery of toolkits/apps, and a hosted MCP endpoint that agents can query to discover servers/tools, plus an llms.txt describing 1,400+ toolkits. This gives agents multiple documented, programmatic paths (REST/OpenAPI, meta-tools, MCP) to discover the catalog. missing for 10: a dedicated 'registry search/list' endpoint example with request/response schema, and independent third-party confirmation of catalog completeness beyond vendor docs.

   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.composio.dev/llms.txt # Composio Documentation > Composio powers 1,400+ toolkits, tool search, con…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialprobed6/10

   Gram documents an official 'platform-mcp' server that lets an agent working through an MCP client find and add MCP servers, manage toolsets, and read observability data programmatically (gram-docs-7, gram-probe-4), and the CLI can install toolsets as MCP servers (gram-docs-4/5). This covers the discovery story via the MCP protocol itself acting as the documented API, but there is no confirmed general-purpose REST/OpenAPI catalog endpoint (the openapi probe returned 404s) and no independent hands-on proof of the discovery flow itself. Missing for 10: a documented REST/OpenAPI registry endpoint, and third-party/hands-on verification that an agent can actually enumerate servers/tools via the platform MCP server.

   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [probe] “official MCP server documented at https://www.speakeasy.com/docs/ai-control-plane/reference/platform-mcp”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.speakeasy.com/openapi.json, https://www.speakeasy.com/swagger.json, https://www.speakeas…”

   proof ↗⚑ flag

Metadata

1. developerInspect a listed server's tools, input schemas, transport, and hosting attributes before connecting to it

   weight 2 · round drawn
   Composio~partialprobed5/10

   Composio's meta-tools and CLI let an agent/developer discover available toolkits and 'inspect schemas' at runtime (composio-docs-2, composio-docs-10, composio-docs-14), and the exposed openapi.json shows schema-level detail is available. However, there's no evidence of a browsable registry entry per listed server that surfaces transport type or hosting attributes before connecting — discovery is described as a runtime/meta-tool action rather than a pre-connection catalog view. Missing for 10: a documented registry/catalog page showing transport (SSE/stdio/hosted) and hosting attributes per toolkit, and independent confirmation that schema inspection works pre-connection as described.

   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed5/10

   Gram's in-dashboard 'Connect' harness lets a developer pick a listed server, authenticate, and chat with a model that calls its tools while inspecting logs, which implies pre-connection tool inspection, and servers are generated from OpenAPI/TypeScript sources implying schema definitions exist. However, no evidence explicitly documents a UI or API for viewing input schemas, transport type, or hosting attributes for a listed server. Missing for 10: explicit documentation of schema viewer, transport metadata display, hosting attribute display, and independent/hands-on confirmation of this inspection workflow.

   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”
   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”

   proof ↗⚑ flag

Playground

1. developerExercise an MCP server's tools from a browser-based inspector or playground before integrating it

   weight 2 · round to Gram (Speakeasy)
   Composio—none0/10

   No evidence pack item describes a browser-based inspector or playground for exercising MCP tools; documentation covers SDK sessions, CLI, meta-tools, and hosted MCP endpoints but never a web UI for pre-integration testing.

   ⚑ flag

   Gram (Speakeasy)✓fullclaimed8/10

   Gram provides an in-dashboard agent harness (gram-docs-8) explicitly described as a way to pick a server, authenticate, chat with a model that invokes tools, and inspect logs before connecting a real client — directly matching the story of exercising tools from a browser-based playground pre-integration. This is first-party documentation with a clear feature name ('Connect') dedicated to this exact workflow. Missing for 10: independent/hands-on user reports confirming the playground works as described, and more detail on tool-level interaction granularity within the UI.

   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”

   proof ↗⚑ flag

Search

1. developerSearch and browse a large catalog of MCP servers and tools by capability before wiring anything into my agent

   weight 3 · round to Composio
   Composio~partialprobed6/10

   Composio documents a large catalog (1,400+ toolkits) and built-in 'tool search' plus meta-tools that let an agent discover available tools/capabilities at runtime before execution, and toolkits can be filtered when creating a session. However, the evidence never shows a dedicated developer-facing browse/search UI or registry page for pre-wiring exploration — discovery is framed as an agent-runtime capability (meta-tools) rather than a human catalog browser. Missing for 10: a documented searchable catalog UI/dashboard for developers, independent hands-on confirmation of browsing by capability, and filtering/faceting details beyond toolkit name.

   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [claimed-docs] “To restrict the set, pass `toolkits` when you create the session.”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.composio.dev/llms.txt # Composio Documentation > Composio powers 1,400+ toolkits, tool search, con…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed3/10

   Gram documents that an admin can 'find and add MCP servers' via the platform MCP client integration, implying some discovery capability, but there is no evidence of a searchable/browsable catalog of MCP servers by capability, catalog size, or a discovery UI — Gram is primarily framed as a control plane for generating/governing your own servers rather than a registry of third-party servers. missing for 10: a documented catalog/search UI, evidence of catalog scale or third-party server listings, and any capability-based filtering/browsing mechanism.

   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”

   proof ↗⚑ flag

Cli

1. developerDrive the publish-and-deploy workflow from an official CLI — authenticate, stage or build a server, push it, and install it into agents

   weight 2 · round to Gram (Speakeasy)
   Composio~partialprobed4/10

   Composio's CLI supports authentication (composio setup) and installing a native plugin into agents like Codex/Claude Code, and sessions can expose a hosted MCP server, but there is no documented workflow for staging/building a custom server artifact and pushing it through the CLI — Composio's MCP server is a pre-built hosted endpoint, not something a developer builds and deploys via CLI. missing for 10: CLI subcommands for building/staging a server, a 'push' or deploy step, and evidence of a full publish pipeline rather than just connect/execute/install.

   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [claimed-docs] “Install the native plugin for Codex or Claude Code: composio setup --target auto”
   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [probe] “official CLI documented at https://docs.composio.dev/docs/cli”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialprobed7/10

   There's a documented, runtime-verified official CLI (gram-docs-4, gram-probe-5, gram-probe-rt-1) that supports installing toolsets as MCP servers into agents, plus docs on building servers from OpenAPI/TS and platform-mcp integration for org admins to add/configure servers from within MCP clients. However, the evidence doesn't clearly walk through explicit CLI subcommands for authenticate/stage/build/push in one coherent flow — most detail is on install and general CLI existence rather than a full documented publish pipeline. Missing for 10: explicit CLI command reference showing login/auth, build/stage, and push subcommands with example output, and independent (non-vendor) confirmation of the full workflow beyond version-check.

   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”

   proof ↗⚑ flag

Frameworks

1. developerPlug the platform's tools into major agent frameworks and SDKs (Vercel AI SDK, LangChain, CrewAI, OpenAI or Anthropic SDKs) with documented adapters

   weight 2 · round to Composio
   Composio~partialprobed6/10

   Composio documents a 'provider' concept explicitly designed as adapters that convert Composio tools into the native tool format for a given framework 'so you don't write glue code,' and runtime probes confirm working npm/pip SDK packages, supporting a multi-framework client model. However, the evidence pack never explicitly names or documents adapters for Vercel AI SDK, LangChain, CrewAI, OpenAI, or Anthropic SDKs specifically. Missing for 10: named documentation pages or examples for each specific framework (Vercel AI SDK, LangChain, CrewAI, OpenAI/Anthropic SDKs), and independent hands-on confirmation that these specific adapters work as advertised.

   • [claimed-docs] “A provider is the adapter that turns Composio tools into the native tool format your framework expects, so you don't write glue code.”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/composio/): `npm install @composio/core` and `uv pip install composio…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram documents MCP server generation, its own CLI, and MCP-client compatibility (Claude Code, Cursor, Codex), but there is no evidence of documented adapters or integration guides for Vercel AI SDK, LangChain, CrewAI, or OpenAI/Anthropic SDKs specifically.

   ⚑ flag

Security

1. platform-engineerRely on the platform scanning or inspecting listed servers for malicious or vulnerable behavior before my agents use them

   weight 2 · round to Gram (Speakeasy)
   Composio—none0/10

   No evidence that Composio scans, audits, or vets listed toolkits/servers for malicious or vulnerable behavior before agents use them; docs cover sessions, auth, CLI, and MCP connectivity but nothing about security scanning of tool listings. The one security-related item (composio-comm-2) concerns a leaked API key exposing tokens, not proactive vetting of toolkit safety, so it does not support this axis.

   ⚑ flag

   Gram (Speakeasy)~partialclaimed5/10

   Gram provides governance controls relevant to tool curation — RBAC/OAuth2.1 access control, full audit/tool-call logging across hosted, tunneled and shadow servers, 'risk policies' management via the platform MCP server, and blocking of unsanctioned shadow MCP servers by default — plus an in-dashboard harness to test a server's tool calls before connecting a real client. However none of this is documented as active scanning for malicious code or known vulnerabilities in listed servers; it's closer to access governance, logging and manual testing than automated security scanning. Missing for 10: explicit vulnerability/malware scanning of server code or behavior, third-party security audit integration, and independent evidence that risk policies actually catch malicious servers.

   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”
   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”
   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”

   proof ↗⚑ flag

Signals

1. developerCompare servers using published quality signals — usage counts, health or liveness checks, scores, or verification badges

   weight 2 · round drawn
   Composio—none0/10

   No evidence that Composio publishes usage counts, health/liveness checks, quality scores, or verification badges for toolkits/servers to help developers compare them; documentation covers sessions, meta-tools, auth, and CLI but not curation/quality signals.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   Gram provides usage/cost tracking (gram-gh-1) and tool-call logs (gram-docs-5), plus governance controls like blocking shadow servers (gram-docs-10), but there is no evidence of published quality signals such as health/liveness checks, verification badges, or comparative quality scores that a developer could use to compare servers side-by-side. Missing for 10: liveness/health-check indicators, explicit quality/verification badges, and any server-comparison or ranking UI.

   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”
   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”

   proof ↗⚑ flag

Tool search

1. ai agentSearch and load tools at runtime through meta-tools instead of having thousands of tool definitions stuffed into my context

   weight 2 · round to Composio
   Composio✓fullprobed9/10

   Composio's core architecture is exactly meta-tools: a fixed small set (7 meta-tools) that discover, authorize, and execute tools on demand rather than dumping thousands of tool definitions into context, documented in detail with sessions, toolkit-scoping, and MCP delivery, and corroborated by a live probe confirming the hosted meta-tool/MCP endpoint is real. Missing for 10: independent third-party benchmarking of context-size savings or hands-on agent trace showing runtime tool search in action.

   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [claimed-docs] “To restrict the set, pass `toolkits` when you create the session.”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's evidence covers MCP server hosting, toolset governance, OAuth, RBAC, and 'skills' loaded on demand, but none of it describes a runtime meta-tool (e.g., search_tools/load_tool) mechanism that lets an agent dynamically discover and load individual tool definitions instead of receiving a large static toolset. The closest analog (SKILL.md manifests loaded on demand) is a different mechanism from runtime tool search across a large tool catalog.

   ⚑ flag