Smithery vs Gram (Speakeasy)

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Gram (Speakeasy)
   Smithery~partialprobed6/10

   llms.txt and docs.md are confirmed 404, so the specific llms.txt convention isn't supported, but Smithery does expose agent-oriented docs alternatives: docs served live over MCP (JSON-RPC initialize succeeds at /docs/mcp) and a machine-readable OpenAPI 3.1 spec at /docs/openapi.json that an agent could consume to understand the API. missing for 10: an actual llms.txt file, and any docs indicating agents are expected to consume these alternate machine-readable formats as a first-class 'llms.txt-style' workflow.

   • [probe] “PROBE llms.txt: HTTP 404 at https://smithery.ai/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://smithery.ai/docs.md”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed8/10

   Speakeasy directly hosts a working llms.txt (HTTP 200, confirmed by probe) plus a docs.md markdown-native page, meaning an agent can be pointed at agent-oriented docs formats; the docs also emphasize AI-native/agent workflows throughout (SKILL.md, platform MCP server, CLI). missing for 10: no evidence of llms.txt existing for Gram product docs specifically (only the root Speakeasy domain), and no independent/hands-on confirmation an agent actually consumed and acted on the llms.txt content.

   • [probe] “PROBE llms.txt: HTTP 200 at https://www.speakeasy.com/llms.txt # Speakeasy > Speakeasy is the AI control plane for the AI-native enterprise…”
   • [probe] “PROBE docs-md: HTTP 200 at https://www.speakeasy.com/docs/ai-control-plane.md # AI Control Plane Documentation {/* * Renders through the …”
   • [claimed-docs] “A skill is a versioned `SKILL.md` manifest: reusable instructions that agents load on demand.”
   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Smithery
   Smithery✓fullprobed7/10

   Smithery ships a CLI (search/add/call tools) and a documented REST API with an OpenAPI 3.1 spec, service tokens scoped for AI agents/automation, and a live registry API queryable without interactive auth — all of which support scripted/headless use in CI pipelines. Runtime probes confirm the CLI installs and runs (`smithery --version`) and the API/registry endpoints respond programmatically. missing for 10: explicit CI/CD pipeline examples or GitHub Actions integration docs, and independent reports of real-world CI usage.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialprobed6/10

   Gram ships an official CLI (verified runnable via runtime probe) that can build/publish MCP servers and toolsets, which supports CI/headless automation, and docs confirm a working command-line reference. However, there is no explicit CI/CD pipeline documentation, no GitHub Actions example, and no headless auth/service-account guidance for unattended runs. Missing for 10: documented CI examples (e.g., GitHub Actions/pipeline configs), non-interactive/service-account authentication flow for the CLI, and independent hands-on confirmation of full automation beyond version-check.

   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to Smithery
   Smithery✓fullprobed8/10

   Smithery's core purpose is connecting/aggregating MCP servers: docs and probes confirm a REST/registry interface, hosted MCP endpoints with OAuth, deep links, uplink for local servers, and CLI/SDK tooling for adding and calling tools across connections, with runtime probes corroborating a live registry and per-server endpoints. Community reports of tool-description injection risks are a security caveat, not evidence the connection/tool-use mechanism fails. missing for 10: independent hands-on confirmation that an AI agent (not just probes) successfully invokes tools end-to-end across multiple aggregated servers, and resolution of the tool-description-injection security concerns.

   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “Deep links provide a seamless way to integrate Smithery MCPs into supported clients. When a user clicks a deep link from our server page, th…”
   • [claimed-docs] “Smithery handles OAuth, token refresh, and connection management automatically.”
   • [claimed-docs] “Connect to multiple servers and aggregate their tools”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullclaimed7/10

   Gram documents fronting/governing third-party MCP servers ('put a governed endpoint in front of an MCP server someone else runs') and an in-dashboard harness to connect, authenticate, and chat with a model that calls tools from a connected server, plus a real-world case (MoonPay, 200+ MCP servers aggregated under the gateway). This directly satisfies plugging external MCP servers in so their tools become usable. Missing for 10: independent/hands-on verification of third-party server integration in production and more detail on tool-call fidelity across diverse upstream servers.

   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”
   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”
   • [claimed-docs] “MoonPay brought 200+ MCP servers under the MCP Gateway, going from proof of concept to a company-wide rollout in 30 days.”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round drawn
   Smithery✓fullprobed9/10

   Smithery is a hosting/registry platform (not itself an agent), so the exception applies and publishing an official MCP server endpoint is a fair, evidenced axis: docs describe OAuth-handled connections, deep linking, uplink, and typed SDKs, and independent probes confirm a live hosted MCP endpoint (server.smithery.ai/exa/mcp) requiring OAuth and even the docs site itself served over MCP (smithery.ai/docs/mcp) with a working CLI install. Missing for 10: no independent third-party report validating that connecting via the hosted server works end-to-end for an actual agent (only probe-level confirmation, not user testimonial), and community findings raise tool-description-injection risk that slightly tempers confidence though it doesn't contradict the connection mechanism itself.

   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “Deep links provide a seamless way to integrate Smithery MCPs into supported clients. When a user clicks a deep link from our server page, th…”
   • [claimed-docs] “Smithery handles OAuth, token refresh, and connection management automatically.”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed9/10

   Gram documents and exposes an official 'Platform MCP' server that lets MCP clients (Claude Code, Cursor, Codex, etc.) connect directly, plus generates/hosts MCP servers as its core product function, and a CLI/runtime probe confirms the tooling actually installs and runs. missing for 10: independent (non-vendor) hands-on confirmation that an external agent successfully connects to and operates the Platform MCP server end-to-end.

   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [probe] “official MCP server documented at https://www.speakeasy.com/docs/ai-control-plane/reference/platform-mcp”
   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to Gram (Speakeasy)
   Smithery~partialprobed6/10

   There is concrete evidence of a working Smithery CLI (npm install @smithery/cli verified, version 4.11.1) with documented commands for searching the registry, adding MCP connections, finding/calling tools, and installing skills. However, the GitHub source is attributed to 'arcadeai-labs/smithery-cli' rather than an obviously first-party Smithery org, leaving some ambiguity about official first-party ownership, and there is no independent/community corroboration of CLI usage. Missing for 10: clearer first-party branding/ownership confirmation, official docs page dedicated to the CLI, and independent hands-on user reports.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [github] “Browse skills on the [Smithery Skills Registry](https://smithery.ai/skills) and install them with the upstream installer”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed8/10

   Gram ships an official CLI documented at reference/command-line, used to install toolsets as MCP servers and keep itself updated, and a runtime probe confirms the binary actually downloads and runs (`gram --version`). missing for 10: independent/third-party review of CLI usage beyond the vendor docs and a single runtime probe, and more detail on the full breadth of CLI commands beyond install/update.

   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to Smithery
   Smithery✓fullprobed8/10

   Smithery exposes a real, documented public API: a registry API (registry.smithery.ai) queryable without keys, a full OpenAPI 3.1.0 spec (35 paths) confirmed live at smithery.ai/docs/openapi.json, and a REST/connection interface with OAuth and token scoping for programmatic/agent use, plus a CLI (smithery mcp/tool commands) for driving it. Initial llms.txt/docs.md probes 404'd but the actual OpenAPI spec and registry endpoints were independently verified live, satisfying 'documented public API' for an AI-native/agentic consumer. missing for 10: no independent third-party corroboration of API stability/versioning, and some docs endpoints (llms.txt) are absent despite the OpenAPI spec existing.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialprobed7/10

   Gram exposes a documented Platform MCP server and CLI that let an AI-native user/agent drive the product programmatically (find/add MCP servers, author skills, manage policies, read logs) without the dashboard, and the CLI was independently verified to install and run. However, a direct probe for a conventional public REST/OpenAPI API (openapi.json, swagger.json, etc.) returned 404s, so there is no evidence of a general-purpose documented HTTP API beyond the MCP/CLI surfaces. Missing for 10: a documented REST/OpenAPI public API endpoint, independent confirmation of the Platform MCP server's functional breadth beyond docs.

   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [probe] “official MCP server documented at https://www.speakeasy.com/docs/ai-control-plane/reference/platform-mcp”
   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.speakeasy.com/openapi.json, https://www.speakeasy.com/swagger.json, https://www.speakeas…”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to Smithery
   Smithery✓fullclaimed7/10

   Smithery docs describe service tokens explicitly scoped to a specific app/connection/user metadata ('list and call tools on connections in my-app where metadata.userId is user-123 — nothing else'), plus encrypted write-only credential storage and namespace-scoped API keys, directly matching the least-privilege credential story for agents. Missing for 10: independent/hands-on verification that scope enforcement actually holds, and documentation of token revocation/expiry/audit controls.

   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”
   • [claimed-docs] “This token can list and call tools on connections in `my-app` where `metadata.userId` is `user-123` — nothing else.”
   • [claimed-docs] “Secure credential storage — Encrypted, write-only storage for API keys and tokens”
   • [claimed-docs] “Namespaces act as a shared workspace. Resources within a namespace can be managed by anyone with the appropriate API key scoped to that name…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed6/10

   Gram supports OAuth 2.0/2.1 integration with external providers, RBAC enforced at team/server/tool level synced to enterprise IDPs, and governs upstream credentials for third-party MCP servers with shadow-server blocking — collectively enabling scoped access control for agent tool use. However, there's no explicit documentation of per-agent least-privilege token minting/vending or fine-grained scope selection at credential-issuance time; the emphasis is on gateway-level RBAC and OAuth pass-through rather than dynamically scoped credential issuance. Missing for 10: explicit per-agent credential scoping/token minting mechanism, documented least-privilege scope configuration UI, and independent verification of enforcement.

   • [claimed-docs] “you can use any OAuth 2.0 provider you have set up (such as Auth0, Okta, Keycloak, or your own OAuth server)”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Smithery
   Smithery✓fullprobed7/10

   Smithery auto-generates a typed TypeScript SDK for every published MCP server (smithery-docs-16), ships an official CLI (smithery-cli on GitHub) verified via live npm install (smithery-probe-rt-4), and exposes a documented OpenAPI 3.1 spec for platform API client generation, confirmed live via probe (smithery-probe-rt-3). This gives AI-native builders concrete, verified official SDK/CLI/API surfaces to build against. Missing for 10: SDKs beyond TypeScript (no Python/other language SDK evidence), and independent third-party corroboration of real-world SDK usage beyond Smithery's own docs/probes.

   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence pack documents Gram's CLI, MCP server distribution, OAuth, skills, and observability features, but nowhere mentions official client SDKs (e.g., Python/TypeScript/Go libraries) for building against Gram's platform itself — this is a fair question for a platform product but is simply unaddressed.

   ⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round to Smithery
   Smithery~partialclaimed6/10

   Smithery's Triggers feature explicitly lets consumers subscribe to upstream events, proxying subscribe calls through to the server—directly matching webhook/event-subscription functionality for agentic use. However, this is documented only at a conceptual level with no depth on webhook delivery mechanics, retry/reliability guarantees, or independent hands-on verification. missing for 10: detailed webhook payload/delivery docs, independent corroboration of triggers working in practice, and coverage of failure/retry semantics.

   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's evidence covers MCP server generation, tool logs, CLI, and RBAC, but nothing describes webhook subscriptions or event-driven push notifications for AI agents; missing for 10: webhook subscription mechanism, event schema/documentation, or any mention of push-based event delivery.

   ⚑ flag

Agentic features

1. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Smithery
   Smithery~partialclaimed3/10

   Smithery's Triggers feature lets an MCP server surface upstream events which Smithery proxies to a consumer, hinting at event-driven automation, but there is no evidence of a scheduler, persistent background execution engine, or the platform itself running autonomous tasks — automation actually running requires an external agent that consumes the trigger. missing for 10: evidence of a scheduling/cron mechanism, autonomous execution loop hosted by Smithery, and confirmation that triggers fire without an active client agent.

   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

2. ai-native userOperate the product with natural-language commands

   weight 2 · round to Gram (Speakeasy)
   Smithery~partialprobed5/10

   Smithery's CLI exposes intent-based discovery ('smithery tool find [query] # Search tools by name or intent') and its docs are themselves served as an MCP endpoint that an agent can query conversationally, showing some natural-language-friendly interaction. However, the primary interface remains structured CLI subcommands (search, add, call) and REST/OpenAPI calls rather than a general natural-language command layer for operating the whole product. Missing for 10: a documented conversational/chat interface or explicit NL parsing across all product operations, and independent evidence of users successfully issuing free-form natural-language commands rather than structured queries.

   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed7/10

   Gram documents a platform MCP server that lets an org admin operate the product (find/add servers, manage risk policies, author skills, read observability) entirely from within an MCP client like Claude Code or Cursor using natural language, without opening the dashboard — this is direct agentic/NL control of the product itself. Missing for 10: independent/hands-on verification that these natural-language admin operations actually execute correctly (only the CLI binary itself was runtime-probed, not the platform-mcp NL workflow).

   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [probe] “official MCP server documented at https://www.speakeasy.com/docs/ai-control-plane/reference/platform-mcp”
   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round to Smithery
   Smithery~partialprobed5/10

   Smithery exposes a full OpenAPI 3.1 spec (35 paths) and even serves its documentation itself as an MCP endpoint that agents can query interactively via JSON-RPC, which supports AI-native exploration of the API surface. However there's no evidence of a classic interactive reference UI (e.g., Swagger/Redoc 'try it out' runnable examples in-browser), and alternate machine-readable doc formats (llms.txt, docs.md) return 404s. missing for 10: an actual interactive/try-it-out API console with executable example requests, evidence of runnable code snippets in docs, independent confirmation of usability of the MCP-served docs.

   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://smithery.ai/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://smithery.ai/docs.md”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram is positioned as an AI control plane/MCP gateway focused on server generation, governance, and observability; there is no evidence of an interactive API reference UI with runnable examples for exploring endpoints, nor documentation of such a feature.

   ⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round to Smithery
   Smithery✓fullprobed8/10

   A direct hands-on probe confirms a live, machine-readable OpenAPI 3.1.0 specification with 35 documented paths at smithery.ai/docs/openapi.json, which an agent could use to generate a client — squarely satisfying the story. Earlier probe attempts at guessed paths (openapi.json at root, swagger.json, etc.) 404'd, but the actual documented location succeeds. Missing for 10: first-party documentation explicitly announcing/linking the OpenAPI spec, and independent community confirmation of its use.

   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://smithery.ai/openapi.json, https://smithery.ai/swagger.json, https://smithery.ai/api/openapi.…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence shows Gram consumes OpenAPI documents as input to build MCP servers, but there is no evidence Gram itself publishes a downloadable machine-readable spec of its own API — the direct probe for openapi.json/swagger.json/well-known paths all returned 404.

   • [probe] “PROBE openapi: all candidate paths 404 (https://www.speakeasy.com/openapi.json, https://www.speakeasy.com/swagger.json, https://www.speakeas…”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Gram (Speakeasy)
   Smithery—none0/10

   Evidence shows Smithery's Uplink lets developers test an in-development MCP server against a real agent before publishing, but there is no mention of a distinct sandbox environment, test data isolation, or any mechanism preventing production data exposure during testing.

   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “You’re developing an MCP server and want to test it against a real agent before publishing.”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed4/10

   Gram provides an in-dashboard 'Connect' harness to test MCP servers with a model before wiring up a real client, which offers some sandbox-like isolation from production agent traffic, but there is no explicit mention of separate sandbox vs. production data/environments, test fixtures, or guarantees that tool calls in this harness avoid hitting live production data sources. missing for 10: explicit sandbox/staging environment concept, isolation guarantees from production data, mock or test data support during agent testing.

   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Smithery—none0/10

   Evidence shows Smithery has an OpenAPI 3.1.0 spec and a documented REST/MCP interface, but nothing in the docs or probes describes API versioning conventions or a deprecation policy for breaking changes.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence pack covers OpenAPI-generated MCP servers, OAuth, governance, CLI, observability, and skills, but contains no documentation of API versioning practices or a deprecation policy for Gram's own APIs/servers.

   ⚑ flag

Multi tenant

1. platform-engineerMaintain separate connected accounts and scoped tokens per end user of my product, so each user's agent acts only on that user's credentials

   weight 3 · round to Smithery
   Smithery~partialcommunity6/10

   Smithery's token-scoping docs show scoped tokens tied to per-user metadata (e.g. a token limited to connections where metadata.userId is 'user-123'), and secure credential storage/automatic refresh are documented, which directly supports per-end-user credential isolation for a platform-engineer building on top of Smithery. However, there's no first-party documentation of a full per-user account management workflow (provisioning, revocation, auditing across many end users) beyond the single scoping example, and independent security research flagged real vulnerabilities (tool description injection, path traversal) that raise doubts about credential-boundary robustness in practice. missing for 10: dedicated multi-tenant/per-end-user account management docs beyond one scoping example, evidence of token revocation/rotation workflows at scale, and independent verification that scoped tokens hold up against the reported security findings.

   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”
   • [claimed-docs] “This token can list and call tools on connections in `my-app` where `metadata.userId` is `user-123` — nothing else.”
   • [claimed-docs] “Automatic token refresh — Credentials stay valid without manual intervention”
   • [claimed-docs] “Secure credential storage — Encrypted, write-only storage for API keys and tokens”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”
   • [community] “Security researchers detailed a path traversal vulnerability in Smithery's MCP server hosting that could be escalated into a supply chain at…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's evidence describes OAuth 2.0 provider integration and team/server/tool-level RBAC synced to an enterprise IDP (gram-docs-2, gram-gh-2), but nothing documents per-end-user connected accounts or scoped tokens so that each of a platform-engineer's own product's end users gets isolated credentials — the RBAC described is org/team-level, not end-user-level multi-tenant credential isolation. Missing for 10: explicit documentation of per-end-user connected-account/token scoping, any multi-tenant credential isolation guide, and evidence of enforcement that one end user's agent cannot use another's credentials.

   • [claimed-docs] “you can use any OAuth 2.0 provider you have set up (such as Auth0, Okta, Keycloak, or your own OAuth server)”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”

   proof ↗⚑ flag

Oauth

1. developerLet the platform run OAuth flows for downstream tools and vault the resulting credentials encrypted, with automatic token refresh

   weight 3 · round to Smithery
   Smithery✓fullprobed7/10

   Docs explicitly describe Smithery handling OAuth flows, encrypted write-only credential storage, and automatic token refresh (smithery-docs-1,2,3,13), and this is corroborated by a live probe showing an OAuth challenge gating a hosted MCP endpoint (smithery-probe-rt-2). missing for 10: no independent hands-on confirmation of encryption-at-rest implementation details or token refresh behavior over time, and community evidence focuses on unrelated security findings rather than corroborating credential vaulting.

   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Automatic token refresh — Credentials stay valid without manual intervention”
   • [claimed-docs] “Secure credential storage — Encrypted, write-only storage for API keys and tokens”
   • [claimed-docs] “Smithery handles OAuth, token refresh, and connection management automatically.”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed5/10

   Docs confirm Gram lets you configure external OAuth 2.0 providers (Auth0, Okta, Keycloak, custom) for downstream tool auth and enforces OAuth2.1/RBAC for platform access, implying it manages OAuth flows for connected servers, but there is no explicit documentation of encrypted credential vaulting or automatic token refresh mechanics. Missing for 10: explicit description of credential encryption at rest, automatic refresh-token handling, and any hands-on/independent verification of these security claims.

   • [claimed-docs] “you can use any OAuth 2.0 provider you have set up (such as Auth0, Okta, Keycloak, or your own OAuth server)”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”

   proof ↗⚑ flag

White label

1. platform-engineerWhite-label the authorization flow with my own branding and my own OAuth app credentials

   weight 1 · round to Gram (Speakeasy)
   Smithery—none0/10

   The evidence pack describes Smithery's own OAuth/credential-management infrastructure (managed auth, token scoping, secure storage) but contains no mention of white-labeling the auth flow with custom branding or bringing your own OAuth app/client credentials to replace Smithery's identity — no BYO-OAuth-app, custom domain, or branding customization feature is documented anywhere.

   ⚑ flag

   Gram (Speakeasy)~partialclaimed6/10

   Gram supports bringing your own OAuth 2.0 provider (Auth0, Okta, Keycloak, or a custom server) so the underlying credentials are fully controlled by the platform-engineer, and RBAC/OAuth2.1 syncs to enterprise IDPs. However, there is no documented evidence of UI-level 'white-labeling' — custom branding/logo/domain on the consent or login screen itself — only backend provider configuration. Missing for 10: evidence of custom branding/theming of the OAuth consent screen or hosted login page, and confirmation that the authorization UI itself (not just the provider) can be rebranded.

   • [claimed-docs] “you can use any OAuth 2.0 provider you have set up (such as Auth0, Okta, Keycloak, or your own OAuth server)”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round drawn
   Smithery—none0/10

   Evidence shows Smithery lets an agent connect to and aggregate tools from multiple servers, and its registry API can be queried/paginated for many entries, but there is no documented batch/bulk operation capability (e.g., bulk tool calls, mass server management, batch CRUD) that would let a user perform bulk actions across many items at once. missing for 10: batch tool-call API, bulk server management commands, mass connection/config updates, any documented multi-item transaction support.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence pack covers MCP server hosting, governance, observability, RBAC, and CLI/skills features, but nowhere describes any bulk or batch operation capability (e.g., bulk approving tools, bulk managing servers, or batch tool invocation across many items). This is a plausible axis for a control-plane product managing many servers/toolsets, but no evidence shows it delivers.

   ⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Smithery
   Smithery~partialclaimed3/10

   Smithery's 'Triggers' feature lets an MCP server surface upstream events and lets a consumer 'activate' a trigger, which is a thin form of event-driven automation, but there's no evidence of a user-facing rule-definition mechanism (e.g., 'if event X then run action Y') or a rules engine — it only proxies subscribe calls. Missing for 10: a documented rule/condition-action authoring interface, examples of automated action execution on trigger events, and independent confirmation the feature works end-to-end.

   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's evidence covers governance, observability, tool logging, RBAC, and MCP server management, but there is no mention of user-definable rule/trigger systems that automatically fire actions in response to events (e.g., event-driven automation, if-this-then-that rules, webhooks-to-action). Risk policies and access controls are governance constructs, not event-triggered automation rules.

   ⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round drawn
   Smithery—none0/10

   Smithery's docs describe on-demand tool connections, OAuth/credential management, and event-driven 'Triggers' that proxy subscribe calls from upstream services, but nothing describes cron-like recurring job scheduling or workflow automation on a timer. Triggers are reactive (event-based), not scheduled/recurring, so the specific capability asked for is unevidenced.

   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”
   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram is an MCP control plane focused on generating, governing, observing, and distributing MCP servers/toolsets/skills, but nothing in the evidence describes scheduling recurring jobs or workflows (e.g., cron-like triggers or automated recurring execution) — it only covers request-time tool calls, logging, and access control.

   ⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round to Gram (Speakeasy)
   Smithery—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Gram (Speakeasy)~partialclaimed4/10

   Gram documents that skills are versioned (`SKILL.md` manifest) and provides audit/observability logs of tool calls, giving some traceability, but there is no evidence of a review workflow or an explicit rollback mechanism for automations/toolsets. missing for 10: explicit review process, rollback/revert functionality, version history UI for toolsets or MCP server configs.

   • [claimed-docs] “A skill is a versioned `SKILL.md` manifest: reusable instructions that agents load on demand.”
   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”

   proof ↗⚑ flag

Bridge

1. developerBridge a locally running MCP server into the platform as a managed remote connection without deploying it

   weight 1 · round to Smithery
   Smithery✓fullclaimed8/10

   Smithery's Uplink feature explicitly exposes an MCP server running on any local machine as a regular Smithery connection, letting developers test/use a locally running server against real agents without deploying it, and this is backed by scoped tokens and managed OAuth/credential handling. missing for 10: independent/hands-on confirmation of Uplink working end-to-end (only first-party docs cited) and no detail on latency/reliability of the bridged local connection.

   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “You’re developing an MCP server and want to test it against a real agent before publishing.”
   • [claimed-docs] “You want to run a private tool for yourself or your team without hosting it.”
   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed5/10

   Docs mention 'tunneled MCP servers' alongside hosted and shadow servers in the observability/tool-logs reference, implying a mechanism to bridge a locally running server into the platform without redeploying it, and the third-party governance guide covers fronting externally-run servers. However, no dedicated guide, CLI command, or hands-on walkthrough for setting up a local-to-remote tunnel is included in the evidence pack. Missing for 10: a dedicated tunnel setup guide/CLI reference, concrete steps for exposing a local dev MCP server, and independent confirmation the tunnel actually works.

   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”
   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”

   proof ↗⚑ flag

Access control

1. platform-engineerRestrict which servers and individual tools an agent may call through allowlists or per-tool access control

   weight 2 · round to Gram (Speakeasy)
   Smithery~partialclaimed6/10

   Smithery's token-scoping feature lets platform engineers issue tokens restricted to specific connections/namespaces (e.g., 'list and call tools on connections in my-app where metadata.userId is user-123 — nothing else'), which functions as a coarse allowlist mechanism, and namespaces further scope API-key access to a workspace. However, the evidence shows scoping is at the connection/namespace level, not explicit per-tool allow/deny lists within a single server. Missing for 10: documented fine-grained per-tool allow/deny configuration, independent/hands-on verification that tool-level restrictions are enforced, and any admin UI or policy examples showing individual tool blocking within an allowed connection.

   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”
   • [claimed-docs] “This token can list and call tools on connections in `my-app` where `metadata.userId` is `user-123` — nothing else.”
   • [claimed-docs] “Namespaces act as a shared workspace. Resources within a namespace can be managed by anyone with the appropriate API key scoped to that name…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullclaimed8/10

   Gram documents team/server/tool-level RBAC enforced with OAuth2.1 synced to enterprise IDPs, and explicitly blocks unsanctioned shadow MCP servers by default, directly matching allowlist/per-tool access control for a platform-engineer persona. This is corroborated by third-party rollout evidence (MoonPay) and governance guides for wrapping third-party MCP servers with access control. Missing for 10: independent hands-on verification of the RBAC/permission enforcement mechanics beyond vendor docs and GitHub README claims.

   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”
   • [claimed-docs] “MoonPay brought 200+ MCP servers under the MCP Gateway, going from proof of concept to a company-wide rollout in 30 days.”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”

   proof ↗⚑ flag

Audit

1. platform-engineerSee a log of every tool call that passed through the platform, with enough detail to audit or replay what an agent did

   weight 2 · round to Gram (Speakeasy)
   Smithery—none0/10

   No evidence describes any audit logging, tool-call history, or replay capability for platform engineers—docs cover connection, credential management, publishing, tokens, and namespaces but never mention logs of tool calls passing through the gateway.

   ⚑ flag

   Gram (Speakeasy)✓fullclaimed8/10

   Gram documents a raw execution log capturing every tool call across hosted, tunneled, shadow MCP servers, skills, and local tools ([gram-docs-5]), plus usage/cost tracking and session deep-dives ([gram-gh-1]) that support audit and replay-style analysis. RBAC/OAuth-scoped access adds context for auditability of who called what ([gram-gh-2]). Missing for 10: independent/hands-on verification of log completeness or replay functionality specifically, and detail on log retention/export format.

   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”
   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”

   proof ↗⚑ flag

Catalog

1. developerReach a deep catalog of third-party apps and APIs (hundreds to thousands of integrations) through the platform

   weight 3 · round to Smithery
   Smithery✓fullprobed8/10

   Smithery is a registry/marketplace connecting to a large catalog of third-party MCP servers (Exa, Context7, Browserbase, Google Sheets, Brave Search, etc.), with a live registry API confirmed via probe returning searchable server entries with useCount/verified metadata, plus hosted per-server MCP endpoints (server.smithery.ai) and CLI search/add tooling. Community evidence confirms real scale (100+ servers scanned) though also raises security concerns about tool description injection, which affects trust/quality but not the breadth-of-catalog claim itself. Missing for 10: independently verified total count of hundreds/thousands of listings and first-party data on catalog size/growth.

   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Exa - Search the live web, access LinkedIn profiles, do deep research, and more”
   • [claimed-docs] “Context7 - Reference the latest docs for most major SDKs and frameworks directly in Cursor or Claude Code”
   • [claimed-docs] “Browserbase - Control a remote web browser using Stagehand”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's documented model is generating MCP servers from your own OpenAPI/TypeScript source or governing a third-party MCP server you already run — there is no evidence of a pre-built catalog of hundreds/thousands of ready-made third-party app/API integrations comparable to a connector marketplace. The MoonPay case study mentions bringing 200+ existing internally-run MCP servers under governance, not a Gram-provided catalog to browse and connect.

   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”
   • [claimed-docs] “MoonPay brought 200+ MCP servers under the MCP Gateway, going from proof of concept to a company-wide rollout in 30 days.”

   proof ↗⚑ flag

Connect

1. developerPoint my MCP-compatible client at a hosted remote server URL and reach downstream tools without deploying any infrastructure

   weight 3 · round drawn
   Smithery✓fullprobed8/10

   Docs and live probes confirm Smithery hosts remote MCP server endpoints (server.smithery.ai/<server>/mcp) that clients can point at directly, with OAuth/token handling, deep links, and typed SDKs generated automatically — no server deployment needed by the developer. Independent probe evidence (smithery-probe-rt-2) shows the hosted endpoint is live and gated by managed OAuth rather than requiring self-hosting infrastructure. missing for 10: independent hands-on developer accounts (outside security research) confirming a smooth end-to-end connect experience beyond documentation and infra probes.

   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”
   • [claimed-docs] “Smithery handles OAuth, token refresh, and connection management automatically.”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed8/10

   Gram's Control Plane auto-generates hosted MCP servers from an OpenAPI doc or TypeScript project, so developers get a remote server URL to point MCP clients at without deploying infra themselves, and this is corroborated by the CLI runtime probe and a real customer rollout (MoonPay's 200+ servers under the MCP Gateway). missing for 10: independent hands-on confirmation of an end-to-end client connecting to a hosted URL (only the CLI tool itself was runtime-verified, not the hosted server reachability).

   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [claimed-docs] “MoonPay brought 200+ MCP servers under the MCP Gateway, going from proof of concept to a company-wide rollout in 30 days.”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”

   proof ↗⚑ flag

Generate

1. developerTurn an existing API definition (OpenAPI) or custom code into a hosted MCP server without writing protocol plumbing

   weight 2 · round to Gram (Speakeasy)
   Smithery~partialclaimed4/10

   Smithery's docs show a hosting/publishing path for MCP servers already built in code (URL method, MCPB bundles, server-card metadata) which removes protocol plumbing for deployment, but there is no evidence of an OpenAPI-spec ingestion or auto-generation feature that turns an existing REST API definition into an MCP server — the publish flow assumes you already have MCP-shaped code or a deployed server. Missing for 10: OpenAPI-to-MCP conversion tooling, any first-party or independent proof of generating a server from an OpenAPI spec, and confirmation that 'custom code' can be arbitrary non-MCP code rather than code already written against the MCP SDK.

   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”
   • [claimed-docs] “For local stdio servers — Smithery distributes a pre-built MCPB bundle that clients download and run locally.”
   • [claimed-docs] “If automatic scanning can’t complete (auth wall, required configuration, or other issues), you can provide server metadata manually via a st…”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed8/10

   Gram's Control Plane generates a hosted MCP server directly from an OpenAPI document or a TypeScript Functions project, with no protocol plumbing required from the developer, and a CLI/runtime probe confirms the tooling actually installs and runs. missing for 10: independent (non-vendor) hands-on account of the OpenAPI-to-MCP generation flow itself, and more detail on custom-code/TypeScript function support beyond the docs mention.

   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”

   proof ↗⚑ flag

Publish

1. developerPublish my own MCP server on the platform for distribution, discovery, and installation by others

   weight 2 · round drawn
   Smithery✓fullprobed8/10

   Docs clearly describe multiple publishing paths (URL method for already-deployed servers, MCPB bundle distribution for local stdio servers, automatic scanning or manual server-card metadata) resulting in a registry listing with generated TypeScript SDK, and the registry API probe confirms servers are discoverable/searchable by others without auth. Security researchers did find vulnerabilities/prompt-injection risks in some published servers, but this reflects on registry quality, not on whether publishing/discovery/installation works. missing for 10: independent hands-on account of a developer successfully publishing a server end-to-end, and clearer detail on install flow verification post-publish.

   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”
   • [claimed-docs] “For local stdio servers — Smithery distributes a pre-built MCPB bundle that clients download and run locally.”
   • [claimed-docs] “If automatic scanning can’t complete (auth wall, required configuration, or other issues), you can provide server metadata manually via a st…”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullprobed8/10

   Gram lets developers generate/host MCP servers from OpenAPI or TypeScript sources on its Control Plane, with governance, RBAC, tool logs, and a CLI/dashboard to publish, discover, and install servers into MCP clients, and the CLI was independently verified to install and run. missing for 10: no independent (non-vendor) evidence of a public discovery/marketplace listing showing other developers actually finding and installing a third party's published server, and no detail on versioning/publishing workflow specifics beyond docs.

   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [claimed-docs] “MoonPay brought 200+ MCP servers under the MCP Gateway, going from proof of concept to a company-wide rollout in 30 days.”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [probe] “official MCP server documented at https://www.speakeasy.com/docs/ai-control-plane/reference/platform-mcp”

   proof ↗⚑ flag

Triggers

1. developerSubscribe my agent to events from connected downstream apps (triggers/webhooks) instead of polling them

   weight 2 · round to Smithery
   Smithery✓fullclaimed7/10

   Smithery documents a Triggers feature that explicitly lets consumers subscribe to event streams surfaced from an upstream service, with Smithery proxying the subscribe call to the server instead of requiring polling. This directly matches the story of subscribing an agent to downstream app events via webhooks/triggers rather than polling. Missing for 10: independent/hands-on confirmation that triggers work reliably in production, and more detail on delivery guarantees or webhook payload handling.

   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's documented capabilities center on generating/govern/observe MCP servers from OpenAPI specs and exposing tools for agents to call (pull-based); there is no mention anywhere in the evidence of webhook subscriptions, event triggers, or push notifications from downstream apps to agents. This is a plausible axis for an MCP gateway/control-plane product, but no evidence shows it delivers it.

   ⚑ flag

Analytics

1. platform-engineerSee usage analytics for the servers and tools my org consumes or publishes — call volumes, users, errors, and cost

   weight 2 · round to Gram (Speakeasy)
   Smithery—none0/10

   The evidence pack documents connection, publishing, token scoping, triggers, and SDK generation, but nothing about a dashboard or API for org-level usage analytics (call volumes, per-user breakdowns, error rates, cost). The only quantitative signal found is a per-server 'useCount' field in the public registry listing, which is not org-scoped consumption/error/cost analytics.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”

   proof ↗⚑ flag

   Gram (Speakeasy)✓fullclaimed7/10

   Gram documents tool-call logs across hosted/tunneled/shadow servers and local tools, and the GitHub repo description explicitly claims usage tracking across teams by tokens/cost, budgets, and tool effectiveness measurement, directly matching call volumes/cost/usage. Errors are implied via raw execution logs but not explicitly broken out as an error-rate metric, and there's no independent/hands-on corroboration of the analytics dashboard itself. Missing for 10: explicit error-rate reporting UI evidence, independent third-party validation of the analytics dashboard beyond vendor docs/GitHub description.

   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”
   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Smithery
   Smithery~partialprobed7/10

   Smithery exposes a documented, live OpenAPI 3.1 spec (35 paths) and a keyless registry API, plus a CLI (`smithery mcp add`, `smithery tool call`) that mirrors core UI actions like discovering, connecting, and invoking MCP servers — showing broad API/CLI parity with the UI for the main workflows (search, connect, publish via URL, call tools). However, no evidence explicitly confirms that secondary UI-only features (team/namespace management, trigger configuration, token scoping setup) are fully API-reachable, and llms.txt/docs.md convenience endpoints 404. missing for 10: explicit confirmation that admin/team features (namespaces, triggers, token scoping) are configurable via the API/CLI rather than only the UI, and independent hands-on verification beyond the vendor's own OpenAPI spec.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”
   • [claimed-docs] “Namespaces act as a shared workspace. Resources within a namespace can be managed by anyone with the appropriate API key scoped to that name…”
   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialprobed6/10

   Gram exposes a Platform MCP server and CLI that let an org admin add/configure MCP servers, author skills, manage risk policies, and read observability data without opening the dashboard (gram-docs-4, gram-docs-7, gram-probe-5), and the CLI is confirmed to actually install and run (gram-probe-rt-1). However, no public REST/OpenAPI surface was found (gram-probe-3: openapi.json paths all 404), so full UI-parity via a general API is unconfirmed. Missing for 10: a documented general-purpose REST/OpenAPI API covering all dashboard actions, and independent confirmation that every UI feature (not just admin/config tasks) is reachable programmatically.

   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.speakeasy.com/openapi.json, https://www.speakeasy.com/swagger.json, https://www.speakeas…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   Smithery—none0/10

   No evidence anywhere in the pack of a data-export feature, account data portability, or open-format export for users leaving the platform; documentation covers connections, tokens, and MCP server publishing but nothing about exporting stored user data (connections, credentials metadata, namespaces) in an open format.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence describes Gram's inputs (OpenAPI specs, SKILL.md manifests) and platform features like RBAC, logging, and CLI, but there is no documented data-export feature letting a user pull out their configured toolsets, logs, policies, or org data in open formats to migrate away. Axis applies to a platform like this but no evidence supports it.

   ⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Smithery—none0/10

   Evidence shows only a GitHub repo for the Smithery CLI tool (smithery-gh-1/2/3) but no license information, and the core Smithery platform (registry, hosting, OAuth handling) is described only via docs as a hosted SaaS with no mention of source availability or open licensing. No evidence establishes that Smithery's product source is published under an open license.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [github] “Browse skills on the [Smithery Skills Registry](https://smithery.ai/skills) and install them with the upstream installer”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence shows a public GitHub repository (github.com/speakeasy-api/gram) exists, but nothing in the pack states or documents an open-source license or that the full source is freely readable/licensed for reuse; docs focus on hosted product features (CLI, MCP servers, RBAC, observability) rather than source availability or licensing terms.

   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   Smithery—none0/10

   Evidence describes Smithery only as a hosted registry/platform (registry.smithery.ai, server.smithery.ai) with OAuth-gated hosted MCP endpoints; 'Uplink' lets users expose their own local MCP server as a connection into the hosted Smithery service, but this is not self-hosting the Smithery platform itself. No docs, GitHub repo, or probes indicate a self-hostable/open-source deployment of the core product.

   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   While Gram's code is hosted on GitHub (gram-gh-1, gram-gh-2), the evidence pack contains no documentation, guide, or deployment instructions describing a supported self-hosted deployment of the core Control Plane — all docs describe the hosted SaaS platform, dashboard, and CLI/skill features rather than self-hosting.

   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”

   proof ↗⚑ flag

Pricing

1. developerPrototype my agent's tool connections on a meaningful free tier before paying anything

   weight 1 · round drawn
   Smithery—none0/10

   The evidence pack contains extensive documentation on Smithery's features, registry, and integrations but no mention of pricing tiers, a free tier, usage limits, or billing at all — the axis is applicable to a hosted MCP platform but simply unevidenced.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence pack contains no mention of pricing tiers, a free tier, or cost thresholds for Gram — nothing describes what is available without payment. This is a fair pricing-plans axis for any commercial dev tool, but there's simply no documentation or citation about free-tier limits or trial terms.

   ⚑ flag

2. developerSee transparent pricing per tool call or connection plus documented rate limits before committing to the platform

   weight 2 · round drawn
   Smithery—none0/10

   No evidence pack item mentions pricing tiers, per-call costs, or documented rate limits anywhere in Smithery's docs, GitHub, or probes; the documentation covers connection, tokens, and hosting but is silent on pricing/rate limits, and repeated probes (llms.txt, docs.md, openapi) surface no pricing/rate-limit schema.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   No evidence pack item mentions pricing, cost per tool call/connection, or documented rate limits; all citations concern architecture, RBAC, skills, and CLI features. Absence of evidence for an applicable and standard pricing-transparency axis means this must be scored none.

   ⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   Smithery—none0/10

   No evidence anywhere in the pack mentions data residency, region selection, or geographic storage controls; Smithery's docs cover credential storage, token scoping, and connection management but not where data is physically stored.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   No evidence in the pack mentions data residency, region selection, or geographic storage controls for Gram; the documentation focuses on governance, RBAC, tool logs, and MCP server hosting but never addresses where data is physically stored.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Smithery—none0/10

   No evidence in the pack addresses AI training data usage, opt-out controls, or any data-privacy policy regarding model training; docs focus on credential storage, OAuth, and connection routing, not training-data opt-out.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's evidence pack covers MCP server governance, RBAC, observability, and tool logging, but nothing addresses whether user/tool-call data is used to train AI models or an opt-out policy for such training — missing for 10: any data-training policy statement, opt-out controls, or model-training disclosure.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Smithery—none0/10

   The evidence pack covers credential storage encryption and token scoping but contains no mention of data retention policies, user-initiated data deletion, or account/data export controls. This is a fair privacy-posture question for a platform managing credentials and connections, so the absence of evidence yields 'none' rather than 'na'.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   The evidence pack covers governance, RBAC, audit logs, OAuth, and shadow-server blocking, but no citation addresses data retention policies, deletion controls, or user data lifecycle management for the AI-native platform.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Smithery—none0/10

   No evidence anywhere in the pack mentions a telemetry/usage-tracking opt-out setting, privacy policy toggle, or data-collection disclosure for Smithery; the docs cover credential security and OAuth but never address analytics/telemetry opt-out.

   ⚑ flag

   Gram (Speakeasy)—none0/10

   No evidence in the pack addresses whether Gram's CLI, dashboard, or platform collects vendor-side telemetry/usage data or offers an opt-out; the only related items describe customer-facing usage analytics (tracking AI agent activity), not the product's own telemetry practices. Missing for 10: any mention of a telemetry setting, opt-out flag, or privacy policy statement about data collection from the CLI/platform itself.

   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”

   proof ↗⚑ flag

Api

1. ai agentQuery the platform's registry or catalog through a documented API to discover servers and tools programmatically

   weight 3 · round to Smithery
   Smithery✓fullprobed9/10

   A live probe confirms an agent can query https://registry.smithery.ai/servers keylessly and get a machine-readable JSON catalog (displayName, qualifiedName, verified, etc.), and a documented OpenAPI 3.1 spec with 35 paths is served at smithery.ai/docs/openapi.json, enabling programmatic client generation. This is corroborated by the smithery-cli's 'mcp search'/'tool find' commands for registry/tool discovery. Missing for 10: no independent third-party report of an agent successfully building a full client purely from the OpenAPI spec, and llms.txt/docs.md convenience endpoints 404.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialprobed6/10

   Gram documents an official 'platform-mcp' server that lets an agent working through an MCP client find and add MCP servers, manage toolsets, and read observability data programmatically (gram-docs-7, gram-probe-4), and the CLI can install toolsets as MCP servers (gram-docs-4/5). This covers the discovery story via the MCP protocol itself acting as the documented API, but there is no confirmed general-purpose REST/OpenAPI catalog endpoint (the openapi probe returned 404s) and no independent hands-on proof of the discovery flow itself. Missing for 10: a documented REST/OpenAPI registry endpoint, and third-party/hands-on verification that an agent can actually enumerate servers/tools via the platform MCP server.

   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [probe] “official MCP server documented at https://www.speakeasy.com/docs/ai-control-plane/reference/platform-mcp”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.speakeasy.com/openapi.json, https://www.speakeasy.com/swagger.json, https://www.speakeas…”

   proof ↗⚑ flag

Metadata

1. developerInspect a listed server's tools, input schemas, transport, and hosting attributes before connecting to it

   weight 2 · round to Smithery
   Smithery~partialprobed6/10

   The registry API (probe-rt-1) exposes machine-readable metadata per server — displayName, qualifiedName, isDeployed, remote, verified — covering hosting/transport attributes, and community security audits (comm-1, comm-2) confirm that tool description fields are visible and inspectable pre-connection (that's how injected instructions were found). The CLI's `tool find` command (gh-2) and per-server typed SDKs generated from tool schemas (docs-16) further support schema visibility, but there is no single documented endpoint or UI view that surfaces full input schemas plus transport plus hosting attributes together in one inspection step. Missing for 10: a documented unified 'server detail' API/UI showing input schemas explicitly, and independent confirmation that transport type (stdio/http/websocket) is surfaced pre-connect.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”
   • [community] “Real examples pulled from Smithery-hosted servers: Context7, Google Sheets, Senzia, and Brave Search tool descriptions contained agent-direc…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed5/10

   Gram's in-dashboard 'Connect' harness lets a developer pick a listed server, authenticate, and chat with a model that calls its tools while inspecting logs, which implies pre-connection tool inspection, and servers are generated from OpenAPI/TypeScript sources implying schema definitions exist. However, no evidence explicitly documents a UI or API for viewing input schemas, transport type, or hosting attributes for a listed server. Missing for 10: explicit documentation of schema viewer, transport metadata display, hosting attribute display, and independent/hands-on confirmation of this inspection workflow.

   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”
   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”

   proof ↗⚑ flag

Playground

1. developerExercise an MCP server's tools from a browser-based inspector or playground before integrating it

   weight 2 · round to Gram (Speakeasy)
   Smithery—none0/10

   The evidence pack documents Smithery's registry, REST connect flow, OAuth/token handling, uplink, deep links, and CLI tool-calling, but nothing describes a browser-based inspector or playground UI for interactively testing a server's tools before integration. Since a registry/marketplace product could plausibly ship such a try-before-integrate UI, the axis applies, but no evidence confirms it exists.

   ⚑ flag

   Gram (Speakeasy)✓fullclaimed8/10

   Gram provides an in-dashboard agent harness (gram-docs-8) explicitly described as a way to pick a server, authenticate, chat with a model that invokes tools, and inspect logs before connecting a real client — directly matching the story of exercising tools from a browser-based playground pre-integration. This is first-party documentation with a clear feature name ('Connect') dedicated to this exact workflow. Missing for 10: independent/hands-on user reports confirming the playground works as described, and more detail on tool-level interaction granularity within the UI.

   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”

   proof ↗⚑ flag

Search

1. developerSearch and browse a large catalog of MCP servers and tools by capability before wiring anything into my agent

   weight 3 · round to Smithery
   Smithery✓fullprobed8/10

   Smithery ships both a CLI (`smithery mcp search`, `smithery tool find [query]`) and a keyless registry REST API (registry.smithery.ai/servers) that returns a machine-readable catalog with names, descriptions, verification status and usage counts, letting developers search/browse servers and tools by capability before connecting; docs also list example servers (Exa, Context7, Browserbase) illustrating capability-based discovery. Missing for 10: no evidence of a rich web UI faceted-browse/filter-by-category experience or independent hands-on review of the discovery UX itself beyond the CLI/API probes.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [claimed-docs] “Exa - Search the live web, access LinkedIn profiles, do deep research, and more”
   • [claimed-docs] “Context7 - Reference the latest docs for most major SDKs and frameworks directly in Cursor or Claude Code”
   • [claimed-docs] “Browserbase - Control a remote web browser using Stagehand”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed3/10

   Gram documents that an admin can 'find and add MCP servers' via the platform MCP client integration, implying some discovery capability, but there is no evidence of a searchable/browsable catalog of MCP servers by capability, catalog size, or a discovery UI — Gram is primarily framed as a control plane for generating/governing your own servers rather than a registry of third-party servers. missing for 10: a documented catalog/search UI, evidence of catalog scale or third-party server listings, and any capability-based filtering/browsing mechanism.

   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [claimed-docs] “Put a governed endpoint in front of an MCP server someone else runs, so access control, audit, and upstream credentials apply without changi…”

   proof ↗⚑ flag

Cli

1. developerDrive the publish-and-deploy workflow from an official CLI — authenticate, stage or build a server, push it, and install it into agents

   weight 2 · round to Gram (Speakeasy)
   Smithery~partialprobed6/10

   Smithery ships an official CLI (@smithery/cli, confirmed installable via npm and version-checked) with commands to search/add/call MCP servers, and docs cover publishing servers (URL method, MCPB bundles, server-card metadata) plus auth via OAuth/service tokens and installing into agents via deep links/uplink. However, evidence does not clearly show a unified CLI-driven build-and-push workflow (e.g., 'smithery build' or 'smithery deploy' subcommands) — the CLI evidence only shows search/add/call, while publish/build steps are documented as web/manual flows rather than CLI-driven. missing for 10: CLI subcommands for authenticate/build/push in one flow, independent hands-on confirmation of end-to-end publish-via-CLI, and clarity on whether install-into-agents is CLI-native vs deep-link/manual.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [github] “Browse skills on the [Smithery Skills Registry](https://smithery.ai/skills) and install them with the upstream installer”
   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”
   • [claimed-docs] “For local stdio servers — Smithery distributes a pre-built MCPB bundle that clients download and run locally.”
   • [claimed-docs] “If automatic scanning can’t complete (auth wall, required configuration, or other issues), you can provide server metadata manually via a st…”
   • [claimed-docs] “Deep links provide a seamless way to integrate Smithery MCPs into supported clients. When a user clicks a deep link from our server page, th…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialprobed7/10

   There's a documented, runtime-verified official CLI (gram-docs-4, gram-probe-5, gram-probe-rt-1) that supports installing toolsets as MCP servers into agents, plus docs on building servers from OpenAPI/TS and platform-mcp integration for org admins to add/configure servers from within MCP clients. However, the evidence doesn't clearly walk through explicit CLI subcommands for authenticate/stage/build/push in one coherent flow — most detail is on install and general CLI existence rather than a full documented publish pipeline. Missing for 10: explicit CLI command reference showing login/auth, build/stage, and push subcommands with example output, and independent (non-vendor) confirmation of the full workflow beyond version-check.

   • [claimed-docs] “install toolsets as MCP servers in AI agents, and keep the CLI itself up to date”
   • [claimed-docs] “Built — the Control Plane generates the server itself from a first-party source: an OpenAPI document or a TypeScript Functions project.”
   • [probe] “official CLI documented at https://www.speakeasy.com/docs/ai-control-plane/reference/command-line”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/gram/): downloaded the official cli@0.16.0 release artifact (gram_dar…”
   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”

   proof ↗⚑ flag

Frameworks

1. developerPlug the platform's tools into major agent frameworks and SDKs (Vercel AI SDK, LangChain, CrewAI, OpenAI or Anthropic SDKs) with documented adapters

   weight 2 · round to Smithery
   Smithery~partialclaimed5/10

   Smithery documents a dedicated Vercel AI SDK integration with automatic OAuth/token handling and multi-server tool aggregation (docs-13/14), and every published server gets an auto-generated typed SDK (docs-16), but the evidence pack shows no documented adapters for LangChain, CrewAI, or OpenAI/Anthropic SDKs specifically. Missing for 10: documented LangChain adapter, documented CrewAI adapter, documented OpenAI/Anthropic SDK adapter, and independent/hands-on confirmation beyond first-party docs.

   • [claimed-docs] “Smithery handles OAuth, token refresh, and connection management automatically.”
   • [claimed-docs] “Connect to multiple servers and aggregate their tools”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram documents MCP server generation, its own CLI, and MCP-client compatibility (Claude Code, Cursor, Codex), but there is no evidence of documented adapters or integration guides for Vercel AI SDK, LangChain, CrewAI, or OpenAI/Anthropic SDKs specifically.

   ⚑ flag

Security

1. platform-engineerRely on the platform scanning or inspecting listed servers for malicious or vulnerable behavior before my agents use them

   weight 2 · round to Gram (Speakeasy)
   Smithery!disputedcontradicted4/10

   Smithery's docs mention 'automatic scanning' of published servers (smithery-docs-11), implying some vetting pipeline, but independent research directly contradicts its effectiveness: scanning the top 100 registry servers found 22 with security findings (4 critical) including tool-description prompt-injection attacks embedded in widely-used servers like Context7 and Brave Search that execute silently on agent connection (smithery-comm-1, smithery-comm-2), and researchers separately found a path-traversal vulnerability in Smithery's own hosting escalatable to a supply-chain attack (smithery-comm-3). Missing for 10: any documented remediation/removal process for flagged malicious servers, transparency on scan methodology or coverage, and evidence the automatic scanning actually catches behavioral/prompt-injection risks rather than just metadata issues.

   • [claimed-docs] “If automatic scanning can’t complete (auth wall, required configuration, or other issues), you can provide server metadata manually via a st…”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”
   • [community] “Real examples pulled from Smithery-hosted servers: Context7, Google Sheets, Senzia, and Brave Search tool descriptions contained agent-direc…”
   • [community] “Security researchers detailed a path traversal vulnerability in Smithery's MCP server hosting that could be escalated into a supply chain at…”

   proof ↗⚑ flag

   Gram (Speakeasy)~partialclaimed5/10

   Gram provides governance controls relevant to tool curation — RBAC/OAuth2.1 access control, full audit/tool-call logging across hosted, tunneled and shadow servers, 'risk policies' management via the platform MCP server, and blocking of unsanctioned shadow MCP servers by default — plus an in-dashboard harness to test a server's tool calls before connecting a real client. However none of this is documented as active scanning for malicious code or known vulnerabilities in listed servers; it's closer to access governance, logging and manual testing than automated security scanning. Missing for 10: explicit vulnerability/malware scanning of server code or behavior, third-party security audit integration, and independent evidence that risk policies actually catch malicious servers.

   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”
   • [claimed-docs] “An org admin working in Claude Code, Cursor, Codex, or another MCP client can find and add MCP servers, finish their setup, put them on a pl…”
   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”
   • [claimed-docs] “An in-dashboard agent harness for testing MCP servers before connecting a real client: pick a server, authenticate, chat with a model that c…”
   • [github] “Team, server, and tool level permissions enforced through RBAC and Oauth2.1. Synced to your enterprise IDP (Okta, Azure AD, Google Workspace…”

   proof ↗⚑ flag

Signals

1. developerCompare servers using published quality signals — usage counts, health or liveness checks, scores, or verification badges

   weight 2 · round to Smithery
   Smithery~partialprobed6/10

   The registry API (probe evidence) exposes useCount and verified fields per server, plus isDeployed/remote status, giving developers concrete usage and verification signals to compare servers; docs also mention automatic scanning during publish which implies some quality vetting. However, there is no documented explicit 'quality score' metric or dedicated health/liveness-check endpoint beyond isDeployed, and independent security scans found unflagged risky servers in the top 100, showing the published signals don't capture all quality concerns. Missing for 10: an explicit scoring/quality-score field, a dedicated liveness/health-check mechanism, and corroboration that verification badges reliably reflect security vetting.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [claimed-docs] “If automatic scanning can’t complete (auth wall, required configuration, or other issues), you can provide server metadata manually via a st…”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram provides usage/cost tracking (gram-gh-1) and tool-call logs (gram-docs-5), plus governance controls like blocking shadow servers (gram-docs-10), but there is no evidence of published quality signals such as health/liveness checks, verification badges, or comparative quality scores that a developer could use to compare servers side-by-side. Missing for 10: liveness/health-check indicators, explicit quality/verification badges, and any server-comparison or ranking UI.

   • [github] “Track AI usage across teams and measure impact with either tokens or cost. Deep dive expensive sessions, create budgets and measure tool eff…”
   • [claimed-docs] “the raw execution log for every tool call the platform observes across hosted MCP servers, tunneled and shadow MCP servers, skills, and loca…”
   • [claimed-docs] “unsanctioned shadow MCP servers are blocked by default”

   proof ↗⚑ flag

Tool search

1. ai agentSearch and load tools at runtime through meta-tools instead of having thousands of tool definitions stuffed into my context

   weight 2 · round to Smithery
   Smithery✓fullprobed7/10

   Smithery's CLI exposes explicit meta-tools — `mcp search`, `tool find [query]`, and `tool call <connection> <tool>` — letting an agent discover and invoke tools by intent at runtime rather than preloading every tool definition, and the registry API is confirmed live and keylessly searchable in a probe returning a machine-readable catalog. missing for 10: no evidence the search/find/call meta-tools are themselves exposed as callable MCP tools inside a live agent session (vs. CLI-level usage), and no independent benchmark showing reduced context footprint from this pattern.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”

   proof ↗⚑ flag

   Gram (Speakeasy)—none0/10

   Gram's evidence covers MCP server hosting, toolset governance, OAuth, RBAC, and 'skills' loaded on demand, but none of it describes a runtime meta-tool (e.g., search_tools/load_tool) mechanism that lets an agent dynamically discover and load individual tool definitions instead of receiving a large static toolset. The closest analog (SKILL.md manifests loaded on demand) is a different mechanism from runtime tool search across a large tool catalog.

   ⚑ flag