Composio vs Smithery

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Composio
   Composio✓fullprobed9/10

   Composio hosts a live llms.txt at docs.composio.dev/llms.txt (verified 200 with agent-oriented summary content), directly satisfying the story of pointing an agent at agent-oriented docs; this is corroborated by a probe rather than just vendor claim. Missing for 10: no independent/community report of an agent actually consuming llms.txt successfully in the wild.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.composio.dev/llms.txt # Composio Documentation > Composio powers 1,400+ toolkits, tool search, con…”

   proof ↗⚑ flag

   Smithery~partialprobed6/10

   llms.txt and docs.md are confirmed 404, so the specific llms.txt convention isn't supported, but Smithery does expose agent-oriented docs alternatives: docs served live over MCP (JSON-RPC initialize succeeds at /docs/mcp) and a machine-readable OpenAPI 3.1 spec at /docs/openapi.json that an agent could consume to understand the API. missing for 10: an actual llms.txt file, and any docs indicating agents are expected to consume these alternate machine-readable formats as a first-class 'llms.txt-style' workflow.

   • [probe] “PROBE llms.txt: HTTP 404 at https://smithery.ai/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://smithery.ai/docs.md”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round drawn
   Composio✓fullprobed7/10

   Composio ships an SDK (Python/TS packages verified to install and import), a REST API (openapi.json), and a CLI, all of which can be scripted/invoked headlessly without a UI — sessions are created programmatically (composio.sessions.create) and tools executed via API/CLI, which is exactly the pattern used in CI automation. Missing for 10: explicit CI/CD pipeline docs or examples (e.g., GitHub Actions integration) and independent confirmation of a fully non-interactive auth flow for headless CI runs.

   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”
   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [probe] “official CLI documented at https://docs.composio.dev/docs/cli”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/composio/): `npm install @composio/core` and `uv pip install composio…”

   proof ↗⚑ flag

   Smithery✓fullprobed7/10

   Smithery ships a CLI (search/add/call tools) and a documented REST API with an OpenAPI 3.1 spec, service tokens scoped for AI agents/automation, and a live registry API queryable without interactive auth — all of which support scripted/headless use in CI pipelines. Runtime probes confirm the CLI installs and runs (`smithery --version`) and the API/registry endpoints respond programmatically. missing for 10: explicit CI/CD pipeline examples or GitHub Actions integration docs, and independent reports of real-world CI usage.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to Smithery
   Composio—none0/10

   All evidence describes Composio exposing its own hosted MCP server/toolkits for agents to consume (composio-docs-4, composio-probe-3, composio-probe-rt-1) — the reverse of this story, which asks whether a user can plug external MCP servers INTO Composio so it can use their tools. No evidence shows Composio importing or connecting to third-party MCP servers as a client.

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”

   proof ↗⚑ flag

   Smithery✓fullprobed8/10

   Smithery's core purpose is connecting/aggregating MCP servers: docs and probes confirm a REST/registry interface, hosted MCP endpoints with OAuth, deep links, uplink for local servers, and CLI/SDK tooling for adding and calling tools across connections, with runtime probes corroborating a live registry and per-server endpoints. Community reports of tool-description injection risks are a security caveat, not evidence the connection/tool-use mechanism fails. missing for 10: independent hands-on confirmation that an AI agent (not just probes) successfully invokes tools end-to-end across multiple aggregated servers, and resolution of the tool-description-injection security concerns.

   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “Deep links provide a seamless way to integrate Smithery MCPs into supported clients. When a user clicks a deep link from our server page, th…”
   • [claimed-docs] “Smithery handles OAuth, token refresh, and connection management automatically.”
   • [claimed-docs] “Connect to multiple servers and aggregate their tools”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round to Smithery
   Composio✓fullprobed8/10

   Composio documents and exposes an official hosted MCP server (session `mcp: true`, and the shared 'Composio Connect' MCP URL) that any MCP-compatible agent/client can point at, and a live runtime probe confirms the endpoint (connect.composio.dev/mcp) is real and gated by managed OAuth as documented. This is a fair and clearly-applicable capability for an integration platform, backed by first-party docs plus independent endpoint verification. Missing for 10: an independent third-party report of a successful end-to-end agent session over this MCP endpoint (only auth-challenge probe, not a full tool-call transcript).

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Combine `mcp: true` with the direct-tools preset to get one MCP URL that serves exactly the tools you list”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Smithery✓fullprobed9/10

   Smithery is a hosting/registry platform (not itself an agent), so the exception applies and publishing an official MCP server endpoint is a fair, evidenced axis: docs describe OAuth-handled connections, deep linking, uplink, and typed SDKs, and independent probes confirm a live hosted MCP endpoint (server.smithery.ai/exa/mcp) requiring OAuth and even the docs site itself served over MCP (smithery.ai/docs/mcp) with a working CLI install. Missing for 10: no independent third-party report validating that connecting via the hosted server works end-to-end for an actual agent (only probe-level confirmation, not user testimonial), and community findings raise tool-description-injection risk that slightly tempers confidence though it doesn't contradict the connection mechanism itself.

   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “Deep links provide a seamless way to integrate Smithery MCPs into supported clients. When a user clicks a deep link from our server page, th…”
   • [claimed-docs] “Smithery handles OAuth, token refresh, and connection management automatically.”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to Composio
   Composio✓fullprobed9/10

   Composio ships an official CLI with documented install/setup for Codex, Claude Code, and terminal use, enabling agents or developers to connect apps, execute tools, inspect schemas, and debug projects locally (composio-docs-10, composio-docs-11, composio-probe-4). missing for 10: no independent/hands-on user report specifically validating the CLI (community evidence relates to unrelated connection issues, not the CLI itself).

   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [claimed-docs] “Install the native plugin for Codex or Claude Code: composio setup --target auto”
   • [probe] “official CLI documented at https://docs.composio.dev/docs/cli”

   proof ↗⚑ flag

   Smithery~partialprobed6/10

   There is concrete evidence of a working Smithery CLI (npm install @smithery/cli verified, version 4.11.1) with documented commands for searching the registry, adding MCP connections, finding/calling tools, and installing skills. However, the GitHub source is attributed to 'arcadeai-labs/smithery-cli' rather than an obviously first-party Smithery org, leaving some ambiguity about official first-party ownership, and there is no independent/community corroboration of CLI usage. Missing for 10: clearer first-party branding/ownership confirmation, official docs page dedicated to the CLI, and independent hands-on user reports.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [github] “Browse skills on the [Smithery Skills Registry](https://smithery.ai/skills) and install them with the upstream installer”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to Composio
   Composio✓fullprobed9/10

   Composio publishes a documented, structured API surface: an OpenAPI spec (verified live via probe), SDK session creation calls, meta-tools, CLI, and MCP endpoints, all documented on docs.composio.dev and confirmed reachable/installable via runtime probes (openapi.json 200, SDK installs succeed, MCP endpoint live behind auth). This gives AI-native users multiple concrete, documented ways (REST API, Python/TS SDK, CLI, MCP) to drive the product programmatically. Missing for 10: independent third-party review validating API completeness/stability beyond vendor docs and probes.

   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”
   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/composio/): `npm install @composio/core` and `uv pip install composio…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”

   proof ↗⚑ flag

   Smithery✓fullprobed8/10

   Smithery exposes a real, documented public API: a registry API (registry.smithery.ai) queryable without keys, a full OpenAPI 3.1.0 spec (35 paths) confirmed live at smithery.ai/docs/openapi.json, and a REST/connection interface with OAuth and token scoping for programmatic/agent use, plus a CLI (smithery mcp/tool commands) for driving it. Initial llms.txt/docs.md probes 404'd but the actual OpenAPI spec and registry endpoints were independently verified live, satisfying 'documented public API' for an AI-native/agentic consumer. missing for 10: no independent third-party corroboration of API stability/versioning, and some docs endpoints (llms.txt) are absent despite the OpenAPI spec existing.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to Smithery
   Composio!disputedcontradicted5/10

   Composio's docs show real scoping controls—restricting a session to specific toolkits, generating an MCP URL for an exact tool subset, and meta-tools that authorize apps on demand ([composio-docs-3],[composio-docs-5],[composio-docs-14])—but independent security research found a leaked Composio key returned live Gmail, GitHub, and CircleCI tokens, indicating credentials are not tightly least-privilege in practice ([composio-comm-2]). This is a concrete hands-on/security counter-example directly on the scoped-credential axis, so the claim is disputed rather than cleanly fulfilled. Missing for 10: no documentation of granular per-scope/permission-level credential issuance (only toolkit/tool-list restriction), and no rebuttal or fix evidence for the leaked-key finding.

   • [claimed-docs] “To restrict the set, pass `toolkits` when you create the session.”
   • [claimed-docs] “Combine `mcp: true` with the direct-tools preset to get one MCP URL that serves exactly the tools you list”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [community] “A leaked Composio key returned live Gmail, GitHub and CircleCI tokens (security research on exposed credentials in Composio's agent integrat…”

   proof ↗⚑ flag

   Smithery✓fullclaimed7/10

   Smithery docs describe service tokens explicitly scoped to a specific app/connection/user metadata ('list and call tools on connections in my-app where metadata.userId is user-123 — nothing else'), plus encrypted write-only credential storage and namespace-scoped API keys, directly matching the least-privilege credential story for agents. Missing for 10: independent/hands-on verification that scope enforcement actually holds, and documentation of token revocation/expiry/audit controls.

   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”
   • [claimed-docs] “This token can list and call tools on connections in `my-app` where `metadata.userId` is `user-123` — nothing else.”
   • [claimed-docs] “Secure credential storage — Encrypted, write-only storage for API keys and tokens”
   • [claimed-docs] “Namespaces act as a shared workspace. Resources within a namespace can be managed by anyone with the appropriate API key scoped to that name…”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Composio
   Composio✓fullprobed8/10

   Composio ships official SDKs for both TypeScript (@composio/core) and Python (composio), verified via real installs that imported cleanly, alongside provider adapters that map tools into native framework formats and OpenAPI-documented endpoints. missing for 10: independent third-party tutorials/showcases building production apps with the SDKs, and no evidence of SDKs beyond Python/TypeScript (e.g. Go, Java).

   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/composio/): `npm install @composio/core` and `uv pip install composio…”
   • [claimed-docs] “A provider is the adapter that turns Composio tools into the native tool format your framework expects, so you don't write glue code.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”

   proof ↗⚑ flag

   Smithery✓fullprobed7/10

   Smithery auto-generates a typed TypeScript SDK for every published MCP server (smithery-docs-16), ships an official CLI (smithery-cli on GitHub) verified via live npm install (smithery-probe-rt-4), and exposes a documented OpenAPI 3.1 spec for platform API client generation, confirmed live via probe (smithery-probe-rt-3). This gives AI-native builders concrete, verified official SDK/CLI/API surfaces to build against. Missing for 10: SDKs beyond TypeScript (no Python/other language SDK evidence), and independent third-party corroboration of real-world SDK usage beyond Smithery's own docs/probes.

   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round to Composio
   Composio✓fullclaimed7/10

   Composio documents a triggers/webhooks system where users register a webhook URL per project and Composio POSTs signed event payloads for every trigger, directly matching the subscribe-to-events story. missing for 10: independent/hands-on confirmation of webhook delivery reliability, event catalog/filtering details, and retry/latency behavior are not evidenced beyond first-party docs.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”

   proof ↗⚑ flag

   Smithery~partialclaimed6/10

   Smithery's Triggers feature explicitly lets consumers subscribe to upstream events, proxying subscribe calls through to the server—directly matching webhook/event-subscription functionality for agentic use. However, this is documented only at a conceptual level with no depth on webhook delivery mechanics, retry/reliability guarantees, or independent hands-on verification. missing for 10: detailed webhook payload/delivery docs, independent corroboration of triggers working in practice, and coverage of failure/retry semantics.

   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”

   proof ↗⚑ flag

Agentic features

1. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Composio
   Composio~partialcommunity5/10

   Composio's triggers feature lets developers register a webhook so Composio pushes event notifications (e.g., new email, PR) that can kick off agent logic without polling, which is a building block for background automation [composio-docs-9]. However, Composio itself doesn't host or run the autonomous agent loop — execution still happens in the developer's own agent/runtime that reacts to the webhook, and there's no evidence of a built-in scheduler or persistent background-running agent. Community evidence also shows integration reliability issues (Instagram connect failing) that would undercut unattended automation [composio-comm-1]. Missing for 10: evidence of Composio actually orchestrating/running an agent autonomously (not just delivering trigger events), a scheduling/cron-like automation primitive, and independent confirmation that background-triggered automations run reliably end-to-end.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”

   proof ↗⚑ flag

   Smithery~partialclaimed3/10

   Smithery's Triggers feature lets an MCP server surface upstream events which Smithery proxies to a consumer, hinting at event-driven automation, but there is no evidence of a scheduler, persistent background execution engine, or the platform itself running autonomous tasks — automation actually running requires an external agent that consumes the trigger. missing for 10: evidence of a scheduling/cron mechanism, autonomous execution loop hosted by Smithery, and confirmation that triggers fire without an active client agent.

   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”

   proof ↗⚑ flag

2. ai-native userOperate the product with natural-language commands

   weight 2 · round drawn
   Composio~partialclaimed5/10

   Composio's CLI plugin integrates with Claude Code/Codex so a coding agent (or its user) can invoke Composio's tools via natural-language prompts inside those agents, and meta-tools let an agent discover/execute tools autonomously. However, Composio itself is an integration/tool-serving layer rather than a natural-language interface — there's no first-party chat/NL command surface of its own, only its embedding inside other agentic tools. Missing for 10: a dedicated natural-language command interface owned by Composio itself, and independent evidence of end-users issuing free-form NL commands successfully.

   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [claimed-docs] “Install the native plugin for Codex or Claude Code: composio setup --target auto”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”

   proof ↗⚑ flag

   Smithery~partialprobed5/10

   Smithery's CLI exposes intent-based discovery ('smithery tool find [query] # Search tools by name or intent') and its docs are themselves served as an MCP endpoint that an agent can query conversationally, showing some natural-language-friendly interaction. However, the primary interface remains structured CLI subcommands (search, add, call) and REST/OpenAPI calls rather than a general natural-language command layer for operating the whole product. Missing for 10: a documented conversational/chat interface or explicit NL parsing across all product operations, and independent evidence of users successfully issuing free-form natural-language commands rather than structured queries.

   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round to Smithery
   Composio—none0/10

   The pack shows an OpenAPI spec exists (composio-probe-2) and standard docs pages, but there is no evidence of an interactive API reference (e.g., Swagger/Redoc-style try-it console) with runnable, in-browser examples — missing for 10: dedicated interactive API explorer, runnable code snippets, evidence of a try-it-now console.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”

   proof ↗⚑ flag

   Smithery~partialprobed5/10

   Smithery exposes a full OpenAPI 3.1 spec (35 paths) and even serves its documentation itself as an MCP endpoint that agents can query interactively via JSON-RPC, which supports AI-native exploration of the API surface. However there's no evidence of a classic interactive reference UI (e.g., Swagger/Redoc 'try it out' runnable examples in-browser), and alternate machine-readable doc formats (llms.txt, docs.md) return 404s. missing for 10: an actual interactive/try-it-out API console with executable example requests, evidence of runnable code snippets in docs, independent confirmation of usability of the MCP-served docs.

   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://smithery.ai/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://smithery.ai/docs.md”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round to Composio
   Composio✓fullprobed9/10

   A live probe confirms Composio serves a valid machine-readable OpenAPI spec at https://docs.composio.dev/openapi.json (HTTP 200, contains 'openapi' key), directly satisfying the story. Missing for 10: no independent third-party confirmation of spec completeness/versioning beyond the probe check.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

   Smithery✓fullprobed8/10

   A direct hands-on probe confirms a live, machine-readable OpenAPI 3.1.0 specification with 35 documented paths at smithery.ai/docs/openapi.json, which an agent could use to generate a client — squarely satisfying the story. Earlier probe attempts at guessed paths (openapi.json at root, swagger.json, etc.) 404'd, but the actual documented location succeeds. Missing for 10: first-party documentation explicitly announcing/linking the OpenAPI spec, and independent community confirmation of its use.

   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://smithery.ai/openapi.json, https://smithery.ai/swagger.json, https://smithery.ai/api/openapi.…”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round drawn
   Composio—none0/10

   Evidence shows a 'remote sandbox' feature (COMPOSIO_REMOTE_WORKBENCH) for running Python code on bulk/large tool responses, and a docs teaser mentioning 'sand[box]', but nothing describes an isolated test environment that lets a user exercise Composio's actual toolkits/connected accounts without touching live production data or APIs — connections like Instagram/Gmail appear to hit real production endpoints (per composio-comm-1/2).

   • [claimed-docs] “COMPOSIO_REMOTE_WORKBENCH** — Run Python in a remote sandbox for bulk operations or processing large tool responses.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.composio.dev/llms.txt # Composio Documentation > Composio powers 1,400+ toolkits, tool search, con…”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”

   proof ↗⚑ flag

   Smithery—none0/10

   Evidence shows Smithery's Uplink lets developers test an in-development MCP server against a real agent before publishing, but there is no mention of a distinct sandbox environment, test data isolation, or any mechanism preventing production data exposure during testing.

   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “You’re developing an MCP server and want to test it against a real agent before publishing.”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Composio—none0/10

   The evidence pack shows API/SDK docs, CLI, MCP endpoints, and an OpenAPI spec, but nowhere mentions API versioning conventions or a documented deprecation policy for breaking changes. Missing for 10: any changelog, version-numbering scheme (e.g. v1/v2 endpoints), or explicit deprecation/sunset policy documentation.

   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

   Smithery—none0/10

   Evidence shows Smithery has an OpenAPI 3.1.0 spec and a documented REST/MCP interface, but nothing in the docs or probes describes API versioning conventions or a deprecation policy for breaking changes.

   ⚑ flag

Multi tenant

1. platform-engineerMaintain separate connected accounts and scoped tokens per end user of my product, so each user's agent acts only on that user's credentials

   weight 3 · round to Composio
   Composio✓fullclaimed7/10

   Composio's session model is built around per-user identity (`user_id`) and each user must complete their own Connect Link OAuth flow, so an agent only executes tools using the connected end-user's own credentials, with callback_url and white-labeling to fit into a platform's UX. Missing for 10: independent/hands-on verification that token scoping is enforced per user in production, and explicit detail on scope-limiting (vs. just per-user connection) mechanics.

   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”
   • [claimed-docs] “The agent posts the Connect Link straight into the conversation. The user clicks it, signs in, and the agent retries the tool and continues.…”
   • [claimed-docs] “To send users back to your app after they connect, pass a `callback_url`”
   • [claimed-docs] “You can swap in your logo and name, then restyle the whole page to match your product.”
   • [claimed-docs] “To restrict the set, pass `toolkits` when you create the session.”

   proof ↗⚑ flag

   Smithery~partialcommunity6/10

   Smithery's token-scoping docs show scoped tokens tied to per-user metadata (e.g. a token limited to connections where metadata.userId is 'user-123'), and secure credential storage/automatic refresh are documented, which directly supports per-end-user credential isolation for a platform-engineer building on top of Smithery. However, there's no first-party documentation of a full per-user account management workflow (provisioning, revocation, auditing across many end users) beyond the single scoping example, and independent security research flagged real vulnerabilities (tool description injection, path traversal) that raise doubts about credential-boundary robustness in practice. missing for 10: dedicated multi-tenant/per-end-user account management docs beyond one scoping example, evidence of token revocation/rotation workflows at scale, and independent verification that scoped tokens hold up against the reported security findings.

   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”
   • [claimed-docs] “This token can list and call tools on connections in `my-app` where `metadata.userId` is `user-123` — nothing else.”
   • [claimed-docs] “Automatic token refresh — Credentials stay valid without manual intervention”
   • [claimed-docs] “Secure credential storage — Encrypted, write-only storage for API keys and tokens”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”
   • [community] “Security researchers detailed a path traversal vulnerability in Smithery's MCP server hosting that could be escalated into a supply chain at…”

   proof ↗⚑ flag

Oauth

1. developerLet the platform run OAuth flows for downstream tools and vault the resulting credentials encrypted, with automatic token refresh

   weight 3 · round to Smithery
   Composio!disputedcontradicted4/10

   Composio's docs describe hosted OAuth via Connect Link with no auth UI to build, callback_url redirects, and white-labeled auth pages (composio-docs-6,7,8), implying it manages and stores credentials for downstream tool auth. However, independent security research found a leaked Composio API key exposed live Gmail, GitHub, and CircleCI tokens, undermining the 'encrypted vault' claim in practice, and a user reported OAuth connect flows failing outright (400 errors) for at least one integration. Missing for 10: explicit documentation of encryption-at-rest mechanics, automatic token-refresh behavior, and any vendor response/fix to the leaked-token incident.

   • [claimed-docs] “The agent posts the Connect Link straight into the conversation. The user clicks it, signs in, and the agent retries the tool and continues.…”
   • [claimed-docs] “To send users back to your app after they connect, pass a `callback_url`”
   • [claimed-docs] “You can swap in your logo and name, then restyle the whole page to match your product.”
   • [community] “A leaked Composio key returned live Gmail, GitHub and CircleCI tokens (security research on exposed credentials in Composio's agent integrat…”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”

   proof ↗⚑ flag

   Smithery✓fullprobed7/10

   Docs explicitly describe Smithery handling OAuth flows, encrypted write-only credential storage, and automatic token refresh (smithery-docs-1,2,3,13), and this is corroborated by a live probe showing an OAuth challenge gating a hosted MCP endpoint (smithery-probe-rt-2). missing for 10: no independent hands-on confirmation of encryption-at-rest implementation details or token refresh behavior over time, and community evidence focuses on unrelated security findings rather than corroborating credential vaulting.

   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Automatic token refresh — Credentials stay valid without manual intervention”
   • [claimed-docs] “Secure credential storage — Encrypted, write-only storage for API keys and tokens”
   • [claimed-docs] “Smithery handles OAuth, token refresh, and connection management automatically.”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”

   proof ↗⚑ flag

White label

1. platform-engineerWhite-label the authorization flow with my own branding and my own OAuth app credentials

   weight 1 · round to Composio
   Composio~partialclaimed6/10

   Composio's docs explicitly describe white-labeling the auth flow (swap logo/name, restyle the page) and building custom callback flows, directly matching the branding half of the story. However, there is no evidence in the pack that platform-engineers can supply their own OAuth app client ID/secret (bring-your-own OAuth credentials) rather than using Composio-managed auth apps. Missing for 10: documentation of custom/BYO OAuth app credential support, independent confirmation that white-labeling works end-to-end in production.

   • [claimed-docs] “You can swap in your logo and name, then restyle the whole page to match your product.”
   • [claimed-docs] “The agent posts the Connect Link straight into the conversation. The user clicks it, signs in, and the agent retries the tool and continues.…”
   • [claimed-docs] “To send users back to your app after they connect, pass a `callback_url`”

   proof ↗⚑ flag

   Smithery—none0/10

   The evidence pack describes Smithery's own OAuth/credential-management infrastructure (managed auth, token scoping, secure storage) but contains no mention of white-labeling the auth flow with custom branding or bringing your own OAuth app/client credentials to replace Smithery's identity — no BYO-OAuth-app, custom domain, or branding customization feature is documented anywhere.

   ⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Composio
   Composio~partialcommunity5/10

   Composio documents parallel tool execution via meta-tools and a remote sandbox workbench explicitly for 'bulk operations or processing large tool responses,' supporting some bulk automation across many items. However, there's no dedicated bulk-operation API, batch endpoint, or hands-on evidence demonstrating bulk operations at scale, and one community report shows a basic connection failure (Instagram 400 error), raising reliability doubts. missing for 10: a documented batch/bulk API or endpoint, concrete examples of bulk operations across many items, independent hands-on verification of bulk workflows succeeding at scale.

   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [claimed-docs] “COMPOSIO_REMOTE_WORKBENCH** — Run Python in a remote sandbox for bulk operations or processing large tool responses.”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”

   proof ↗⚑ flag

   Smithery—none0/10

   Evidence shows Smithery lets an agent connect to and aggregate tools from multiple servers, and its registry API can be queried/paginated for many entries, but there is no documented batch/bulk operation capability (e.g., bulk tool calls, mass server management, batch CRUD) that would let a user perform bulk actions across many items at once. missing for 10: batch tool-call API, bulk server management commands, mass connection/config updates, any documented multi-item transaction support.

   ⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Composio
   Composio~partialclaimed5/10

   Composio's Triggers system delivers event notifications to a registered webhook (signed, verifiable), which lets a user's own system react to events — but Composio itself doesn't appear to expose a first-party rule-definition or condition/action automation builder; the 'trigger action automatically' logic must be implemented by the user's external service consuming the webhook. missing for 10: a documented rules/automation engine (conditions, filters, chained actions) inside Composio itself, and independent evidence of end-to-end automatic action execution rather than just event delivery.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”

   proof ↗⚑ flag

   Smithery~partialclaimed3/10

   Smithery's 'Triggers' feature lets an MCP server surface upstream events and lets a consumer 'activate' a trigger, which is a thin form of event-driven automation, but there's no evidence of a user-facing rule-definition mechanism (e.g., 'if event X then run action Y') or a rules engine — it only proxies subscribe calls. Missing for 10: a documented rule/condition-action authoring interface, examples of automated action execution on trigger events, and independent confirmation the feature works end-to-end.

   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round drawn
   Composio—none0/10

   Composio documents event-driven triggers delivered via webhooks (composio-docs-9), but there is no evidence of a scheduling primitive for recurring jobs/workflows (e.g., cron-style automation) — triggers fire on external events, not on a time-based recurring schedule. Missing for 10: any documented scheduler, cron syntax, or recurring-job API, and independent confirmation that scheduled workflows actually run.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”

   proof ↗⚑ flag

   Smithery—none0/10

   Smithery's docs describe on-demand tool connections, OAuth/credential management, and event-driven 'Triggers' that proxy subscribe calls from upstream services, but nothing describes cron-like recurring job scheduling or workflow automation on a timer. Triggers are reactive (event-based), not scheduled/recurring, so the specific capability asked for is unevidenced.

   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”
   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”

   proof ↗⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round drawn
   Composio—none0/10

   The evidence pack covers Composio's tool/session/MCP/auth infrastructure but contains no mention of versioning automations, review workflows, or rollback of workflows/agent configurations. No changelog, version history, diff/review, or revert capability is documented anywhere in the pack.

   ⚑ flag

   Smithery—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

Bridge

1. developerBridge a locally running MCP server into the platform as a managed remote connection without deploying it

   weight 1 · round to Smithery
   Composio—none0/10

   Composio's docs describe the platform hosting its own MCP server (sessions with mcp:true, Composio Connect shared URL) so agents can consume Composio's tools via MCP, but there is no evidence of the reverse flow — registering or tunneling a developer's own locally-running MCP server into Composio to be exposed as a managed remote connection.

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Smithery✓fullclaimed8/10

   Smithery's Uplink feature explicitly exposes an MCP server running on any local machine as a regular Smithery connection, letting developers test/use a locally running server against real agents without deploying it, and this is backed by scoped tokens and managed OAuth/credential handling. missing for 10: independent/hands-on confirmation of Uplink working end-to-end (only first-party docs cited) and no detail on latency/reliability of the bridged local connection.

   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “You’re developing an MCP server and want to test it against a real agent before publishing.”
   • [claimed-docs] “You want to run a private tool for yourself or your team without hosting it.”
   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”

   proof ↗⚑ flag

Access control

1. platform-engineerRestrict which servers and individual tools an agent may call through allowlists or per-tool access control

   weight 2 · round to Composio
   Composio✓fullclaimed7/10

   Composio's docs describe restricting a session to specific toolkits (composio-docs-3) and combining `mcp:true` with a direct-tools preset to serve an MCP URL with exactly the listed tools (composio-docs-5), directly enabling allowlist-style, per-tool/server access control for an agent. Missing for 10: independent/hands-on verification of enforcement, granular per-user or role-based policy controls, and audit/logging of denied calls.

   • [claimed-docs] “To restrict the set, pass `toolkits` when you create the session.”
   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Combine `mcp: true` with the direct-tools preset to get one MCP URL that serves exactly the tools you list”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”

   proof ↗⚑ flag

   Smithery~partialclaimed6/10

   Smithery's token-scoping feature lets platform engineers issue tokens restricted to specific connections/namespaces (e.g., 'list and call tools on connections in my-app where metadata.userId is user-123 — nothing else'), which functions as a coarse allowlist mechanism, and namespaces further scope API-key access to a workspace. However, the evidence shows scoping is at the connection/namespace level, not explicit per-tool allow/deny lists within a single server. Missing for 10: documented fine-grained per-tool allow/deny configuration, independent/hands-on verification that tool-level restrictions are enforced, and any admin UI or policy examples showing individual tool blocking within an allowed connection.

   • [claimed-docs] “Service tokens let you safely expose Smithery to browsers, mobile apps, and AI agents without leaking your API key.”
   • [claimed-docs] “This token can list and call tools on connections in `my-app` where `metadata.userId` is `user-123` — nothing else.”
   • [claimed-docs] “Namespaces act as a shared workspace. Resources within a namespace can be managed by anyone with the appropriate API key scoped to that name…”

   proof ↗⚑ flag

Audit

1. platform-engineerSee a log of every tool call that passed through the platform, with enough detail to audit or replay what an agent did

   weight 2 · round drawn
   Composio—none0/10

   The evidence pack describes sessions, meta-tools, MCP hosting, auth flows, and a webhook mechanism for triggers, but nowhere documents a queryable audit log of tool calls with replay detail for platform engineers. Webhook triggers (composio-docs-9) cover external events, not a comprehensive tool-call audit trail.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”

   proof ↗⚑ flag

   Smithery—none0/10

   No evidence describes any audit logging, tool-call history, or replay capability for platform engineers—docs cover connection, credential management, publishing, tokens, and namespaces but never mention logs of tool calls passing through the gateway.

   ⚑ flag

Catalog

1. developerReach a deep catalog of third-party apps and APIs (hundreds to thousands of integrations) through the platform

   weight 3 · round to Composio
   Composio✓fullprobed9/10

   Composio's own docs and llms.txt claim 1,400+ toolkits reachable via meta-tools, sessions, MCP, and CLI, and probes confirm the docs/OpenAPI/MCP endpoint are live and functioning (though gated by auth), plus SDK packages install cleanly. One community report shows an isolated integration failure (Instagram OAuth) which doesn't contradict the breadth of the catalog itself. Missing for 10: independent third-party verification of the actual '1,400+' count beyond vendor claims.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.composio.dev/llms.txt # Composio Documentation > Composio powers 1,400+ toolkits, tool search, con…”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/composio/): `npm install @composio/core` and `uv pip install composio…”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”

   proof ↗⚑ flag

   Smithery✓fullprobed8/10

   Smithery is a registry/marketplace connecting to a large catalog of third-party MCP servers (Exa, Context7, Browserbase, Google Sheets, Brave Search, etc.), with a live registry API confirmed via probe returning searchable server entries with useCount/verified metadata, plus hosted per-server MCP endpoints (server.smithery.ai) and CLI search/add tooling. Community evidence confirms real scale (100+ servers scanned) though also raises security concerns about tool description injection, which affects trust/quality but not the breadth-of-catalog claim itself. Missing for 10: independently verified total count of hundreds/thousands of listings and first-party data on catalog size/growth.

   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Exa - Search the live web, access LinkedIn profiles, do deep research, and more”
   • [claimed-docs] “Context7 - Reference the latest docs for most major SDKs and frameworks directly in Cursor or Claude Code”
   • [claimed-docs] “Browserbase - Control a remote web browser using Stagehand”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”

   proof ↗⚑ flag

Connect

1. developerPoint my MCP-compatible client at a hosted remote server URL and reach downstream tools without deploying any infrastructure

   weight 3 · round drawn
   Composio✓fullprobed8/10

   Composio documents a hosted MCP server reachable via a shared URL (Composio Connect) or per-session mcp:true flag, requiring no self-hosted infrastructure, and a live probe confirms the endpoint at connect.composio.dev/mcp is active and auth-gated as documented. Missing for 10: independent hands-on confirmation of a successful authenticated tool call through the hosted endpoint beyond the 401 challenge probe.

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Combine `mcp: true` with the direct-tools preset to get one MCP URL that serves exactly the tools you list”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Smithery✓fullprobed8/10

   Docs and live probes confirm Smithery hosts remote MCP server endpoints (server.smithery.ai/<server>/mcp) that clients can point at directly, with OAuth/token handling, deep links, and typed SDKs generated automatically — no server deployment needed by the developer. Independent probe evidence (smithery-probe-rt-2) shows the hosted endpoint is live and gated by managed OAuth rather than requiring self-hosting infrastructure. missing for 10: independent hands-on developer accounts (outside security research) confirming a smooth end-to-end connect experience beyond documentation and infra probes.

   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”
   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”
   • [claimed-docs] “Smithery handles OAuth, token refresh, and connection management automatically.”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”

   proof ↗⚑ flag

Generate

1. developerTurn an existing API definition (OpenAPI) or custom code into a hosted MCP server without writing protocol plumbing

   weight 2 · round to Smithery
   Composio—none0/10

   Evidence shows Composio can expose its own pre-built toolkits as hosted MCP servers (via `mcp:true` sessions or Composio Connect) and ships an OpenAPI spec for its own API, but there is no evidence of a feature letting developers upload a custom OpenAPI definition or their own code and have Composio auto-generate a hosted MCP server from it — the toolkits appear to be Composio-curated integrations, not user-supplied API/code onboarding.

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”

   proof ↗⚑ flag

   Smithery~partialclaimed4/10

   Smithery's docs show a hosting/publishing path for MCP servers already built in code (URL method, MCPB bundles, server-card metadata) which removes protocol plumbing for deployment, but there is no evidence of an OpenAPI-spec ingestion or auto-generation feature that turns an existing REST API definition into an MCP server — the publish flow assumes you already have MCP-shaped code or a deployed server. Missing for 10: OpenAPI-to-MCP conversion tooling, any first-party or independent proof of generating a server from an OpenAPI spec, and confirmation that 'custom code' can be arbitrary non-MCP code rather than code already written against the MCP SDK.

   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”
   • [claimed-docs] “For local stdio servers — Smithery distributes a pre-built MCPB bundle that clients download and run locally.”
   • [claimed-docs] “If automatic scanning can’t complete (auth wall, required configuration, or other issues), you can provide server metadata manually via a st…”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”

   proof ↗⚑ flag

Publish

1. developerPublish my own MCP server on the platform for distribution, discovery, and installation by others

   weight 2 · round to Smithery
   Composio—none0/10

   Composio's docs describe hosting MCP servers built from its own toolkits/sessions and exposing a shared Composio Connect MCP endpoint for consuming tools, plus a CLI for local tool use — all about consuming Composio-provided tools, not about a developer publishing a custom MCP server for others to discover and install. No evidence of a publish/marketplace/registry flow for third-party-authored MCP servers.

   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Smithery✓fullprobed8/10

   Docs clearly describe multiple publishing paths (URL method for already-deployed servers, MCPB bundle distribution for local stdio servers, automatic scanning or manual server-card metadata) resulting in a registry listing with generated TypeScript SDK, and the registry API probe confirms servers are discoverable/searchable by others without auth. Security researchers did find vulnerabilities/prompt-injection risks in some published servers, but this reflects on registry quality, not on whether publishing/discovery/installation works. missing for 10: independent hands-on account of a developer successfully publishing a server end-to-end, and clearer detail on install flow verification post-publish.

   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”
   • [claimed-docs] “For local stdio servers — Smithery distributes a pre-built MCPB bundle that clients download and run locally.”
   • [claimed-docs] “If automatic scanning can’t complete (auth wall, required configuration, or other issues), you can provide server metadata manually via a st…”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”

   proof ↗⚑ flag

Triggers

1. developerSubscribe my agent to events from connected downstream apps (triggers/webhooks) instead of polling them

   weight 2 · round drawn
   Composio✓fullclaimed7/10

   Composio documents a triggers/webhooks system where developers register one webhook URL per project and Composio POSTs signed trigger events from connected apps, directly enabling event subscription instead of polling. However, missing for 10: independent/hands-on confirmation of trigger reliability, details on per-app trigger configuration granularity, and evidence of delivery guarantees or retry semantics.

   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”

   proof ↗⚑ flag

   Smithery✓fullclaimed7/10

   Smithery documents a Triggers feature that explicitly lets consumers subscribe to event streams surfaced from an upstream service, with Smithery proxying the subscribe call to the server instead of requiring polling. This directly matches the story of subscribing an agent to downstream app events via webhooks/triggers rather than polling. Missing for 10: independent/hands-on confirmation that triggers work reliably in production, and more detail on delivery guarantees or webhook payload handling.

   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”

   proof ↗⚑ flag

Analytics

1. platform-engineerSee usage analytics for the servers and tools my org consumes or publishes — call volumes, users, errors, and cost

   weight 2 · round drawn
   Composio—none0/10

   Evidence covers sessions, MCP hosting, auth, triggers, CLI, and SDK installation, but nothing in the pack describes any analytics/usage dashboard, call volume metrics, per-user breakdowns, error rates, or cost reporting for consumed or published tools/servers.

   ⚑ flag

   Smithery—none0/10

   The evidence pack documents connection, publishing, token scoping, triggers, and SDK generation, but nothing about a dashboard or API for org-level usage analytics (call volumes, per-user breakdowns, error rates, cost). The only quantitative signal found is a per-server 'useCount' field in the public registry listing, which is not org-scoped consumption/error/cost analytics.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Smithery
   Composio~partialprobed6/10

   Composio ships a full OpenAPI-documented API, SDK, CLI, and hosted MCP server covering sessions, auth, triggers, and tool execution (composio-docs-1,9,10; composio-probe-2,3,4), suggesting an API-first architecture where most UI-configurable actions (auth, webhooks, toolkits, white-labeling) have API/CLI equivalents. However there's no explicit documentation asserting full 1:1 parity between the dashboard UI and the API, and a community report shows at least one integration (Instagram connect) failing via the API path with a 400 error, suggesting some workflows may not work as smoothly as advertised. Missing for 10: an explicit parity statement or audit showing every UI feature (e.g. dashboard analytics, project settings) is API-accessible, and independent confirmation that failures like the Instagram case are edge cases rather than systemic gaps.

   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”
   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”
   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “official CLI documented at https://docs.composio.dev/docs/cli”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”

   proof ↗⚑ flag

   Smithery~partialprobed7/10

   Smithery exposes a documented, live OpenAPI 3.1 spec (35 paths) and a keyless registry API, plus a CLI (`smithery mcp add`, `smithery tool call`) that mirrors core UI actions like discovering, connecting, and invoking MCP servers — showing broad API/CLI parity with the UI for the main workflows (search, connect, publish via URL, call tools). However, no evidence explicitly confirms that secondary UI-only features (team/namespace management, trigger configuration, token scoping setup) are fully API-reachable, and llms.txt/docs.md convenience endpoints 404. missing for 10: explicit confirmation that admin/team features (namespaces, triggers, token scoping) are configurable via the API/CLI rather than only the UI, and independent hands-on verification beyond the vendor's own OpenAPI spec.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”
   • [claimed-docs] “Namespaces act as a shared workspace. Resources within a namespace can be managed by anyone with the appropriate API key scoped to that name…”
   • [claimed-docs] “Triggers let an MCP server surface events from its upstream service. When a consumer activates a trigger via Smithery, Smithery proxies the …”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   Composio—none0/10

   No evidence anywhere in the pack of a data export feature, open-format export, or account portability tooling; documentation covers sessions, MCP, auth, CLI, and triggers but nothing about exporting user data or configurations. The axis is fair for a SaaS integration platform holding connection/auth/trigger data, but no evidence supports it.

   ⚑ flag

   Smithery—none0/10

   No evidence anywhere in the pack of a data-export feature, account data portability, or open-format export for users leaving the platform; documentation covers connections, tokens, and MCP server publishing but nothing about exporting stored user data (connections, credentials metadata, namespaces) in an open format.

   ⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   Composio—none0/10

   No evidence of an open-license public source repository; evidence only covers docs, CLI, SDK packages (which are published but not shown to be open-licensed source), and hosted MCP/cloud infrastructure. Composio appears to be primarily a closed hosted SaaS platform, with no mention of a license or public repo for its core source.

   ⚑ flag

   Smithery—none0/10

   Evidence shows only a GitHub repo for the Smithery CLI tool (smithery-gh-1/2/3) but no license information, and the core Smithery platform (registry, hosting, OAuth handling) is described only via docs as a hosted SaaS with no mention of source availability or open licensing. No evidence establishes that Smithery's product source is published under an open license.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [github] “Browse skills on the [Smithery Skills Registry](https://smithery.ai/skills) and install them with the upstream installer”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   Composio—none0/10

   All evidence describes Composio as a hosted/cloud SaaS (sessions, hosted MCP endpoint, cloud webhook delivery) with no documentation or mention of a self-hosted deployment option; community evidence even references 'your cloud solution.' Self-hosting is a fair axis for this type of platform product, but no evidence shows it is supported.

   • [claimed-docs] “session = composio.sessions.create(user_id="user_123")”
   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Composio delivers every event to one destination you control: your webhook URL. You register it once per project, and Composio `POST`s every…”
   • [community] “I tried using your cloud solution to test this and I couldnt pass the connect to Instagram through Composio. I got a 400 error. I checked Re…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Smithery—none0/10

   Evidence describes Smithery only as a hosted registry/platform (registry.smithery.ai, server.smithery.ai) with OAuth-gated hosted MCP endpoints; 'Uplink' lets users expose their own local MCP server as a connection into the hosted Smithery service, but this is not self-hosting the Smithery platform itself. No docs, GitHub repo, or probes indicate a self-hostable/open-source deployment of the core product.

   • [claimed-docs] “Uplink exposes an MCP server running on any machine as a regular Smithery connection.”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://server.smithery.ai/exa/mcp returned HTTP 401 with an OAuth challenge — the hoste…”

   proof ↗⚑ flag

Pricing

1. developerPrototype my agent's tool connections on a meaningful free tier before paying anything

   weight 1 · round drawn
   Composio—none0/10

   The evidence pack contains no mention of pricing tiers, free tier limits, or cost structure for Composio; it only covers technical docs (sessions, MCP, CLI, auth) and unrelated community/security notes. Pricing-plans is a fair axis for a SaaS developer platform, but there is no evidence it delivers a meaningful free tier.

   ⚑ flag

   Smithery—none0/10

   The evidence pack contains extensive documentation on Smithery's features, registry, and integrations but no mention of pricing tiers, a free tier, usage limits, or billing at all — the axis is applicable to a hosted MCP platform but simply unevidenced.

   ⚑ flag

2. developerSee transparent pricing per tool call or connection plus documented rate limits before committing to the platform

   weight 2 · round drawn
   Composio—none0/10

   The evidence pack contains no pricing pages, per-tool-call cost tables, or documented rate limits anywhere in the docs, probes, or community items — only technical usage docs and unrelated auth/security anecdotes. Missing for 10: any pricing page or per-call cost breakdown, any documented rate-limit thresholds, any plan comparison before commitment.

   ⚑ flag

   Smithery—none0/10

   No evidence pack item mentions pricing tiers, per-call costs, or documented rate limits anywhere in Smithery's docs, GitHub, or probes; the documentation covers connection, tokens, and hosting but is silent on pricing/rate limits, and repeated probes (llms.txt, docs.md, openapi) surface no pricing/rate-limit schema.

   ⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   Composio—none0/10

   No evidence in the pack addresses data residency, region selection, or storage location controls for Composio's hosted services; nothing mentions where data is stored or any regional configuration options.

   ⚑ flag

   Smithery—none0/10

   No evidence anywhere in the pack mentions data residency, region selection, or geographic storage controls; Smithery's docs cover credential storage, token scoping, and connection management but not where data is physically stored.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Composio—none0/10

   No evidence in the pack addresses data usage for AI model training, opt-out settings, or privacy controls related to training data; documentation covers sessions, MCP, auth, and CLI but nothing about training-data exclusion policies.

   ⚑ flag

   Smithery—none0/10

   No evidence in the pack addresses AI training data usage, opt-out controls, or any data-privacy policy regarding model training; docs focus on credential storage, OAuth, and connection routing, not training-data opt-out.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Composio—none0/10

   The evidence pack contains no documentation or mention of data retention policies, data deletion APIs, or user-controlled data lifecycle management for Composio — only session/auth/tool-execution mechanics are covered.

   ⚑ flag

   Smithery—none0/10

   The evidence pack covers credential storage encryption and token scoping but contains no mention of data retention policies, user-initiated data deletion, or account/data export controls. This is a fair privacy-posture question for a platform managing credentials and connections, so the absence of evidence yields 'none' rather than 'na'.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Composio—none0/10

   No evidence pack item addresses a telemetry/usage-tracking opt-out mechanism, privacy settings, or data collection policy for Composio; the docs cover sessions, MCP, auth, CLI, and triggers but nothing about telemetry controls.

   ⚑ flag

   Smithery—none0/10

   No evidence anywhere in the pack mentions a telemetry/usage-tracking opt-out setting, privacy policy toggle, or data-collection disclosure for Smithery; the docs cover credential security and OAuth but never address analytics/telemetry opt-out.

   ⚑ flag

Api

1. ai agentQuery the platform's registry or catalog through a documented API to discover servers and tools programmatically

   weight 3 · round to Smithery
   Composio✓fullprobed8/10

   Composio exposes a documented OpenAPI spec (probe confirms HTTP 200 with openapi key), meta-tools for programmatic discovery of toolkits/apps, and a hosted MCP endpoint that agents can query to discover servers/tools, plus an llms.txt describing 1,400+ toolkits. This gives agents multiple documented, programmatic paths (REST/OpenAPI, meta-tools, MCP) to discover the catalog. missing for 10: a dedicated 'registry search/list' endpoint example with request/response schema, and independent third-party confirmation of catalog completeness beyond vendor docs.

   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.composio.dev/llms.txt # Composio Documentation > Composio powers 1,400+ toolkits, tool search, con…”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”

   proof ↗⚑ flag

   Smithery✓fullprobed9/10

   A live probe confirms an agent can query https://registry.smithery.ai/servers keylessly and get a machine-readable JSON catalog (displayName, qualifiedName, verified, etc.), and a documented OpenAPI 3.1 spec with 35 paths is served at smithery.ai/docs/openapi.json, enabling programmatic client generation. This is corroborated by the smithery-cli's 'mcp search'/'tool find' commands for registry/tool discovery. Missing for 10: no independent third-party report of an agent successfully building a full client purely from the OpenAPI spec, and llms.txt/docs.md convenience endpoints 404.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [probe] “PROBE openapi (2026-09-05): GET https://smithery.ai/docs/openapi.json returned HTTP 200 with an OpenAPI 3.1.0 description (35 documented pat…”
   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”

   proof ↗⚑ flag

Metadata

1. developerInspect a listed server's tools, input schemas, transport, and hosting attributes before connecting to it

   weight 2 · round to Smithery
   Composio~partialprobed5/10

   Composio's meta-tools and CLI let an agent/developer discover available toolkits and 'inspect schemas' at runtime (composio-docs-2, composio-docs-10, composio-docs-14), and the exposed openapi.json shows schema-level detail is available. However, there's no evidence of a browsable registry entry per listed server that surfaces transport type or hosting attributes before connecting — discovery is described as a runtime/meta-tool action rather than a pre-connection catalog view. Missing for 10: a documented registry/catalog page showing transport (SSE/stdio/hosted) and hosting attributes per toolkit, and independent confirmation that schema inspection works pre-connection as described.

   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”

   proof ↗⚑ flag

   Smithery~partialprobed6/10

   The registry API (probe-rt-1) exposes machine-readable metadata per server — displayName, qualifiedName, isDeployed, remote, verified — covering hosting/transport attributes, and community security audits (comm-1, comm-2) confirm that tool description fields are visible and inspectable pre-connection (that's how injected instructions were found). The CLI's `tool find` command (gh-2) and per-server typed SDKs generated from tool schemas (docs-16) further support schema visibility, but there is no single documented endpoint or UI view that surfaces full input schemas plus transport plus hosting attributes together in one inspection step. Missing for 10: a documented unified 'server detail' API/UI showing input schemas explicitly, and independent confirmation that transport type (stdio/http/websocket) is surfaced pre-connect.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”
   • [community] “Real examples pulled from Smithery-hosted servers: Context7, Google Sheets, Senzia, and Brave Search tool descriptions contained agent-direc…”

   proof ↗⚑ flag

Playground

1. developerExercise an MCP server's tools from a browser-based inspector or playground before integrating it

   weight 2 · round drawn
   Composio—none0/10

   No evidence pack item describes a browser-based inspector or playground for exercising MCP tools; documentation covers SDK sessions, CLI, meta-tools, and hosted MCP endpoints but never a web UI for pre-integration testing.

   ⚑ flag

   Smithery—none0/10

   The evidence pack documents Smithery's registry, REST connect flow, OAuth/token handling, uplink, deep links, and CLI tool-calling, but nothing describes a browser-based inspector or playground UI for interactively testing a server's tools before integration. Since a registry/marketplace product could plausibly ship such a try-before-integrate UI, the axis applies, but no evidence confirms it exists.

   ⚑ flag

Search

1. developerSearch and browse a large catalog of MCP servers and tools by capability before wiring anything into my agent

   weight 3 · round to Smithery
   Composio~partialprobed6/10

   Composio documents a large catalog (1,400+ toolkits) and built-in 'tool search' plus meta-tools that let an agent discover available tools/capabilities at runtime before execution, and toolkits can be filtered when creating a session. However, the evidence never shows a dedicated developer-facing browse/search UI or registry page for pre-wiring exploration — discovery is framed as an agent-runtime capability (meta-tools) rather than a human catalog browser. Missing for 10: a documented searchable catalog UI/dashboard for developers, independent hands-on confirmation of browsing by capability, and filtering/faceting details beyond toolkit name.

   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [claimed-docs] “To restrict the set, pass `toolkits` when you create the session.”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.composio.dev/llms.txt # Composio Documentation > Composio powers 1,400+ toolkits, tool search, con…”

   proof ↗⚑ flag

   Smithery✓fullprobed8/10

   Smithery ships both a CLI (`smithery mcp search`, `smithery tool find [query]`) and a keyless registry REST API (registry.smithery.ai/servers) that returns a machine-readable catalog with names, descriptions, verification status and usage counts, letting developers search/browse servers and tools by capability before connecting; docs also list example servers (Exa, Context7, Browserbase) illustrating capability-based discovery. Missing for 10: no evidence of a rich web UI faceted-browse/filter-by-category experience or independent hands-on review of the discovery UX itself beyond the CLI/API probes.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [claimed-docs] “Exa - Search the live web, access LinkedIn profiles, do deep research, and more”
   • [claimed-docs] “Context7 - Reference the latest docs for most major SDKs and frameworks directly in Cursor or Claude Code”
   • [claimed-docs] “Browserbase - Control a remote web browser using Stagehand”

   proof ↗⚑ flag

Cli

1. developerDrive the publish-and-deploy workflow from an official CLI — authenticate, stage or build a server, push it, and install it into agents

   weight 2 · round to Smithery
   Composio~partialprobed4/10

   Composio's CLI supports authentication (composio setup) and installing a native plugin into agents like Codex/Claude Code, and sessions can expose a hosted MCP server, but there is no documented workflow for staging/building a custom server artifact and pushing it through the CLI — Composio's MCP server is a pre-built hosted endpoint, not something a developer builds and deploys via CLI. missing for 10: CLI subcommands for building/staging a server, a 'push' or deploy step, and evidence of a full publish pipeline rather than just connect/execute/install.

   • [claimed-docs] “The Composio CLI gives coding agents and terminals a local tool surface. Codex, Claude Code, or a person at the terminal can connect apps, e…”
   • [claimed-docs] “Install the native plugin for Codex or Claude Code: composio setup --target auto”
   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [claimed-docs] “Use Composio Connect when you already have an MCP-compatible client and want the shared Composio MCP URL, without creating an SDK session.”
   • [probe] “official CLI documented at https://docs.composio.dev/docs/cli”

   proof ↗⚑ flag

   Smithery~partialprobed6/10

   Smithery ships an official CLI (@smithery/cli, confirmed installable via npm and version-checked) with commands to search/add/call MCP servers, and docs cover publishing servers (URL method, MCPB bundles, server-card metadata) plus auth via OAuth/service tokens and installing into agents via deep links/uplink. However, evidence does not clearly show a unified CLI-driven build-and-push workflow (e.g., 'smithery build' or 'smithery deploy' subcommands) — the CLI evidence only shows search/add/call, while publish/build steps are documented as web/manual flows rather than CLI-driven. missing for 10: CLI subcommands for authenticate/build/push in one flow, independent hands-on confirmation of end-to-end publish-via-CLI, and clarity on whether install-into-agents is CLI-native vs deep-link/manual.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [github] “Browse skills on the [Smithery Skills Registry](https://smithery.ai/skills) and install them with the upstream installer”
   • [claimed-docs] “If you’ve already deployed an MCP server elsewhere, you can publish it directly on Smithery via the URL method.”
   • [claimed-docs] “For local stdio servers — Smithery distributes a pre-built MCPB bundle that clients download and run locally.”
   • [claimed-docs] “If automatic scanning can’t complete (auth wall, required configuration, or other issues), you can provide server metadata manually via a st…”
   • [claimed-docs] “Deep links provide a seamless way to integrate Smithery MCPs into supported clients. When a user clicks a deep link from our server page, th…”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/smithery/): a JSON-RPC initialize POST to https://smithery.ai/docs/mc…”

   proof ↗⚑ flag

Frameworks

1. developerPlug the platform's tools into major agent frameworks and SDKs (Vercel AI SDK, LangChain, CrewAI, OpenAI or Anthropic SDKs) with documented adapters

   weight 2 · round to Composio
   Composio~partialprobed6/10

   Composio documents a 'provider' concept explicitly designed as adapters that convert Composio tools into the native tool format for a given framework 'so you don't write glue code,' and runtime probes confirm working npm/pip SDK packages, supporting a multi-framework client model. However, the evidence pack never explicitly names or documents adapters for Vercel AI SDK, LangChain, CrewAI, OpenAI, or Anthropic SDKs specifically. Missing for 10: named documentation pages or examples for each specific framework (Vercel AI SDK, LangChain, CrewAI, OpenAI/Anthropic SDKs), and independent hands-on confirmation that these specific adapters work as advertised.

   • [claimed-docs] “A provider is the adapter that turns Composio tools into the native tool format your framework expects, so you don't write glue code.”
   • [probe] “PROBE runtime (recorded 2026-09-04, see data/mcp-infrastructure/proofs/composio/): `npm install @composio/core` and `uv pip install composio…”
   • [probe] “PROBE openapi: HTTP 200 at https://docs.composio.dev/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

   Smithery~partialclaimed5/10

   Smithery documents a dedicated Vercel AI SDK integration with automatic OAuth/token handling and multi-server tool aggregation (docs-13/14), and every published server gets an auto-generated typed SDK (docs-16), but the evidence pack shows no documented adapters for LangChain, CrewAI, or OpenAI/Anthropic SDKs specifically. Missing for 10: documented LangChain adapter, documented CrewAI adapter, documented OpenAI/Anthropic SDK adapter, and independent/hands-on confirmation beyond first-party docs.

   • [claimed-docs] “Smithery handles OAuth, token refresh, and connection management automatically.”
   • [claimed-docs] “Connect to multiple servers and aggregate their tools”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”
   • [claimed-docs] “Smithery gives you a simple REST interface for connecting to MCP servers. Instead of implementing the MCP protocol directly, handling OAuth …”

   proof ↗⚑ flag

Security

1. platform-engineerRely on the platform scanning or inspecting listed servers for malicious or vulnerable behavior before my agents use them

   weight 2 · round to Smithery
   Composio—none0/10

   No evidence that Composio scans, audits, or vets listed toolkits/servers for malicious or vulnerable behavior before agents use them; docs cover sessions, auth, CLI, and MCP connectivity but nothing about security scanning of tool listings. The one security-related item (composio-comm-2) concerns a leaked API key exposing tokens, not proactive vetting of toolkit safety, so it does not support this axis.

   ⚑ flag

   Smithery!disputedcontradicted4/10

   Smithery's docs mention 'automatic scanning' of published servers (smithery-docs-11), implying some vetting pipeline, but independent research directly contradicts its effectiveness: scanning the top 100 registry servers found 22 with security findings (4 critical) including tool-description prompt-injection attacks embedded in widely-used servers like Context7 and Brave Search that execute silently on agent connection (smithery-comm-1, smithery-comm-2), and researchers separately found a path-traversal vulnerability in Smithery's own hosting escalatable to a supply-chain attack (smithery-comm-3). Missing for 10: any documented remediation/removal process for flagged malicious servers, transparency on scan methodology or coverage, and evidence the automatic scanning actually catches behavioral/prompt-injection risks rather than just metadata issues.

   • [claimed-docs] “If automatic scanning can’t complete (auth wall, required configuration, or other issues), you can provide server metadata manually via a st…”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”
   • [community] “Real examples pulled from Smithery-hosted servers: Context7, Google Sheets, Senzia, and Brave Search tool descriptions contained agent-direc…”
   • [community] “Security researchers detailed a path traversal vulnerability in Smithery's MCP server hosting that could be escalated into a supply chain at…”

   proof ↗⚑ flag

Signals

1. developerCompare servers using published quality signals — usage counts, health or liveness checks, scores, or verification badges

   weight 2 · round to Smithery
   Composio—none0/10

   No evidence that Composio publishes usage counts, health/liveness checks, quality scores, or verification badges for toolkits/servers to help developers compare them; documentation covers sessions, meta-tools, auth, and CLI but not curation/quality signals.

   ⚑ flag

   Smithery~partialprobed6/10

   The registry API (probe evidence) exposes useCount and verified fields per server, plus isDeployed/remote status, giving developers concrete usage and verification signals to compare servers; docs also mention automatic scanning during publish which implies some quality vetting. However, there is no documented explicit 'quality score' metric or dedicated health/liveness-check endpoint beyond isDeployed, and independent security scans found unflagged risky servers in the top 100, showing the published signals don't capture all quality concerns. Missing for 10: an explicit scoring/quality-score field, a dedicated liveness/health-check mechanism, and corroboration that verification badges reliably reflect security vetting.

   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [claimed-docs] “If automatic scanning can’t complete (auth wall, required configuration, or other issues), you can provide server metadata manually via a st…”
   • [community] “Scanned the top 100 servers on Smithery's registry; 22 of 100 flagged at least one security finding (28 total, 4 critical, 24 high), most co…”

   proof ↗⚑ flag

Tool search

1. ai agentSearch and load tools at runtime through meta-tools instead of having thousands of tool definitions stuffed into my context

   weight 2 · round to Composio
   Composio✓fullprobed9/10

   Composio's core architecture is exactly meta-tools: a fixed small set (7 meta-tools) that discover, authorize, and execute tools on demand rather than dumping thousands of tool definitions into context, documented in detail with sessions, toolkit-scoping, and MCP delivery, and corroborated by a live probe confirming the hosted meta-tool/MCP endpoint is real. Missing for 10: independent third-party benchmarking of context-size savings or hands-on agent trace showing runtime tool search in action.

   • [claimed-docs] “A session gives your agent meta tools, a small fixed set that discover, authenticate, and execute tools at runtime”
   • [claimed-docs] “To restrict the set, pass `toolkits` when you create the session.”
   • [claimed-docs] “Composio exposes **7 meta-tools** that let the agent discover what's available, authorize apps on demand, and execute tools across apps in p…”
   • [claimed-docs] “Opt into MCP by passing `mcp: true` when you create the session. The session then exposes its hosted MCP server.”
   • [probe] “PROBE mcp-endpoint (2026-09-05): POST initialize to https://connect.composio.dev/mcp returned HTTP 401 with an OAuth challenge — the hosted …”
   • [probe] “official MCP server documented at https://docs.composio.dev/docs/composio-connect”

   proof ↗⚑ flag

   Smithery✓fullprobed7/10

   Smithery's CLI exposes explicit meta-tools — `mcp search`, `tool find [query]`, and `tool call <connection> <tool>` — letting an agent discover and invoke tools by intent at runtime rather than preloading every tool definition, and the registry API is confirmed live and keylessly searchable in a probe returning a machine-readable catalog. missing for 10: no evidence the search/find/call meta-tools are themselves exposed as callable MCP tools inside a live agent session (vs. CLI-level usage), and no independent benchmark showing reduced context footprint from this pattern.

   • [github] “smithery mcp search [term] # Search the Smithery registry smithery mcp add <url> # Add an MCP server connectio…”
   • [github] “smithery tool find [query] # Search tools by name or intent smithery tool call <connection> <tool> [args] # Call a tool”
   • [probe] “PROBE registry-api (2026-09-05): GET https://registry.smithery.ai/servers?pageSize=3 with NO API key returned HTTP 200 with a machine-readab…”
   • [claimed-docs] “Every MCP server published on Smithery gets a typed TypeScript SDK generated from its tool and trigger schemas.”

   proof ↗⚑ flag