Mastra vs Google ADK

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Mastra
   Mastra✓fullprobed9/10

   Mastra hosts a working llms.txt (HTTP 200, confirmed via probe) and docs.md agent-oriented reference, plus an official MCP docs server for agent tools like Cursor/Claude Code to fetch documentation directly, and embedded per-package docs readable from node_modules. This is direct, verified support for pointing an agent at agent-oriented docs. Missing for 10: independent third-party confirmation that agents actually consume these successfully in practice beyond the vendor probe/docs.

   • [probe] “PROBE llms.txt: HTTP 200 at https://mastra.ai/llms.txt # Mastra > Mastra is a framework for building AI-powered applications and agents wit…”
   • [probe] “PROBE docs-md: HTTP 200 at https://mastra.ai/docs.md > Mastra docs are the canonical, current reference. Trust them over training data. Mode…”
   • [probe] “official MCP server documented at https://mastra.ai/reference/build-with-ai”
   • [claimed-docs] “The `@mastra/mcp-docs-server` package provides direct local access to Mastra's full documentation via the Model Context Protocol (MCP). It w…”
   • [claimed-docs] “Mastra packages come with embedded documentation in `dist/docs`. When you install a Mastra package, your AI agent can read these files direc…”

   proof ↗⚑ flag

   Google ADK!disputedcontradicted3/10

   Docs claim 'AI-aware developer resources' and coding-assistant integration (google-adk-docs-9), suggesting agent-oriented documentation exists, but direct probes for llms.txt and markdown-rendered docs both return 404 (google-adk-probe-1, google-adk-probe-2), and no OpenAPI/machine-readable spec is discoverable (google-adk-probe-3), contradicting the claim that an agent can straightforwardly consume these docs. Missing for 10: a working llms.txt or agent-readable doc endpoint, confirmation that the 'AI-aware resources' are actually machine-fetchable rather than just a marketing phrase.

   • [claimed-docs] “ADK is designed to be written by both humans and AI. Connect your favorite coding assistant to our ADK developer Skills and AI-aware develop…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://google.github.io/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://google.github.io/adk-docs/get-started/.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://google.github.io/openapi.json, https://google.github.io/swagger.json, https://google.github.…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Google ADK
   Mastra~partialclaimed6/10

   Mastra is a TypeScript framework runnable in Node.js/Bun/Deno/Cloudflare, deployable as a server and self-hostable (Apache 2.0), which supports headless/CI use, and it exposes cron-scheduled workflows/agents and programmatic workflow results (status/errors) suitable for automation pipelines. However, there is no explicit documentation of a CLI flag or guide for running in CI, no CI/CD pipeline examples, and no dedicated 'headless mode' or automation-testing docs. missing for 10: explicit CI/CD integration guide, documented headless/non-interactive CLI usage, and independent evidence of running Mastra in automated pipelines.

   • [claimed-docs] “Mastra can run against any of these runtime environments: - Node.js `v22.13.0` or later - Bun - Deno - Cloudflare”
   • [claimed-docs] “Mastra applications can be deployed to any Node.js-compatible environment. You can deploy a Mastra server or integrate with an existing web …”
   • [claimed-docs] “Mastra applications can be deployed to any Node.js-compatible environment.”
   • [claimed-docs] “Self host your Mastra projects ... Free Apache 2.0 licensed $0/ month ... Build and host agents anywhere”
   • [claimed-docs] “Declare a `schedule` field on a workflow and Mastra will fire it on the cron you specify.”
   • [claimed-docs] “A schedule runs an agent on a cron cadence.”
   • [claimed-docs] “the result object contains the status and any errors that occurred.”

   proof ↗⚑ flag

   Google ADK✓fullclaimed8/10

   ADK provides a CLI (`adk run`, `adk web`, `adk eval`, `adk deploy docker`) that supports headless invocation and scripted evaluation, plus containerized deployment for CI/production pipelines. missing for 10: explicit CI pipeline examples (e.g. GitHub Actions), independent third-party confirmation of headless CI usage.

   • [github] “adk run path/to/my_agent”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [github] “adk eval \ samples_for_testing/hello_world \ samples_for_testing/hello_world/hello_world_eval_set_001.evalset.json”
   • [github] “adk deploy docker --with_ui <agent-folder>”
   • [claimed-docs] “You can manually package your Agent into a container image and then run it in any environment that supports container images.”
   • [claimed-docs] “This approach involves creating individual test files, each representing a single, simple agent-model interaction (a session).”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round drawn
   Mastra✓fullclaimed8/10

   Mastra's docs explicitly state agents can load tools from remote MCP servers to expand capabilities (mastra-docs-3), directly matching the story of plugging in MCP servers to use their tools. This is corroborated by first-party framework design (agents/tools architecture) though independent hands-on confirmation of MCP client usage specifically is thin. Missing for 10: independent/community hands-on verification of consuming external MCP servers, and more detail on configuration/auth for remote MCP connections.

   • [claimed-docs] “You can also load tools from remote MCP servers to expand an agent's capabilities.”
   • [claimed-docs] “Import `createTool` from `@mastra/core/tools` and define a tool with an `id`, `description`, `inputSchema`, `outputSchema`, and `execute` fu…”
   • [github] “Build autonomous agents that use LLMs and tools to solve open-ended tasks. Agents reason about goals, decide which tools to use, and iterate…”

   proof ↗⚑ flag

   Google ADK✓fullclaimed8/10

   Docs explicitly state an ADK agent can act as an MCP client and use tools provided by external MCP servers, directly matching the story. missing for 10: independent/hands-on corroboration beyond first-party docs, and more detail on multi-server configuration or auth handling.

   • [claimed-docs] “An ADK agent can act as an MCP client and use tools provided by external MCP servers.”
   • [claimed-docs] “Exposing ADK Tools via an MCP Server: How to build an MCP server that wraps ADK tools, making them accessible to any MCP client.”
   • [claimed-docs] “How to build an MCP server that wraps ADK tools, making them accessible to any MCP client.”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round to Mastra
   Mastra✓fullprobed8/10

   Mastra ships an official MCP server (@mastra/mcp-docs-server) documented at mastra.ai/reference/build-with-ai, confirmed by probe, which agents like Cursor, Windsurf, Cline, Claude Code, VS Code, or Codex can connect to, and Mastra also supports authoring MCP servers to expose agents/tools. missing for 10: independent hands-on third-party verification of connecting to the MCP server and broader detail on its full tool surface beyond docs access.

   • [claimed-docs] “The `@mastra/mcp-docs-server` package provides direct local access to Mastra's full documentation via the Model Context Protocol (MCP). It w…”
   • [claimed-docs] “The `@mastra/mcp-docs-server` package provides direct local access to Mastra's full documentation via the Model Context Protocol (MCP).”
   • [probe] “official MCP server documented at https://mastra.ai/reference/build-with-ai”
   • [github] “Author Model Context Protocol servers, exposing agents, tools, and other structured resources via the MCP interface.”
   • [claimed-docs] “You can also load tools from remote MCP servers to expand an agent's capabilities.”

   proof ↗⚑ flag

   Google ADK~partialclaimed6/10

   ADK's official docs explicitly document how to expose ADK tools via an MCP server ('build an MCP server that wraps ADK tools, making them accessible to any MCP client'), showing the framework supports the server side of MCP, not just being an MCP client. However, this is a build-your-own-server guide rather than a turnkey, pre-hosted official MCP endpoint, so it requires developer setup work. Missing for 10: a ready-made hosted/official MCP server endpoint, independent hands-on confirmation that the generated server works reliably with third-party MCP clients.

   • [claimed-docs] “Exposing ADK Tools via an MCP Server: How to build an MCP server that wraps ADK tools, making them accessible to any MCP client.”
   • [claimed-docs] “How to build an MCP server that wraps ADK tools, making them accessible to any MCP client.”
   • [claimed-docs] “An ADK agent can act as an MCP client and use tools provided by external MCP servers.”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to Google ADK
   Mastra~partialclaimed6/10

   Docs reference creating a first agent 'with a single command' and running a local Studio dev server, implying an official CLI (e.g., `mastra dev`), but no evidence pack item explicitly documents CLI subcommands, installation, or full command reference. Missing for 10: explicit CLI command documentation/reference page, list of supported commands, independent hands-on confirmation of CLI usage.

   • [claimed-docs] “Mastra is a TypeScript framework for building AI agents and applications. Create your first agent with a single command and start building.”
   • [claimed-docs] “Create your first agent with a single command and start building.”
   • [claimed-docs] “Open Studio at [`http://localhost:4111`](http://localhost:4111) to test your agent and inspect its runs.”

   proof ↗⚑ flag

   Google ADK✓fullclaimed8/10

   ADK ships an official CLI (`adk run`, `adk web`, `adk eval`, `adk deploy docker`) documented in the GitHub repo with concrete command examples, plus docs reference an "Agents CLI" for scaffolding/build/test/deploy workflows tailored to AI-native/agentic use. Missing for 10: independent third-party hands-on review of the CLI's AI-native ergonomics beyond first-party docs/repo.

   • [github] “adk run path/to/my_agent”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [github] “adk eval \ samples_for_testing/hello_world \ samples_for_testing/hello_world/hello_world_eval_set_001.evalset.json”
   • [github] “adk deploy docker --with_ui <agent-folder>”
   • [claimed-docs] “Go from idea to coded ADK agent in minutes. Use your favorite AI-enabled developer environment to scaffold, build, test, evaluate, and deplo…”
   • [claimed-docs] “Migrate existing agents and workflows to ADK with Agents CLI.”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round drawn
   Mastra~partialprobed6/10

   Mastra's TypeScript framework API (createAgent, createTool, createWorkflow, etc.) is extensively documented and even exposed to AI agents via an MCP docs server and llms.txt/docs.md endpoints, letting an AI-native user drive it programmatically. However, a probe for a standard machine-readable public API spec (OpenAPI/Swagger) returned 404 on all candidate paths, so there's no confirmed formal REST API contract beyond the SDK-level docs. missing for 10: a published OpenAPI/Swagger spec or equivalent formal API contract, independent third-party confirmation of API completeness.

   • [claimed-docs] “Import `createTool` from `@mastra/core/tools` and define a tool with an `id`, `description`, `inputSchema`, `outputSchema`, and `execute` fu…”
   • [claimed-docs] “tools MUST be defined via `createTool()` with `id`, `description`, `inputSchema` (zod), `execute()`”
   • [claimed-docs] “Agents use LLMs and tools to solve open-ended tasks. They reason about goals and decide which tools to use.”
   • [claimed-docs] “Composing **steps** with `createWorkflow` to define the execution flow.”
   • [claimed-docs] “Mastra packages come with embedded documentation in `dist/docs`. When you install a Mastra package, your AI agent can read these files direc…”
   • [probe] “PROBE llms.txt: HTTP 200 at https://mastra.ai/llms.txt # Mastra > Mastra is a framework for building AI-powered applications and agents wit…”
   • [probe] “PROBE docs-md: HTTP 200 at https://mastra.ai/docs.md > Mastra docs are the canonical, current reference. Trust them over training data. Mode…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://mastra.ai/openapi.json, https://mastra.ai/swagger.json, https://mastra.ai/api/openapi.json, …”
   • [probe] “official MCP server documented at https://mastra.ai/reference/build-with-ai”

   proof ↗⚑ flag

   Google ADK~partialprobed6/10

   ADK is a Python framework/CLI (adk run, adk web, adk eval, adk deploy) with documented programmatic APIs for building and driving agents, plus MCP client/server support, but there is no evidence of a formal public REST/OpenAPI-style API surface — probes for openapi/swagger specs and llms.txt all 404. missing for 10: a documented public HTTP/OpenAPI API spec, independent third-party confirmation of programmatic drivability beyond first-party docs.

   • [claimed-docs] “Create your first Python ADK agent in minutes.”
   • [github] “adk run path/to/my_agent”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [github] “adk eval \ samples_for_testing/hello_world \ samples_for_testing/hello_world/hello_world_eval_set_001.evalset.json”
   • [github] “adk deploy docker --with_ui <agent-folder>”
   • [probe] “PROBE openapi: all candidate paths 404 (https://google.github.io/openapi.json, https://google.github.io/swagger.json, https://google.github.…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://google.github.io/llms.txt”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   Mastra—none0/10

   Evidence shows tool-call approval gating (mastra-docs-31) and enterprise RBAC/SSO/IAM controls (mastra-docs-26), but nothing documents a mechanism for issuing scoped or least-privilege API credentials/keys specifically to an agent. Missing for 10: any documentation of per-agent credential scoping, secrets vault integration, or least-privilege API key issuance.

   • [claimed-docs] “Enterprise controls RBAC, SSO, IAM, and network policy integration.”
   • [claimed-docs] “Mark a tool with `requireApproval: true`, then check for the `tool-call-approval` chunk in the stream to approve or decline the action”

   proof ↗⚑ flag

   Google ADK—none0/10

   No evidence in the pack describes issuing scoped or least-privilege API credentials/tokens for agents; the docs cover tools, MCP, workflows, deployment, and evaluation but nothing about credential scoping or permission management for agent identities.

   ⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Google ADK
   Mastra✓fullprobed8/10

   Mastra is itself a first-party TypeScript SDK/framework (@mastra/core, @mastra/mcp-docs-server, etc.) with extensive official documentation covering agents, tools, memory, workflows, and model routing to 40+ providers, and community comments confirm real-world usage building on it as an SDK. Missing for 10: no evidence of official SDKs in other languages (e.g., Python) or a formal API reference/OpenAPI spec (probe found no openapi.json), and independent corroboration is limited to community sentiment rather than technical SDK conformance testing.

   • [claimed-docs] “Mastra is a TypeScript framework for building AI agents and applications. Create your first agent with a single command and start building.”
   • [github] “Model routing: Connect to 40+ providers through one standard interface. Use models from OpenAI, Anthropic, Gemini, and more.”
   • [github] “Build autonomous agents that use LLMs and tools to solve open-ended tasks. Agents reason about goals, decide which tools to use, and iterate…”
   • [claimed-docs] “Import `createTool` from `@mastra/core/tools` and define a tool with an `id`, `description`, `inputSchema`, `outputSchema`, and `execute` fu…”
   • [claimed-docs] “tools MUST be defined via `createTool()` with `id`, `description`, `inputSchema` (zod), `execute()`”
   • [claimed-docs] “Mastra can run against any of these runtime environments: - Node.js `v22.13.0` or later - Bun - Deno - Cloudflare”
   • [community] “Happy Mastra user here! Strikes the right balance between letting me build with higher level abstractions but providing lower level controls…”
   • [community] “I've been building with Mastra for a couple of weeks now and loving it, so congratulations on reaching 1.0! It's built on top of Vercel AI e…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://mastra.ai/openapi.json, https://mastra.ai/swagger.json, https://mastra.ai/api/openapi.json, …”

   proof ↗⚑ flag

   Google ADK✓fullclaimed9/10

   Google ADK is itself an official Python SDK/framework with extensive first-party documentation, code examples, CLI tooling (adk run/web/eval/deploy), and a public GitHub repo, giving AI-native developers a fully documented, official SDK to build against. Minor gap — missing for 10: independent third-party corroboration beyond vendor docs/repo, and llms.txt/OpenAPI probes returned 404s suggesting some machine-readable doc surfaces are incomplete.

   • [claimed-docs] “Create your first Python ADK agent in minutes.”
   • [claimed-docs] “Building an agent with just a model, instructions, and tools is a great place to start for most developers.”
   • [claimed-docs] “agent = Agent( name="researcher", model="gemini-flash-latest", instruction="You help users research topics thoroughly.", too…”
   • [github] “Agent Config: Build agents without code.”
   • [github] “adk run path/to/my_agent”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [claimed-docs] “ADK is designed to be written by both humans and AI. Connect your favorite coding assistant to our ADK developer Skills and AI-aware develop…”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Mastra—none0/10

   Evidence shows PubSub eventing, workflow suspend/resume awaiting an API callback, and cron-based scheduling, but no documentation of a webhook subscription mechanism for AI-native users to register and receive external events. Missing for 10: any explicit webhook registration/subscription API, incoming webhook trigger docs, or example of an agent/workflow subscribing to external webhook events.

   • [claimed-docs] “Pause a workflow at any step to collect additional data, wait for an API callback, throttle a costly operation, or request human-in-the-loop…”
   • [claimed-docs] “Events flow through PubSub, which means a client can disconnect and reconnect without missing chunks.”
   • [claimed-docs] “A schedule runs an agent on a cron cadence.”

   proof ↗⚑ flag

   Google ADK—none0/10

   ADK's evidence shows only in-process callbacks/hooks for agent execution lifecycle, not an external webhook subscription mechanism; no docs mention registering webhook URLs or event push notifications. Missing for 10: any webhook registration API, outbound event delivery docs, or third-party confirmation of webhook support.

   • [claimed-docs] “Callbacks: Hook into specific events during an agent's execution lifecycle to add logging, monitoring, or custom side-effects without alteri…”

   proof ↗⚑ flag

Agentic features

1. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Mastra
   Mastra✓fullclaimed8/10

   Mastra supports scheduled/cron-triggered agents and workflows (mastra-docs-43, mastra-docs-47), durable goals that persist across loop iterations (mastra-docs-46), background tasks that don't block the agentic loop (mastra-docs-45), suspend/resume with persisted state for long-running processes (mastra-gh-4, mastra-docs-12, mastra-docs-39), and self-hostable deployment for continuous background operation (mastra-docs-11, mastra-docs-18). missing for 10: independent/hands-on verification specifically of background/scheduled autonomous runs (community evidence covers general framework use, not background automation specifically), and no evidence of built-in alerting/monitoring for unattended failures.

   • [claimed-docs] “Declare a `schedule` field on a workflow and Mastra will fire it on the cron you specify.”
   • [claimed-docs] “A schedule runs an agent on a cron cadence.”
   • [claimed-docs] “A goal is a durable, thread-scoped objective: a standing instruction the agent keeps working toward across loop iterations until a judge mod…”
   • [claimed-docs] “Background tasks let an agent dispatch a long-running tool call without blocking the agentic loop.”
   • [github] “Suspend an agent or workflow and await user input or approval before resuming. Mastra uses storage to remember execution state, so you can p…”
   • [claimed-docs] “Pause a workflow at any step to collect additional data, wait for an API callback, throttle a costly operation, or request human-in-the-loop…”
   • [claimed-docs] “Snapshots capture all the information needed to resume a workflow from exactly where it left off”
   • [claimed-docs] “Mastra applications can be deployed to any Node.js-compatible environment. You can deploy a Mastra server or integrate with an existing web …”
   • [claimed-docs] “Self host your Mastra projects ... Free Apache 2.0 licensed $0/ month ... Build and host agents anywhere”

   proof ↗⚑ flag

   Google ADK~partialclaimed6/10

   ADK supports deployable, auto-scaling agent runtimes (Cloud Run, GKE, Agent Runtime) and workflow orchestration with retries, state, and scheduling-like execution (fan-out/fan-in, loops), enabling agents to run unattended once deployed. However, evidence does not show explicit scheduling/triggers (e.g., cron-like autonomous kick-off) or a dedicated 'background automation' mode distinct from deployment. missing for 10: explicit trigger/schedule mechanism for autonomous background runs, independent evidence of long-running unattended operation, and confirmation of persistent background execution outside a deploy/response cycle.

   • [claimed-docs] “Agent Runtime is a fully managed auto-scaling service on Google Cloud specifically designed for deploying, managing, and scaling AI agents b…”
   • [claimed-docs] “Cloud Run is a managed auto-scaling compute platform on Google Cloud that enables you to run your agent as a container-based application.”
   • [claimed-docs] “GKE is a good option if you need more control over the deployment as well as for running Open Models.”
   • [github] “Workflow Runtime: A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan…”
   • [github] “A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan-out/fan-in, loops…”
   • [claimed-docs] “In ADK, any agent application that has more than one agent or executable Node is considered a workflow.”

   proof ↗⚑ flag

2. ai-native userOperate the product with natural-language commands

   weight 2 · round to Google ADK
   Mastra~partialprobed4/10

   Mastra is a code-first TypeScript framework; there's no direct evidence of a natural-language command interface for operating the framework itself. Its main AI-native affordances are indirect: an MCP docs-server so coding assistants (Cursor, Claude Code, etc.) can read Mastra docs and scaffold code via NL, and a local Studio UI for inspecting/testing agent runs, but neither is a natural-language 'operate Mastra' interface. missing for 10: a documented NL command/chat interface for controlling the framework itself, evidence of Studio accepting free-form NL operational commands, independent hands-on confirmation of NL-driven operation.

   • [claimed-docs] “The `@mastra/mcp-docs-server` package provides direct local access to Mastra's full documentation via the Model Context Protocol (MCP). It w…”
   • [claimed-docs] “The `@mastra/mcp-docs-server` package provides direct local access to Mastra's full documentation via the Model Context Protocol (MCP).”
   • [claimed-docs] “Mastra packages come with embedded documentation in `dist/docs`. When you install a Mastra package, your AI agent can read these files direc…”
   • [claimed-docs] “Open Studio at [`http://localhost:4111`](http://localhost:4111) to test your agent and inspect its runs.”
   • [probe] “official MCP server documented at https://mastra.ai/reference/build-with-ai”

   proof ↗⚑ flag

   Google ADK~partialclaimed5/10

   ADK docs claim it is designed to be built and operated via AI coding assistants (Agent Config for no-code agent building, 'Agents CLI' for AI-enabled dev environments to scaffold/build/test/deploy) which supports some natural-language-driven operation, but the primary operating surface is a traditional CLI (adk run/web/eval/deploy) and Python code, not direct NL commands to the tool itself. Missing for 10: concrete example of natural-language command controlling ADK end-to-end, independent/hands-on confirmation that Agent Config or coding-assistant integration works as a full NL interface.

   • [claimed-docs] “ADK is designed to be written by both humans and AI. Connect your favorite coding assistant to our ADK developer Skills and AI-aware develop…”
   • [claimed-docs] “Go from idea to coded ADK agent in minutes. Use your favorite AI-enabled developer environment to scaffold, build, test, evaluate, and deplo…”
   • [github] “Agent Config: Build agents without code.”
   • [github] “Agent Config: Build agents without code. Check out the Agent Config feature.”
   • [github] “Build agents without code. Check out the Agent Config feature.”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Mastra—none0/10

   Evidence shows Mastra has markdown-based docs (docs.md, llms.txt) and a local dev Studio for testing agents, but no interactive API reference with runnable/embedded examples (e.g., a Swagger/OpenAPI-style playground) is documented, and probes for OpenAPI specs all 404'd.

   • [probe] “PROBE llms.txt: HTTP 200 at https://mastra.ai/llms.txt # Mastra > Mastra is a framework for building AI-powered applications and agents wit…”
   • [probe] “PROBE docs-md: HTTP 200 at https://mastra.ai/docs.md > Mastra docs are the canonical, current reference. Trust them over training data. Mode…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://mastra.ai/openapi.json, https://mastra.ai/swagger.json, https://mastra.ai/api/openapi.json, …”
   • [claimed-docs] “Open Studio at [`http://localhost:4111`](http://localhost:4111) to test your agent and inspect its runs.”

   proof ↗⚑ flag

   Google ADK—none0/10

   The evidence pack shows standard docs, code snippets, and CLI examples, but no interactive/runnable API reference (e.g., a Swagger/OpenAPI explorer or live code sandbox); probes for openapi.json and similar endpoints explicitly returned 404s.

   • [probe] “PROBE openapi: all candidate paths 404 (https://google.github.io/openapi.json, https://google.github.io/swagger.json, https://google.github.…”
   • [probe] “PROBE docs-md: HTTP 404 at https://google.github.io/adk-docs/get-started/.md”
   • [claimed-docs] “agent = Agent( name="researcher", model="gemini-flash-latest", instruction="You help users research topics thoroughly.", too…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   Mastra—none0/10

   Mastra deploys servers/agents but explicit probes for OpenAPI/swagger endpoints all returned 404, and no docs mention a downloadable machine-readable API spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://mastra.ai/openapi.json, https://mastra.ai/swagger.json, https://mastra.ai/api/openapi.json, …”

   proof ↗⚑ flag

   Google ADK—none0/10

   No evidence of a downloadable OpenAPI/machine-readable spec for ADK; explicit probes for openapi.json/swagger.json and llms.txt all return 404, indicating no such spec is published.

   • [probe] “PROBE llms.txt: HTTP 404 at https://google.github.io/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://google.github.io/adk-docs/get-started/.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://google.github.io/openapi.json, https://google.github.io/swagger.json, https://google.github.…”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Google ADK
   Mastra~partialclaimed4/10

   Mastra supports local self-hosting and running against separate dev/runtime environments (mastra-docs-11, mastra-docs-18), plus a local Studio for testing agents at localhost:4111 (mastra-docs-50), which implies developers can iterate without touching production. However there is no explicit sandbox/staging environment feature, no documented separation of test vs production data stores, and no first-party 'sandbox mode' or test-data isolation guidance. missing for 10: explicit sandbox environment/test-data isolation feature, documented staging vs production separation, independent confirmation of safe non-prod testing.

   • [claimed-docs] “Self host your Mastra projects ... Free Apache 2.0 licensed $0/ month ... Build and host agents anywhere”
   • [claimed-docs] “Mastra applications can be deployed to any Node.js-compatible environment. You can deploy a Mastra server or integrate with an existing web …”
   • [claimed-docs] “Open Studio at [`http://localhost:4111`](http://localhost:4111) to test your agent and inspect its runs.”
   • [claimed-docs] “Mastra can run against any of these runtime environments: - Node.js `v22.13.0` or later - Bun - Deno - Cloudflare”

   proof ↗⚑ flag

   Google ADK~partialclaimed5/10

   ADK supports local dev/test workflows (adk run, adk web, adk eval, local evaluation with test files and eval sets) that inherently run against a local/dev environment rather than production, and offline/disconnected deployment is mentioned. However, there's no explicit documentation of a dedicated 'sandbox' environment or data isolation guarantee distinct from production. missing for 10: explicit sandbox/staging environment docs, explicit statement that test runs are isolated from production data/state, independent confirmation of this isolation.

   • [claimed-docs] “This approach involves creating individual test files, each representing a single, simple agent-model interaction (a session).”
   • [claimed-docs] “This approach involves creating individual test files, each representing a single, simple agent-model interaction (a session). It's most eff…”
   • [claimed-docs] “Expected Intermediate Tool Use Trajectory: The tool calls we expect the agent to make in order to respond correctly to the user query.”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [github] “adk eval \ samples_for_testing/hello_world \ samples_for_testing/hello_world/hello_world_eval_set_001.evalset.json”
   • [claimed-docs] “This is a good option if you prefer to run offline or disconnected, or otherwise in a system that has no connection to Google Cloud.”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Mastra—none0/10

   No evidence of a versioned API scheme or documented deprecation policy; OpenAPI spec probes all 404 and no docs reference API versioning or deprecation practices.

   • [probe] “PROBE openapi: all candidate paths 404 (https://mastra.ai/openapi.json, https://mastra.ai/swagger.json, https://mastra.ai/api/openapi.json, …”

   proof ↗⚑ flag

   Google ADK—none0/10

   No evidence in the pack mentions API versioning schemes or a documented deprecation policy for ADK; probes for OpenAPI specs and related docs all returned 404s. missing for 10: versioning scheme documentation, deprecation policy, changelog/migration guides, semantic versioning commitments.

   • [probe] “PROBE llms.txt: HTTP 404 at https://google.github.io/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://google.github.io/adk-docs/get-started/.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://google.github.io/openapi.json, https://google.github.io/swagger.json, https://google.github.…”

   proof ↗⚑ flag

Agent authoring

1. developerDefine an agent with typed custom tools in a few lines of code

   weight 3 · round to Mastra
   Mastra✓fullclaimed9/10

   Mastra provides a clear createTool() API with typed inputSchema/outputSchema (zod) and execute function, directly attachable to agents, matching the 'typed custom tools in a few lines of code' story; docs show agent creation is a single command plus tool wiring is minimal boilerplate. missing for 10: independent hands-on code sample demonstrating the exact few-lines flow rather than just docs description.

   • [claimed-docs] “Import `createTool` from `@mastra/core/tools` and define a tool with an `id`, `description`, `inputSchema`, `outputSchema`, and `execute` fu…”
   • [claimed-docs] “tools MUST be defined via `createTool()` with `id`, `description`, `inputSchema` (zod), `execute()`”
   • [claimed-docs] “Agents use LLMs and tools to solve open-ended tasks. They reason about goals and decide which tools to use.”
   • [claimed-docs] “Agents use tools to call APIs or query databases.”
   • [claimed-docs] “Create your first agent with a single command and start building.”

   proof ↗⚑ flag

   Google ADK~partialclaimed6/10

   The docs show a concrete few-line example of defining an Agent with a model, instructions, and a tools list (google-adk-docs-22), and other docs confirm tools are a core, optional component of agent definition (google-adk-docs-2, google-adk-docs-13). However, the evidence never shows a custom Python tool function with type hints/typed parameters being defined and passed in — only a prebuilt tool (google_search) is used in the example. Missing for 10: an explicit example of writing a custom typed tool function, and documentation of automatic schema/type inference from function signatures.

   • [claimed-docs] “agent = Agent( name="researcher", model="gemini-flash-latest", instruction="You help users research topics thoroughly.", too…”
   • [claimed-docs] “Building an agent with just a model, instructions, and tools is a great place to start for most developers.”
   • [claimed-docs] “The basic components of an Agent are an artificial intelligence (AI) model, task instructions, and optionally, a set of tools to be used by …”

   proof ↗⚑ flag

Ai buildability

1. ai-native userHave a coding agent scaffold a new agent project from an official CLI or template in one command

   weight 2 · round to Google ADK
   Mastra~partialclaimed6/10

   Docs explicitly state 'Create your first agent with a single command and start building,' indicating an official CLI/one-command scaffolding path, and Mastra also ships embedded docs/MCP docs-server so coding agents can understand its APIs. However, there's no explicit evidence of a dedicated scaffold template repo, no hands-on/community confirmation of the CLI experience for agent-driven scaffolding, and no detail on flags/templates variety. Missing for 10: independent confirmation of the one-command scaffold working end-to-end, details on official templates, and evidence of an agent (not just a human) invoking the CLI successfully.

   • [claimed-docs] “Mastra is a TypeScript framework for building AI agents and applications. Create your first agent with a single command and start building.”
   • [claimed-docs] “Create your first agent with a single command and start building.”
   • [claimed-docs] “Mastra packages come with embedded documentation in `dist/docs`. When you install a Mastra package, your AI agent can read these files direc…”
   • [claimed-docs] “The `@mastra/mcp-docs-server` package provides direct local access to Mastra's full documentation via the Model Context Protocol (MCP). It w…”

   proof ↗⚑ flag

   Google ADK✓fullclaimed8/10

   ADK docs explicitly advertise an official 'Agents CLI' to scaffold, build, test, evaluate, and deploy agents in minutes, and the GitHub README shows concrete one-line commands (adk run, adk web, adk deploy) plus a no-code 'Agent Config' template feature for scaffolding agents. This directly matches the ai-native scaffolding story via an official CLI/template workflow. Missing for 10: independent/hands-on confirmation of the one-command scaffold experience beyond first-party docs.

   • [claimed-docs] “Go from idea to coded ADK agent in minutes. Use your favorite AI-enabled developer environment to scaffold, build, test, evaluate, and deplo…”
   • [claimed-docs] “Migrate existing agents and workflows to ADK with Agents CLI.”
   • [github] “adk run path/to/my_agent”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [github] “Agent Config: Build agents without code. Check out the Agent Config feature.”
   • [github] “Build agents without code. Check out the Agent Config feature.”

   proof ↗⚑ flag

2. ai-native userRun the framework's example agents headlessly from a terminal so an agent can verify what it just built

   weight 2 · round to Google ADK
   Mastra—none0/10

   The evidence shows a single-command project scaffold (mastra-docs-1/13) and a web-based Studio for testing agents at localhost:4111 (mastra-docs-50), but nothing documents a headless terminal invocation of example agents for automated self-verification. missing for 10: a documented CLI command to run/test agents non-interactively, evidence of scripted/headless agent execution, and confirmation this works without the Studio UI.

   • [claimed-docs] “Mastra is a TypeScript framework for building AI agents and applications. Create your first agent with a single command and start building.”
   • [claimed-docs] “Open Studio at [`http://localhost:4111`](http://localhost:4111) to test your agent and inspect its runs.”
   • [claimed-docs] “Mastra can run against any of these runtime environments: - Node.js `v22.13.0` or later - Bun - Deno - Cloudflare”

   proof ↗⚑ flag

   Google ADK✓fullclaimed7/10

   ADK provides a documented CLI (`adk run path/to/my_agent`) to run agents headlessly from a terminal, plus `adk eval` for automated verification of agent behavior against eval sets, matching the 'verify what it just built' use case for an ai-native/agentic workflow. Missing for 10: explicit confirmation that shipped 'example agents' (vs. user-authored ones) work with this flow, and independent/hands-on corroboration beyond the official repo docs.

   • [github] “adk run path/to/my_agent”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [github] “adk eval \ samples_for_testing/hello_world \ samples_for_testing/hello_world/hello_world_eval_set_001.evalset.json”
   • [claimed-docs] “This approach involves creating individual test files, each representing a single, simple agent-model interaction (a session).”

   proof ↗⚑ flag

3. ai-native userRely on strict typing and schema validation so a coding agent catches its own mistakes at build time

   weight 2 · round to Mastra
   Mastra✓fullclaimed6/10

   Mastra is a TypeScript-native framework where tools must be defined via createTool() with typed inputSchema/outputSchema (Zod), agents support structured output matching a schema, and workflow steps use type-safe logic — giving strong compile-time/schema-level guarantees an agent's tool calls and outputs conform to expected shapes. Missing for 10: explicit documentation framing this as 'catching mistakes at build time,' independent/hands-on evidence confirming compile-time error catching in practice, and detail on how validation failures are surfaced to the agent.

   • [claimed-docs] “Import `createTool` from `@mastra/core/tools` and define a tool with an `id`, `description`, `inputSchema`, `outputSchema`, and `execute` fu…”
   • [claimed-docs] “tools MUST be defined via `createTool()` with `id`, `description`, `inputSchema` (zod), `execute()`”
   • [claimed-docs] “Structured output lets an agent return an object that matches the shape defined by a schema instead of returning text.”
   • [claimed-docs] “Workflow steps can call agents to use LLM reasoning or call tools for type-safe logic.”
   • [claimed-docs] “the result object contains the status and any errors that occurred.”

   proof ↗⚑ flag

   Google ADK—none0/10

   The evidence pack covers ADK's agent orchestration, deployment, and evaluation features, but contains no mention of strict typing, schema validation, or build-time error detection for tool/agent definitions — the evaluation features described (docs-20, docs-21, docs-25) are runtime test-set based, not compile/build-time type checks.

   ⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Mastra
   Mastra~partialclaimed4/10

   Mastra provides workflow primitives like `.parallel()` for simultaneous step execution and background/async tool dispatch that could be used to build bulk-processing pipelines, but there is no dedicated 'bulk operation' feature, batch API, or documentation describing operating over many items at once as a first-class capability. missing for 10: explicit bulk/batch API or UI for operating on many items simultaneously, documented examples of bulk item processing, independent evidence of this pattern being used in practice.

   • [claimed-docs] “Use `.parallel()` to run steps simultaneously.”
   • [github] “use Mastra's graph-based workflow engine to orchestrate complex multi-step processes. Mastra workflows use an intuitive syntax for control f…”
   • [claimed-docs] “Background tasks let an agent dispatch a long-running tool call without blocking the agentic loop.”
   • [claimed-docs] “Composing **steps** with `createWorkflow` to define the execution flow.”

   proof ↗⚑ flag

   Google ADK~partialclaimed3/10

   ADK's Workflow Runtime offers fan-out/fan-in and loop constructs that could be used by developers to build bulk-item processing pipelines, but there is no documented built-in 'bulk operations' feature or example for end users acting across many items at once. Missing for 10: explicit bulk-operation tooling/UI, documented examples of processing many items in one call, and evidence of end-user (not just developer-framework) bulk workflows.

   • [github] “Workflow Runtime: A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan…”
   • [claimed-docs] “you can use the ADK development framework to expand them into workflows, which allow you to combine and orchestrate multiple agents and code…”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Google ADK
   Mastra~partialclaimed6/10

   Mastra supports automation triggers via cron-scheduled workflows/agents ("Declare a schedule field on a workflow and Mastra will fire it on the cron you specify", "A schedule runs an agent on a cron cadence") and event-driven execution through its PubSub system and background tasks, which allow actions to fire without manual intervention. However, evidence doesn't show a general-purpose rule/trigger engine for arbitrary custom events (e.g., webhook-based or condition-based triggers beyond cron/schedule), so the automation-depth story is only partially evidenced. Missing for 10: documentation of arbitrary event-trigger definitions (not just cron schedules), webhook/external-event triggers, and independent/hands-on confirmation of this automation behavior in production.

   • [claimed-docs] “Declare a `schedule` field on a workflow and Mastra will fire it on the cron you specify.”
   • [claimed-docs] “A schedule runs an agent on a cron cadence.”
   • [claimed-docs] “Events flow through PubSub, which means a client can disconnect and reconnect without missing chunks.”
   • [claimed-docs] “Background tasks let an agent dispatch a long-running tool call without blocking the agentic loop.”
   • [claimed-docs] “Use dynamic workflows when users, agents, visual editors, or external systems need to create workflows without changing application code or …”

   proof ↗⚑ flag

   Google ADK✓fullclaimed7/10

   ADK explicitly supports event-driven automation via Callbacks ("Hook into specific events during an agent's execution lifecycle... without altering core agent logic") and a Workflow Runtime graph engine with routing, retry, fan-out/fan-in and dynamic nodes for triggering actions on execution events, matching the story of defining rules that fire on events. missing for 10: independent/hands-on evidence of callback-triggered rules in production use, and more detail on condition-based rule syntax beyond docs summaries.

   • [claimed-docs] “Callbacks: Hook into specific events during an agent's execution lifecycle to add logging, monitoring, or custom side-effects without alteri…”
   • [github] “Workflow Runtime: A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan…”
   • [github] “A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan-out/fan-in, loops…”
   • [claimed-docs] “In ADK, any agent application that has more than one agent or executable Node is considered a workflow.”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round to Mastra
   Mastra✓fullclaimed7/10

   Mastra docs explicitly support cron-based scheduling: workflows can declare a `schedule` field that fires on a specified cron, and agents can likewise be run on a cron cadence, directly enabling recurring jobs/workflows. Missing for 10: independent/hands-on corroboration of scheduling in production, and details on schedule management (pause/resume, monitoring) beyond the single doc lines.

   • [claimed-docs] “Declare a `schedule` field on a workflow and Mastra will fire it on the cron you specify.”
   • [claimed-docs] “A schedule runs an agent on a cron cadence.”
   • [github] “use Mastra's graph-based workflow engine to orchestrate complex multi-step processes. Mastra workflows use an intuitive syntax for control f…”
   • [claimed-docs] “Workflows let you define complex task sequences with clear, structured steps instead of relying on one agent to reason through the entire pr…”

   proof ↗⚑ flag

   Google ADK—none0/10

   The evidence covers agent/workflow orchestration, deployment targets (Cloud Run, GKE, Agent Runtime), and evaluation, but nothing describes scheduling, cron-like triggers, or recurring execution of jobs/workflows. Absence of evidence for this applicable automation-depth capability yields 'none'.

   ⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round drawn
   Mastra—none0/10

   Mastra is a developer framework for building agents/workflows, not an automations product with version history, review/approval, or rollback of automations themselves. Evidence covers workflow suspend/resume, snapshots, and time-travel re-execution of workflow steps, but there is no evidence of versioning automation definitions, a review/approval workflow for changes, or rolling back to a prior automation version. missing for 10: version control of automation/workflow definitions, change review/approval process, rollback to previous automation versions.

   • [github] “Suspend an agent or workflow and await user input or approval before resuming. Mastra uses storage to remember execution state, so you can p…”
   • [claimed-docs] “Pause a workflow at any step to collect additional data, wait for an API callback, throttle a costly operation, or request human-in-the-loop…”
   • [claimed-docs] “Snapshots capture all the information needed to resume a workflow from exactly where it left off”
   • [claimed-docs] “Time travel allows you to re-execute a workflow starting from any specific step, using either stored snapshot data or custom context you pro…”

   proof ↗⚑ flag

   Google ADK—none0/10

   ADK is a framework for building agents (code, workflows, tools, deployment) but the evidence pack shows no version control, review, or rollback mechanism for automations themselves — no changelog/versioning UI, no approval/review workflow for agent definitions, no rollback feature. Agent code could theoretically be tracked via external git, but ADK itself provides no such capability in the evidence. Missing for 10: any versioning system, review/approval workflow, or rollback capability for automations.

   ⚑ flag

Deployment

1. engineering-leadDeploy an agent to a managed runtime and call it as an API endpoint

   weight 2 · round to Google ADK
   Mastra~partialprobed6/10

   Mastra docs confirm agents can be deployed as a server/API and self-hosted on Node.js-compatible runtimes (mastra-docs-18, mastra-docs-25, mastra-docs-11), and Studio provides local run/test endpoints (mastra-docs-50). However there's no first-party 'managed runtime' (Mastra Cloud/PaaS) evidence in this pack, no documented deployment API spec (openapi probe returned 404s), and no independent confirmation of a hosted call-as-API-endpoint experience — missing for 10: managed/hosted runtime offering, official deployment API reference, independent hands-on verification of calling a deployed agent as an endpoint.

   • [claimed-docs] “Mastra applications can be deployed to any Node.js-compatible environment. You can deploy a Mastra server or integrate with an existing web …”
   • [claimed-docs] “Mastra applications can be deployed to any Node.js-compatible environment.”
   • [claimed-docs] “Self host your Mastra projects ... Free Apache 2.0 licensed $0/ month ... Build and host agents anywhere”
   • [claimed-docs] “Open Studio at [`http://localhost:4111`](http://localhost:4111) to test your agent and inspect its runs.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://mastra.ai/openapi.json, https://mastra.ai/swagger.json, https://mastra.ai/api/openapi.json, …”

   proof ↗⚑ flag

   Google ADK✓fullclaimed8/10

   ADK docs explicitly describe deploying agents to a fully managed, auto-scaling Agent Engine/Agent Runtime on Google Cloud, plus alternative managed options like Cloud Run and GKE, with the stated purpose being to make the agent 'accessed, queried, and used in production' as an API endpoint. Missing for 10: no explicit hands-on/independent confirmation of the API contract (e.g., request/response schema) or third-party verification of endpoint behavior beyond first-party docs.

   • [claimed-docs] “Agent Runtime is a fully managed auto-scaling service on Google Cloud specifically designed for deploying, managing, and scaling AI agents b…”
   • [claimed-docs] “Cloud Run is a managed auto-scaling compute platform on Google Cloud that enables you to run your agent as a container-based application.”
   • [claimed-docs] “GKE is a good option if you need more control over the deployment as well as for running Open Models.”
   • [claimed-docs] “Once you've built and tested your agent using ADK, the next step is to deploy it so it can be accessed, queried, and used in production”
   • [claimed-docs] “You can manually package your Agent into a container image and then run it in any environment that supports container images.”

   proof ↗⚑ flag

2. engineering-leadRun my agents entirely on my own infrastructure with no dependence on the vendor's platform

   weight 2 · round drawn
   Mastra✓fullcommunity8/10

   Mastra is Apache 2.0 licensed, self-hostable for $0/month, deployable to any Node.js-compatible environment or runtime (Node, Bun, Deno, Cloudflare), and explicitly markets 'build and host agents anywhere' with no vendor lock-in for core hosting. missing for 10: no independent case study confirming a production fully self-hosted deployment, and a community note flags the license restricts reselling as a hosted service (not a self-hosting restriction, but a licensing nuance worth noting).

   • [claimed-docs] “Self host your Mastra projects ... Free Apache 2.0 licensed $0/ month ... Build and host agents anywhere”
   • [claimed-docs] “Build and host agents anywhere”
   • [claimed-docs] “Mastra can run against any of these runtime environments: - Node.js `v22.13.0` or later - Bun - Deno - Cloudflare”
   • [claimed-docs] “Mastra applications can be deployed to any Node.js-compatible environment. You can deploy a Mastra server or integrate with an existing web …”
   • [claimed-docs] “Mastra applications can be deployed to any Node.js-compatible environment.”
   • [community] “"You may not provide the software to third parties as a hosted or managed service" - The Elastic v2 license isn't actually open source like …”

   proof ↗⚑ flag

   Google ADK✓fullclaimed8/10

   ADK is an open-source framework (google/adk-python) that supports running agents locally via `adk run`/`adk web`, packaging into containers with `adk deploy docker`, deploying to any container-supporting environment, and explicitly documents an offline/disconnected mode with no Google Cloud connection required, alongside optional managed services like Agent Runtime/Cloud Run/GKE. missing for 10: independent/hands-on confirmation of fully vendor-free operation (e.g., third-party report of running ADK completely offline with non-Google models) and clarity on whether any telemetry/model calls still phone home by default.

   • [claimed-docs] “This is a good option if you prefer to run offline or disconnected, or otherwise in a system that has no connection to Google Cloud.”
   • [claimed-docs] “You can manually package your Agent into a container image and then run it in any environment that supports container images.”
   • [github] “adk deploy docker --with_ui <agent-folder>”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [claimed-docs] “ADK can work with almost any generative AI model. The framework provides easy access to Gemini as well as other leading models, and we provi…”
   • [claimed-docs] “GKE is a good option if you need more control over the deployment as well as for running Open Models.”

   proof ↗⚑ flag

Portability

1. developerSwap the underlying LLM provider or model without rewriting my agent

   weight 3 · round drawn
   Mastra✓fullclaimed8/10

   Mastra documents model routing through a single standard interface connecting to 40+ providers (OpenAI, Anthropic, Gemini, etc.), which directly enables swapping the underlying LLM without rewriting agent logic, and community feedback corroborates ease of building agents this way. missing for 10: no explicit hands-on example showing a provider swap in an existing agent config, and no independent benchmark/confirmation of zero-code-change swaps.

   • [github] “Model routing: Connect to 40+ providers through one standard interface. Use models from OpenAI, Anthropic, Gemini, and more.”
   • [github] “Connect to 40+ providers through one standard interface. Use models from OpenAI, Anthropic, Gemini, and more.”
   • [claimed-docs] “Agents use LLMs and tools to solve open-ended tasks. They reason about goals and decide which tools to use.”

   proof ↗⚑ flag

   Google ADK✓fullclaimed8/10

   Docs explicitly state ADK works with 'almost any generative AI model,' providing adapters for Gemini and many other model providers including locally running models, and the agent definition just takes a `model` string parameter (e.g., 'gemini-flash-latest'), implying swapping providers is a config change rather than a rewrite. Missing for 10: independent/hands-on confirmation that switching providers requires no code changes to agent logic, and no explicit list of supported non-Gemini providers with concrete migration examples.

   • [claimed-docs] “ADK can work with almost any generative AI model. The framework provides easy access to Gemini as well as other leading models, and we provi…”
   • [claimed-docs] “agent = Agent( name="researcher", model="gemini-flash-latest", instruction="You help users research topics thoroughly.", too…”
   • [claimed-docs] “The basic components of an Agent are an artificial intelligence (AI) model, task instructions, and optionally, a set of tools to be used by …”

   proof ↗⚑ flag

Evals

1. engineering-leadScore agent quality with built-in evals and run them as part of CI

   weight 2 · round to Google ADK
   Mastra~partialclaimed5/10

   Mastra ships built-in evals/scorers that score agent quality using model-graded, rule-based, and statistical methods, plus live evaluations during runtime (mastra-docs-24, mastra-docs-49, mastra-docs-7). However, there is no evidence describing how to run these evals as part of a CI pipeline (e.g., CLI test runner, GitHub Actions integration, pass/fail gating). Missing for 10: documentation or examples of invoking scorers/evals in CI, CI-specific tooling or exit-code/test-runner support, and independent confirmation of CI usage.

   • [claimed-docs] “Live evaluations allow you to automatically score AI outputs in real-time as your agents and workflows operate.”
   • [claimed-docs] “Scorers help bridge this gap by providing quantifiable metrics for measuring agent quality.”
   • [claimed-docs] “Scorers are automated tests that evaluate Agents outputs using model-graded, rule-based, and statistical methods.”
   • [claimed-docs] “Mastra's observability system gives you visibility into every agent run, workflow step, tool call, and model interaction.”

   proof ↗⚑ flag

   Google ADK✓fullclaimed8/10

   ADK ships a first-party evaluation framework with groundtruth and rubric-based metrics, expected tool-use trajectories, evalset.json test files, and a documented CLI command (`adk eval <agent> <evalset>`) that can be scripted/invoked headlessly, which is exactly the shape needed for CI integration. Missing for 10: explicit CI/CD pipeline documentation (e.g., a GitHub Actions example) and independent/third-party corroboration of running adk eval in CI.

   • [claimed-docs] “This approach involves creating individual test files, each representing a single, simple agent-model interaction (a session).”
   • [claimed-docs] “ADK provides both groundtruth based and rubric based tool use evaluation metrics.”
   • [claimed-docs] “This approach involves creating individual test files, each representing a single, simple agent-model interaction (a session). It's most eff…”
   • [claimed-docs] “Expected Intermediate Tool Use Trajectory: The tool calls we expect the agent to make in order to respond correctly to the user query.”
   • [github] “adk eval \ samples_for_testing/hello_world \ samples_for_testing/hello_world/hello_world_eval_set_001.evalset.json”

   proof ↗⚑ flag

Testing

1. developerUnit-test agents with mocked models and tools

   weight 2 · round to Google ADK
   Mastra—none0/10

   Evidence shows Mastra has evals/scorers (mastra-docs-24, mastra-docs-49) and a local Studio for inspecting agent runs (mastra-docs-50), but nothing describes unit-testing agents with mocked models or mocked tools, dependency injection for models, or test utilities/harnesses for isolating agent logic from real LLM calls.

   ⚑ flag

   Google ADK~partialclaimed5/10

   ADK docs describe a test-file based evaluation approach explicitly described as 'a form of unit testing' for single agent-model interactions, with expected tool-use trajectories and groundtruth/rubric metrics plus an `adk eval` CLI — but none of this evidence explicitly describes mocking models or tools (e.g., swapping in fake LLM responses or stub tool implementations) for isolated unit tests. Missing for 10: explicit mocked-model/mocked-tool test fixtures or APIs, independent/hands-on confirmation of mocking support, and unit-test framework integration examples (e.g., pytest with mock objects).

   • [claimed-docs] “This approach involves creating individual test files, each representing a single, simple agent-model interaction (a session).”
   • [claimed-docs] “This approach involves creating individual test files, each representing a single, simple agent-model interaction (a session). It's most eff…”
   • [claimed-docs] “Expected Intermediate Tool Use Trajectory: The tool calls we expect the agent to make in order to respond correctly to the user query.”
   • [claimed-docs] “ADK provides both groundtruth based and rubric based tool use evaluation metrics.”
   • [github] “adk eval \ samples_for_testing/hello_world \ samples_for_testing/hello_world/hello_world_eval_set_001.evalset.json”

   proof ↗⚑ flag

Tracing

1. developerTrace every LLM call and tool invocation of an agent run in an observability UI

   weight 3 · round to Mastra
   Mastra✓fullclaimed8/10

   Docs explicitly describe an observability system giving visibility into every agent run, workflow step, tool call, and model interaction, plus a local Studio UI at localhost:4111 to inspect agent runs, matching the story closely. Missing for 10: no independent/hands-on corroboration of the observability UI's tracing depth, and no detail on trace export/integration with third-party observability backends.

   • [claimed-docs] “Mastra's observability system gives you visibility into every agent run, workflow step, tool call, and model interaction.”
   • [claimed-docs] “Open Studio at [`http://localhost:4111`](http://localhost:4111) to test your agent and inspect its runs.”
   • [claimed-docs] “Live evaluations allow you to automatically score AI outputs in real-time as your agents and workflows operate.”

   proof ↗⚑ flag

   Google ADK~partialclaimed5/10

   ADK ships a built-in development Web UI explicitly for testing, evaluating, and debugging agents, and provides callbacks to hook into execution lifecycle events for logging/monitoring, which together imply some run-level visibility into tool and model calls. However, the evidence never explicitly describes a trace view showing each LLM call and tool invocation of a run, nor mentions integration with tracing standards (e.g., OpenTelemetry) or a dedicated observability dashboard beyond the dev/eval UI. Missing for 10: explicit documentation of per-call tracing UI, tool-invocation-level trace inspection, and any third-party/hands-on confirmation of this granularity.

   • [github] “A built-in development UI to help you test, evaluate, debug, and showcase your agent(s).”
   • [github] “Web UI (supports multi-agent directories or pointing directly to a single agent folder)”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [claimed-docs] “Callbacks: Hook into specific events during an agent's execution lifecycle to add logging, monitoring, or custom side-effects without alteri…”

   proof ↗⚑ flag

Guardrails

1. developerAttach input/output guardrails that validate, transform, or block unsafe content

   weight 3 · round to Mastra
   Mastra✓fullclaimed7/10

   Mastra docs explicitly describe 'Processors' that transform, validate, or control messages passing through an agent, plus a PromptInjectionDetector for scanning/blocking unsafe input, and tool-call approval gating via requireApproval. This directly matches input/output guardrail validation, transformation, and blocking. missing for 10: no independent/hands-on corroboration of guardrail behavior, no detail on output-side blocking/transform examples, and no evidence of configurable custom guardrail policies beyond the named built-ins.

   • [claimed-docs] “The `PromptInjectionDetector()` scans user messages for prompt injection, jailbreak attempts, and system override patterns.”
   • [claimed-docs] “Processors transform, validate, or control messages as they pass through an agent.”
   • [claimed-docs] “Mark a tool with `requireApproval: true`, then check for the `tool-call-approval` chunk in the stream to approve or decline the action”

   proof ↗⚑ flag

   Google ADK~partialclaimed5/10

   ADK exposes general extensibility hooks—Callbacks to intercept execution events for custom logic/side-effects, Plugins for pre-packaged behaviors, and a Tool Confirmation (HITL) flow that can guard tool execution—which developers could use to build input/output guardrails, but there is no dedicated 'guardrails' feature, built-in content-safety/validation API, or example showing blocking/transforming unsafe content end-to-end. Missing for 10: explicit guardrail/validation API or moderation integration, documented examples of blocking/transforming unsafe input or output, and any third-party/community confirmation of this pattern in practice.

   • [claimed-docs] “Callbacks: Hook into specific events during an agent's execution lifecycle to add logging, monitoring, or custom side-effects without alteri…”
   • [claimed-docs] “Plugins: Integrate complex, pre-packaged behaviors and third-party services directly into your agent's workflow.”
   • [github] “Tool Confirmation: A tool confirmation flow (HITL) that can guard tool execution with explicit confirmation and custom input.”
   • [github] “A tool confirmation flow (HITL) that can guard tool execution with explicit confirmation and custom input.”

   proof ↗⚑ flag

2. engineering-leadRestrict what an agent may do with fine-grained tool permissions and sandboxed execution

   weight 2 · round to Mastra
   Mastra✓fullclaimed7/10

   Mastra docs show per-tool approval gating (`requireApproval: true` with `tool-call-approval` stream chunks), a 'Code mode' that runs multi-tool computations in an isolated sandbox, and enterprise RBAC/IAM/network-policy controls — directly covering both fine-grained tool permissions and sandboxed execution. Missing for 10: independent/hands-on verification of the sandbox's isolation guarantees, and detail on how granular (per-tool vs per-agent) permission scoping actually works in practice beyond the single approval flag.

   • [claimed-docs] “Mark a tool with `requireApproval: true`, then check for the `tool-call-approval` chunk in the stream to approve or decline the action”
   • [claimed-docs] “Code mode lets an agent run multi-tool computations in an isolated sandbox and return the result as a single, more accurate response.”
   • [claimed-docs] “Processors transform, validate, or control messages as they pass through an agent.”
   • [claimed-docs] “Enterprise controls RBAC, SSO, IAM, and network policy integration.”
   • [claimed-docs] “The `PromptInjectionDetector()` scans user messages for prompt injection, jailbreak attempts, and system override patterns.”

   proof ↗⚑ flag

   Google ADK~partialclaimed4/10

   ADK provides a Tool Confirmation (HITL) flow that can gate tool execution with explicit confirmation/custom input, plus callbacks/plugins hooks to intercept agent actions, giving some control over agent behavior. However there is no evidence of fine-grained per-tool permission policies or an actual sandboxed execution environment for code/tool runs. Missing for 10: explicit sandboxing of tool/code execution, a permissions/ACL system scoping tool access, and independent verification of these guardrails in practice.

   • [github] “Tool Confirmation: A tool confirmation flow (HITL) that can guard tool execution with explicit confirmation and custom input.”
   • [github] “A tool confirmation flow (HITL) that can guard tool execution with explicit confirmation and custom input.”
   • [claimed-docs] “Callbacks: Hook into specific events during an agent's execution lifecycle to add logging, monitoring, or custom side-effects without alteri…”
   • [claimed-docs] “Plugins: Integrate complex, pre-packaged behaviors and third-party services directly into your agent's workflow.”

   proof ↗⚑ flag

Approval flows

1. developerPause an agent mid-run for human input or approval and resume with the human's decision

   weight 3 · round drawn
   Mastra✓fullclaimed8/10

   Mastra has documented first-class suspend/resume for workflows and agents, explicitly for human-in-the-loop approval: 'Suspend an agent or workflow and await user input or approval before resuming' with persisted state (mastra-gh-4), a dedicated suspend-and-resume docs page (mastra-docs-12), snapshot-based resume ('Snapshots capture all the information needed to resume a workflow from exactly where it left off', mastra-docs-39), and tool-level approval gating via requireApproval and tool-call-approval chunks (mastra-docs-31). missing for 10: independent/hands-on developer confirmation of the pause/resume-with-human-decision flow working in practice, and more detail on how the resumed human decision is injected back into agent state.

   • [github] “Suspend an agent or workflow and await user input or approval before resuming. Mastra uses storage to remember execution state, so you can p…”
   • [claimed-docs] “Pause a workflow at any step to collect additional data, wait for an API callback, throttle a costly operation, or request human-in-the-loop…”
   • [claimed-docs] “Snapshots capture all the information needed to resume a workflow from exactly where it left off”
   • [claimed-docs] “Mark a tool with `requireApproval: true`, then check for the `tool-call-approval` chunk in the stream to approve or decline the action”
   • [claimed-docs] “The workflow can then either resume or bail based on the input received.”

   proof ↗⚑ flag

   Google ADK✓fullclaimed8/10

   ADK explicitly documents a Tool Confirmation flow described as HITL that can 'guard tool execution with explicit confirmation and custom input,' plus a Workflow Runtime and Task API both explicitly listing human-in-the-loop support with state management for pausing and resuming execution. This directly matches pausing mid-run for human approval and resuming with the decision, though missing for 10: a concrete end-to-end code example showing pause/resume state persistence and independent third-party corroboration beyond vendor GitHub README claims.

   • [github] “Tool Confirmation: A tool confirmation flow (HITL) that can guard tool execution with explicit confirmation and custom input.”
   • [github] “A tool confirmation flow (HITL) that can guard tool execution with explicit confirmation and custom input.”
   • [github] “Workflow Runtime: A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan…”
   • [github] “A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan-out/fan-in, loops…”
   • [github] “Task API: Structured agent-to-agent delegation with multi-turn task mode, single-turn controlled output, mixed delegation patterns, human-in…”
   • [github] “Structured agent-to-agent delegation with multi-turn task mode, single-turn controlled output, mixed delegation patterns, human-in-the-loop,…”

   proof ↗⚑ flag

2. engineering-leadRequire human approval before specific sensitive tool calls execute

   weight 2 · round drawn
   Mastra✓fullclaimed8/10

   Mastra explicitly supports marking a tool with requireApproval: true and checking for a tool-call-approval chunk to approve or decline the action before it executes, plus general suspend/resume for workflows to await human input/approval. missing for 10: independent/hands-on corroboration of the requireApproval mechanism in production use, and detail on approval UI/audit trail beyond the docs snippet.

   • [claimed-docs] “Mark a tool with `requireApproval: true`, then check for the `tool-call-approval` chunk in the stream to approve or decline the action”
   • [github] “Suspend an agent or workflow and await user input or approval before resuming. Mastra uses storage to remember execution state, so you can p…”
   • [claimed-docs] “Pause a workflow at any step to collect additional data, wait for an API callback, throttle a costly operation, or request human-in-the-loop…”

   proof ↗⚑ flag

   Google ADK✓fullclaimed8/10

   ADK explicitly documents a 'Tool Confirmation' HITL flow that guards tool execution with explicit confirmation and custom input, plus broader human-in-the-loop support in its workflow/task orchestration engines, directly matching the story of requiring approval before sensitive tool calls execute. Missing for 10: no independent/hands-on validation or detailed walkthrough of configuring per-tool approval policies beyond the feature summary.

   • [github] “Tool Confirmation: A tool confirmation flow (HITL) that can guard tool execution with explicit confirmation and custom input.”
   • [github] “A tool confirmation flow (HITL) that can guard tool execution with explicit confirmation and custom input.”
   • [github] “Workflow Runtime: A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan…”
   • [github] “A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan-out/fan-in, loops…”
   • [github] “Task API: Structured agent-to-agent delegation with multi-turn task mode, single-turn controlled output, mixed delegation patterns, human-in…”
   • [github] “Structured agent-to-agent delegation with multi-turn task mode, single-turn controlled output, mixed delegation patterns, human-in-the-loop,…”

   proof ↗⚑ flag

Memory

1. developerTrim, summarize, or filter conversation history to keep an agent inside its context window

   weight 2 · round to Google ADK
   Mastra—none0/10

   The evidence describes Mastra's Memory system only in general terms (remembering messages/tool results, multi-user threads) but never mentions any mechanism for trimming, summarizing, or filtering conversation history to manage context window size. Absence of evidence for this specific, applicable capability yields none rather than na.

   • [claimed-docs] “Memory enables your agent to remember user messages and agent replies, and tool results across interactions, giving it the context it needs …”
   • [claimed-docs] “Memory enables your agent to remember user messages and agent replies, and tool results across interactions, giving it the context it needs …”
   • [claimed-docs] “Mastra agents can be configured to store message history.”
   • [claimed-docs] “Memory gives your agent access to earlier messages and tool results.”

   proof ↗⚑ flag

   Google ADK✓fullclaimed7/10

   ADK docs explicitly state it "automatically filters irrelevant events, summarizes older conversational turns, lazy-loads artifacts, and tracks token usage," directly addressing trimming/summarizing/filtering to manage context window, reinforced by mention of designing for AI context window limits. Missing for 10: no code example/API reference showing how a developer configures or customizes this summarization/filtering behavior, and no independent/hands-on corroboration beyond first-party docs.

   • [claimed-docs] “ADK automatically filters irrelevant events, summarizes older conversational turns, lazy-loads artifacts, and tracks token usage.”
   • [claimed-docs] “Use prebuilt or custom Agent Skills to extend agent capabilities in a way that works efficiently inside AI context window limits.”

   proof ↗⚑ flag

2. developerGive agents long-term memory that persists across sessions and threads

   weight 2 · round to Mastra
   Mastra✓fullclaimed8/10

   Mastra's docs explicitly describe a Memory system that stores message history and tool results 'across interactions' to keep agents consistent, with thread-scoped and multi-user thread support, plus 'goals' as durable thread-scoped objectives persisting across loop iterations, indicating persistence across sessions/threads. Missing for 10: no independent/hands-on verification of long-term persistence across actual separate sessions, and no detail on storage backends or recall/retrieval mechanics (e.g., vector search, working vs semantic memory) in the evidence.

   • [claimed-docs] “Memory enables your agent to remember user messages and agent replies, and tool results across interactions, giving it the context it needs …”
   • [claimed-docs] “Multi-user threads: Share one thread between multiple users.”
   • [claimed-docs] “Memory enables your agent to remember user messages and agent replies, and tool results across interactions, giving it the context it needs …”
   • [claimed-docs] “Mastra agents can be configured to store message history.”
   • [claimed-docs] “Memory gives your agent access to earlier messages and tool results.”
   • [claimed-docs] “A goal is a durable, thread-scoped objective: a standing instruction the agent keeps working toward across loop iterations until a judge mod…”

   proof ↗⚑ flag

   Google ADK~partialclaimed3/10

   The docs mention session-based interactions and automatic context management (filtering irrelevant events, summarizing older turns, tracking token usage) but there is no explicit evidence of a dedicated long-term memory service or store that persists agent knowledge across separate sessions/threads. missing for 10: explicit memory/session-store API docs, cross-session persistence guarantees, first-party examples of retrieving memory in a new thread.

   • [claimed-docs] “ADK automatically filters irrelevant events, summarizes older conversational turns, lazy-loads artifacts, and tracks token usage.”
   • [claimed-docs] “This approach involves creating individual test files, each representing a single, simple agent-model interaction (a session).”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Google ADK
   Mastra~partialprobed4/10

   Mastra is fundamentally code-first: agents, workflows, tools, memory, and evals are all defined and invoked via the TypeScript API (mastra-docs-2, mastra-docs-14, mastra-docs-21), and the Studio UI (mastra-docs-50) is described only as a way to 'test your agent and inspect its runs,' implying it surfaces API-driven functionality rather than adding UI-exclusive capability. However, there is no explicit documentation asserting full parity between Studio and the API, and a probe for a public OpenAPI/swagger spec returned 404s across all candidate paths (mastra-probe-3), leaving the scope of any hosted/API surface unverified. Missing for 10: explicit statement or docs page enumerating API endpoints equivalent to every Studio UI action, an OpenAPI/API reference confirming completeness, and independent confirmation that no Studio-only feature exists.

   • [claimed-docs] “Open Studio at [`http://localhost:4111`](http://localhost:4111) to test your agent and inspect its runs.”
   • [claimed-docs] “Import `createTool` from `@mastra/core/tools` and define a tool with an `id`, `description`, `inputSchema`, `outputSchema`, and `execute` fu…”
   • [claimed-docs] “tools MUST be defined via `createTool()` with `id`, `description`, `inputSchema` (zod), `execute()`”
   • [claimed-docs] “Composing **steps** with `createWorkflow` to define the execution flow.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://mastra.ai/openapi.json, https://mastra.ai/swagger.json, https://mastra.ai/api/openapi.json, …”

   proof ↗⚑ flag

   Google ADK~partialprobed5/10

   ADK is primarily a code-first Python framework where agents are built and orchestrated programmatically (Agent(), Workflow Runtime, Task API), and the CLI (adk run/web/eval/deploy) exposes most dev-loop actions including the same UI functions, suggesting reasonable parity between programmatic/CLI and the built-in dev UI. However, there's no evidence of a documented REST/OpenAPI API for driving the dev UI's specific features programmatically, and probes show no OpenAPI spec or llms.txt discoverability. missing for 10: explicit API/CLI parity documentation for every dev-UI feature (debug, evaluate, showcase), a published OpenAPI/REST spec, and confirmation that UI-only actions (e.g. visual debugging, showcase mode) are fully scriptable.

   • [github] “A built-in development UI to help you test, evaluate, debug, and showcase your agent(s).”
   • [github] “Web UI (supports multi-agent directories or pointing directly to a single agent folder)”
   • [github] “adk run path/to/my_agent”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [github] “adk eval \ samples_for_testing/hello_world \ samples_for_testing/hello_world/hello_world_eval_set_001.evalset.json”
   • [github] “adk deploy docker --with_ui <agent-folder>”
   • [probe] “PROBE openapi: all candidate paths 404 (https://google.github.io/openapi.json, https://google.github.io/swagger.json, https://google.github.…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://google.github.io/llms.txt”

   proof ↗⚑ flag

2. ai-native userRead the product's source under an open license

   weight 2 · round to Google ADK
   Mastra!disputedcontradicted4/10

   Mastra's own pricing page claims the self-hosted project is "Free Apache 2.0 licensed," suggesting a fully permissive open-source license, but a community comment directly disputes this, quoting license text that forbids offering the software as a hosted/managed service and asserting it is actually Elastic License v2, not truly open source. This is a concrete, on-topic contradiction between vendor claim and community report rather than mere skepticism. Missing for 10: a resolved/authoritative statement of the actual current license (e.g., LICENSE file content) and independent confirmation of source availability terms.

   • [claimed-docs] “Self host your Mastra projects ... Free Apache 2.0 licensed $0/ month ... Build and host agents anywhere”
   • [claimed-docs] “Self host your Mastra projects”
   • [community] “"You may not provide the software to third parties as a hosted or managed service" - The Elastic v2 license isn't actually open source like …”

   proof ↗⚑ flag

   Google ADK✓fullclaimed7/10

   The evidence repeatedly links to the public GitHub repository https://github.com/google/adk-python, which hosts the full source code and CLI (adk run, adk web, adk eval, adk deploy) that AI-native users can read and inspect directly. Missing for 10: an explicit citation of the license file/type (e.g., Apache-2.0) confirming the open-license terms, and independent third-party confirmation of licensing.

   • [github] “Agent Config: Build agents without code.”
   • [github] “A built-in development UI to help you test, evaluate, debug, and showcase your agent(s).”
   • [github] “adk run path/to/my_agent”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”

   proof ↗⚑ flag

3. ai-native userSelf-host the core product

   weight 3 · round to Google ADK
   Mastra!disputedcontradicted5/10

   Mastra's own pricing docs state you can self-host projects for free under an 'Apache 2.0' license and deploy to any Node.js-compatible environment, which supports the self-host story. However, a hands-on community comment directly disputes the licensing claim, stating the actual license is Elastic License v2, not Apache 2.0, and explicitly prohibits providing the software to third parties as a hosted/managed service — a concrete contradiction of the openness claim tied to self-hosting. Missing for 10: an authoritative current license file confirming which license actually applies, and clarification on hosting restrictions for multi-tenant use.

   • [claimed-docs] “Self host your Mastra projects ... Free Apache 2.0 licensed $0/ month ... Build and host agents anywhere”
   • [claimed-docs] “Build and host agents anywhere”
   • [claimed-docs] “Self host your Mastra projects”
   • [claimed-docs] “Mastra applications can be deployed to any Node.js-compatible environment. You can deploy a Mastra server or integrate with an existing web …”
   • [community] “"You may not provide the software to third parties as a hosted or managed service" - The Elastic v2 license isn't actually open source like …”

   proof ↗⚑ flag

   Google ADK✓fullclaimed8/10

   ADK is an open-source framework (github.com/google/adk-python) that can be run entirely locally via `adk run`/`adk web`, packaged into containers, and deployed offline/disconnected from Google Cloud, evidencing full self-hosting capability without requiring the vendor's managed service. Missing for 10: no independent third-party report confirming a full self-hosted production deployment, and no explicit self-hosted infra requirements/scaling guidance beyond container packaging.

   • [claimed-docs] “This is a good option if you prefer to run offline or disconnected, or otherwise in a system that has no connection to Google Cloud.”
   • [claimed-docs] “You can manually package your Agent into a container image and then run it in any environment that supports container images.”
   • [github] “adk run path/to/my_agent”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [github] “adk deploy docker --with_ui <agent-folder>”
   • [claimed-docs] “GKE is a good option if you need more control over the deployment as well as for running Open Models.”

   proof ↗⚑ flag

Multi agent

1. developerOrchestrate multiple agents — handoffs, subagents, or crews — inside one workflow

   weight 3 · round to Google ADK
   Mastra✓fullcommunity7/10

   Mastra's graph-based workflow engine explicitly supports steps that call different agents, with `.branch()`, `.parallel()`, and `.then()` control flow, enabling orchestration of multiple agents/subagents within a single workflow, plus suspend/resume for handoff-like human-in-the-loop points. Missing for 10: a dedicated named multi-agent 'crew'/'network' primitive and independent case-study evidence of complex multi-agent orchestration succeeding at scale (one community review notes workflow branching logic felt 'clunky').

   • [github] “use Mastra's graph-based workflow engine to orchestrate complex multi-step processes. Mastra workflows use an intuitive syntax for control f…”
   • [claimed-docs] “Composing **steps** with `createWorkflow` to define the execution flow.”
   • [claimed-docs] “Use `.parallel()` to run steps simultaneously.”
   • [claimed-docs] “Workflow steps can call agents to use LLM reasoning or call tools for type-safe logic.”
   • [claimed-docs] “Workflows let you define complex task sequences with clear, structured steps instead of relying on one agent to reason through the entire pr…”
   • [github] “Suspend an agent or workflow and await user input or approval before resuming. Mastra uses storage to remember execution state, so you can p…”
   • [community] “I worked with Mastra for three months and it is awesome... it felt clunky working with workflows and branching logic with non LLM agents... …”

   proof ↗⚑ flag

   Google ADK✓fullclaimed9/10

   ADK explicitly supports multi-agent orchestration: workflows are defined as any application with more than one agent/node, with a graph-based Workflow Runtime supporting routing, fan-out/fan-in, loops, nested workflows, and a Task API for structured agent-to-agent delegation including multi-turn task mode and mixed delegation patterns; the CLI/Web UI explicitly supports multi-agent directories. missing for 10: independent third-party hands-on validation of complex multi-agent orchestration at scale.

   • [claimed-docs] “you can use the ADK development framework to expand them into workflows, which allow you to combine and orchestrate multiple agents and code…”
   • [claimed-docs] “In ADK, any agent application that has more than one agent or executable Node is considered a workflow.”
   • [github] “Workflow Runtime: A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan…”
   • [github] “Task API: Structured agent-to-agent delegation with multi-turn task mode, single-turn controlled output, mixed delegation patterns, human-in…”
   • [github] “Structured agent-to-agent delegation with multi-turn task mode, single-turn controlled output, mixed delegation patterns, human-in-the-loop,…”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”

   proof ↗⚑ flag

Workflow control

1. developerCompose agents into an explicit graph or workflow with branching, loops, and parallel steps

   weight 2 · round to Google ADK
   Mastra~partialcommunity7/10

   Mastra's graph-based workflow engine explicitly supports `.then()`, `.branch()`, `.parallel()` control flow, workflow state sharing across steps, suspend/resume, and dynamic workflow composition, directly matching the story (mastra-gh-3, mastra-docs-21, mastra-docs-35, mastra-docs-36, mastra-docs-38). However, a hands-on community report describes branching logic with non-LLM agents as 'clunky,' leading the user to build custom branching workarounds after weeks of frustration (mastra-comm-7), tempering the otherwise strong first-party documentation. Missing for 10: independent benchmarks or more hands-on validation of loop/branch robustness beyond one mixed community report, and clearer first-party examples of loops specifically (only branch/parallel/then are explicitly named).

   • [github] “use Mastra's graph-based workflow engine to orchestrate complex multi-step processes. Mastra workflows use an intuitive syntax for control f…”
   • [claimed-docs] “Workflows let you define complex task sequences with clear, structured steps instead of relying on one agent to reason through the entire pr…”
   • [claimed-docs] “Composing **steps** with `createWorkflow` to define the execution flow.”
   • [claimed-docs] “Workflow state lets you share values across steps without passing them through every step's inputSchema and outputSchema.”
   • [claimed-docs] “Use `.parallel()` to run steps simultaneously.”
   • [claimed-docs] “Use dynamic workflows when users, agents, visual editors, or external systems need to create workflows without changing application code or …”
   • [community] “I worked with Mastra for three months and it is awesome... it felt clunky working with workflows and branching logic with non LLM agents... …”

   proof ↗⚑ flag

   Google ADK✓fullclaimed9/10

   ADK provides a dedicated graph-based Workflow Runtime with explicit support for routing, fan-out/fan-in (parallel), loops, retry, nested workflows, and dynamic nodes, plus structured Task API for agent delegation and workflow nodes—directly matching branching/loops/parallel composition; docs also describe 'graph-based architectures with explicit execution paths.' Missing for 10: independent/hands-on third-party validation beyond vendor docs and GitHub README.

   • [github] “Workflow Runtime: A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan…”
   • [github] “A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan-out/fan-in, loops…”
   • [github] “Task API: Structured agent-to-agent delegation with multi-turn task mode, single-turn controlled output, mixed delegation patterns, human-in…”
   • [github] “Structured agent-to-agent delegation with multi-turn task mode, single-turn controlled output, mixed delegation patterns, human-in-the-loop,…”
   • [claimed-docs] “Weave deterministic code with adaptive AI reasoning. Orchestrate complex tasks through structured, graph-based architectures, with explicit …”
   • [claimed-docs] “In ADK, any agent application that has more than one agent or executable Node is considered a workflow.”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round to Mastra
   Mastra~partialclaimed4/10

   Mastra is self-hostable under Apache 2.0 and can be deployed to 'any Node.js-compatible environment' or 'anywhere,' which implicitly lets a user control where data is stored by choosing their own infrastructure/region. However, there is no explicit region/residency selection feature, no data-storage location controls, and no documentation addressing compliance/residency requirements directly. Missing for 10: explicit region/residency configuration options, documentation on data storage locations for any hosted offering, and compliance certifications tied to residency.

   • [claimed-docs] “Self host your Mastra projects ... Free Apache 2.0 licensed $0/ month ... Build and host agents anywhere”
   • [claimed-docs] “Build and host agents anywhere”
   • [claimed-docs] “Self host your Mastra projects”
   • [claimed-docs] “Mastra applications can be deployed to any Node.js-compatible environment. You can deploy a Mastra server or integrate with an existing web …”
   • [claimed-docs] “Mastra applications can be deployed to any Node.js-compatible environment.”

   proof ↗⚑ flag

   Google ADK—none0/10

   ADK is a framework that can be deployed via Cloud Run, GKE, or self-hosted/offline (google-adk-docs-7, google-adk-docs-14, google-adk-docs-19), which implies developers control infrastructure location, but there is no explicit documentation about data residency, region selection, or storage location controls for agent data.

   • [claimed-docs] “This is a good option if you prefer to run offline or disconnected, or otherwise in a system that has no connection to Google Cloud.”
   • [claimed-docs] “Cloud Run is a managed auto-scaling compute platform on Google Cloud that enables you to run your agent as a container-based application.”
   • [claimed-docs] “GKE is a good option if you need more control over the deployment as well as for running Open Models.”

   proof ↗⚑ flag

2. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Mastra—none0/10

   Mastra's docs describe memory/storage of messages and threads (mastra-docs-4, mastra-docs-22) and self-hosting (mastra-docs-11), but no evidence describes explicit data retention policies, TTLs, or deletion/erasure APIs for stored memory, threads, or logs. Self-hosting implies infrastructural control but the evidence pack contains no documented retention/deletion controls a user could invoke.

   • [claimed-docs] “Memory enables your agent to remember user messages and agent replies, and tool results across interactions, giving it the context it needs …”
   • [claimed-docs] “Mastra agents can be configured to store message history.”
   • [claimed-docs] “Self host your Mastra projects ... Free Apache 2.0 licensed $0/ month ... Build and host agents anywhere”

   proof ↗⚑ flag

   Google ADK—none0/10

   The evidence describes ADK as a self-hosted/deployable agent framework (Cloud Run, GKE, offline/disconnected deployment) but contains no documentation of explicit data retention policies, session/state deletion APIs, or user-facing controls for purging stored data. missing for 10: explicit retention/deletion controls, session data lifecycle docs, any privacy/compliance statements about stored artifacts or memory.

   • [claimed-docs] “This is a good option if you prefer to run offline or disconnected, or otherwise in a system that has no connection to Google Cloud.”
   • [claimed-docs] “ADK automatically filters irrelevant events, summarizes older conversational turns, lazy-loads artifacts, and tracks token usage.”

   proof ↗⚑ flag

3. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Mastra—none0/10

   No evidence in the pack addresses telemetry, usage tracking, or opt-out settings for Mastra; while the framework is self-hostable, there's nothing documenting a telemetry opt-out mechanism.

   ⚑ flag

   Google ADK—none0/10

   No evidence in the pack addresses telemetry collection or an opt-out mechanism for ADK; the docs cover agent building, deployment, evaluation, and workflows but never mention usage tracking or privacy controls. This is a fair axis for a developer framework/SDK, but absence of evidence means it counts as none. missing for 10: any mention of telemetry collection, an opt-out flag/env var, or a privacy policy describing data tracking.

   ⚑ flag

Durable state

1. developerCheckpoint agent state so a run can resume exactly where it left off after a crash or restart

   weight 3 · round to Mastra
   Mastra✓fullclaimed8/10

   Mastra's workflow engine explicitly persists execution state via storage and snapshots that "capture all the information needed to resume a workflow from exactly where it left off," supporting indefinite pause/resume and even step-level time travel from stored snapshots. This directly matches checkpoint/resume semantics needed after a crash or restart, backed by first-party docs across suspend/resume, snapshots, and time-travel features. missing for 10: no independent/hands-on evidence specifically demonstrating recovery after a process crash (vs. planned suspend), and no detail on storage backend guarantees (e.g., durability across restarts of the host process itself).

   • [github] “Suspend an agent or workflow and await user input or approval before resuming. Mastra uses storage to remember execution state, so you can p…”
   • [claimed-docs] “Pause a workflow at any step to collect additional data, wait for an API callback, throttle a costly operation, or request human-in-the-loop…”
   • [claimed-docs] “Snapshots capture all the information needed to resume a workflow from exactly where it left off”
   • [claimed-docs] “Time travel allows you to re-execute a workflow starting from any specific step, using either stored snapshot data or custom context you pro…”
   • [claimed-docs] “The workflow can then either resume or bail based on the input received.”

   proof ↗⚑ flag

   Google ADK—none0/10

   Evidence only mentions generic 'state management' as one feature in the workflow runtime engine, with no documentation of session/state persistence, checkpointing, or resuming an agent run after a crash or restart. Missing for 10: explicit checkpoint/save-state API, resume-from-crash mechanism, persistence backend documentation, and any hands-on confirmation of durable resumption.

   • [github] “Workflow Runtime: A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan…”
   • [github] “A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan-out/fan-in, loops…”
   • [claimed-docs] “ADK automatically filters irrelevant events, summarizes older conversational turns, lazy-loads artifacts, and tracks token usage.”

   proof ↗⚑ flag

2. engineering-leadRun long-lived agents durably across process restarts and deploys, natively or via durable-execution integrations

   weight 2 · round to Mastra
   Mastra✓fullclaimed7/10

   Mastra documents storage-backed suspend/resume and workflow snapshots explicitly designed to persist execution state so agents/workflows can 'pause indefinitely and resume where you left off,' plus time-travel re-execution from stored snapshots and cron-based scheduling — all native durability primitives rather than third-party integrations. Missing for 10: explicit statement that this survives process crashes/redeploys (only implied), no mention of pluggable durable-execution engines (e.g., Temporal/Inngest) as an alternative, and no independent/hands-on verification of restart durability.

   • [github] “Suspend an agent or workflow and await user input or approval before resuming. Mastra uses storage to remember execution state, so you can p…”
   • [claimed-docs] “Pause a workflow at any step to collect additional data, wait for an API callback, throttle a costly operation, or request human-in-the-loop…”
   • [claimed-docs] “Snapshots capture all the information needed to resume a workflow from exactly where it left off”
   • [claimed-docs] “Time travel allows you to re-execute a workflow starting from any specific step, using either stored snapshot data or custom context you pro…”
   • [claimed-docs] “Declare a `schedule` field on a workflow and Mastra will fire it on the cron you specify.”
   • [claimed-docs] “Memory enables your agent to remember user messages and agent replies, and tool results across interactions, giving it the context it needs …”

   proof ↗⚑ flag

   Google ADK~partialclaimed3/10

   ADK's Workflow Runtime mentions 'state management' and 'retry' in its graph-based execution engine, and deployment docs describe scalable hosting (Agent Runtime, Cloud Run), but there is no explicit evidence of session/state persistence surviving process restarts or redeploys, nor any named durable-execution integration (e.g., Temporal, Cloud Workflows checkpointing). Missing for 10: documented durable state store or checkpoint/resume mechanism, explicit claim of surviving restarts/redeploys, and any third-party durable-execution integration.

   • [github] “Workflow Runtime: A graph-based execution engine for composing deterministic execution flows for agentic apps, with support for routing, fan…”
   • [claimed-docs] “Agent Runtime is a fully managed auto-scaling service on Google Cloud specifically designed for deploying, managing, and scaling AI agents b…”
   • [claimed-docs] “Cloud Run is a managed auto-scaling compute platform on Google Cloud that enables you to run your agent as a container-based application.”
   • [claimed-docs] “ADK automatically filters irrelevant events, summarizes older conversational turns, lazy-loads artifacts, and tracks token usage.”

   proof ↗⚑ flag

Streaming

1. developerStream tokens and intermediate agent events (tool calls, steps) to my UI in real time

   weight 3 · round to Mastra
   Mastra✓fullclaimed8/10

   Docs explicitly describe real-time incremental streaming of agent/workflow output, tool-call approval chunks appearing in the stream, AI SDK-compatible stream conversion, and observability into every agent run, workflow step, and tool call—covering tokens plus intermediate tool/step events. Missing for 10: independent/hands-on confirmation of streaming behavior and concrete UI integration examples beyond docs claims.

   • [claimed-docs] “Mastra supports real-time, incremental responses from agents and workflows, allowing users to see output as it's generated instead of waitin…”
   • [claimed-docs] “Mastra supports real-time, incremental responses from agents and workflows, allowing users to see output as it’s generated instead of waitin…”
   • [claimed-docs] “Use `toAISdkStream()` and `toAISdkMessages()` to convert Mastra streams and stored messages to AI SDK-compatible formats.”
   • [claimed-docs] “Mark a tool with `requireApproval: true`, then check for the `tool-call-approval` chunk in the stream to approve or decline the action”
   • [claimed-docs] “Mastra's observability system gives you visibility into every agent run, workflow step, tool call, and model interaction.”
   • [claimed-docs] “Events flow through PubSub, which means a client can disconnect and reconnect without missing chunks.”

   proof ↗⚑ flag

   Google ADK~partialclaimed4/10

   The evidence shows a built-in Web/dev UI (`adk web`) for testing/debugging agents and a Callbacks mechanism to hook into execution-lifecycle events (tool calls, steps), which implies some visibility into intermediate agent activity, but nothing explicitly documents token-level streaming to a custom UI (no mention of SSE/websocket/streaming API). missing for 10: explicit documentation of real-time token streaming API/protocol, evidence of streaming tool-call/step events to an arbitrary UI beyond the built-in dev UI, independent confirmation of streaming behavior.

   • [github] “A built-in development UI to help you test, evaluate, debug, and showcase your agent(s).”
   • [github] “Web UI (supports multi-agent directories or pointing directly to a single agent folder)”
   • [github] “adk run path/to/my_agent # Web UI (supports multi-agent directories or pointing directly to a single agent folder) adk web path/to/agents_d…”
   • [claimed-docs] “Callbacks: Hook into specific events during an agent's execution lifecycle to add logging, monitoring, or custom side-effects without alteri…”

   proof ↗⚑ flag

Structured output

1. developerGet schema-validated structured output from an agent, with automatic retries when validation fails

   weight 3 · round to Mastra
   Mastra~partialclaimed5/10

   Docs confirm agents can return schema-validated structured output instead of text (mastra-docs-30), but no evidence describes an automatic retry mechanism when validation fails. missing for 10: explicit documentation of retry-on-validation-failure behavior, hands-on/community confirmation of retry reliability.

   • [claimed-docs] “Structured output lets an agent return an object that matches the shape defined by a schema instead of returning text.”

   proof ↗⚑ flag

   Google ADK—none0/10

   No evidence in the pack mentions schema-validated structured output (e.g., Pydantic output_schema) or automatic retry-on-validation-failure behavior for ADK agents; the evidence covers agent setup, tools, workflows, deployment, and evaluation but not structured output validation. Missing for 10: any mention of output schema enforcement, structured output configuration, or validation-retry mechanism.

   ⚑ flag