LangGraph vs CrewAI

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round drawn
   LangGraph✓fullprobed9/10

   Probes confirm a live llms.txt at docs.langchain.com (HTTP 200) with a documentation index, and individual doc pages are available in agent-friendly .md format (e.g., overview.md) that explicitly reference the llms.txt index for further crawling — this is exactly the agent-oriented docs pattern the story asks for. Missing for 10: no independent/community confirmation of an agent actually consuming llms.txt successfully in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.langchain.com/llms.txt # Docs by LangChain > Documentation for LangSmith, Fleet, and our open sour…”
   • [probe] “PROBE docs-md: HTTP 200 at https://docs.langchain.com/oss/python/langgraph/overview.md > ## Documentation Index > Fetch the complete documen…”

   proof ↗⚑ flag

   CrewAI✓fullprobed9/10

   CrewAI's docs site serves a live llms.txt (probe confirms HTTP 200 with a full page index) and per-page .md variants, plus explicit doc instructions to copy a setup prompt that 'points the agent at the right docs' for Claude Code, Codex, Cursor, etc. missing for 10: no independent/community confirmation of an agent actually consuming llms.txt successfully in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.crewai.com/llms.txt # CrewAI - [Edge (199 pages)](https://docs.crewai.com/_llms/en/edge.md): Docum…”
   • [probe] “PROBE docs-md: HTTP 200 at https://docs.crewai.com/en/installation.md > ## Documentation Index > Fetch the complete documentation index at: …”
   • [claimed-docs] “Copy a ready-to-paste setup prompt for Claude Code, Codex, Cursor, or any coding agent. It installs the official CrewAI skills, checks the C…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to CrewAI
   LangGraph~partialclaimed6/10

   LangGraph is a pip-installable Python library with a programmatic graph API (stream/astream, Command, checkpointers) and a CLI that builds/runs an Agent Server locally, all of which support non-interactive, scriptable execution suitable for CI/automation. However, there is no explicit documentation or example of running LangGraph in a CI pipeline or headless automation context specifically. Missing for 10: explicit CI/automation guide or example, documented headless/non-interactive invocation patterns, and independent evidence of real-world CI usage.

   • [github] “pip install -U langgraph”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally.”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”
   • [claimed-docs] “LangGraph graphs expose the stream (sync) and astream (async) methods to yield streamed outputs as iterators.”
   • [claimed-docs] “Checkpointers persist a thread's graph state as checkpoints. Use them for short-term, thread-scoped memory, including conversation continuit…”

   proof ↗⚑ flag

   CrewAI✓fullprobed8/10

   CrewAI ships an official CLI (create/train/run/test/deploy) that is scriptable for CI, plus a testing subcommand (`crewai test`) for automated performance checks, and a REST API (kickoff/status/resume endpoints) enabling fully headless invocation from external automation systems. Missing for 10: no explicit CI pipeline example (e.g. GitHub Actions) or independent report confirming headless CI usage in the wild.

   • [claimed-docs] “The CrewAI CLI provides a set of commands to interact with CrewAI, allowing you to create, train, run, and manage crews & flows”
   • [claimed-docs] “We added the CLI command crewai test to make it easy to test your crew. This command will run your crew for a specified number of iterations…”
   • [claimed-docs] “We added the CLI command `crewai test` to make it easy to test your crew.”
   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”
   • [probe] “official CLI documented at https://docs.crewai.com/en/concepts/cli”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to CrewAI
   LangGraph✓fullclaimed6/10

   Docs confirm LangChain/LangGraph agents can consume MCP servers via MCPAdapter, which discovers a server's tools and adapts them into LangChain tools for use inside graphs. Missing for 10: deeper first-party walkthrough/code example of wiring an MCP server into a LangGraph agent, and independent/community corroboration of this working in practice.

   • [claimed-docs] “LangChain agents call tools defined on MCP servers through MCPAdapter, which discovers a server's tools and adapts them into LangChain tools…”

   proof ↗⚑ flag

   CrewAI✓fullclaimed8/10

   CrewAI documents a first-party MCP integration via the `mcps` field on agents, supporting both string references and structured configs for seamless MCP tool integration, directly matching the story. Missing for 10: independent/hands-on community corroboration of MCP usage and more detail on connection/auth mechanics beyond the DSL description.

   • [claimed-docs] “Use the mcps field directly on agents for seamless MCP tool integration. The DSL supports both string references... and structured configura…”
   • [claimed-docs] “Use the `mcps` field directly on agents for seamless MCP tool integration.”
   • [claimed-docs] “Use the `mcps` field directly on agents for seamless MCP tool integration. The DSL supports both **string references** (for quick setup) and…”
   • [claimed-docs] “Use the `mcps` field directly on agents for seamless MCP tool integration. The DSL supports both **string references** (for quick setup) and…”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round drawn
   LangGraph—none0/10

   Evidence shows LangGraph/LangChain agents can act as MCP clients (via MCPAdapter, discovering and calling tools from external MCP servers), but there is no evidence LangGraph itself exposes an official MCP server that other agents could connect to. As a framework/platform (not itself an agent), shipping an official MCP server is a fair axis, but nothing in the evidence pack shows this capability.

   • [claimed-docs] “LangChain agents call tools defined on MCP servers through MCPAdapter, which discovers a server's tools and adapts them into LangChain tools…”

   proof ↗⚑ flag

   CrewAI—none0/10

   CrewAI documents only client-side MCP integration (an `mcps` field letting CrewAI agents call out to external MCP servers), but there is no evidence of CrewAI itself exposing an official MCP server that other agents could connect to. As a framework/platform, this axis applies, but absence of a documented server-mode offering means it is not delivered.

   • [claimed-docs] “Use the mcps field directly on agents for seamless MCP tool integration. The DSL supports both string references... and structured configura…”
   • [claimed-docs] “Use the `mcps` field directly on agents for seamless MCP tool integration.”
   • [claimed-docs] “Use the `mcps` field directly on agents for seamless MCP tool integration. The DSL supports both **string references** (for quick setup) and…”
   • [claimed-docs] “Use the `mcps` field directly on agents for seamless MCP tool integration. The DSL supports both **string references** (for quick setup) and…”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to LangGraph
   LangGraph✓fullprobed9/10

   LangGraph ships an official CLI (LangGraph CLI) documented for building and running the Agent Server locally, exposing API endpoints for runs, threads, assistants, etc., with supporting services like managed DB for checkpointing — confirmed by first-party docs and a live probe of the doc page. missing for 10: no independent/community hands-on confirmation of CLI usage beyond vendor docs.

   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally.”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”
   • [probe] “official CLI documented at https://docs.langchain.com/langsmith/cli”

   proof ↗⚑ flag

   CrewAI✓fullprobed8/10

   CrewAI ships an official CLI (installed via `uv tool install crewai`) documented to create, train, test, and run crews & flows, with dedicated commands like `crewai test`, confirmed by both docs and a live probe of the docs page. missing for 10: independent/hands-on third-party validation of the CLI itself (community evidence covers the framework broadly, not CLI usage specifically) and no changelog/version history showing CLI maturity over time.

   • [claimed-docs] “The CrewAI CLI provides a set of commands to interact with CrewAI, allowing you to create, train, run, and manage crews & flows”
   • [claimed-docs] “We added the CLI command crewai test to make it easy to test your crew. This command will run your crew for a specified number of iterations…”
   • [claimed-docs] “Run the following command to install crewai CLI: uv tool install crewai”
   • [claimed-docs] “Create a new crew, flow, tool, skill, or template project.”
   • [claimed-docs] “These older commands still work but print a yellow deprecation warning.”
   • [probe] “official CLI documented at https://docs.crewai.com/en/concepts/cli”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to LangGraph
   LangGraph~partialprobed7/10

   LangGraph's Python API (graph construction, streaming, persistence, interrupts) is extensively documented, and the LangGraph CLI/Agent Server exposes REST endpoints for runs, threads, and assistants (docs-30, docs-37), giving programmatic/API access beyond just an SDK. However, a probe for a formal OpenAPI/swagger spec returned 404 on all candidate paths, and community comments note documentation gaps and breaking changes, suggesting the 'public API' is real but not as formally discoverable as a REST-first product. Missing for 10: a published OpenAPI/swagger spec or API reference, and independent confirmation the Agent Server API is stable/production-documented rather than CLI-only.

   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langc…”
   • [probe] “official CLI documented at https://docs.langchain.com/langsmith/cli”
   • [community] “I read the article but have yet to understand why someone would want to use a framework that introduces meaningless abstractions that are no…”
   • [claimed-docs] “LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components: State, Nodes, Edges.”
   • [claimed-docs] “LangGraph graphs expose the stream (sync) and astream (async) methods to yield streamed outputs as iterators.”

   proof ↗⚑ flag

   CrewAI~partialprobed6/10

   CrewAI documents a REST API for driving crew execution (POST /kickoff, GET /status, POST /resume) under its Enterprise/Edge docs, and the whole framework is Python-native and scriptable via CLI/SDK, giving AI-native users multiple documented entry points. However, this API surface is only exposed for Enterprise (AMP) deployments rather than a universally public API, and a direct probe for a discoverable OpenAPI/Swagger spec returned 404s, indicating no formal machine-readable API contract. Missing for 10: a public OpenAPI/swagger spec, confirmation the REST API is available outside the paid Enterprise tier, and independent hands-on confirmation of API usage.

   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”
   • [claimed-docs] “Deploy your crews to a managed infrastructure with a few clicks... Access your deployed crews via REST API for integration with existing sys…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.crewai.com/openapi.json, https://docs.crewai.com/swagger.json, https://docs.crewai.com/…”
   • [claimed-docs] “The CrewAI CLI provides a set of commands to interact with CrewAI, allowing you to create, train, run, and manage crews & flows”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   LangGraph—none0/10

   No evidence in the pack addresses issuing scoped or least-privilege API credentials for an agent; LangGraph's docs cover orchestration, persistence, streaming, memory, and deployment but nothing about credential scoping or permission-limited API keys.

   ⚑ flag

   CrewAI—none0/10

   No documentation or evidence shows CrewAI issuing scoped/least-privilege API credentials per agent; tools/LLM/MCP integration docs describe capability wiring but not credential scoping. A community comment even notes agents typically run with 'full credentials' access, reinforcing the absence of this feature.

   • [community] “On a CrewAI-based personal email agent: 'Local personal agents are going to be a major app type... They run locally on my computer with my f…”
   • [claimed-docs] “CrewAI tools empower agents with capabilities ranging from web searching and data analysis to collaboration and delegating tasks among cowor…”
   • [claimed-docs] “A tool in CrewAI is a skill or function that agents can utilize to perform various actions. This includes tools from the [CrewAI Toolkit] an…”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to LangGraph
   LangGraph✓fullprobed8/10

   LangGraph itself is shipped as an official, well-documented SDK/package (pip install langgraph) with extensive first-party API docs (graph API, persistence, streaming, interrupts), an official CLI/Agent Server, and GitHub-hosted source, all confirming it is a legitimate SDK for building AI-native agent systems. Missing for 10: evidence of official SDKs beyond Python (e.g., JS/TS parity claims) and independent hands-on validation of SDK API stability (community notes mention breaking changes/documentation gaps).

   • [github] “pip install -U langgraph”
   • [github] “LangGraph is a low-level orchestration framework for building, managing, and deploying long-running, stateful agents.”
   • [claimed-docs] “At its core, LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components”
   • [claimed-docs] “LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components: State, Nodes, Edges.”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [probe] “official CLI documented at https://docs.langchain.com/langsmith/cli”
   • [community] “I read the article but have yet to understand why someone would want to use a framework that introduces meaningless abstractions that are no…”

   proof ↗⚑ flag

   CrewAI~partialprobed6/10

   CrewAI is itself a Python SDK/framework for building agents (Agent, Task, Crew, Flow classes) and integrates with LLM provider native SDKs, plus a REST API for kickoff/status/resume, which supports SDK-style programmatic access. However, there is no evidence of official SDKs in multiple languages (e.g., JS/TS, Go) beyond Python, no versioned SDK release notes, and no independent corroboration of SDK stability/quality from third-party developers. missing for 10: multi-language official SDKs, independent developer corroboration of SDK reliability, formal SDK versioning/changelog beyond docs.

   • [claimed-docs] “CrewAI integrates with multiple LLM providers through providers native sdks, giving you the flexibility to choose the right model for your s…”
   • [claimed-docs] “CrewAI integrates with multiple LLM providers through providers native sdks, giving you the flexibility to choose the right model for your s…”
   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”
   • [claimed-docs] “Build collaborative AI agents, crews, and flows — production ready from day one.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.crewai.com/openapi.json, https://docs.crewai.com/swagger.json, https://docs.crewai.com/…”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round to CrewAI
   LangGraph—none0/10

   Evidence covers streaming, checkpointing, interrupts, and an Agent Server exposing API endpoints, but nowhere mentions webhook subscriptions or push-based event notifications for external systems. missing for 10: any documentation of a webhook registration/subscription mechanism, delivery guarantees, or event-push API.

   ⚑ flag

   CrewAI~partialclaimed5/10

   CrewAI's enterprise tier (AMP) explicitly advertises 'Webhook Streaming: Stream real-time events and updates to your systems' alongside a REST API for kickoff/status/resume, indicating a webhook subscription mechanism exists. However, this is gated behind the paid CrewAI AMP/Enterprise product rather than the open-source core, and no detail is given on event types, subscription setup, or payload schema. Missing for 10: documentation of specific webhook event types/payloads, open-source (non-enterprise) support, and independent/hands-on confirmation the webhook feature works as described.

   • [claimed-docs] “Webhook Streaming Stream real-time events and updates to your systems”
   • [claimed-docs] “Stream real-time events and updates to your systems”
   • [claimed-docs] “Deploy your crews to a managed infrastructure with a few clicks... Access your deployed crews via REST API for integration with existing sys…”
   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”

   proof ↗⚑ flag

Agentic features

1. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to CrewAI
   LangGraph~partialcommunity7/10

   LangGraph explicitly supports durable, long-running agent execution that persists through failures and resumes automatically, with checkpointing, human-in-the-loop interrupts, and a CLI/Agent Server for production deployment (langgraph-gh-6, langgraph-docs-13, langgraph-docs-30). This covers the core of 'autonomous background automation' but it is a low-level orchestration framework requiring developers to build and deploy the graph themselves rather than a turnkey scheduler/trigger system, and community feedback notes rough edges in streaming/persistence implementation (langgraph-comm-13). Missing for 10: built-in scheduling/trigger mechanisms for kicking off automations, independent hands-on verification of unattended long-running runs, and clearer distinction of 'autonomous' (no human) vs human-in-the-loop operation.

   • [github] “Durable execution — Build agents that persist through failures and can run for extended periods, automatically resuming from exactly where t…”
   • [github] “Production-ready deployment — Deploy sophisticated agent systems confidently with scalable infrastructure designed to handle the unique chal…”
   • [claimed-docs] “In production, use a checkpointer backed by a database: from langgraph.checkpoint.postgres import PostgresSaver”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “Checkpointing keeps your place: the checkpointer writes the exact graph state so you can resume later, even when in an error state.”
   • [community] “The one thing I wish was better developed is persistence and streaming - they give sample code to stream, but it's essentially a complete im…”

   proof ↗⚑ flag

   CrewAI✓fullcommunity7/10

   CrewAI agents/crews/flows are explicitly designed as autonomous units that make decisions, run tasks sequentially or hierarchically, and can be deployed to managed infrastructure with monitoring, webhooks, and REST API kickoff/resume endpoints—enabling background autonomous execution. Community evidence (HN Gmail automation example) confirms real-world use of CrewAI for unattended background automations. missing for 10: independent verification of long-running/scheduled background execution reliability, and no first-party docs on scheduling/triggers beyond kickoff API and webhooks.

   • [claimed-docs] “In the CrewAI framework, an `Agent` is an autonomous unit that can: * Perform specific tasks * Make decisions based on its role and goal * U…”
   • [claimed-docs] “A crew in crewAI represents a collaborative group of agents working together to achieve a set of tasks.”
   • [claimed-docs] “Flows allow developers to combine and coordinate coding tasks and Crews efficiently, providing a robust framework for building sophisticated…”
   • [claimed-docs] “Deploy your crews to a managed infrastructure with a few clicks... Access your deployed crews via REST API for integration with existing sys…”
   • [claimed-docs] “Webhook Streaming Stream real-time events and updates to your systems”
   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”
   • [community] “Show HN: built AI agents with CrewAI to automate an old Gmail account—categorizing emails, drafting replies, Slack alerts for high-priority …”

   proof ↗⚑ flag

2. ai-native userOperate the product with natural-language commands

   weight 2 · round drawn
   LangGraph—none0/10

   LangGraph is a code-first orchestration framework (graphs, nodes, edges, checkpointers, CLI for running a server) with no evidence of any natural-language command interface for operating the product itself — developers configure and run it via Python/JS APIs and CLI flags, not NL prompts. Missing for 10: any documented chat/NL interface, NL-driven graph builder, or NL-based CLI/administration capability.

   • [claimed-docs] “At its core, LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components”
   • [claimed-docs] “LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components: State, Nodes, Edges.”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally.”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”

   proof ↗⚑ flag

   CrewAI—none0/10

   CrewAI is operated via Python code, YAML config, and a traditional CLI (create/train/run/test) or a drag-and-drop Visual Builder — none of which constitute natural-language command operation of the product itself. Evidence shows agents can be given natural-language goals within a crew, but that's the framework's internal task model, not the user's interface to the product.

   • [claimed-docs] “The CrewAI CLI provides a set of commands to interact with CrewAI, allowing you to create, train, run, and manage crews & flows”
   • [claimed-docs] “Create a new crew, flow, tool, skill, or template project.”
   • [claimed-docs] “CrewAI AMP includes a Visual Agent Builder that simplifies agent creation and configuration without writing code. Design your agents visuall…”
   • [claimed-docs] “The Visual Task Builder enables: Drag-and-drop task creation, Visual task dependencies and flow, Real-time testing and validation”
   • [claimed-docs] “Crew Studio Create and customize crews using a no-code/low-code interface”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   LangGraph—none0/10

   The evidence pack shows conventional markdown documentation and a probe confirming no OpenAPI/interactive API spec is published (all candidate paths 404). There is no mention of an interactive API reference or runnable examples/playground anywhere in the docs or GitHub materials.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langc…”

   proof ↗⚑ flag

   CrewAI—none0/10

   Evidence shows only static API reference pages (e.g., kickoff/status/resume endpoints) and markdown-based docs, not an interactive, runnable API explorer. A probe explicitly found no OpenAPI/Swagger spec at standard locations, indicating no interactive console for testing API calls exists.

   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.crewai.com/openapi.json, https://docs.crewai.com/swagger.json, https://docs.crewai.com/…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   LangGraph—none0/10

   While LangGraph's Agent Server is documented as exposing REST API endpoints for runs, threads, and assistants (langgraph-docs-30/37), no evidence shows a downloadable machine-readable spec (OpenAPI/Swagger) — a direct probe for openapi.json, swagger.json, and related paths returned 404 on all candidates (langgraph-probe-3). missing for 10: a documented OpenAPI/Swagger endpoint or downloadable spec file, any doc page referencing 'openapi' or 'swagger' for the Agent Server, confirmation from the actual running server rather than just the docs site.

   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langc…”

   proof ↗⚑ flag

   CrewAI—none0/10

   CrewAI documents REST-style API endpoints (kickoff, status, resume) for its Enterprise/Edge offering, suggesting an API surface exists, but a direct probe for machine-readable spec files (openapi.json, swagger.json, etc.) returned 404 on all candidate paths, and no documentation links to a downloadable OpenAPI/Swagger spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.crewai.com/openapi.json, https://docs.crewai.com/swagger.json, https://docs.crewai.com/…”
   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to LangGraph
   LangGraph~partialclaimed4/10

   LangGraph docs show a local/dev path (LangGraph CLI running the Agent Server locally, in-memory or dev checkpointers) distinct from a production database-backed checkpointer, which implies a way to iterate locally without touching production data, but there is no explicit 'sandbox environment' or test-data-isolation feature documented. missing for 10: no dedicated sandbox/staging environment concept, no explicit guidance on isolating test data from production, no independent confirmation that local runs are safely isolated from production stores.

   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”
   • [claimed-docs] “In production, use a checkpointer backed by a database: from langgraph.checkpoint.postgres import PostgresSaver”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally.”

   proof ↗⚑ flag

   CrewAI—none0/10

   CrewAI's `crewai test` command (crewai-docs-9/27/40) runs performance-metric iterations on a crew, but no evidence describes a distinct sandbox environment or mechanism to isolate test runs from production data/systems. Enterprise deployment docs mention managed infra and APIs but nothing about sandbox vs. production data separation.

   ⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round to CrewAI
   LangGraph—none0/10

   The evidence pack contains no documentation of API versioning scheme or a formal deprecation policy for LangGraph's APIs; only general framework descriptions and a community complaint that the framework 'often introduces breaking changes' without being well documented, which is unrelated to any specific versioning/deprecation guarantee. This is an applicable axis for a developer framework/API, but no supporting evidence exists.

   • [community] “I read the article but have yet to understand why someone would want to use a framework that introduces meaningless abstractions that are no…”

   proof ↗⚑ flag

   CrewAI~partialprobed3/10

   There is only glancing evidence: CLI commands show deprecation warnings when replaced (crewai-docs-26), and the docs site exposes multiple version snapshots (Edge, v1.15.21) suggesting some versioning of releases (crewai-probe-1). However, no formal documented deprecation policy for the REST/enterprise API or SDK, no version-numbering scheme (e.g., v1/v2 REST endpoints), and no OpenAPI spec was found (crewai-probe-3 shows 404s for openapi.json), meaning API contracts aren't clearly versioned or governed by a public deprecation timeline. Missing for 10: explicit API versioning scheme (e.g., /v1/, /v2/ endpoints), a published deprecation/sunset policy document, and a machine-readable API spec (OpenAPI) confirming version support windows.

   • [claimed-docs] “These older commands still work but print a yellow deprecation warning.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.crewai.com/llms.txt # CrewAI - [Edge (199 pages)](https://docs.crewai.com/_llms/en/edge.md): Docum…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.crewai.com/openapi.json, https://docs.crewai.com/swagger.json, https://docs.crewai.com/…”

   proof ↗⚑ flag

Agent authoring

1. developerDefine an agent with typed custom tools in a few lines of code

   weight 3 · round to CrewAI
   LangGraph~partialcommunity4/10

   Evidence confirms LangGraph nodes/tools can be freely custom-coded (comm-10) and that tool integration exists via MCPAdapter (docs-22), implying developers can define custom tools, but the pack lacks any concrete code example showing typed tool definitions or a 'few lines of code' walkthrough for tool creation. Missing for 10: a documented tool-definition API/decorator with type hints, a minimal code snippet, and independent confirmation of ease-of-use for typed tools.

   • [community] “In langgraph nodes are just functions that can do whatever you want... you don't have to use langchain tools or ToolNode with langgraph, you…”
   • [claimed-docs] “LangChain agents call tools defined on MCP servers through MCPAdapter, which discovers a server's tools and adapts them into LangChain tools…”
   • [claimed-docs] “At its core, LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components”
   • [claimed-docs] “LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components: State, Nodes, Edges.”

   proof ↗⚑ flag

   CrewAI~partialclaimed6/10

   CrewAI docs show agents can be defined with tools and integrate with CrewAI Toolkit/LangChain tools (crewai-docs-6, crewai-docs-52), and tools are described as skills/functions agents can use, implying a simple decorator/class-based definition, but the evidence pack lacks a concrete code example showing typed tool schemas (e.g., pydantic-based args_schema) or a minimal few-line snippet demonstrating this end-to-end. missing for 10: an explicit few-line code sample defining a typed custom tool (with input schema) and attaching it to an agent, plus independent/hands-on confirmation of ease-of-use for typed tools.

   • [claimed-docs] “CrewAI tools empower agents with capabilities ranging from web searching and data analysis to collaboration and delegating tasks among cowor…”
   • [claimed-docs] “A tool in CrewAI is a skill or function that agents can utilize to perform various actions. This includes tools from the [CrewAI Toolkit] an…”
   • [claimed-docs] “In the CrewAI framework, an `Agent` is an autonomous unit that can: * Perform specific tasks * Make decisions based on its role and goal * U…”
   • [claimed-docs] “Perform specific tasks * Make decisions based on its role and goal * Use tools to accomplish objectives * Communicate and collaborate with o…”

   proof ↗⚑ flag

Ai buildability

1. ai-native userHave a coding agent scaffold a new agent project from an official CLI or template in one command

   weight 2 · round to CrewAI
   LangGraph~partialprobed4/10

   An official LangGraph CLI is documented (installable, used to build/run the Agent Server locally), which is the kind of official tool a scaffold command would live in, but the evidence never shows a specific one-command project/template scaffolding action (e.g., `langgraph new`) — only server build/run functionality is described. missing for 10: explicit scaffold/template command documentation, a first-command quickstart example, independent confirmation it works as a one-command project generator.

   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally.”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”
   • [probe] “official CLI documented at https://docs.langchain.com/langsmith/cli”

   proof ↗⚑ flag

   CrewAI✓fullprobed8/10

   CrewAI ships an official CLI (`crewai create`) to scaffold new crew/flow/tool/template projects in one command, installed via `uv tool install crewai`, and docs even provide a ready-to-paste setup prompt for coding agents like Claude Code/Cursor/Codex to bootstrap and drive the CLI. This directly matches the story of an ai-native user having a coding agent scaffold a new agent project from an official CLI. missing for 10: independent/hands-on confirmation of the scaffold command's actual output structure and no community verification of the agent-driven setup-prompt workflow.

   • [claimed-docs] “The CrewAI CLI provides a set of commands to interact with CrewAI, allowing you to create, train, run, and manage crews & flows”
   • [claimed-docs] “Create a new crew, flow, tool, skill, or template project.”
   • [claimed-docs] “Run the following command to install crewai CLI: uv tool install crewai”
   • [claimed-docs] “Copy a ready-to-paste setup prompt for Claude Code, Codex, Cursor, or any coding agent. It installs the official CrewAI skills, checks the C…”
   • [probe] “official CLI documented at https://docs.crewai.com/en/concepts/cli”

   proof ↗⚑ flag

2. ai-native userRun the framework's example agents headlessly from a terminal so an agent can verify what it just built

   weight 2 · round to CrewAI
   LangGraph~partialprobed4/10

   LangGraph ships a CLI that runs an Agent Server locally and graphs expose sync/async invoke and stream methods that can be called headlessly from a terminal or script, which technically enables scripted verification runs. However, there is no evidence of a curated set of 'example agents' meant for headless self-verification, nor any documented workflow where an agent inspects its own build via terminal output. Missing for 10: dedicated example-agent scripts/quickstarts, explicit headless verification/testing workflow, and any first-party or community confirmation that agents use this for self-check.

   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally.”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”
   • [claimed-docs] “LangGraph graphs expose the stream (sync) and astream (async) methods to yield streamed outputs as iterators.”
   • [probe] “official CLI documented at https://docs.langchain.com/langsmith/cli”

   proof ↗⚑ flag

   CrewAI~partialprobed5/10

   CrewAI ships a CLI (`crewai run`, `crewai test`) that can run crews/agents headlessly from a terminal and provide performance metrics, which an agent could invoke to verify its own build, but there is no documented 'example agents' bundled specifically for self-verification nor evidence of an agent-oriented headless verification workflow. missing for 10: a documented example/template agent meant for headless self-check, explicit guidance on running examples non-interactively for verification purposes, and independent confirmation of this exact workflow.

   • [claimed-docs] “The CrewAI CLI provides a set of commands to interact with CrewAI, allowing you to create, train, run, and manage crews & flows”
   • [claimed-docs] “We added the CLI command crewai test to make it easy to test your crew. This command will run your crew for a specified number of iterations…”
   • [claimed-docs] “Create a new crew, flow, tool, skill, or template project.”
   • [claimed-docs] “The CrewAI CLI provides a set of commands to interact with CrewAI, allowing you to create, train, run, and manage crews & flows.”
   • [claimed-docs] “We added the CLI command `crewai test` to make it easy to test your crew. This command will run your crew for a specified number of iteratio…”
   • [probe] “official CLI documented at https://docs.crewai.com/en/concepts/cli”

   proof ↗⚑ flag

3. ai-native userRely on strict typing and schema validation so a coding agent catches its own mistakes at build time

   weight 2 · round to LangGraph
   LangGraph~partialclaimed3/10

   The only relevant evidence is a passing mention that the graph builder performs 'basic checks on the structure of your graph (no orphaned nodes, etc.)' at compile time, which is a thin form of build-time validation but not strict typing or schema validation of agent outputs/tools. No evidence describes typed state schemas, Pydantic/TypedDict validation, or static type-checking catching agent mistakes. missing for 10: explicit schema/type validation for node inputs-outputs, evidence of build-time type errors being caught, independent confirmation of this behavior in practice.

   • [claimed-docs] “It provides a few basic checks on the structure of your graph (no orphaned nodes, etc). It is also where you can specify runtime args like c…”

   proof ↗⚑ flag

   CrewAI—none0/10

   The evidence pack covers CrewAI's agents, tasks, flows, memory, CLI, and enterprise features but contains no mention of strict typing, schema validation, or build-time error catching for agent outputs or task definitions. Nothing describes Pydantic-based output schemas, type-checked task I/O, or static validation tooling that would let an agent catch its own mistakes before runtime.

   ⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to CrewAI
   LangGraph—none0/10

   The evidence pack describes graph orchestration, streaming, checkpointing, memory, and human-in-the-loop features, and mentions internal parallel execution within a single graph (Pregel/BSP model), but there is no documentation of a bulk/batch API for invoking the graph across many independent items or records at once (e.g., a .batch()/.abatch() method or bulk import/export tooling). Missing for 10: explicit batch invocation API, bulk data import/export tooling, or evidence of processing many independent items concurrently as a first-class feature.

   • [community] “LangGraph implements a variant of the Pregel/BSP algorithm for orchestrating workflows with cycles (ie. not DAGs) and parallelism without da…”
   • [claimed-docs] “LangGraph graphs expose the stream (sync) and astream (async) methods to yield streamed outputs as iterators.”
   • [claimed-docs] “It exposes graph execution through stream modes such as updates, values, messages, custom, checkpoints, tasks, and debug.”

   proof ↗⚑ flag

   CrewAI~partialcommunity5/10

   CrewAI's task/flow/crew orchestration model supports building agents that iterate over many items programmatically, and a documented community example shows exactly this — an agent bulk-categorizing, drafting, and auto-deleting many emails in an inbox (crewai-comm-1). However, there is no first-party documentation of a dedicated bulk/batch-processing API (e.g., a 'run for each item' or batch endpoint) in the evidence pack, only sequential/hierarchical single-task execution concepts. missing for 10: explicit bulk/batch API or 'for-each' primitive in official docs, independent benchmarks of bulk-scale runs, guardrails/rate-limit guidance for large batch jobs.

   • [community] “Show HN: built AI agents with CrewAI to automate an old Gmail account—categorizing emails, drafting replies, Slack alerts for high-priority …”
   • [claimed-docs] “process = Process.sequential # or Process.hierarchical”
   • [claimed-docs] “Sequential: Tasks are executed in the order they are defined * Hierarchical: Tasks are as”
   • [claimed-docs] “Flows allow you to create structured, event-driven workflows. They provide a seamless way to connect multiple tasks, manage state, and contr…”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to CrewAI
   LangGraph~partialclaimed4/10

   LangGraph's graph model (State, Nodes, Edges) and interrupts allow conditional routing and pausing at specific points, which can act like internal rules driving actions as state changes, and its Agent Server exposes API endpoints for runs/threads that could be invoked on external events. However, the evidence pack contains no explicit documentation of an event-trigger system (e.g., webhooks, schedules, external event listeners, conditional-edge rule definitions) that automatically fires actions outside of manually invoked graph runs. Missing for 10: explicit conditional-edge/rule syntax, documented external event triggers (webhook/cron), and evidence of automatic action firing without a user-initiated run.

   • [claimed-docs] “At its core, LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components”
   • [claimed-docs] “By composing Nodes and Edges, you can create complex, looping workflows that evolve the state over time.”
   • [claimed-docs] “LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components: State, Nodes, Edges.”
   • [claimed-docs] “Interrupts allow you to pause graph execution at specific points and wait for external input before continuing.”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”

   proof ↗⚑ flag

   CrewAI~partialcommunity6/10

   CrewAI's Flows provide event-driven workflow orchestration (state changes, task chaining) and webhook streaming/event delivery to external systems, plus decorators like @after_kickoff for triggering follow-up actions, and a community example shows event-triggered automation (Slack alerts, auto-delete) built with CrewAI. However, this is more a code-defined workflow/agent framework than a declarative 'if-event-then-rule' automation engine, and a competitor explicitly notes CrewAI lacks persistent state/deterministic hooks for pausing/controlling agents that some rule-based automation needs. missing for 10: a first-party declarative rule-trigger system (e.g., condition-action rule builder), independent verification of robustness of event triggers beyond one community project, and confirmation of deterministic hook/state support contested by competitor.

   • [claimed-docs] “Flows allow you to create structured, event-driven workflows. They provide a seamless way to connect multiple tasks, manage state, and contr…”
   • [claimed-docs] “Flows allow you to create structured, event-driven workflows. They provide a seamless way to connect multiple tasks, manage state, and contr…”
   • [claimed-docs] “Webhook Streaming Stream real-time events and updates to your systems”
   • [community] “Show HN: built AI agents with CrewAI to automate an old Gmail account—categorizing emails, drafting replies, Slack alerts for high-priority …”
   • [community] “Competitor (Portia) founder's take: 'We think Crew AI is great. It's just that a lot of businesses we spoke to want the autonomy/decision ma…”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round drawn
   LangGraph—none0/10

   No evidence in the pack mentions cron-style scheduling, recurring triggers, or time-based/periodic job execution; LangGraph's docs focus on persistence, checkpointing, interrupts, streaming, and durable execution but not scheduled/recurring workflow invocation.

   ⚑ flag

   CrewAI—none0/10

   CrewAI's docs describe crews, flows, tasks, memory, CLI, and an enterprise kickoff/resume API, but nothing in the evidence pack mentions cron-style scheduling, recurring triggers, or any built-in job scheduler for automatically re-running crews/flows on a timer.

   ⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round to LangGraph
   LangGraph~partialclaimed5/10

   LangGraph's checkpointer/time-travel and interrupt features let developers pause for human review and resume or roll back to prior graph states, and LangSmith tracing gives visibility into execution paths, covering 'review' and partial 'rollback'. However there's no evidence of an explicit versioning system for automations (e.g., named/versioned assistant deployments, diffing or rollback UI) beyond code-level state checkpoints. missing for 10: explicit automation/version management (e.g., versioned assistants/deployments), a UI for reviewing/rolling back workflow versions, independent hands-on confirmation of rollback working in production.

   • [claimed-docs] “Checkpointers persist a thread's graph state as checkpoints. Use them for short-term, thread-scoped memory, including conversation continuit…”
   • [claimed-docs] “Checkpointing keeps your place: the checkpointer writes the exact graph state so you can resume later, even when in an error state.”
   • [github] “Seamlessly incorporate human oversight by inspecting and modifying agent state at any point during execution.”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “Trace and compare these workflow patterns with LangSmith... Follow the tracing quickstart to see how data flows through each step.”

   proof ↗⚑ flag

   CrewAI~partialcommunity2/10

   CrewAI's docs note that agent/task configs are defined in YAML which 'allows you to version control your agent settings,' implying git-based versioning of automation definitions, but there is no first-party feature for reviewing changes or rolling back a running crew/flow to a prior state. A competitor explicitly contrasts CrewAI's lack of persistent state/deterministic pause-and-resume hooks, further underscoring the absence of built-in rollback mechanics. missing for 10: dedicated review workflow (diff/approval) for automation changes, built-in rollback/undo of crew or flow state, versioned deployment history beyond generic YAML+git.

   • [claimed-docs] “The YAML configuration allows you to: * Version control your agent settings * Easily switch between different models”
   • [community] “Competitor (Portia) founder's take: 'We think Crew AI is great. It's just that a lot of businesses we spoke to want the autonomy/decision ma…”

   proof ↗⚑ flag

Deployment

1. engineering-leadDeploy an agent to a managed runtime and call it as an API endpoint

   weight 2 · round drawn
   LangGraph~partialcommunity6/10

   Docs confirm a LangGraph CLI/Agent Server that exposes API endpoints for runs, threads, and assistants with managed checkpointing/storage (langgraph-docs-30/37/23), and GitHub claims 'production-ready deployment' with scalable infrastructure for stateful agents (langgraph-gh-9). However, a community engineer explicitly asks how to deploy LangGraph as a production API beyond 'langgraph serve' locally, suggesting the managed/production deployment path is not fully clear from hands-on experience (langgraph-comm-7). Missing for 10: independent hands-on confirmation of a hosted managed cloud runtime (vs. local CLI server), and details on production SLAs/scaling beyond marketing claims.

   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally.”
   • [github] “Production-ready deployment — Deploy sophisticated agent systems confidently with scalable infrastructure designed to handle the unique chal…”
   • [community] “How can one deploy LangGraph as an API (with production like features)? I have worked with langgraph serve to deploy locally, but are there …”

   proof ↗⚑ flag

   CrewAI~partialclaimed6/10

   CrewAI AMP/Enterprise explicitly offers one-click deployment to managed infrastructure and exposes deployed crews via REST API (kickoff/status/resume endpoints), satisfying the core ask. However, this is a separate paid product (AMP) from the open-source framework, requires additional setup, and evidence lacks independent/hands-on confirmation or details on scaling, auth, or SLAs. Missing for 10: independent/hands-on verification of the managed deployment and API in production, pricing/limits transparency, and confirmation this works outside the enterprise tier.

   • [claimed-docs] “Deploy your crews to a managed infrastructure with a few clicks”
   • [claimed-docs] “Deploy your crews to a managed infrastructure and monitor their execution in real-time.”
   • [claimed-docs] “Deploy your crews to a managed infrastructure with a few clicks... Access your deployed crews via REST API for integration with existing sys…”
   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”
   • [claimed-docs] “CrewAI AMP extends the power of the open-source framework with features designed for production deployments, collaboration, and scalability.…”

   proof ↗⚑ flag

2. engineering-leadRun my agents entirely on my own infrastructure with no dependence on the vendor's platform

   weight 2 · round drawn
   LangGraph✓fullcommunity7/10

   LangGraph is open-source, pip-installable, and includes a CLI to build/run the Agent Server locally with self-managed checkpointing via Postgres or other backends, meaning agents can run fully on self-hosted infra without the vendor's managed platform. However, evidence pack emphasizes LangSmith for tracing/debugging and doesn't explicitly discuss self-hosting at scale or full platform parity without LangSmith. missing for 10: independent verification of large-scale self-hosted production deployments, explicit statement that all deployment features (e.g., cron/scheduling, multi-tenant auth) work without LangSmith/LangGraph Platform, and clearer separation of open-source vs paid-platform features.

   • [github] “pip install -U langgraph”
   • [claimed-docs] “In production, use a checkpointer backed by a database: from langgraph.checkpoint.postgres import PostgresSaver”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally.”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”
   • [community] “How can one deploy LangGraph as an API (with production like features)? I have worked with langgraph serve to deploy locally, but are there …”

   proof ↗⚑ flag

   CrewAI✓fullcommunity7/10

   CrewAI is an open-source Python framework installable via pip/uv with a local CLI to create, run, and test crews entirely in the developer's own environment, with no forced dependency on CrewAI's hosted platform — the 'AMP'/enterprise offerings (managed infra, Crew Studio, tool repository) are explicitly optional add-ons layered on top of the OSS core. missing for 10: no independent hands-on confirmation of fully air-gapped/self-hosted production runs at scale, and no explicit documentation contrasting self-hosted vs vendor-dependent feature parity.

   • [claimed-docs] “Run the following command to install crewai CLI: uv tool install crewai”
   • [claimed-docs] “Run the following command to install `crewai` CLI: uv tool install crewai ”
   • [claimed-docs] “The CrewAI CLI provides a set of commands to interact with CrewAI, allowing you to create, train, run, and manage crews & flows”
   • [claimed-docs] “The CrewAI CLI provides a set of commands to interact with CrewAI, allowing you to create, train, run, and manage crews & flows.”
   • [claimed-docs] “CrewAI AMP extends the power of the open-source framework with features designed for production deployments, collaboration, and scalability.…”
   • [claimed-docs] “Deploy your crews to a managed infrastructure with a few clicks”
   • [community] “On a CrewAI-based personal email agent: 'Local personal agents are going to be a major app type... They run locally on my computer with my f…”

   proof ↗⚑ flag

Portability

1. developerSwap the underlying LLM provider or model without rewriting my agent

   weight 3 · round to CrewAI
   LangGraph—none0/10

   The evidence pack contains no documentation or examples showing that LangGraph nodes use a provider-agnostic model interface (e.g., a single call that can swap between OpenAI, Anthropic, etc. without code changes); it focuses on graph structure, checkpointing, streaming, and human-in-the-loop features, not model abstraction. A stray community comment about 'bring your own keys' apps is too thin and non-technical to establish this capability. missing for 10: any docs on a unified chat-model interface, model-swap examples, or provider abstraction demonstrating no-rewrite portability.

   • [community] “The use case where they are helpful is 'bring your own keys' apps... The abstraction is very much worth it for me. That said: I migrated fro…”

   proof ↗⚑ flag

   CrewAI✓fullclaimed7/10

   CrewAI docs explicitly state it integrates with multiple LLM providers via their native SDKs and that YAML-based agent configuration lets developers 'easily switch between different models' without rewriting agent logic, directly matching the deployment-portability story. missing for 10: independent/hands-on confirmation that switching providers requires zero code changes, and documentation of any provider-specific feature gaps that could break portability.

   • [claimed-docs] “CrewAI integrates with multiple LLM providers through providers native sdks, giving you the flexibility to choose the right model for your s…”
   • [claimed-docs] “The YAML configuration allows you to: * Version control your agent settings * Easily switch between different models”
   • [claimed-docs] “CrewAI integrates with multiple LLM providers through providers native sdks, giving you the flexibility to choose the right model for your s…”

   proof ↗⚑ flag

Evals

1. engineering-leadScore agent quality with built-in evals and run them as part of CI

   weight 2 · round to CrewAI
   LangGraph—none0/10

   The evidence pack shows LangGraph provides tracing/visualization via LangSmith and debugging tools, but contains no mention of built-in evals, scoring agent quality, or running evals as part of CI. Evals appear to be a separate LangSmith capability not documented here.

   ⚑ flag

   CrewAI~partialclaimed4/10

   CrewAI ships a `crewai test` CLI command that runs the crew for N iterations and reports performance metrics, plus observability/tracing for execution times and token usage, which could be scripted into CI. However, there's no documented eval framework (accuracy/quality scoring, rubrics, pass/fail thresholds) or first-party CI integration guidance. Missing for 10: a dedicated eval scoring API/rubric beyond generic performance metrics, and explicit CI pipeline documentation/examples.

   • [claimed-docs] “We added the CLI command crewai test to make it easy to test your crew. This command will run your crew for a specified number of iterations…”
   • [claimed-docs] “We added the CLI command `crewai test` to make it easy to test your crew.”
   • [claimed-docs] “We added the CLI command `crewai test` to make it easy to test your crew. This command will run your crew for a specified number of iteratio…”
   • [claimed-docs] “Performance Monitoring: Track agent execution times, token usage, and resource consumption”
   • [claimed-docs] “Track agent execution times, token usage, and resource consumption”
   • [github] “Tracing & Observability: Monitor and track your AI agents and workflows in real-time, including metrics, logs, and traces.”

   proof ↗⚑ flag

Testing

1. developerUnit-test agents with mocked models and tools

   weight 2 · round drawn
   LangGraph—none0/10

   No evidence pack item documents unit-testing patterns, mocking of models/tools, or a testing framework/utilities for LangGraph agents; the closest is a community remark that nodes are plain functions you can implement however you like, which only implies testability rather than demonstrating it.

   • [community] “In langgraph nodes are just functions that can do whatever you want... you don't have to use langchain tools or ToolNode with langgraph, you…”

   proof ↗⚑ flag

   CrewAI—none0/10

   The evidence shows a `crewai test` CLI command that runs a crew for several iterations and reports performance metrics, but this is an integration-style evaluation run, not unit testing with mocked models/tools. There is no documentation of dependency injection, mock LLM/tool interfaces, or pytest-style fixtures for isolating agent logic from real model/tool calls.

   • [claimed-docs] “We added the CLI command crewai test to make it easy to test your crew. This command will run your crew for a specified number of iterations…”
   • [claimed-docs] “We added the CLI command `crewai test` to make it easy to test your crew.”
   • [claimed-docs] “We added the CLI command `crewai test` to make it easy to test your crew. This command will run your crew for a specified number of iteratio…”

   proof ↗⚑ flag

Tracing

1. developerTrace every LLM call and tool invocation of an agent run in an observability UI

   weight 3 · round to LangGraph
   LangGraph✓fullcommunity8/10

   LangGraph integrates with LangSmith to provide tracing and debugging UI that visualizes execution paths, captures state transitions, and provides runtime metrics for agent runs, with docs explicitly directing users to trace and compare workflow patterns via the tracing quickstart. missing for 10: no independent/hands-on confirmation of trace fidelity for LLM calls and tool invocations specifically, and one community comment notes streaming/observability implementation is left partly to the client.

   • [github] “Debugging with LangSmith — Gain deep visibility into complex agent behavior with visualization tools that trace execution paths, capture sta…”
   • [github] “Gain deep visibility into complex agent behavior with visualization tools that trace execution paths, capture state transitions, and provide…”
   • [claimed-docs] “Trace and compare these workflow patterns with LangSmith... Follow the tracing quickstart to see how data flows through each step.”
   • [claimed-docs] “It exposes graph execution through stream modes such as updates, values, messages, custom, checkpoints, tasks, and debug.”
   • [community] “The one thing I wish was better developed is persistence and streaming - they give sample code to stream, but it's essentially a complete im…”

   proof ↗⚑ flag

   CrewAI~partialclaimed7/10

   CrewAI documents built-in observability with tracing of metrics, logs, and traces for agent/tool calls (crewai-gh-2, crewai-docs-11/29), and CrewAI AMP/enterprise offers a real-time monitoring dashboard (crewai-docs-30/55). However, the evidence is entirely first-party doc claims with no independent/hands-on confirmation of the observability UI's granularity (e.g., per-LLM-call or per-tool-invocation trace views) or screenshots demonstrating it works as described. missing for 10: independent/hands-on validation of the trace UI, detail on per-call/tool-invocation granularity, and evidence of third-party observability integrations (e.g., Langfuse/Arize) being verified in practice.

   • [github] “Tracing & Observability: Monitor and track your AI agents and workflows in real-time, including metrics, logs, and traces.”
   • [claimed-docs] “Performance Monitoring: Track agent execution times, token usage, and resource consumption”
   • [claimed-docs] “Track agent execution times, token usage, and resource consumption”
   • [claimed-docs] “Deploy your crews to a managed infrastructure and monitor their execution in real-time.”
   • [claimed-docs] “CrewAI AMP extends the power of the open-source framework with features designed for production deployments, collaboration, and scalability.…”

   proof ↗⚑ flag

Guardrails

1. developerAttach input/output guardrails that validate, transform, or block unsafe content

   weight 3 · round drawn
   LangGraph—none0/10

   The evidence describes LangGraph's general graph/node architecture, persistence, interrupts, and human-in-the-loop features, but nothing documents a guardrails feature (input/output validation, content moderation, or blocking unsafe content). While nodes are flexible functions (allowing a developer to hand-roll such logic), there is no first-party guardrails API, validator, or moderation integration cited.

   • [community] “In langgraph nodes are just functions that can do whatever you want... you don't have to use langchain tools or ToolNode with langgraph, you…”
   • [claimed-docs] “At its core, LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components”
   • [claimed-docs] “By composing Nodes and Edges, you can create complex, looping workflows that evolve the state over time.”

   proof ↗⚑ flag

   CrewAI—none0/10

   The evidence pack contains no documentation of a guardrail mechanism for validating, transforming, or blocking agent input/output content — no task-level or agent-level guardrail parameter, content filter, or safety-check API is mentioned anywhere in the docs. In fact, a competitor explicitly claims 'where Crew AI lacks that' regarding dial-up/down guardrails, and a community critique flags CrewAI's lack of safeguards against destructive actions (e.g., no guardrail preventing permanent email deletion on misclassification).

   • [community] “Competitor (Portia) founder's take: 'We think Crew AI is great. It's just that a lot of businesses we spoke to want the autonomy/decision ma…”
   • [community] “Commenter critique of a CrewAI-based email agent: 'Why not archive vs. the destructive delete... Seems like a default aggressive stance,' ra…”

   proof ↗⚑ flag

2. engineering-leadRestrict what an agent may do with fine-grained tool permissions and sandboxed execution

   weight 2 · round to CrewAI
   LangGraph—none0/10

   The evidence pack documents human-in-the-loop interrupts, checkpointing, and custom node logic (langgraph-docs-4, langgraph-gh-2), but nowhere describes fine-grained per-tool permission scoping or sandboxed/isolated execution environments for agent actions. Community notes even mention nodes/tools are 'whatever you want' custom code (langgraph-comm-10), implying no built-in permissioning or sandbox layer is provided by the framework itself.

   • [claimed-docs] “Interrupts allow you to pause graph execution at specific points and wait for external input before continuing.”
   • [github] “Seamlessly incorporate human oversight by inspecting and modifying agent state at any point during execution.”
   • [github] “Human-in-the-loop — Seamlessly incorporate human oversight by inspecting and modifying agent state at any point during execution.”
   • [community] “In langgraph nodes are just functions that can do whatever you want... you don't have to use langchain tools or ToolNode with langgraph, you…”
   • [community] “Almost none of those things are part of the LangGraph framework? LangGraph does the scheduling, checkpointing, state management, etc. All of…”

   proof ↗⚑ flag

   CrewAI~partialcommunity3/10

   CrewAI lets you assign specific tools to a given Agent/Task (crewai-docs-2, crewai-docs-6), which is a rudimentary form of restricting what an agent can do, but there is no documentation of fine-grained permission scopes, execution sandboxing, or resource isolation. A competitor's founder explicitly states CrewAI 'lacks' dial-up/down guardrails and deterministic pause hooks (crewai-comm-4), and a community example shows a CrewAI agent auto-deleting emails with no safety confirmation, drawing criticism for the lack of guardrails (crewai-comm-2). missing for 10: sandboxed/isolated execution environment, granular permission policies per tool/action, first-party guardrail/approval mechanisms, and evidence rebutting the community claim that such guardrails are absent.

   • [claimed-docs] “a "Task" is a specific assignment completed by an "Agent". Tasks provide all necessary details for execution, such as a description, the age…”
   • [claimed-docs] “CrewAI tools empower agents with capabilities ranging from web searching and data analysis to collaboration and delegating tasks among cowor…”
   • [community] “Commenter critique of a CrewAI-based email agent: 'Why not archive vs. the destructive delete... Seems like a default aggressive stance,' ra…”
   • [community] “Competitor (Portia) founder's take: 'We think Crew AI is great. It's just that a lot of businesses we spoke to want the autonomy/decision ma…”

   proof ↗⚑ flag

Approval flows

1. developerPause an agent mid-run for human input or approval and resume with the human's decision

   weight 3 · round to LangGraph
   LangGraph✓fullclaimed9/10

   LangGraph has a dedicated interrupts feature explicitly designed to pause graph execution and wait for external input, with resumption via re-invoking the graph with a Command object carrying the human's decision; this is backed by checkpointer-based persistence for durability across pauses, and GitHub docs explicitly list 'Human-in-the-loop' as a core capability allowing inspection/modification of agent state mid-execution. Missing for 10: independent hands-on developer account specifically validating the interrupt/resume workflow (community evidence discusses persistence/streaming generally but not this exact HITL pause-resume flow).

   • [claimed-docs] “Interrupts allow you to pause graph execution at specific points and wait for external input before continuing.”
   • [claimed-docs] “you resume execution by re-invoking the graph using Command, which then becomes the return value of the interrupt() call from inside the nod…”
   • [claimed-docs] “when you're ready to continue, you resume execution by re-invoking the graph using Command, which then becomes the return value of the inter…”
   • [claimed-docs] “Checkpointing keeps your place: the checkpointer writes the exact graph state so you can resume later, even when in an error state.”
   • [claimed-docs] “Checkpointers persist a thread's graph state as checkpoints. Use them for short-term, thread-scoped memory, including conversation continuit…”
   • [github] “Seamlessly incorporate human oversight by inspecting and modifying agent state at any point during execution.”
   • [github] “Human-in-the-loop — Seamlessly incorporate human oversight by inspecting and modifying agent state at any point during execution.”

   proof ↗⚑ flag

   CrewAI~partialcommunity5/10

   CrewAI's Edge API explicitly documents kickoff/status/resume endpoints, including 'Resume crew execution with human feedback' (crewai-docs-56/57), which matches the pause-for-approval-and-resume story. However, this exists mainly as a REST-level control-plane feature (not deeply documented with in-process human_input examples), and a competitor's public claim explicitly disputes CrewAI's pause/resume robustness ('deterministic hooks so you can pause an agent... where Crew AI lacks that' — crewai-comm-4), creating real doubt about how mature this capability is in practice. missing for 10: first-party walkthrough/code sample of pausing mid-task for human approval, independent hands-on confirmation that resume-with-feedback works reliably, and rebuttal to the competitor's lacks-that claim.

   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”
   • [community] “Competitor (Portia) founder's take: 'We think Crew AI is great. It's just that a lot of businesses we spoke to want the autonomy/decision ma…”

   proof ↗⚑ flag

2. engineering-leadRequire human approval before specific sensitive tool calls execute

   weight 2 · round to LangGraph
   LangGraph✓fullcommunity8/10

   LangGraph's interrupt() mechanism explicitly lets a graph pause execution at any node (e.g., a node calling a sensitive tool) and wait for external input, resuming only via Command re-invocation — this is the standard pattern for gating tool calls on human approval, and checkpointers back this with durable state. GitHub feature list and docs independently confirm 'Human-in-the-loop — seamlessly incorporate human oversight... at any point during execution,' and community commentary (HN) corroborates LangGraph as providing 'a state machine framework for human in the loop.' missing for 10: a first-party worked example specifically gating a tool-call node (vs. generic interrupt points), and independent hands-on validation of the approval-before-tool-call pattern.

   • [claimed-docs] “Interrupts allow you to pause graph execution at specific points and wait for external input before continuing.”
   • [claimed-docs] “you resume execution by re-invoking the graph using Command, which then becomes the return value of the interrupt() call from inside the nod…”
   • [claimed-docs] “when you're ready to continue, you resume execution by re-invoking the graph using Command, which then becomes the return value of the inter…”
   • [claimed-docs] “Checkpointing keeps your place: the checkpointer writes the exact graph state so you can resume later, even when in an error state.”
   • [github] “Seamlessly incorporate human oversight by inspecting and modifying agent state at any point during execution.”
   • [github] “Human-in-the-loop — Seamlessly incorporate human oversight by inspecting and modifying agent state at any point during execution.”
   • [community] “I think the main thing LangGraph adds is a state machine framework for human in the loop with time travel... you won't have to make your own…”

   proof ↗⚑ flag

   CrewAI!disputedcontradicted4/10

   CrewAI's Enterprise/Edge API exposes a POST /resume endpoint described as 'Resume crew execution with human feedback,' implying some human-in-the-loop pause/resume mechanism, but this is a crew-level checkpoint, not a documented gate that specifically intercepts sensitive tool calls before execution. A competitor founder (Portia) explicitly states in an independent HN discussion that 'Crew AI lacks' persistent state and deterministic hooks to pause an agent mid-run, directly contradicting the vendor's implied human-in-the-loop guardrail capability. Missing for 10: first-party docs describing per-tool or per-action approval gating, a documented tool-level 'require_confirmation' or callback mechanism, and independent corroboration that the resume API actually functions as a sensitive-action gate rather than a generic crew pause.

   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [community] “Competitor (Portia) founder's take: 'We think Crew AI is great. It's just that a lot of businesses we spoke to want the autonomy/decision ma…”

   proof ↗⚑ flag

Memory

1. developerTrim, summarize, or filter conversation history to keep an agent inside its context window

   weight 2 · round drawn
   LangGraph—none0/10

   Evidence describes LangGraph's persistence/checkpointing and short-term vs long-term memory model, but nothing in the pack documents specific mechanisms to trim, summarize, or filter conversation history to manage context window size. Missing for 10: any mention of message trimming utilities, summarization nodes/chains, or history-filtering APIs, and independent confirmation these features work as intended.

   • [claimed-docs] “Short-term memory (thread-level persistence) enables agents to track multi-turn conversations.”
   • [claimed-docs] “Add short-term memory as a part of your agent's state to enable multi-turn conversations.”
   • [claimed-docs] “Short-term memory (thread-level persistence) enables agents to track multi-turn conversations. To add short-term memory:”
   • [claimed-docs] “Checkpointers persist a thread's graph state as checkpoints. Use them for short-term, thread-scoped memory, including conversation continuit…”
   • [claimed-docs] “Stores persist application-defined data outside the graph state. Use them for long-term, cross-thread memory, including user preferences, fa…”

   proof ↗⚑ flag

   CrewAI—none0/10

   CrewAI's memory docs describe a retrieval system with composite scoring (semantic + recency + importance) for long-term memory recall, but there is no evidence of features to trim, summarize, or filter conversation history specifically to keep an agent within its context window during execution.

   • [claimed-docs] “CrewAI provides a unified memory system -- a single Memory class... supports adaptive-depth recall with composite scoring that blends semant…”
   • [claimed-docs] “Memory uses an LLM to analyze content when saving (inferring scope, categories, and importance) and supports adaptive-depth recall with comp…”
   • [claimed-docs] “CrewAI provides a **unified memory system** -- a single `Memory` class that replaces separate short-term, long-term, entity, and external me…”
   • [claimed-docs] “Retrieve -- results ranked by composite score (semantic + recency + importance)”

   proof ↗⚑ flag

2. developerGive agents long-term memory that persists across sessions and threads

   weight 2 · round drawn
   LangGraph✓fullcommunity8/10

   LangGraph documents a dedicated Store abstraction explicitly for 'long-term, cross-thread memory' (user preferences, facts, shared knowledge) separate from thread-scoped checkpointers, with guidance to back it with production databases (e.g., Postgres) and docs explicitly stating 'Add long-term memory to store user-specific or application-level data across sessions.' GitHub README also markets 'long-term persistent memory across sessions' as a core feature. Missing for 10: independent/hands-on corroboration of cross-thread memory at scale — one community comment vaguely notes persistence 'could be better developed,' but this is not a concrete failure report.

   • [claimed-docs] “Stores persist application-defined data outside the graph state. Use them for long-term, cross-thread memory, including user preferences, fa…”
   • [claimed-docs] “In production, use a checkpointer backed by a database: from langgraph.checkpoint.postgres import PostgresSaver”
   • [claimed-docs] “Add long-term memory to store user-specific or application-level data across sessions.”
   • [github] “Comprehensive memory — Create truly stateful agents with both short-term working memory for ongoing reasoning and long-term persistent memor…”
   • [community] “The one thing I wish was better developed is persistence and streaming - they give sample code to stream, but it's essentially a complete im…”

   proof ↗⚑ flag

   CrewAI✓fullclaimed8/10

   CrewAI's docs describe a unified Memory system that persists agent interactions with semantic, recency, and importance-based recall, explicitly replacing separate short-term/long-term/entity memory types with a single persistent API. This directly supports giving agents durable memory across executions. Missing for 10: explicit documentation confirming memory persists across separate sessions/threads (vs. within a single crew run) and independent/hands-on verification of long-term persistence.

   • [claimed-docs] “CrewAI provides a unified memory system -- a single Memory class... supports adaptive-depth recall with composite scoring that blends semant…”
   • [claimed-docs] “Memory uses an LLM to analyze content when saving (inferring scope, categories, and importance) and supports adaptive-depth recall with comp…”
   • [claimed-docs] “CrewAI provides a **unified memory system** -- a single `Memory` class... Memory uses an LLM to analyze content when saving (inferring scope…”
   • [claimed-docs] “CrewAI provides a **unified memory system** -- a single `Memory` class that replaces separate short-term, long-term, entity, and external me…”
   • [claimed-docs] “Retrieve -- results ranked by composite score (semantic + recency + importance)”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round drawn
   LangGraph~partialprobed4/10

   The LangGraph CLI/Agent Server exposes API endpoints for runs, threads, assistants, etc., suggesting programmatic access mirrors what LangGraph Studio UI shows, but there's no explicit documentation confirming full feature parity between the Studio UI and the API. missing for 10: explicit parity documentation, a public OpenAPI spec (probe found only 404s), and hands-on confirmation that every UI action (e.g., time-travel, breakpoints, state edits) is scriptable via API.

   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langc…”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally.”

   proof ↗⚑ flag

   CrewAI~partialprobed4/10

   CrewAI exposes execution-oriented API endpoints (POST /kickoff, /resume, GET /status) and enterprise REST access to deployed crews, so core run/monitor actions are API-reachable, but the no-code Visual Agent Builder / Crew Studio and Tool Repository features are marketed as UI-only conveniences with no documented API equivalent for building/configuring agents that mirrors the visual tools, and a direct probe for a public OpenAPI/swagger spec returned 404s, indicating no confirmed comprehensive API surface matching UI capabilities. missing for 10: documented API parity for visual agent/task building and tool repository management, a public OpenAPI spec confirming full UI-equivalent API coverage, independent confirmation of API completeness.

   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”
   • [claimed-docs] “Deploy your crews to a managed infrastructure with a few clicks... Access your deployed crews via REST API for integration with existing sys…”
   • [claimed-docs] “Crew Studio Create and customize crews using a no-code/low-code interface”
   • [claimed-docs] “Tool Repository Publish and install tools to enhance your crews’ capabilities”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.crewai.com/openapi.json, https://docs.crewai.com/swagger.json, https://docs.crewai.com/…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to LangGraph
   LangGraph~partialclaimed4/10

   LangGraph is open-source and self-hosted, and its persistence layer explicitly supports standard, user-controlled databases (e.g., PostgresSaver) rather than a proprietary hosted store, giving users inherent access to their own state/checkpoint data. However, there is no explicit documentation of an export feature, data-format guarantees, or a supported 'leave with your data' workflow beyond the fact that storage backends are pluggable/open. Missing for 10: documented export/import tooling, explicit open-format (e.g., JSON/CSV) data dumps, and any first-party or community confirmation of a clean migration/export path.

   • [claimed-docs] “In production, use a checkpointer backed by a database: from langgraph.checkpoint.postgres import PostgresSaver”
   • [claimed-docs] “Stores persist application-defined data outside the graph state. Use them for long-term, cross-thread memory, including user preferences, fa…”
   • [claimed-docs] “Checkpointers persist a thread's graph state as checkpoints. Use them for short-term, thread-scoped memory, including conversation continuit…”
   • [github] “LangGraph is a low-level orchestration framework for building, managing, and deploying long-running, stateful agents.”

   proof ↗⚑ flag

   CrewAI—none0/10

   While CrewAI's core framework is open-source and configs are local YAML (implying some inherent portability), the evidence pack contains no explicit data-export feature, no documented way to export memory/agent state in open formats, and no mention of account/data portability for the hosted AMP/Enterprise offering. Deployment and REST API docs focus on running crews, not exporting user data to leave the platform.

   ⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round to CrewAI
   LangGraph~partialclaimed5/10

   Evidence confirms LangGraph's source is publicly hosted on GitHub (langchain-ai/langgraph) with install instructions and repo links, implying the code is readable, but no evidence pack item explicitly states or cites an open-source license (e.g., MIT/Apache) for the repo. missing for 10: explicit license file/citation, confirmation of license terms, any independent verification of licensing terms.

   • [github] “pip install -U langgraph”
   • [github] “LangGraph is a low-level orchestration framework for building, managing, and deploying long-running, stateful agents.”
   • [github] “Low-level orchestration framework for building stateful agents.”

   proof ↗⚑ flag

   CrewAI~partialclaimed6/10

   The evidence confirms CrewAI's core is described as an 'open-source framework' (crewai-docs-55) and the source is hosted publicly on GitHub (crewai-gh-1, crewai-gh-2), supporting that AI-native users can read the source. However, no explicit license file or license type (e.g., MIT) is cited in the evidence pack. Missing for 10: explicit license identification (e.g., LICENSE file content or SPDX tag), independent confirmation of license terms, and any documentation page explicitly stating open-source licensing terms.

   • [claimed-docs] “CrewAI AMP extends the power of the open-source framework with features designed for production deployments, collaboration, and scalability.…”
   • [github] “It gives developers autonomous agent collaboration through Crews and precise, event-driven control through Flows.”
   • [github] “Tracing & Observability: Monitor and track your AI agents and workflows in real-time, including metrics, logs, and traces.”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round drawn
   LangGraph✓fullclaimed8/10

   LangGraph core is a pip-installable open-source library (langgraph-gh-5) with a CLI to build and run the Agent Server locally (langgraph-docs-23, langgraph-docs-30, langgraph-docs-37), and supports production-grade self-hosted persistence via PostgresSaver (langgraph-docs-13), confirming a fully self-hostable core product outside any managed SaaS. missing for 10: explicit license/self-hosting infra docs (scaling, containerization) and independent hands-on confirmation of self-hosting beyond CLI docs.

   • [github] “pip install -U langgraph”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally.”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”
   • [claimed-docs] “In production, use a checkpointer backed by a database: from langgraph.checkpoint.postgres import PostgresSaver”

   proof ↗⚑ flag

   CrewAI✓fullclaimed8/10

   CrewAI's core is an open-source Python framework (confirmed by GitHub repo and docs noting 'CrewAI AMP extends the power of the open-source framework'), installable and runnable entirely locally via `uv tool install crewai`/pip, with no dependency on a hosted service for core Crew/Flow/Agent execution. This directly satisfies self-hosting for an AI-native/developer persona. Missing for 10: no dedicated self-hosted production deployment guide (e.g., Docker/Kubernetes) or independent report of large-scale self-hosted operation.

   • [claimed-docs] “CrewAI AMP extends the power of the open-source framework with features designed for production deployments, collaboration, and scalability.…”
   • [claimed-docs] “Run the following command to install crewai CLI: uv tool install crewai”
   • [claimed-docs] “Run the following command to install `crewai` CLI: uv tool install crewai ”
   • [github] “It gives developers autonomous agent collaboration through Crews and precise, event-driven control through Flows.”
   • [claimed-docs] “Build collaborative AI agents, crews, and flows — production ready from day one.”

   proof ↗⚑ flag

Multi agent

1. developerOrchestrate multiple agents — handoffs, subagents, or crews — inside one workflow

   weight 3 · round to CrewAI
   LangGraph✓fullcommunity8/10

   LangGraph explicitly documents multi-agent orchestration patterns (handoffs, subagents/crews) via its graph-api and multi-agent docs, letting developers embed agent patterns as nodes, mix deterministic/agentic steps, and use Command/interrupts for handoffs, all within one stateful graph with persistence and streaming. Community evidence corroborates it as a legitimate stateful orchestration engine (not just a wrapper) supporting cycles/parallelism. Missing for 10: no hands-on demonstration of a specific named multi-agent 'crew' example or independent benchmark of handoff reliability at scale.

   • [claimed-docs] “Build bespoke execution flows with LangGraph, mixing deterministic logic and agentic behavior. Embed other patterns as nodes in your workflo…”
   • [claimed-docs] “Custom workflow: Build bespoke execution flows with LangGraph, mixing deterministic logic and agentic behavior. Embed other patterns as node…”
   • [claimed-docs] “Custom workflow — Build bespoke execution flows with LangGraph, mixing deterministic logic and agentic behavior. Embed other patterns as nod…”
   • [claimed-docs] “Here are the main patterns for building multi-agent systems, each suited to different use cases”
   • [claimed-docs] “At its core, LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components”
   • [claimed-docs] “LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components: State, Nodes, Edges.”
   • [claimed-docs] “Interrupts allow you to pause graph execution at specific points and wait for external input before continuing.”
   • [claimed-docs] “when you're ready to continue, you resume execution by re-invoking the graph using Command, which then becomes the return value of the inter…”
   • [community] “LangGraph implements a variant of the Pregel/BSP algorithm for orchestrating workflows with cycles (ie. not DAGs) and parallelism without da…”
   • [community] “LangGraph is different. It is a legitimate piece of workflow software and not a wrapper framework. Now, when it comes to workflow there are …”

   proof ↗⚑ flag

   CrewAI✓fullcommunity9/10

   CrewAI's core architecture is explicitly built for multi-agent orchestration: Agents, Tasks, and Crews collaborate with sequential/hierarchical processes and delegation, plus Flows for event-driven coordination across multiple crews/tasks (crewai-docs-3,4,20,21,35,36,46,48). A hands-on community build (Gmail triage agent with multiple collaborating CrewAI agents) corroborates real-world multi-agent use (crewai-comm-1), and a competitor's critique concerns guardrails/state persistence, not whether orchestration works at all (crewai-comm-4). missing for 10: independent large-scale benchmarking of complex multi-crew handoffs beyond docs and one hands-on example.

   • [claimed-docs] “process = Process.sequential # or Process.hierarchical”
   • [claimed-docs] “Flows allow you to create structured, event-driven workflows. They provide a seamless way to connect multiple tasks, manage state, and contr…”
   • [claimed-docs] “A crew in crewAI represents a collaborative group of agents working together to achieve a set of tasks.”
   • [claimed-docs] “Flows allow developers to combine and coordinate coding tasks and Crews efficiently, providing a robust framework for building sophisticated…”
   • [claimed-docs] “A crew in crewAI represents a collaborative group of agents working together to achieve a set of tasks. Each crew defines the strategy for t…”
   • [claimed-docs] “Flows allow you to create structured, event-driven workflows. They provide a seamless way to connect multiple tasks, manage state, and contr…”
   • [claimed-docs] “Perform specific tasks * Make decisions based on its role and goal * Use tools to accomplish objectives * Communicate and collaborate with o…”
   • [claimed-docs] “Sequential: Tasks are executed in the order they are defined * Hierarchical: Tasks are as”
   • [community] “Show HN: built AI agents with CrewAI to automate an old Gmail account—categorizing emails, drafting replies, Slack alerts for high-priority …”
   • [community] “Competitor (Portia) founder's take: 'We think Crew AI is great. It's just that a lot of businesses we spoke to want the autonomy/decision ma…”

   proof ↗⚑ flag

Workflow control

1. developerCompose agents into an explicit graph or workflow with branching, loops, and parallel steps

   weight 2 · round to LangGraph
   LangGraph✓fullcommunity9/10

   LangGraph's core model is explicitly graph-based (State, Nodes, Edges) with support for loops/cycles, branching, and parallelism via its Pregel/BSP execution model, confirmed both by docs and independent community technical commentary. missing for 10: no first-party hands-on benchmark of parallel-branch execution at scale, and community notes some friction with built-in parallelism complicating debugging.

   • [claimed-docs] “By composing Nodes and Edges, you can create complex, looping workflows that evolve the state over time.”
   • [claimed-docs] “LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components: State, Nodes, Edges.”
   • [claimed-docs] “Workflows have predetermined code paths and are designed to operate in a certain order.”
   • [community] “LangGraph implements a variant of the Pregel/BSP algorithm for orchestrating workflows with cycles (ie. not DAGs) and parallelism without da…”
   • [community] “by predeclaring the structure, you can show debugging UI of the full graph, even if you've only executed part of it... The downside is that …”
   • [community] “Hot take #1: For experienced developers, framework abstractions can add unnecessary complexity. Hot take #2: Built-in parallelism, while pro…”
   • [claimed-docs] “At its core, LangGraph models agent workflows as graphs. You define the behavior of your agents using three key components”

   proof ↗⚑ flag

   CrewAI~partialclaimed6/10

   CrewAI's Flows feature is explicitly documented as an event-driven workflow system that lets developers 'connect multiple tasks, manage state, and control the flow of execution,' and Crews support sequential/hierarchical process graphs — both point toward explicit workflow composition. However, the evidence never documents specific branching (routers/conditionals), loop, or parallel-step primitives, so the graph capability is implied rather than concretely shown. missing for 10: explicit documentation of branching/router constructs, loop constructs, and parallel-step execution within Flows, plus independent hands-on confirmation of these mechanics.

   • [claimed-docs] “Flows allow you to create structured, event-driven workflows. They provide a seamless way to connect multiple tasks, manage state, and contr…”
   • [claimed-docs] “Flows allow developers to combine and coordinate coding tasks and Crews efficiently, providing a robust framework for building sophisticated…”
   • [claimed-docs] “Flows allow you to create structured, event-driven workflows. They provide a seamless way to connect multiple tasks, manage state, and contr…”
   • [claimed-docs] “Sequential: Tasks are executed in the order they are defined * Hierarchical: Tasks are as”
   • [claimed-docs] “Flows make it super easy to manage and share state between different tasks in your workflow.”
   • [github] “It gives developers autonomous agent collaboration through Crews and precise, event-driven control through Flows.”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   LangGraph—none0/10

   The evidence pack describes LangGraph's checkpointing/persistence architecture (Postgres checkpointer, Agent Server with managed database) but contains no documentation of region selection, data residency controls, or geographic deployment options for stored data. Since LangGraph offers a hosted Agent Server/deployment platform, region/residency is a fair question, but nothing in the pack addresses it.

   • [claimed-docs] “In production, use a checkpointer backed by a database: from langgraph.checkpoint.postgres import PostgresSaver”
   • [claimed-docs] “LangGraph CLI is a command-line tool for building and running the Agent Server locally. The resulting server exposes all API endpoints for r…”
   • [claimed-docs] “LangGraph CLI** is a command-line tool for building and running the [Agent Server](/langsmith/agent-server) locally. The resulting server ex…”

   proof ↗⚑ flag

   CrewAI—none0/10

   No evidence in the pack mentions data residency, region selection, or storage location controls for CrewAI (open-source framework or AMP/enterprise offering); deployment docs describe managed infrastructure but never region choice or compliance certifications.

   ⚑ flag

2. ai-native userControl data retention and deletion

   weight 2 · round drawn
   LangGraph—none0/10

   The evidence describes checkpointers and stores that persist conversation state and long-term memory (e.g., via Postgres), but nothing in the pack documents any deletion API, TTL/retention policy, or user-facing control to purge stored threads/state. As a self-hosted framework the user technically owns the database, but no LangGraph-specific retention/deletion mechanism is evidenced.

   • [claimed-docs] “Checkpointers persist a thread's graph state as checkpoints. Use them for short-term, thread-scoped memory, including conversation continuit…”
   • [claimed-docs] “Stores persist application-defined data outside the graph state. Use them for long-term, cross-thread memory, including user preferences, fa…”
   • [claimed-docs] “In production, use a checkpointer backed by a database: from langgraph.checkpoint.postgres import PostgresSaver”
   • [claimed-docs] “Checkpointers persist a thread's graph state as checkpoints. Use them for short-term, thread-scoped memory... Stores persist application-def…”

   proof ↗⚑ flag

   CrewAI—none0/10

   The evidence pack contains no documentation of data retention policies, deletion controls, or privacy/compliance settings for CrewAI's memory or storage systems. Community items describe an agent programmed to delete emails as a task behavior, not CrewAI providing user-facing retention/deletion controls over its own stored data.

   ⚑ flag

3. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   LangGraph—none0/10

   No evidence in the pack addresses telemetry, usage tracking, or an opt-out mechanism for LangGraph itself; the docs focus on orchestration, memory, streaming, and deployment, and LangSmith tracing is presented as an opt-in observability feature rather than a telemetry opt-out control.

   ⚑ flag

   CrewAI—none0/10

   No evidence in the pack mentions telemetry collection or an opt-out/disable-tracking mechanism for CrewAI; observability docs describe monitoring agent performance but say nothing about product usage telemetry or opt-out controls.

   ⚑ flag

Durable state

1. developerCheckpoint agent state so a run can resume exactly where it left off after a crash or restart

   weight 3 · round to LangGraph
   LangGraph✓fullclaimed9/10

   LangGraph's checkpointer system explicitly persists exact graph state per thread, enabling resume after error/crash ('checkpointing keeps your place... even when in an error state'), with production-grade backends like PostgresSaver documented and GitHub README explicitly touting 'durable execution' that resumes 'exactly where they left off' after failures. This is a well-documented, core feature with clear technical backing across multiple doc pages; missing for 10: independent hands-on verification of crash-recovery behavior beyond docs/marketing claims.

   • [claimed-docs] “Checkpointers persist a thread's graph state as checkpoints. Use them for short-term, thread-scoped memory, including conversation continuit…”
   • [claimed-docs] “In production, use a checkpointer backed by a database: from langgraph.checkpoint.postgres import PostgresSaver”
   • [claimed-docs] “Checkpointing keeps your place: the checkpointer writes the exact graph state so you can resume later, even when in an error state.”
   • [github] “Build agents that persist through failures and can run for extended periods, automatically resuming from exactly where they left off.”
   • [github] “Durable execution — Build agents that persist through failures and can run for extended periods, automatically resuming from exactly where t…”

   proof ↗⚑ flag

   CrewAI!disputedcontradicted4/10

   Docs describe Flows managing/sharing state across tasks and a unified Memory system for persistence (crewai-docs-4, crewai-docs-36, crewai-docs-49, crewai-docs-5, crewai-docs-50), plus an enterprise API with kickoff/status/resume endpoints (crewai-docs-56, crewai-docs-57), but the resume endpoint is described for human-feedback loops, not crash recovery. A competitor's founder explicitly claims 'Crew AI lacks' persistent state and deterministic pause/resume hooks that their own product has (crewai-comm-4), directly contradicting the notion of reliable crash-resume checkpointing. Missing for 10: first-party documentation of an explicit crash-recovery/checkpoint-resume mechanism, and independent hands-on confirmation that a run can resume exactly where it left off after a crash.

   • [claimed-docs] “Flows allow you to create structured, event-driven workflows. They provide a seamless way to connect multiple tasks, manage state, and contr…”
   • [claimed-docs] “Flows allow you to create structured, event-driven workflows. They provide a seamless way to connect multiple tasks, manage state, and contr…”
   • [claimed-docs] “Flows make it super easy to manage and share state between different tasks in your workflow.”
   • [claimed-docs] “CrewAI provides a unified memory system -- a single Memory class... supports adaptive-depth recall with composite scoring that blends semant…”
   • [claimed-docs] “CrewAI provides a **unified memory system** -- a single `Memory` class that replaces separate short-term, long-term, entity, and external me…”
   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”
   • [community] “Competitor (Portia) founder's take: 'We think Crew AI is great. It's just that a lot of businesses we spoke to want the autonomy/decision ma…”

   proof ↗⚑ flag

2. engineering-leadRun long-lived agents durably across process restarts and deploys, natively or via durable-execution integrations

   weight 2 · round to LangGraph
   LangGraph✓fullclaimed8/10

   LangGraph explicitly advertises 'Durable execution' as a core feature, with checkpointers (including production Postgres-backed checkpointers) that persist thread state so agents 'automatically resume from exactly where they left off' after failures, and interrupts that preserve execution state even in error conditions. This directly matches the engineering-lead's requirement for durable, restart-resilient long-running agents. Missing for 10: independent hands-on verification of actual crash/restart recovery in production, and explicit coverage of third-party durable-execution integrations (e.g., Temporal) beyond LangGraph's native mechanism.

   • [github] “Durable execution — Build agents that persist through failures and can run for extended periods, automatically resuming from exactly where t…”
   • [github] “Build agents that persist through failures and can run for extended periods, automatically resuming from exactly where they left off.”
   • [claimed-docs] “Checkpointers persist a thread's graph state as checkpoints. Use them for short-term, thread-scoped memory, including conversation continuit…”
   • [claimed-docs] “In production, use a checkpointer backed by a database: from langgraph.checkpoint.postgres import PostgresSaver”
   • [claimed-docs] “Checkpointing keeps your place: the checkpointer writes the exact graph state so you can resume later, even when in an error state.”
   • [github] “Production-ready deployment — Deploy sophisticated agent systems confidently with scalable infrastructure designed to handle the unique chal…”

   proof ↗⚑ flag

   CrewAI!disputedcontradicted3/10

   CrewAI's enterprise/edge API exposes kickoff/status/resume endpoints suggesting some ability to pause and resume crew execution (crewai-docs-56, crewai-docs-57), but there is no documentation of durable state persistence across process restarts/deploys or integration with a durable-execution engine (e.g., Temporal, Restate). A competitor explicitly claims 'Crew AI lacks' persistent state and deterministic pause hooks that their own product offers (crewai-comm-4), directly contradicting any durability claim. missing for 10: native crash/restart recovery mechanism, documented durable-execution integration, independent verification of the resume API's persistence guarantees.

   • [claimed-docs] “POST /resume](https://docs.crewai.com/edge/en/api-reference/resume.md): Resume crew execution with human feedback”
   • [claimed-docs] “POST /kickoff](https://docs.crewai.com/edge/en/api-reference/kickoff.md): Start a crew execution - [GET /status/{kickoff_id}](https://docs.c…”
   • [community] “Competitor (Portia) founder's take: 'We think Crew AI is great. It's just that a lot of businesses we spoke to want the autonomy/decision ma…”

   proof ↗⚑ flag

Streaming

1. developerStream tokens and intermediate agent events (tool calls, steps) to my UI in real time

   weight 3 · round to LangGraph
   LangGraph~partialcommunity7/10

   LangGraph docs clearly document multiple stream modes including 'messages' for token streaming and 'updates'/'debug'/'tasks' for intermediate node/tool events, plus separate iterators per projection via stream/astream, directly supporting real-time UI streaming of tokens and agent steps. However, a community report notes the streaming implementation is minimal sample code that each client must fully reimplement, indicating real-world integration effort beyond the docs. missing for 10: independent hands-on confirmation of smooth tool-call/step event streaming in a UI, and clearer first-party UI integration examples beyond sample code.

   • [claimed-docs] “It exposes graph execution through stream modes such as updates, values, messages, custom, checkpoints, tasks, and debug.”
   • [claimed-docs] “Event streaming gives you separate iterators per projection (messages, values, subgraphs, output) so you can consume them independently”
   • [claimed-docs] “LangGraph graphs expose the stream (sync) and astream (async) methods to yield streamed outputs as iterators.”
   • [claimed-docs] “It exposes graph execution through stream modes such as `updates`, `values`, `messages`, `custom`, `checkpoints`, `tasks`, and `debug`.”
   • [claimed-docs] “Event streaming gives you separate iterators per projection (messages, values, subgraphs, output) so you can consume them independently inst…”
   • [community] “The one thing I wish was better developed is persistence and streaming - they give sample code to stream, but it's essentially a complete im…”

   proof ↗⚑ flag

   CrewAI~partialclaimed4/10

   Docs mention 'Webhook Streaming' and real-time execution monitoring, but these are enterprise (CrewAI AMP) features aimed at systems integration, not documented developer-facing token streaming or step/tool-call event hooks for building a live UI. No evidence of an open-source streaming API, callback for individual tokens, or intermediate agent step events exposed to a UI. Missing for 10: token-level streaming API/SDK, documented callback hooks for tool calls/steps, and any independent/hands-on confirmation of real-time UI event feeds.

   • [claimed-docs] “Stream real-time events and updates to your systems”
   • [claimed-docs] “Webhook Streaming Stream real-time events and updates to your systems”
   • [claimed-docs] “Performance Monitoring: Track agent execution times, token usage, and resource consumption”
   • [claimed-docs] “Track agent execution times, token usage, and resource consumption”
   • [claimed-docs] “CrewAI AMP extends the power of the open-source framework with features designed for production deployments, collaboration, and scalability.…”

   proof ↗⚑ flag

Structured output

1. developerGet schema-validated structured output from an agent, with automatic retries when validation fails

   weight 3 · round drawn
   LangGraph—none0/10

   The evidence pack covers persistence, streaming, human-in-the-loop, checkpointing, and multi-agent workflows, but contains no mention of structured output, schema validation, or automatic retries on validation failure for LangGraph agents.

   ⚑ flag

   CrewAI—none0/10

   Evidence pack has no mention of Pydantic/schema output validation or automatic retry-on-validation-failure mechanisms for structured outputs; it covers agents, tasks, memory, tools, CLI, and enterprise features but nothing about structured output validation or retries.

   ⚑ flag