Agent Frameworks & SDKs — procurement report
ProductArena · rankings as of 2026-09-16 · evidence as of 2026-09-16 · 9 products · 51 judged requirements · 459 judged cells
Methodology: Every product is judged against a shared taxonomy of user stories using cited evidence — hands-on probes > repository code > independent community sources > vendor claims — never opinion. Full writeup: https://ultrametric.ai/productarena/methodology
Leaderboard
| # | Product | PA Score | Coverage score | Applicable cells | Confidence |
|---|---|---|---|---|---|
| 1 | Claude Agent SDK | 38.1 | 38.4 | 50/51 | C |
| 2 | OpenAI Agents SDK | 37.2 | 40.0 | 47/51 | B |
| 3 | Pydantic AI | 36.2 | 41.8 | 45/51 | C |
| 4 | Google ADK | 34.8 | 40.1 | 48/51 | A |
| 5 | smolagents | 33.4 | 34.8 | 42/51 | A |
| 6 | CrewAI | 32.9 | 32.4 | 50/51 | B |
| 7 | Mastra | 31.7 | 45.1 | 48/51 | A |
| 8 | LangGraph | 27.0 | 33.5 | 49/51 | A |
| 9 | AutoGen | 26.4 | 24.5 | 48/51 | B |
PA Score = agent-readiness blend (see methodology). Coverage score = weighted share of judged requirements met. Confidence = how much of the score rests on tested vs claimed evidence (A–D).
Uncertainty note
This arena is currently a close race: Claude Agent SDK (38.1) vs OpenAI Agents SDK (37.2), a gap of 0.9 PA Score. The ordering was re-checked with extra judge samples: 34 decisive cells were triple-judged and 8 came back unstable. Treat the #1/#2 ordering as contested — shortlist both.
Buyer checklist (RFP)
The arena's 51 judged user stories as requirements, grouped by theme. Priorities mirror the story weights our scoring uses (3 = must-have, 2 = should-have, 1 = nice-to-have). Interactive version with per-requirement verdicts for the top products: /arena/agent-frameworks/checklist
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
- ai-native userPlug MCP servers into this product so it can use their toolsmust-have
- ai-native userConnect an agent via an official MCP servermust-have
- ai-native userDrive the product through a documented public APImust-have
- ai-native userDelegate tasks to a built-in AI assistant inside the productmust-have
- ai-native userPoint an agent at llms.txt or agent-oriented docsshould-have
- ai-native userRun the product headlessly / in CI for automationshould-have
- ai-native userUse an official CLIshould-have
- ai-native userIssue scoped/least-privilege API credentials for an agentshould-have
- ai-native userBuild against official SDKsshould-have
- ai-native userSubscribe to events via webhooksshould-have
- ai-native userGet AI-generated insights and suggestions from my data inside the productshould-have
- ai-native userSet up automations that run autonomously in the backgroundshould-have
- ai-native userOperate the product with natural-language commandsshould-have
- ai-native userExplore an interactive API reference with runnable examplesshould-have
- ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)should-have
- ai-native userRely on versioned APIs with a documented deprecation policyshould-have
- ai-native userTest against a sandbox environment without touching production datanice-to-have
Agents tools — stories about agents tools in this arenaAgents tools
Stories about agents tools in this arena
- developerDefine an agent with typed custom tools in a few lines of codemust-have
- ai-native userHave a coding agent scaffold a new agent project from an official CLI or template in one commandshould-have
- ai-native userRun the framework's example agents headlessly from a terminal so an agent can verify what it just builtshould-have
- ai-native userRely on strict typing and schema validation so a coding agent catches its own mistakes at build timeshould-have
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
- ai-native userDefine rules that trigger actions automatically on eventsmust-have
- ai-native userPerform bulk operations across many items at onceshould-have
- ai-native userSchedule recurring jobs or workflowsshould-have
- ai-native userVersion, review, and roll back my automationsnice-to-have
Deployment portability — stories about deployment portability in this arenaDeployment portability
Stories about deployment portability in this arena
- developerSwap the underlying LLM provider or model without rewriting my agentmust-have
- engineering-leadDeploy an agent to a managed runtime and call it as an API endpointshould-have
- engineering-leadRun my agents entirely on my own infrastructure with no dependence on the vendor's platformshould-have
Evals observability — stories about evals observability in this arenaEvals observability
Stories about evals observability in this arena
- developerTrace every LLM call and tool invocation of an agent run in an observability UImust-have
- engineering-leadScore agent quality with built-in evals and run them as part of CIshould-have
- developerUnit-test agents with mocked models and toolsshould-have
Guardrails safety — stories about guardrails safety in this arenaGuardrails safety
Stories about guardrails safety in this arena
- developerAttach input/output guardrails that validate, transform, or block unsafe contentmust-have
- engineering-leadRestrict what an agent may do with fine-grained tool permissions and sandboxed executionshould-have
Human in the loop — stories about human in the loop in this arenaHuman in the loop
Stories about human in the loop in this arena
- developerPause an agent mid-run for human input or approval and resume with the human's decisionmust-have
- engineering-leadRequire human approval before specific sensitive tool calls executeshould-have
Memory context — stories about memory context in this arenaMemory context
Stories about memory context in this arena
- developerTrim, summarize, or filter conversation history to keep an agent inside its context windowshould-have
- developerGive agents long-term memory that persists across sessions and threadsshould-have
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
- ai-native userExport all of my data in open formats and leavemust-have
- ai-native userSelf-host the core productmust-have
- ai-native userDo everything through the API that I can do in the UIshould-have
- ai-native userRead the product's source under an open licenseshould-have
Orchestration multi agent — stories about orchestration multi agent in this arenaOrchestration multi agent
Stories about orchestration multi agent in this arena
- developerOrchestrate multiple agents — handoffs, subagents, or crews — inside one workflowmust-have
- developerCompose agents into an explicit graph or workflow with branching, loops, and parallel stepsshould-have
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
- ai-native userPrevent my data from being used to train AI modelsmust-have
- ai-native userChoose where my data is stored (region/residency)should-have
- ai-native userControl data retention and deletionshould-have
- ai-native userOpt out of telemetry and usage trackingshould-have
State durability — stories about state durability in this arenaState durability
Stories about state durability in this arena
- developerCheckpoint agent state so a run can resume exactly where it left off after a crash or restartmust-have
- engineering-leadRun long-lived agents durably across process restarts and deploys, natively or via durable-execution integrationsshould-have
Streaming output — stories about streaming output in this arenaStreaming output
Stories about streaming output in this arena
- developerStream tokens and intermediate agent events (tool calls, steps) to my UI in real timemust-have
- developerGet schema-validated structured output from an agent, with automatic retries when validation failsmust-have
Appendix: recorded probes
Hands-on probe recordings — transcripts/videos a human can replay, the strongest evidence tier. Watch them at https://ultrametric.ai/productarena/proofs
- Claude Agent SDK
claude --versionterminal · recorded 2026-09-04 · exit 0
Cite as: ProductArena by Ultrametric Inc, Agent Frameworks & SDKs arena, rankings as of 2026-09-16 — https://ultrametric.ai/productarena/arena/agent-frameworks
License: © 2026 Ultrametric Inc. Brief quotation of individual verdicts, scores, or evidence excerpts is permitted with attribution to "ProductArena by Ultrametric Inc (ultrametric.ai/productarena)", as is use of the data to evaluate, contest, or contribute corrections. Bulk copying, redistribution, or use to build competing datasets requires prior written permission (see DATA-LICENSE in the repository).
No liability: rankings, verdicts, and scores are research outputs derived from the cited evidence at a point in time, provided "as is", without warranties. Ultrametric Inc accepts no responsibility for procurement, purchasing, or other decisions made in reliance on them — verify against the cited evidence before acting (https://ultrametric.ai/productarena/terms).