Skip to content
ProductArena

How OpenAI Agents SDK’s scores are calculated

The full audit trail, recomputed from the verdict data at build time through the same code that produced the leaderboard: verdict × quality × story weight per cell, cells sum to dimension scores, dimensions blend into the PA Score. Every number on the product page is reproducible from this page alone; for why the formula looks like this, see the methodology.

verdict factors: full ×1.0 · partial ×0.6 · disputed ×0.3 · none ×0.0 · n/a excluded from both sides · cell points = weight × quality × factor · cell max = weight × 10

PA Score37/100

Agent-ready 55.0 × 0.30 = 16.50

API quality 10.0 × 0.20 = 2.00

Openness 60.0 × 0.20 = 12.00

Built-in AI 35.4 × 0.15 = 5.31

Automation 9.0 × 0.15 = 1.35

(16.50 + 2.00 + 12.00 + 5.31 + 1.35) ÷ (0.30 + 0.20 + 0.20 + 0.15 + 0.15) = 37.16 ÷ 1.00 = 37.2

Scores are stored to 1 decimal; the product page’s pills round to whole numbers for display. Each dimension below shows the stories, verdicts, and cited evidence behind its number.

Agent-ready55.0/100×0.30 of the PA blend

Outside-in: can YOUR agent reach and drive this product — API, MCP, CLI, headless runs, agent docs.

Point an agent at llms.txt or agent-oriented docsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [probe] https://developers.openai.com/llms.txt“PROBE llms.txt: HTTP 200 at https://developers.openai.com/llms.txt # OpenAI Developers > Complete documentation hub for OpenAI API, Ads, Plugins, Workspace Agents, Codex, Agentic Commerc”
  • [probe] https://developers.openai.com/api/docs/guides/agents.md“PROBE docs-md: HTTP 200 at https://developers.openai.com/api/docs/guides/agents.md # Agents > For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages ar”
  • [claimed-docs] https://openai.github.io/openai-agents-python/tools/“The WebSearchTool lets an agent search the web.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/mcp/“The Agents Python SDK understands multiple MCP transports. This lets you reuse existing MCP servers or build your own to expose filesystem, HTTP, or connector backed tools to an agent.”

Run the product headlessly / in CI for automationweight 2

2 (weight) × 7 (quality) × 1.0 (full) = 14.0 of 20 max

  • [claimed-docs] https://openai.github.io/openai-agents-python/running_agents/“You can run agents via the Runner class. You have 3 options: Runner.run(), Runner.run_sync(), Runner.run_streamed()”
  • [claimed-docs] https://openai.github.io/openai-agents-python/running_agents/“You have 3 options: Runner.run(), Runner.run_sync(), Runner.run_streamed()”
  • [claimed-docs] https://openai.github.io/openai-agents-python/running_agents/“You can run agents via the `Runner` class. You have 3 options: `Runner.run()`... `Runner.run_sync()`... `Runner.run_streamed()`”
  • [claimed-docs] https://openai.github.io/openai-agents-python/tracing/“The Agents SDK includes built-in tracing, collecting a comprehensive record of events during an agent run: LLM generations, tool calls, handoffs, guardrails, and even custom events that occur.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/sessions/“The Agents SDK provides built-in session memory to automatically maintain conversation history across multiple agent runs, eliminating the need to manually handle `.to_input_list()` between turns.”

Plug MCP servers into this product so it can use their toolsweight 3

3 (weight) × 9 (quality) × 1.0 (full) = 27.0 of 30 max

  • [claimed-docs] https://openai.github.io/openai-agents-python/mcp/“The Agents Python SDK understands multiple MCP transports. This lets you reuse existing MCP servers or build your own to expose filesystem, HTTP, or connector backed tools to an agent.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/mcp/“The Agents Python SDK understands multiple MCP transports. This lets you reuse existing MCP servers or build your own”
  • [claimed-docs] https://openai.github.io/openai-agents-python/tools/“Use OpenAI-managed tools (web search, file search, code interpreter, hosted MCP, image generation)”

Connect an agent via an official MCP serverweight 3

n/a — not applicable to this product: excluded from numerator and denominator

  • [claimed-docs] https://openai.github.io/openai-agents-python/mcp/“The Agents Python SDK understands multiple MCP transports. This lets you reuse existing MCP servers or build your own to expose filesystem, HTTP, or connector backed tools to an agent.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/mcp/“The Agents Python SDK understands multiple MCP transports. This lets you reuse existing MCP servers or build your own”
  • [claimed-docs] https://openai.github.io/openai-agents-python/tools/“Use OpenAI-managed tools (web search, file search, code interpreter, hosted MCP, image generation)”

Use an official CLIweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://developers.openai.com/openapi.json“PROBE openapi: all candidate paths 404 (https://developers.openai.com/openapi.json, https://developers.openai.com/swagger.json, https://developers.openai.com/api/openapi.json, https://developers.openai.com/.well-known/openapi.json)”

Drive the product through a documented public APIweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

  • [claimed-docs] https://openai.github.io/openai-agents-python/running_agents/“You can run agents via the Runner class. You have 3 options: Runner.run(), Runner.run_sync(), Runner.run_streamed()”
  • [claimed-docs] https://openai.github.io/openai-agents-python/agents/“An agent is a large language model (LLM) configured with instructions, tools, and optional runtime behavior such as handoffs, guardrails, and structured outputs.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/tools/“FunctionTool instances: wrap any Python function as a tool.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/handoffs/“Handoffs allow an agent to delegate tasks to another agent. This is particularly useful in scenarios where different agents specialize in distinct areas.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/tracing/“The Agents SDK includes built-in tracing, collecting a comprehensive record of events during an agent run: LLM generations, tool calls, handoffs, guardrails, and even custom events that occur.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/sessions/“The Agents SDK provides built-in session memory to automatically maintain conversation history across multiple agent runs, eliminating the need to manually handle `.to_input_list()` between turns.”
  • [github] https://github.com/openai/openai-agents-python“It is provider-agnostic, supporting the OpenAI Responses and Chat Completions APIs, as well as 100+ other LLMs.”
  • [community] https://hn.algolia.com/api/v1/items/44353964“OpenAI SDK is much simpler to use and understand than langchain etc. I havent deployed really complicated agents, but I have a number of simple use cases that work just fine”
  • [probe] https://developers.openai.com/openapi.json“PROBE openapi: all candidate paths 404 (https://developers.openai.com/openapi.json, https://developers.openai.com/swagger.json, https://developers.openai.com/api/openapi.json, https://developers.openai.com/.well-known/openapi.json)”

Issue scoped/least-privilege API credentials for an agentweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Build against official SDKsweight 2

2 (weight) × 9 (quality) × 1.0 (full) = 18.0 of 20 max

  • [claimed-docs] https://openai.github.io/openai-agents-python/agents/“An agent is a large language model (LLM) configured with instructions, tools, and optional runtime behavior such as handoffs, guardrails, and structured outputs.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/running_agents/“You can run agents via the Runner class. You have 3 options: Runner.run(), Runner.run_sync(), Runner.run_streamed()”
  • [claimed-docs] https://openai.github.io/openai-agents-python/tracing/“The Agents SDK includes built-in tracing, collecting a comprehensive record of events during an agent run: LLM generations, tool calls, handoffs, guardrails, and even custom events that occur.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/mcp/“The Agents Python SDK understands multiple MCP transports. This lets you reuse existing MCP servers or build your own to expose filesystem, HTTP, or connector backed tools to an agent.”
  • [github] https://github.com/openai/openai-agents-python“It is provider-agnostic, supporting the OpenAI Responses and Chat Completions APIs, as well as 100+ other LLMs.”
  • [community] https://hn.algolia.com/api/v1/items/44353964“OpenAI SDK is much simpler to use and understand than langchain etc. I havent deployed really complicated agents, but I have a number of simple use cases that work just fine”

Subscribe to events via webhooksweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Agent-ready = 99.0 ÷ 180 × 100 = 55.0

API quality10.0/100×0.20 of the PA blend

The programmable surface once an agent is there — machine-readable spec, interactive docs, sandbox, versioning discipline.

Explore an interactive API reference with runnable examplesweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://developers.openai.com/openapi.json“PROBE openapi: all candidate paths 404 (https://developers.openai.com/openapi.json, https://developers.openai.com/swagger.json, https://developers.openai.com/api/openapi.json, https://developers.openai.com/.well-known/openapi.json)”
  • [probe] https://developers.openai.com/llms.txt“PROBE llms.txt: HTTP 200 at https://developers.openai.com/llms.txt # OpenAI Developers > Complete documentation hub for OpenAI API, Ads, Plugins, Workspace Agents, Codex, Agentic Commerc”
  • [probe] https://developers.openai.com/api/docs/guides/agents.md“PROBE docs-md: HTTP 200 at https://developers.openai.com/api/docs/guides/agents.md # Agents > For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages ar”

Download a machine-readable API spec (OpenAPI or equivalent)weight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://developers.openai.com/openapi.json“PROBE openapi: all candidate paths 404 (https://developers.openai.com/openapi.json, https://developers.openai.com/swagger.json, https://developers.openai.com/api/openapi.json, https://developers.openai.com/.well-known/openapi.json)”

Test against a sandbox environment without touching production dataweight 1

1 (weight) × 7 (quality) × 1.0 (full) = 7.0 of 10 max

  • [claimed-docs] https://openai.github.io/openai-agents-python/“Sandbox agents: Run specialists inside real isolated workspaces. Sandbox agents support manifest-defined files, sandbox client selection, and resumable sandbox sessions.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/agents/“If the agent should run inside an isolated workspace with manifest-defined files and sandbox-native capabilities, read Sandbox agent concepts.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/tools/“OpenAI offers a few built-in tools when using the OpenAIResponsesModel... The WebSearchTool lets an agent search the web. The FileSearchTool allows retrieving information from your OpenAI Vector Stores. The CodeInterpreterTool lets the LLM execute code in a sandboxed environment.”

Rely on versioned APIs with a documented deprecation policyweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://developers.openai.com/openapi.json“PROBE openapi: all candidate paths 404 (https://developers.openai.com/openapi.json, https://developers.openai.com/swagger.json, https://developers.openai.com/api/openapi.json, https://developers.openai.com/.well-known/openapi.json)”

API quality = 7.0 ÷ 70 × 100 = 10.0

Openness60.0/100×0.20 of the PA blend

Can you leave, inspect, or self-host — data export, open source, portability.

Do everything through the API that I can do in the UIweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Export all of my data in open formats and leaveweight 3

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Read the product's source under an open licenseweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

  • [github] https://github.com/openai/openai-agents-python“It is provider-agnostic, supporting the OpenAI Responses and Chat Completions APIs, as well as 100+ other LLMs.”
  • [github] https://github.com/openai/openai-agents-python“Voice agents: Build voice pipelines that combine speech-to-text, an agent workflow, and text-to-speech”
  • [github] https://github.com/openai/openai-agents-python“Realtime agents: Build powerful voice agents with `gpt-realtime-2.1` and full agent features”
  • [community] https://hn.algolia.com/api/v1/items/44353964“OpenAI SDK is much simpler to use and understand than langchain etc. I havent deployed really complicated agents, but I have a number of simple use cases that work just fine”

Self-host the core productweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

  • [github] https://github.com/openai/openai-agents-python“It is provider-agnostic, supporting the OpenAI Responses and Chat Completions APIs, as well as 100+ other LLMs.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/running_agents/“You can run agents via the Runner class. You have 3 options: Runner.run(), Runner.run_sync(), Runner.run_streamed()”
  • [claimed-docs] https://openai.github.io/openai-agents-python/running_agents/“You can run agents via the `Runner` class. You have 3 options: `Runner.run()`... `Runner.run_sync()`... `Runner.run_streamed()`”
  • [claimed-docs] https://openai.github.io/openai-agents-python/models/“if you want to consistently use a specific model for all agents that do not set a custom model, set the `OPENAI_DEFAULT_MODEL` environment variable before running your agents.”

Openness = 30.0 ÷ 50 × 100 = 60.0

Built-in AI35.4/100×0.15 of the PA blend

Inside-out: how agentic the product itself is for its users — built-in assistants, autonomous features.

Get AI-generated insights and suggestions from my data inside the productweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Set up automations that run autonomously in the backgroundweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://openai.github.io/openai-agents-python/running_agents/“You can run agents via the Runner class. You have 3 options: Runner.run(), Runner.run_sync(), Runner.run_streamed()”
  • [claimed-docs] https://openai.github.io/openai-agents-python/sessions/“The Agents SDK provides built-in session memory to automatically maintain conversation history across multiple agent runs, eliminating the need to manually handle `.to_input_list()` between turns.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/tracing/“The Agents SDK includes built-in tracing, collecting a comprehensive record of events during an agent run: LLM generations, tool calls, handoffs, guardrails, and even custom events that occur.”

Delegate tasks to a built-in AI assistant inside the productweight 3

3 (weight) × 6 (quality) × 0.6 (partial) = 10.8 of 30 max

  • [claimed-docs] https://openai.github.io/openai-agents-python/handoffs/“Handoffs allow an agent to delegate tasks to another agent. This is particularly useful in scenarios where different agents specialize in distinct areas.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/running_agents/“You can run agents via the Runner class. You have 3 options: Runner.run(), Runner.run_sync(), Runner.run_streamed()”
  • [claimed-docs] https://openai.github.io/openai-agents-python/tools/“Agents as tools: expose an agent as a callable tool without a full handoff.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/agents/“An agent is a large language model (LLM) configured with instructions, tools, and optional runtime behavior such as handoffs, guardrails, and structured outputs.”

Operate the product with natural-language commandsweight 2

2 (weight) × 7 (quality) × 1.0 (full) = 14.0 of 20 max

  • [claimed-docs] https://openai.github.io/openai-agents-python/agents/“An agent is a large language model (LLM) configured with instructions, tools, and optional runtime behavior such as handoffs, guardrails, and structured outputs.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/running_agents/“You can run agents via the Runner class. You have 3 options: Runner.run(), Runner.run_sync(), Runner.run_streamed()”
  • [claimed-docs] https://openai.github.io/openai-agents-python/running_agents/“You can run agents via the `Runner` class. You have 3 options: `Runner.run()`... `Runner.run_sync()`... `Runner.run_streamed()`”
  • [claimed-docs] https://openai.github.io/openai-agents-python/“Agents, which are LLMs equipped with instructions and tools”
  • [claimed-docs] https://openai.github.io/openai-agents-python/streaming/“Streaming lets you subscribe to updates of the agent run as it proceeds.”

Built-in AI = 24.8 ÷ 70 × 100 = 35.4

Automation9.0/100×0.15 of the PA blend

Depth of automation primitives — rules, scheduling, bulk operations, webhooks.

Perform bulk operations across many items at onceweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Define rules that trigger actions automatically on eventsweight 3

3 (weight) × 4 (quality) × 0.6 (partial) = 7.2 of 30 max

  • [claimed-docs] https://openai.github.io/openai-agents-python/guardrails/“Guardrails enable you to do checks and validations of user input and agent output.”
  • [claimed-docs] https://openai.github.io/openai-agents-js/guides/human-in-the-loop/“When a tool call requires approval, the SDK pauses the run, returns interruptions, and lets you resume later from the same RunState.”
  • [claimed-docs] https://openai.github.io/openai-agents-js/guides/human-in-the-loop/“When a tool call requires approval, the SDK pauses the run, returns `interruptions`, and lets you resume later from the same `RunState`.”
  • [claimed-docs] https://openai.github.io/openai-agents-python/streaming/“Streaming lets you subscribe to updates of the agent run as it proceeds. This can be useful for showing the end-user progress updates and partial responses.”

Schedule recurring jobs or workflowsweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Version, review, and roll back my automationsweight 1

1 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 10 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Automation = 7.2 ÷ 80 × 100 = 9.0