Skip to content
ProductArena

How Helicone’s scores are calculated

The full audit trail, recomputed from the verdict data at build time through the same code that produced the leaderboard: verdict × quality × story weight per cell, cells sum to dimension scores, dimensions blend into the PA Score. Every number on the product page is reproducible from this page alone; for why the formula looks like this, see the methodology.

verdict factors: full ×1.0 · partial ×0.6 · disputed ×0.3 · none ×0.0 · n/a excluded from both sides · cell points = weight × quality × factor · cell max = weight × 10

PA Score34/100

Agent-ready 42.9 × 0.30 = 12.87

API quality 36.9 × 0.20 = 7.38

Openness 30.0 × 0.20 = 6.00

Built-in AI 24.0 × 0.15 = 3.60

Automation 29.0 × 0.15 = 4.35

(12.87 + 7.38 + 6.00 + 3.60 + 4.35) ÷ (0.30 + 0.20 + 0.20 + 0.15 + 0.15) = 34.20 ÷ 1.00 = 34.2

Scores are stored to 1 decimal; the product page’s pills round to whole numbers for display. Each dimension below shows the stories, verdicts, and cited evidence behind its number.

Agent-ready42.9/100×0.30 of the PA blend

Outside-in: can YOUR agent reach and drive this product — API, MCP, CLI, headless runs, agent docs.

Point an agent at llms.txt or agent-oriented docsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [probe] https://docs.helicone.ai/llms.txt“PROBE llms.txt: HTTP 200 at https://docs.helicone.ai/llms.txt # Helicone OSS LLM Observability - [Quickstart](https://docs.helicone.ai/getting-started/quick-start.md): Get your firs”
  • [probe] https://docs.helicone.ai/getting-started/quick-start.md“PROBE docs-md: HTTP 200 at https://docs.helicone.ai/getting-started/quick-start.md > ## Documentation Index > Fetch the complete documentation index at: https://docs.helicone.ai/llms.txt > Use this file”
  • [probe] https://docs.helicone.ai/swagger.json“PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”

Run the product headlessly / in CI for automationweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [github] https://github.com/Helicone/helicone“Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
  • [claimed-docs] https://docs.helicone.ai/features/webhooks“Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI activity with external systems.”
  • [claimed-docs] https://docs.helicone.ai/rest/request/post-v1requestquery“Get Requests (Point Queries)”
  • [claimed-docs] https://docs.helicone.ai/getting-started/self-host/overview“Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
  • [probe] https://docs.helicone.ai/swagger.json“PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”

Plug MCP servers into this product so it can use their toolsweight 3

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Connect an agent via an official MCP serverweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Use an official CLIweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Drive the product through a documented public APIweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

  • [claimed-docs] https://docs.helicone.ai/rest/request/post-v1requestquery“Get Requests (Point Queries)”
  • [claimed-docs] https://www.helicone.ai/pricing“HQL (Query Language)”
  • [probe] https://docs.helicone.ai/swagger.json“PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
  • [probe] https://docs.helicone.ai/llms.txt“PROBE llms.txt: HTTP 200 at https://docs.helicone.ai/llms.txt # Helicone OSS LLM Observability - [Quickstart](https://docs.helicone.ai/getting-started/quick-start.md): Get your firs”
  • [github] https://github.com/Helicone/helicone“AI Gateway: Access 100+ AI models with 1 API key through the OpenAI API with intelligent routing and automatic fallbacks.”

Issue scoped/least-privilege API credentials for an agentweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.helicone.ai/getting-started/quick-start“Want more control? You can bring your own provider keys instead.”
  • [community] https://news.ycombinator.com/item?id=35279155“Hmm, so to integrate I have to basically send my api key to you on every request? Not great”

Build against official SDKsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://docs.helicone.ai/getting-started/quick-start“Use the familiar OpenAI SDK to access 100+ LLM models across OpenAI, Anthropic, Google, and more with automatic logging, observability, and fallbacks built in.”
  • [github] https://github.com/Helicone/helicone“Quick integration: One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
  • [github] https://github.com/Helicone/helicone“One-line of code to log all your requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more.”
  • [community] https://news.ycombinator.com/item?id=35279155“We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why requests were failing... Love how easy it was to integrate too--just one line to swap out the OpenAI API with theirs.”
  • [community] https://news.ycombinator.com/item?id=35279155“Happy Helicone customer here. It's a dead simple setup. It's great to have the extra charts and logging to debug issues and make sure all is running well.”
  • [community] https://news.ycombinator.com/item?id=42806254“Your onboarding is impressive, one of the few products where 'get set up in one line of code' is true.”

Subscribe to events via webhooksweight 2

2 (weight) × 7 (quality) × 1.0 (full) = 14.0 of 20 max

  • [claimed-docs] https://docs.helicone.ai/features/webhooks“Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI activity with external systems.”
  • [claimed-docs] https://docs.helicone.ai/features/webhooks“Only requests matching ALL specified properties will trigger webhooks.”
  • [claimed-docs] https://docs.helicone.ai/features/webhooks“Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”

Agent-ready = 77.2 ÷ 180 × 100 = 42.9

API quality36.9/100×0.20 of the PA blend

The programmable surface once an agent is there — machine-readable spec, interactive docs, sandbox, versioning discipline.

Explore an interactive API reference with runnable examplesweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

  • [probe] https://docs.helicone.ai/swagger.json“PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
  • [claimed-docs] https://docs.helicone.ai/rest/request/post-v1requestquery“Get Requests (Point Queries)”
  • [github] https://github.com/Helicone/helicone“Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
  • [github] https://github.com/Helicone/helicone“Rapidly test and iterate on prompts, sessions and traces in our UI.”

Download a machine-readable API spec (OpenAPI or equivalent)weight 2

2 (weight) × 9 (quality) × 1.0 (full) = 18.0 of 20 max

  • [probe] https://docs.helicone.ai/swagger.json“PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
  • [claimed-docs] https://docs.helicone.ai/rest/request/post-v1requestquery“Get Requests (Point Queries)”

Test against a sandbox environment without touching production dataweight 1

1 (weight) × 3 (quality) × 0.6 (partial) = 1.8 of 10 max

  • [github] https://github.com/Helicone/helicone“Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
  • [github] https://github.com/Helicone/helicone“Rapidly test and iterate on prompts, sessions and traces in our UI.”
  • [claimed-docs] https://docs.helicone.ai/getting-started/self-host/overview“Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”

Rely on versioned APIs with a documented deprecation policyweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://docs.helicone.ai/swagger.json“PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
  • [claimed-docs] https://docs.helicone.ai/rest/request/post-v1requestquery“Get Requests (Point Queries)”

API quality = 25.8 ÷ 70 × 100 = 36.9

Openness30.0/100×0.20 of the PA blend

Can you leave, inspect, or self-host — data export, open source, portability.

Do everything through the API that I can do in the UIweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://docs.helicone.ai/rest/request/post-v1requestquery“Get Requests (Point Queries)”
  • [claimed-docs] https://www.helicone.ai/pricing“HQL (Query Language)”
  • [probe] https://docs.helicone.ai/swagger.json“PROBE openapi: HTTP 200 at https://docs.helicone.ai/swagger.json — contains "openapi" key”
  • [github] https://github.com/Helicone/helicone“Playground: Rapidly test and iterate on prompts, sessions and traces in our UI.”
  • [claimed-docs] https://docs.helicone.ai/features/advanced-usage/prompts/overview“Test and deploy prompt changes instantly without rebuilding or redeploying your application”
  • [claimed-docs] https://docs.helicone.ai/features/webhooks“Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI activity with external systems.”

Export all of my data in open formats and leaveweight 3

3 (weight) × 5 (quality) × 0.6 (partial) = 9.0 of 30 max

  • [claimed-docs] https://docs.helicone.ai/rest/request/post-v1requestquery“Get Requests (Point Queries)”
  • [github] https://github.com/Helicone/helicone“Export to PostHog in one-line for custom dashboards”
  • [claimed-docs] https://docs.helicone.ai/getting-started/self-host/overview“Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
  • [claimed-docs] https://docs.helicone.ai/getting-started/self-host/overview“Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
  • [community] https://news.ycombinator.com/item?id=35279155“Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typically considered an open source license.”

Read the product's source under an open licenseweight 2

2 (weight) × 5 (quality) × 0.3 (disputed) = 3.0 of 20 max

  • [claimed-docs] https://docs.helicone.ai/getting-started/self-host/overview“Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
  • [claimed-docs] https://docs.helicone.ai/getting-started/self-host/overview“Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
  • [claimed-docs] https://www.helicone.ai/pricing“Helicone gives you more provider flexibility, is open-source, and scales more cost-effectively.”
  • [community] https://news.ycombinator.com/item?id=35279155“Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typically considered an open source license.”

Self-host the core productweight 3

3 (weight) × 6 (quality) × 0.6 (partial) = 10.8 of 30 max

  • [claimed-docs] https://docs.helicone.ai/getting-started/self-host/overview“Helicone offers multiple deployment methods to suit your infrastructure and scalability needs.”
  • [claimed-docs] https://docs.helicone.ai/getting-started/self-host/overview“Docker Compose: Ideal for quick setups, local development, or small-scale deployments without complex infrastructure requirements.”
  • [claimed-docs] https://www.helicone.ai/pricing“Helicone gives you more provider flexibility, is open-source, and scales more cost-effectively.”
  • [community] https://news.ycombinator.com/item?id=35279155“Congrats on the launch on launch! I noticed you are referring to the project as open source while using the commons clause, which isn't typically considered an open source license.”
  • [community] https://news.ycombinator.com/item?id=48903110“Discussion of Helicone's architecture allowing users to write SQL directly to a shared ClickHouse instance, with commenters noting noisy-neighbor query performance risks in shared multi-tenant databases.”

Openness = 30.0 ÷ 100 × 100 = 30.0

Built-in AI24.0/100×0.15 of the PA blend

Inside-out: how agentic the product itself is for its users — built-in assistants, autonomous features.

Get AI-generated insights and suggestions from my data inside the productweight 2

2 (weight) × 4 (quality) × 0.6 (partial) = 4.8 of 20 max

  • [claimed-docs] https://docs.helicone.ai/features/webhooks“Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”

Set up automations that run autonomously in the backgroundweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Delegate tasks to a built-in AI assistant inside the productweight 3

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Operate the product with natural-language commandsweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Built-in AI = 4.8 ÷ 20 × 100 = 24.0

Automation29.0/100×0.15 of the PA blend

Depth of automation primitives — rules, scheduling, bulk operations, webhooks.

Perform bulk operations across many items at onceweight 2

2 (weight) × 4 (quality) × 0.6 (partial) = 4.8 of 20 max

  • [claimed-docs] https://www.helicone.ai/pricing“HQL (Query Language)”
  • [claimed-docs] https://docs.helicone.ai/rest/request/post-v1requestquery“Get Requests (Point Queries)”
  • [github] https://github.com/Helicone/helicone“Export to PostHog in one-line for custom dashboards”

Define rules that trigger actions automatically on eventsweight 3

3 (weight) × 5 (quality) × 0.6 (partial) = 9.0 of 30 max

  • [claimed-docs] https://docs.helicone.ai/features/webhooks“Webhooks provide instant notifications when LLM requests complete, allowing you to automate workflows, score responses, and integrate AI activity with external systems.”
  • [claimed-docs] https://docs.helicone.ai/features/webhooks“Only requests matching ALL specified properties will trigger webhooks.”
  • [claimed-docs] https://docs.helicone.ai/features/alerts“Helicone Alerts let you monitor error rates and costs on LLM requests to catch issues before they impact users.”
  • [claimed-docs] https://docs.helicone.ai/features/webhooks“Real-time evaluation: Automatically score and evaluate LLM responses for quality, safety, and relevance”

Schedule recurring jobs or workflowsweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Version, review, and roll back my automationsweight 1

1 (weight) × 6 (quality) × 0.6 (partial) = 3.6 of 10 max

  • [claimed-docs] https://docs.helicone.ai/features/advanced-usage/prompts/overview“Track every change, compare versions, and rollback instantly if something goes wrong”
  • [claimed-docs] https://docs.helicone.ai/features/advanced-usage/prompts/overview“Test and deploy prompt changes instantly without rebuilding or redeploying your application”
  • [claimed-docs] https://docs.helicone.ai/features/advanced-usage/prompts/overview“Use your prompt instantly by referencing its ID in your AI Gateway. No code changes, no rebuilds.”

Automation = 17.4 ÷ 60 × 100 = 29.0