Skip to content
ProductArena

How Sierra’s scores are calculated

The full audit trail, recomputed from the verdict data at build time through the same code that produced the leaderboard: verdict × quality × story weight per cell, cells sum to dimension scores, dimensions blend into the PA Score. Every number on the product page is reproducible from this page alone; for why the formula looks like this, see the methodology.

verdict factors: full ×1.0 · partial ×0.6 · disputed ×0.3 · none ×0.0 · n/a excluded from both sides · cell points = weight × quality × factor · cell max = weight × 10

PA Score14/100

Agent-ready 10.0 × 0.30 = 3.00

API quality 4.3 × 0.20 = 0.86

Openness 0.0 × 0.20 = 0.00

Built-in AI 53.3 × 0.15 = 7.99

Automation 15.0 × 0.15 = 2.25

(3.00 + 0.86 + 0.00 + 7.99 + 2.25) ÷ (0.30 + 0.20 + 0.20 + 0.15 + 0.15) = 14.10 ÷ 1.00 = 14.1

Scores are stored to 1 decimal; the product page’s pills round to whole numbers for display. Each dimension below shows the stories, verdicts, and cited evidence behind its number.

Agent-ready10.0/100×0.30 of the PA blend

Outside-in: can YOUR agent reach and drive this product — API, MCP, CLI, headless runs, agent docs.

Point an agent at llms.txt or agent-oriented docsweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://sierra.ai/llms.txt“PROBE llms.txt: HTTP 404 at https://sierra.ai/llms.txt”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

Run the product headlessly / in CI for automationweight 2

2 (weight) × 4 (quality) × 0.6 (partial) = 4.8 of 20 max

  • [claimed-docs] https://sierra.ai/product/agent-sdk“write customer journeys as code, track changes, and build sophisticated logic without giving up our development workflows”
  • [claimed-docs] https://sierra.ai/blog/publish-to-chatgpt“Publish to ChatGPT with one click: Make your agent available on ChatGPT with the push of a button, or via CI/CD.”
  • [probe] https://sierra.ai/llms.txt“PROBE llms.txt: HTTP 404 at https://sierra.ai/llms.txt”
  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

Plug MCP servers into this product so it can use their toolsweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Connect an agent via an official MCP serverweight 3

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Use an official CLIweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://sierra.ai/product/agent-sdk“write customer journeys as code, track changes, and build sophisticated logic without giving up our development workflows”
  • [probe] https://sierra.ai/llms.txt“PROBE llms.txt: HTTP 404 at https://sierra.ai/llms.txt”
  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”

Drive the product through a documented public APIweight 3

3 (weight) × 4 (quality) × 0.6 (partial) = 7.2 of 30 max

  • [claimed-docs] https://sierra.ai/product/agent-sdk“write customer journeys as code, track changes, and build sophisticated logic without giving up our development workflows”
  • [claimed-docs] https://sierra.ai/product/agent-sdk“Understand and rapidly adjust agent behavior by inspecting API calls, logic traces, and more.”
  • [probe] https://sierra.ai/llms.txt“PROBE llms.txt: HTTP 404 at https://sierra.ai/llms.txt”
  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

Issue scoped/least-privilege API credentials for an agentweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

Build against official SDKsweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

  • [claimed-docs] https://sierra.ai/product/agent-sdk“write customer journeys as code, track changes, and build sophisticated logic without giving up our development workflows”
  • [claimed-docs] https://sierra.ai/product/agent-sdk“Understand and rapidly adjust agent behavior by inspecting API calls, logic traces, and more.”
  • [claimed-docs] https://sierra.ai/product/agent-sdk“Verify your agent performs as expected across a wide range of scenarios and avoid regressions.”
  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

Subscribe to events via webhooksweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

Agent-ready = 18.0 ÷ 180 × 100 = 10.0

API quality4.3/100×0.20 of the PA blend

The programmable surface once an agent is there — machine-readable spec, interactive docs, sandbox, versioning discipline.

Explore an interactive API reference with runnable examplesweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://sierra.ai/llms.txt“PROBE llms.txt: HTTP 404 at https://sierra.ai/llms.txt”
  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

Download a machine-readable API spec (OpenAPI or equivalent)weight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://sierra.ai/llms.txt“PROBE llms.txt: HTTP 404 at https://sierra.ai/llms.txt”
  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

Test against a sandbox environment without touching production dataweight 1

1 (weight) × 5 (quality) × 0.6 (partial) = 3.0 of 10 max

  • [claimed-docs] https://sierra.ai/product/agent-sdk“Verify your agent performs as expected across a wide range of scenarios and avoid regressions.”
  • [claimed-docs] https://sierra.ai/blog/release-governance-guardrails-for-agents-at-scale“Agent Checks and Simulations catching problems proactively, merge approval workflows putting a person in the loop, and split traffic releases rolling changes out gradually.”

Rely on versioned APIs with a documented deprecation policyweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://sierra.ai/llms.txt“PROBE llms.txt: HTTP 404 at https://sierra.ai/llms.txt”
  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

API quality = 3.0 ÷ 70 × 100 = 4.3

Openness0.0/100×0.20 of the PA blend

Can you leave, inspect, or self-host — data export, open source, portability.

Do everything through the API that I can do in the UIweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://sierra.ai/product/agent-sdk“write customer journeys as code, track changes, and build sophisticated logic without giving up our development workflows”
  • [claimed-docs] https://sierra.ai/product/agent-studio“Empower teams to build and manage agents—no code required”
  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

Export all of my data in open formats and leaveweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

  • [probe] https://sierra.ai/llms.txt“PROBE llms.txt: HTTP 404 at https://sierra.ai/llms.txt”
  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

Read the product's source under an open licenseweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://sierra.ai/resources/research/tau-bench“PROBE runtime (recorded 2026-09-10): Sierra's agent benchmark is genuinely open — the official sierra-research/tau-bench repository README fetched keylessly from GitHub, the published benchmark for evaluating conversational agents on simulated user interactions.”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”
  • [probe] https://sierra.ai/llms.txt“PROBE llms.txt: HTTP 404 at https://sierra.ai/llms.txt”
  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”

Self-host the core productweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”
  • [probe] https://sierra.ai/llms.txt“PROBE llms.txt: HTTP 404 at https://sierra.ai/llms.txt”
  • [probe] https://sierra.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://sierra.ai/openapi.json, https://sierra.ai/swagger.json, https://sierra.ai/api/openapi.json, https://sierra.ai/.well-known/openapi.json)”

Openness = 0.0 ÷ 100 × 100 = 0.0

Built-in AI53.3/100×0.15 of the PA blend

Inside-out: how agentic the product itself is for its users — built-in assistants, autonomous features.

Get AI-generated insights and suggestions from my data inside the productweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://sierra.ai/product/explorer“Ask any question about your customer experience in natural language, and Explorer identifies the answer across thousands of real conversations.”
  • [claimed-docs] https://sierra.ai/product/explorer“Explorer automatically delivers a weekly briefing on key trends, emerging issues, and recommendations — without you having to ask.”
  • [claimed-docs] https://sierra.ai/product/insights“Click any data point on a report to launch Explorer and instantly investigate what's driving that trend.”
  • [claimed-docs] https://sierra.ai/product/insights“Understand the reasoning behind every agent action or answer—view knowledge sources, systems accessed, and more.”
  • [community] https://hn.algolia.com/api/v1/items/39358925“Having seen this set up for a customer, what is interesting is how it works with internal APIs for customers. It's all point and click in a simple UI -- add a UI, describe how the agent can work with it, and get the agent set up and working.”
  • [community] https://hn.algolia.com/api/v1/items/39358925“The magic isn't in a new LLM technology, it is in reliably productionizing a solution for real-world problems... fill in the gap of missing web forms for processes that customers don't yet have access to, that currently fall to human support teams.”

Set up automations that run autonomously in the backgroundweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://sierra.ai/product/explorer“Explorer automatically delivers a weekly briefing on key trends, emerging issues, and recommendations — without you having to ask.”
  • [claimed-docs] https://sierra.ai/product/agent-sdk“Build once and deploy across any channel—chat, phone, email, SMS, and messaging.”
  • [claimed-docs] https://sierra.ai/product/live-assist“Guide care reps through next steps—whether in chat or on calls—automatically capturing and updating customer context as work progresses.”
  • [community] https://hn.algolia.com/api/v1/items/39358925“Having seen this set up for a customer, what is interesting is how it works with internal APIs for customers. It's all point and click in a simple UI -- add a UI, describe how the agent can work with it, and get the agent set up and working.”

Delegate tasks to a built-in AI assistant inside the productweight 3

3 (weight) × 6 (quality) × 0.6 (partial) = 10.8 of 30 max

  • [claimed-docs] https://sierra.ai/product/ghostwriter“Build or modify agents by describing how you want them to behave. Update workflows, systems integrations, guardrails, tone, and style with simple prompts.”
  • [claimed-docs] https://sierra.ai/product/ghostwriter“Upload SOPs, raw transcripts, or even audio interviews of subject matter experts to create customer journeys from scratch.”
  • [claimed-docs] https://sierra.ai/product/agent-studio“Define step-by-step workflows for your agent from scratch, or use AI to instantly generate journeys from your existing operating procedures.”

Operate the product with natural-language commandsweight 2

2 (weight) × 7 (quality) × 1.0 (full) = 14.0 of 20 max

  • [claimed-docs] https://sierra.ai/product/ghostwriter“Build or modify agents by describing how you want them to behave. Update workflows, systems integrations, guardrails, tone, and style with simple prompts.”
  • [claimed-docs] https://sierra.ai/product/ghostwriter“Upload SOPs, raw transcripts, or even audio interviews of subject matter experts to create customer journeys from scratch.”
  • [claimed-docs] https://sierra.ai/product/explorer“Ask any question about your customer experience in natural language, and Explorer identifies the answer across thousands of real conversations.”
  • [community] https://hn.algolia.com/api/v1/items/39358925“Having seen this set up for a customer, what is interesting is how it works with internal APIs for customers. It's all point and click in a simple UI -- add a UI, describe how the agent can work with it, and get the agent set up and working.”

Built-in AI = 48.0 ÷ 90 × 100 = 53.3

Automation15.0/100×0.15 of the PA blend

Depth of automation primitives — rules, scheduling, bulk operations, webhooks.

Perform bulk operations across many items at onceweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Define rules that trigger actions automatically on eventsweight 3

3 (weight) × 5 (quality) × 0.6 (partial) = 9.0 of 30 max

  • [claimed-docs] https://sierra.ai/product/agent-studio“Define step-by-step workflows for your agent from scratch, or use AI to instantly generate journeys from your existing operating procedures.”
  • [claimed-docs] https://sierra.ai/blog/release-governance-guardrails-for-agents-at-scale“Agent Checks and Simulations catching problems proactively, merge approval workflows putting a person in the loop, and split traffic releases rolling changes out gradually.”
  • [claimed-docs] https://sierra.ai/product/live-assist“Initiate workflows directly from a conversation—no tab-switching, no referencing instructions, no lost context.”

Schedule recurring jobs or workflowsweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Version, review, and roll back my automationsweight 1

1 (weight) × 5 (quality) × 0.6 (partial) = 3.0 of 10 max

  • [claimed-docs] https://sierra.ai/product/agent-sdk“write customer journeys as code, track changes, and build sophisticated logic without giving up our development workflows”
  • [claimed-docs] https://sierra.ai/blog/release-governance-guardrails-for-agents-at-scale“Agent Checks and Simulations catching problems proactively, merge approval workflows putting a person in the loop, and split traffic releases rolling changes out gradually.”
  • [probe] https://docs.sierra.ai/“PROBE runtime finding (recorded 2026-09-10): docs.sierra.ai has no agent-legible surface — /llms.txt resolves (HTTP 200) to the login SPA's text/html shell rather than a plain-text index; the docs and Agent SDK reference are login-gated to contracted customers.”

Automation = 12.0 ÷ 80 × 100 = 15.0