Skip to content
ProductArena

How LangSmith’s scores are calculated

The full audit trail, recomputed from the verdict data at build time through the same code that produced the leaderboard: verdict × quality × story weight per cell, cells sum to dimension scores, dimensions blend into the PA Score. Every number on the product page is reproducible from this page alone; for why the formula looks like this, see the methodology.

verdict factors: full ×1.0 · partial ×0.6 · disputed ×0.3 · none ×0.0 · n/a excluded from both sides · cell points = weight × quality × factor · cell max = weight × 10

PA Score23/100

Agent-ready 35.8 × 0.30 = 10.74

API quality 4.3 × 0.20 = 0.86

Openness 24.0 × 0.20 = 4.80

Built-in AI 20.7 × 0.15 = 3.10

Automation 22.5 × 0.15 = 3.38

(10.74 + 0.86 + 4.80 + 3.10 + 3.38) ÷ (0.30 + 0.20 + 0.20 + 0.15 + 0.15) = 22.88 ÷ 1.00 = 22.9

Scores are stored to 1 decimal; the product page’s pills round to whole numbers for display. Each dimension below shows the stories, verdicts, and cited evidence behind its number.

Agent-ready35.8/100×0.30 of the PA blend

Outside-in: can YOUR agent reach and drive this product — API, MCP, CLI, headless runs, agent docs.

Point an agent at llms.txt or agent-oriented docsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [probe] https://docs.langchain.com/llms.txt“PROBE llms.txt: HTTP 200 at https://docs.langchain.com/llms.txt # Docs by LangChain > Documentation for LangSmith, Fleet, and our open source packages. Each section index below lists”
  • [probe] https://docs.langchain.com/langsmith/home.md“PROBE docs-md: HTTP 200 at https://docs.langchain.com/langsmith/home.md > ## Documentation Index > Fetch the complete documentation index at: https://docs.langchain.com/llms.txt > Use this fil”

Run the product headlessly / in CI for automationweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.”
  • [claimed-docs] https://docs.langchain.com/langsmith/trace-with-opentelemetry“LangSmith supports OpenTelemetry-based tracing, allowing you to send traces from any OpenTelemetry-compatible application.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Run evaluations on curated datasets during development to compare versions, benchmark performance, and catch regressions.”
  • [claimed-docs] https://www.langchain.com/langsmith/observability“Trace your preferred framework or integrate LangSmith with any agent stack using our Python, Typescript, Go, or Java SDKs.”
  • [github] https://github.com/langchain-ai/langsmith-sdk“client = wrap_openai(openai.Client())”

Plug MCP servers into this product so it can use their toolsweight 3

n/a — not applicable to this product: excluded from numerator and denominator

  • [probe] https://github.com/langchain-ai/langsmith-mcp-server“official MCP server documented at https://github.com/langchain-ai/langsmith-mcp-server”

Connect an agent via an official MCP serverweight 3

3 (weight) × 4 (quality) × 0.6 (partial) = 7.2 of 30 max

  • [probe] https://github.com/langchain-ai/langsmith-mcp-server“official MCP server documented at https://github.com/langchain-ai/langsmith-mcp-server”

Use an official CLIweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Drive the product through a documented public APIweight 3

3 (weight) × 6 (quality) × 0.6 (partial) = 10.8 of 30 max

  • [claimed-docs] https://docs.langchain.com/langsmith/home“Filter, export, share, and compare traces via the UI or API.”
  • [claimed-docs] https://www.langchain.com/langsmith/observability“Trace your preferred framework or integrate LangSmith with any agent stack using our Python, Typescript, Go, or Java SDKs.”
  • [claimed-docs] https://docs.langchain.com/langsmith/home“Automate workflows with rules, webhooks, and online evaluations.”
  • [probe] https://docs.langchain.com/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langchain.com/api/openapi.json, https://docs.langchain.com/.well-known/openapi.json)”

Issue scoped/least-privilege API credentials for an agentweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Build against official SDKsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [claimed-docs] https://www.langchain.com/langsmith/observability“Trace your preferred framework or integrate LangSmith with any agent stack using our Python, Typescript, Go, or Java SDKs.”
  • [github] https://github.com/langchain-ai/langsmith-sdk“client = wrap_openai(openai.Client())”
  • [claimed-docs] https://docs.langchain.com/langsmith/trace-with-opentelemetry“LangSmith supports OpenTelemetry-based tracing, allowing you to send traces from any OpenTelemetry-compatible application.”
  • [community] https://news.ycombinator.com/item?id=42485113“Langsmith only shows the llm inferences. Also it hardly works with other frameworks like crewai for instance.”

Subscribe to events via webhooksweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://docs.langchain.com/langsmith/home“Automate workflows with rules, webhooks, and online evaluations.”
  • [claimed-docs] https://docs.langchain.com/langsmith/alerts“Alerts can route to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
  • [claimed-docs] https://docs.langchain.com/langsmith/alerts“Alerts can [route](#step-4-configure-notification-channel) to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
  • [claimed-docs] https://docs.langchain.com/langsmith/alerts“LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”

Agent-ready = 64.4 ÷ 180 × 100 = 35.8

API quality4.3/100×0.20 of the PA blend

The programmable surface once an agent is there — machine-readable spec, interactive docs, sandbox, versioning discipline.

Explore an interactive API reference with runnable examplesweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://docs.langchain.com/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langchain.com/api/openapi.json, https://docs.langchain.com/.well-known/openapi.json)”

Download a machine-readable API spec (OpenAPI or equivalent)weight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://docs.langchain.com/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langchain.com/api/openapi.json, https://docs.langchain.com/.well-known/openapi.json)”
  • [claimed-docs] https://docs.langchain.com/langsmith/home“Filter, export, share, and compare traces via the UI or API.”

Test against a sandbox environment without touching production dataweight 1

1 (weight) × 5 (quality) × 0.6 (partial) = 3.0 of 10 max

  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Create a dataset with examples from manually curated test cases, historical production traces, or synthetic data generation.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Run evaluations on curated datasets during development to compare versions, benchmark performance, and catch regressions.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Create a [dataset](/langsmith/manage-datasets) with [examples](/langsmith/evaluation-concepts#examples) from manually curated test cases, historical production traces, or synthetic data generation.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Execute your application on the dataset to create an [experiment](/langsmith/evaluation-concepts#experiment). Configure [repetitions, concurrency, and caching](/langsmith/experiment-configuration) to optimize runs.”

Rely on versioned APIs with a documented deprecation policyweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://docs.langchain.com/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langchain.com/api/openapi.json, https://docs.langchain.com/.well-known/openapi.json)”

API quality = 3.0 ÷ 70 × 100 = 4.3

Openness24.0/100×0.20 of the PA blend

Can you leave, inspect, or self-host — data export, open source, portability.

Do everything through the API that I can do in the UIweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

  • [claimed-docs] https://docs.langchain.com/langsmith/home“Filter, export, share, and compare traces via the UI or API.”
  • [claimed-docs] https://www.langchain.com/langsmith/observability“Trace your preferred framework or integrate LangSmith with any agent stack using our Python, Typescript, Go, or Java SDKs.”
  • [probe] https://docs.langchain.com/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.langchain.com/openapi.json, https://docs.langchain.com/swagger.json, https://docs.langchain.com/api/openapi.json, https://docs.langchain.com/.well-known/openapi.json)”
  • [community] https://news.ycombinator.com/item?id=48063206“I built this because LangSmith needs a cloud account to see my own traces.”

Export all of my data in open formats and leaveweight 3

3 (weight) × 4 (quality) × 0.6 (partial) = 7.2 of 30 max

  • [claimed-docs] https://docs.langchain.com/langsmith/home“Filter, export, share, and compare traces via the UI or API.”
  • [claimed-docs] https://docs.langchain.com/langsmith/self-hosted“Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable LangSmith Deployment to deploy and manage agents through the LangSmith UI.”
  • [community] https://news.ycombinator.com/item?id=48063206“I built this because LangSmith needs a cloud account to see my own traces.”
  • [community] https://news.ycombinator.com/item?id=42485113“LangSmith is excellent, but my usage is quite minimal, and I would prefer a locally hosted version that is easy to customize.”

Read the product's source under an open licenseweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [github] https://github.com/langchain-ai/langsmith-sdk“client = wrap_openai(openai.Client())”
  • [probe] https://github.com/langchain-ai/langsmith-mcp-server“official MCP server documented at https://github.com/langchain-ai/langsmith-mcp-server”
  • [claimed-docs] https://docs.langchain.com/langsmith/self-hosted“Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable LangSmith Deployment to deploy and manage agents through the LangSmith UI.”

Self-host the core productweight 3

3 (weight) × 6 (quality) × 0.6 (partial) = 10.8 of 30 max

  • [claimed-docs] https://docs.langchain.com/langsmith/self-hosted“Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable LangSmith Deployment to deploy and manage agents through the LangSmith UI.”
  • [claimed-docs] https://docs.langchain.com/langsmith/self-hosted“Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering.”
  • [claimed-docs] https://docs.langchain.com/langsmith/self-hosted“Host an instance of LangSmith in your own infrastructure for [observability](/langsmith/observability), [evaluation](/langsmith/evaluation), and [prompt engineering](/langsmith/prompt-context-hub#prompts).”
  • [community] https://news.ycombinator.com/item?id=48063206“I built this because LangSmith needs a cloud account to see my own traces.”

Openness = 24.0 ÷ 100 × 100 = 24.0

Built-in AI20.7/100×0.15 of the PA blend

Inside-out: how agentic the product itself is for its users — built-in assistants, autonomous features.

Get AI-generated insights and suggestions from my data inside the productweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

  • [claimed-docs] https://docs.langchain.com/langsmith/home“Automatically detect recurring issues in your traces, diagnose their root cause, and resolve them with LangSmith Engine.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Create evaluators to score performance: Human review, Code rules, LLM-as-judge, Pairwise comparison”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Set up evaluators to run automatically on production traces: safety checks, format validation, quality heuristics, and reference-free LLM-as-judge. Apply filters and sampling rates to control costs.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Human review * Code rules * LLM-as-judge * Pairwise comparison”
  • [community] https://news.ycombinator.com/item?id=42485113“Langsmith only shows the llm inferences. Also it hardly works with other frameworks like crewai for instance.”

Set up automations that run autonomously in the backgroundweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://docs.langchain.com/langsmith/home“Automate workflows with rules, webhooks, and online evaluations.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Set up evaluators to run automatically on production traces: safety checks, format validation, quality heuristics, and reference-free LLM-as-judge. Apply filters and sampling rates to control costs.”
  • [claimed-docs] https://docs.langchain.com/langsmith/alerts“LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”
  • [claimed-docs] https://docs.langchain.com/langsmith/alerts“Alerts can route to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
  • [claimed-docs] https://docs.langchain.com/langsmith/self-hosted“Host an instance of LangSmith in your own infrastructure for observability, evaluation, and prompt engineering. You can optionally enable LangSmith Deployment to deploy and manage agents through the LangSmith UI.”

Delegate tasks to a built-in AI assistant inside the productweight 3

3 (weight) × 3 (quality) × 0.6 (partial) = 5.4 of 30 max

  • [claimed-docs] https://docs.langchain.com/langsmith/home“Automatically detect recurring issues in your traces, diagnose their root cause, and resolve them with LangSmith Engine.”

Operate the product with natural-language commandsweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://github.com/langchain-ai/langsmith-mcp-server“official MCP server documented at https://github.com/langchain-ai/langsmith-mcp-server”

Built-in AI = 18.6 ÷ 90 × 100 = 20.7

Automation22.5/100×0.15 of the PA blend

Depth of automation primitives — rules, scheduling, bulk operations, webhooks.

Perform bulk operations across many items at onceweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://docs.langchain.com/langsmith/home“Filter, export, share, and compare traces via the UI or API.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Create a dataset with examples from manually curated test cases, historical production traces, or synthetic data generation.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Create a [dataset](/langsmith/manage-datasets) with [examples](/langsmith/evaluation-concepts#examples) from manually curated test cases, historical production traces, or synthetic data generation.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Execute your application on the dataset to create an [experiment](/langsmith/evaluation-concepts#experiment). Configure [repetitions, concurrency, and caching](/langsmith/experiment-configuration) to optimize runs.”
  • [github] https://github.com/langchain-ai/langsmith-sdk“client = wrap_openai(openai.Client())”

Define rules that trigger actions automatically on eventsweight 3

3 (weight) × 6 (quality) × 0.6 (partial) = 10.8 of 30 max

  • [claimed-docs] https://docs.langchain.com/langsmith/home“Automate workflows with rules, webhooks, and online evaluations.”
  • [claimed-docs] https://docs.langchain.com/langsmith/alerts“LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”
  • [claimed-docs] https://docs.langchain.com/langsmith/alerts“Alerts can route to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”
  • [claimed-docs] https://docs.langchain.com/langsmith/alerts“LangSmith provides threshold-based alerting on the following metrics:”
  • [claimed-docs] https://docs.langchain.com/langsmith/alerts“Alerts can [route](#step-4-configure-notification-channel) to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”

Schedule recurring jobs or workflowsweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.langchain.com/langsmith/home“Automate workflows with rules, webhooks, and online evaluations.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Set up evaluators to run automatically on production traces: safety checks, format validation, quality heuristics, and reference-free LLM-as-judge. Apply filters and sampling rates to control costs.”
  • [claimed-docs] https://docs.langchain.com/langsmith/evaluation“Evaluate real user interactions in real-time to detect issues and measure quality on live traffic.”

Version, review, and roll back my automationsweight 1

1 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 10 max

  • [claimed-docs] https://docs.langchain.com/langsmith/home“Automate workflows with rules, webhooks, and online evaluations.”
  • [claimed-docs] https://docs.langchain.com/langsmith/alerts“LangSmith provides threshold-based alerting on the following metrics: Run Count, Cost, Errors, Feedback Score, Latency”
  • [claimed-docs] https://docs.langchain.com/langsmith/alerts“Alerts can route to Slack, PagerDuty, Dynatrace, or any HTTP endpoint via webhook.”

Automation = 18.0 ÷ 80 × 100 = 22.5