How W&B Weave’s scores are calculated

Point an agent at llms.txt or agent-oriented docsweight 2

2 (weight) × 9 (quality) × 1.0 (full) = 18.0 of 20 max

• [probe] https://docs.wandb.ai/llms.txt“PROBE llms.txt: HTTP 200 at https://docs.wandb.ai/llms.txt # Weights & Biases Documentation - [Products (407 pages)](https://docs.wandb.ai/_llms/en/products.md): Documentation fo”
• [probe] https://docs.wandb.ai/weave.md“PROBE docs-md: HTTP 200 at https://docs.wandb.ai/weave.md > ## Documentation Index > Fetch the complete documentation index at: https://docs.wandb.ai/llms.txt > Use this file to”
• [probe] https://docs.wandb.ai/openapi.json“PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
• [probe] https://github.com/wandb/wandb-mcp-server“official MCP server documented at https://github.com/wandb/wandb-mcp-server”

Run the product headlessly / in CI for automationweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

• [claimed-docs] https://docs.wandb.ai/weave/guides/tracking/tracing“When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, inputs, outputs, and execution metadata.”
• [claimed-docs] https://docs.wandb.ai/weave/reference/service-api“Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
• [claimed-docs] https://docs.wandb.ai/weave/guides/tracking/otel“Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
• [github] https://github.com/wandb/weave“Log and debug language model inputs, outputs, and traces”

Plug MCP servers into this product so it can use their toolsweight 3

n/a — not applicable to this product: excluded from numerator and denominator

• [claimed-docs] https://wandb.ai/site/weave/“Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, and execute automatic iteration loops on their own”
• [probe] https://github.com/wandb/wandb-mcp-server“official MCP server documented at https://github.com/wandb/wandb-mcp-server”

Connect an agent via an official MCP serverweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

• [claimed-docs] https://wandb.ai/site/weave/“Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, and execute automatic iteration loops on their own”
• [probe] https://github.com/wandb/wandb-mcp-server“official MCP server documented at https://github.com/wandb/wandb-mcp-server”

Use an official CLIweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

• [probe] https://docs.wandb.ai/models/ref/cli“official CLI documented at https://docs.wandb.ai/models/ref/cli”

Drive the product through a documented public APIweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

• [claimed-docs] https://docs.wandb.ai/weave/reference/service-api“Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
• [probe] https://docs.wandb.ai/openapi.json“PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
• [probe] https://github.com/wandb/wandb-mcp-server“official MCP server documented at https://github.com/wandb/wandb-mcp-server”
• [claimed-docs] https://wandb.ai/site/weave/“Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, and execute automatic iteration loops on their own”
• [claimed-docs] https://docs.wandb.ai/weave/guides/tracking/tracing“When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, inputs, outputs, and execution metadata.”

Issue scoped/least-privilege API credentials for an agentweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Build against official SDKsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

• [claimed-docs] https://docs.wandb.ai/weave/guides/tracking/tracing“When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, inputs, outputs, and execution metadata.”
• [github] https://github.com/wandb/weave“You can trace any function using `weave.op` - from api calls to OpenAI, Anthropic, Google AI Studio etc to generation calls from Hugging Face and other open source models to any other validation functions or data transformations”
• [claimed-docs] https://docs.wandb.ai/weave/reference/service-api“Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
• [claimed-docs] https://docs.wandb.ai/weave“Trace and collect metrics about your agents built with popular SDKs and harnesses using Weave OTel-compatible SDK”
• [probe] https://docs.wandb.ai/openapi.json“PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
• [probe] https://github.com/wandb/wandb-mcp-server“official MCP server documented at https://github.com/wandb/wandb-mcp-server”
• [probe] https://docs.wandb.ai/models/ref/cli“official CLI documented at https://docs.wandb.ai/models/ref/cli”

Subscribe to events via webhooksweight 2

2 (weight) × 4 (quality) × 0.6 (partial) = 4.8 of 20 max

• [claimed-docs] https://wandb.ai/site/weave/“Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteration.”
• [claimed-docs] https://wandb.ai/site/weave/“Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications and trigger webhook automations”
• [claimed-docs] https://wandb.ai/site/weave/“Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts route what matters through Slack notifications and trigger webhook automations”

Explore an interactive API reference with runnable examplesweight 2

2 (weight) × 3 (quality) × 0.6 (partial) = 3.6 of 20 max

• [claimed-docs] https://docs.wandb.ai/weave/reference/service-api“Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
• [probe] https://docs.wandb.ai/openapi.json“PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”

Download a machine-readable API spec (OpenAPI or equivalent)weight 2

2 (weight) × 9 (quality) × 1.0 (full) = 18.0 of 20 max

• [probe] https://docs.wandb.ai/openapi.json“PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
• [claimed-docs] https://docs.wandb.ai/weave/reference/service-api“Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”

Test against a sandbox environment without touching production dataweight 1

1 (weight) × 3 (quality) × 0.6 (partial) = 1.8 of 10 max

• [claimed-docs] https://docs.wandb.ai/weave/guides/core-types/evaluations“the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more scoring functions.”
• [claimed-docs] https://docs.wandb.ai/weave/guides/tools/playground“With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
• [claimed-docs] https://wandb.ai/site/weave/“You can test new LLMs and custom models against production traces, assessing their performance for your specific use cases.”

Rely on versioned APIs with a documented deprecation policyweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

• [probe] https://docs.wandb.ai/openapi.json“PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
• [claimed-docs] https://docs.wandb.ai/weave/reference/service-api“Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”

Do everything through the API that I can do in the UIweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

• [claimed-docs] https://docs.wandb.ai/weave/reference/service-api“Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
• [probe] https://docs.wandb.ai/openapi.json“PROBE openapi: HTTP 200 at https://docs.wandb.ai/openapi.json — contains "openapi" key”
• [claimed-docs] https://docs.wandb.ai/weave/guides/tools/playground“With features like prompt editing, message retrying, and model comparison, Playground helps you test and improve your LLM applications.”
• [claimed-docs] https://wandb.ai/site/weave/“Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”
• [claimed-docs] https://wandb.ai/site/weave/“Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteration.”

Export all of my data in open formats and leaveweight 3

3 (weight) × 3 (quality) × 0.6 (partial) = 5.4 of 30 max

• [claimed-docs] https://docs.wandb.ai/weave/reference/service-api“Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
• [claimed-docs] https://docs.wandb.ai/weave/guides/tracking/otel“Weave supports import of OpenTelemetry compatible trace data through a dedicated endpoint.”
• [claimed-docs] https://docs.wandb.ai/weave/guides/tracking/otel“Use this integration when you want to instrument your application with the OpenTelemetry standard and have those traces appear alongside your other Weave data, without replacing your existing OTel-based observability pipeline.”

Read the product's source under an open licenseweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

• [github] https://github.com/wandb/weave“You can trace any function using `weave.op` - from api calls to OpenAI, Anthropic, Google AI Studio etc to generation calls from Hugging Face and other open source models to any other validation functions or data transformations”
• [github] https://github.com/wandb/weave“Decorate all the functions you want to trace, this will generate a trace tree of the inputs and outputs of all your functions”
• [github] https://github.com/wandb/weave“Log and debug language model inputs, outputs, and traces”
• [github] https://github.com/wandb/weave“Build rigorous, apples-to-apples evaluations for language model use cases”

Self-host the core productweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Get AI-generated insights and suggestions from my data inside the productweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

• [claimed-docs] https://docs.wandb.ai/weave/guides/evaluation/scorers“Weave comes with predefined scorers and local SLM scorers that you can use right away, including: Hallucination detection, Summarization quality, Embedding similarity”
• [claimed-docs] https://wandb.ai/site/weave/“Weave Guardrails offers pre-built scorers for safety and quality to support responsible AI. Safety scorers include toxicity, bias, PII detection, and hallucinations”
• [claimed-docs] https://wandb.ai/site/weave/“Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”
• [claimed-docs] https://wandb.ai/site/weave/“Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts route what matters through Slack notifications and trigger webhook automations”
• [claimed-docs] https://docs.wandb.ai/weave/guides/evaluation/scorers“Custom Scorers let you encode evaluation criteria that are specific to your use case, beyond what the built-in Scorers cover.”

Set up automations that run autonomously in the backgroundweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

• [claimed-docs] https://wandb.ai/site/weave/“Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteration.”
• [claimed-docs] https://wandb.ai/site/weave/“Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications and trigger webhook automations”
• [claimed-docs] https://wandb.ai/site/weave/“Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts route what matters through Slack notifications and trigger webhook automations”
• [claimed-docs] https://wandb.ai/site/weave/“Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, and execute automatic iteration loops on their own”

Delegate tasks to a built-in AI assistant inside the productweight 3

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Operate the product with natural-language commandsweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

• [claimed-docs] https://wandb.ai/site/weave/“Using W&B skills and the MCP server, coding agents like Claude Code connect to Weave. They can read live production data, run evaluations, and execute automatic iteration loops on their own”
• [probe] https://github.com/wandb/wandb-mcp-server“official MCP server documented at https://github.com/wandb/wandb-mcp-server”

Perform bulk operations across many items at onceweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

• [claimed-docs] https://docs.wandb.ai/weave/guides/core-types/evaluations“the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more scoring functions.”
• [claimed-docs] https://docs.wandb.ai/weave/guides/core-types/evaluations“The core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples.”
• [claimed-docs] https://docs.wandb.ai/weave/guides/core-types/evaluations“This guide shows you how to set up and run evaluations in Weave so you can measure your LLM application’s performance against repeatable test cases, compare changes over time, and identify regressions.”
• [claimed-docs] https://docs.wandb.ai/weave/reference/service-api“Use the Weave Service API, which provides REST endpoints for programmatic access to Weave functionality.”
• [claimed-docs] https://wandb.ai/site/weave/“Aggregate evaluations into leaderboards featuring the best performers and share them across your organization.”

Define rules that trigger actions automatically on eventsweight 3

3 (weight) × 5 (quality) × 0.6 (partial) = 9.0 of 30 max

• [claimed-docs] https://wandb.ai/site/weave/“Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteration.”
• [claimed-docs] https://wandb.ai/site/weave/“Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications and trigger webhook automations”
• [claimed-docs] https://wandb.ai/site/weave/“Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts route what matters through Slack notifications and trigger webhook automations”
• [claimed-docs] https://wandb.ai/site/weave/“Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving.”

Schedule recurring jobs or workflowsweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Version, review, and roll back my automationsweight 1

1 (weight) × 3 (quality) × 0.6 (partial) = 1.8 of 10 max

• [claimed-docs] https://docs.wandb.ai/weave/guides/tracking/tracing“When you decorate a function with `@weave.op()` (Python) or wrap it with `weave.op()` (TypeScript), Weave automatically captures its code, inputs, outputs, and execution metadata.”
• [claimed-docs] https://docs.wandb.ai/weave/guides/core-types/evaluations“the core of the workflow is the _`Evaluation` object_, which defines: A `Dataset` or list of dictionaries for test examples. One or more scoring functions.”
• [claimed-docs] https://wandb.ai/site/weave/“Alerts route what matters through Slack notifications and trigger webhook automations, turning every production insight into a fast iteration.”
• [claimed-docs] https://wandb.ai/site/weave/“Built-in and custom signals automatically capture and classify agent interactions... Alerts route what matters through Slack notifications and trigger webhook automations”
• [claimed-docs] https://wandb.ai/site/weave/“Built-in and custom signals automatically capture and classify agent interactions, so you always know how your agents are behaving. Alerts route what matters through Slack notifications and trigger webhook automations”