Rank #4 of 8 in LLM Evals & Observability
Access
Install
npm install @helicone/helpers
vs Langfuse
vs LangSmith
vs Braintrust
vs Arize Phoenix
vs W&B Weave
vs Galileo
vs CekuraShowcase
Verified integrations
Connections to other tracked products — hover a chip for the verbatim evidence quote behind it.
By theme — the product's score on each story themeBy theme
Agenticness — how well agents can access and operate the productAgenticnessevidence →
How well agents can access and operate the product
Alerting dashboards — stories about alerting dashboards in this arenaAlerting dashboardsevidence →
Stories about alerting dashboards in this arena
Automation depth — how much of the product can run unattendedAutomation depthevidence →
How much of the product can run unattended
Cost monitoring — stories about cost monitoring in this arenaCost monitoringevidence →
Stories about cost monitoring in this arena
Data access export — stories about data access export in this arenaData access exportevidence →
Stories about data access export in this arena
Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasetsevidence →
Measuring quality — datasets, eval runs, regression tracking
Openness — open source, data portability, and self-hosting storiesOpennessevidence →
Open source, data portability, and self-hosting stories
Privacy posture — data-handling and privacy storiesPrivacy postureevidence →
Data-handling and privacy stories
Prompt management — stories about prompt management in this arenaPrompt managementevidence →
Stories about prompt management in this arena
Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentationevidence →
Instrumenting code and tracing requests end to end
Story verdicts — every judged story with its evidenceStory verdicts
What’s free: 3 free · 0 paid · 0 enterprise · 31 not stated in evidence
Follow the green: where the map greys out is where Helicone stops today. ✓ full · ~ partial · ! disputed · — none · n/a not applicable.
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
API surface
Drive the product through a documented public API
✓8/10
unlocks → Scoped API keys · MCP server · Versioning policy · Official CLI
Subscribe to events via webhooks
✓7/10
Build against official SDKs
✓8/10
Issue scoped/least-privilege API credentials for an agent
—0/10
Connect an agent via an official MCP server
—–
Download a machine-readable API spec (OpenAPI or equivalent)
✓9/10
unlocks → MCP server
Rely on versioned APIs with a documented deprecation policy
—0/10
Test against a sandbox environment without touching production data
~3/10
Explore an interactive API reference with runnable examples
~5/10
Docs for agents
Point an agent at llms.txt or agent-oriented docs
✓8/10
Agentic features
Delegate tasks to a built-in AI assistant inside the product
n/an/a
Operate the product with natural-language commands
n/an/a
Plug MCP servers into this product so it can use their tools
n/an/a
Get AI-generated insights and suggestions from my data inside the product
~4/10
Set up automations that run autonomously in the background
n/an/a
Alerting dashboards — stories about alerting dashboards in this arenaAlerting dashboards
Stories about alerting dashboards in this arena
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
Cost monitoring — stories about cost monitoring in this arenaCost monitoring
Stories about cost monitoring in this arena
Data access export — stories about data access export in this arenaData access export
Stories about data access export in this arena
Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets
Measuring quality — datasets, eval runs, regression tracking
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
Prompt management — stories about prompt management in this arenaPrompt management
Stories about prompt management in this arena
Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation
Instrumenting code and tracing requests end to end
Have an agent query my traces, metrics, and eval results through an API or MCP server to debug my app
~5/10
Mask or redact sensitive data before it is stored in traces
—–
Instrument apps in both Python and JS/TS with officially supported SDKs
~6/10
Sorted by importance (agentic first) (high → low) · 52/52 stories · click a row’s chevron for the rationale and evidence
Drive the product through a documented public API G Agent access | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 3 | full | 8/10 | Tprobed | |
Connect an agent via an official MCP server G Agent access | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 3 | none | untested | none yet | |
Delegate tasks to a built-in AI assistant inside the product G Agentic features | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 3 | n/a | untested | none yet | |
Plug MCP servers into this product so it can use their tools G Agent access | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 3 | n/a | untested | none yet | |
Download a machine-readable API spec (OpenAPI or equivalent) G Api quality | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | full | 9/10 | Tprobed | |
Build against official SDKs G Agent access | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | full | 8/10 | Xcommunity | |
Point an agent at llms.txt or agent-oriented docs G Agent access | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | full | 8/10 | Tprobed | |
Subscribe to events via webhooks G Agent access | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | full | 7/10 | Cclaimed | |
Run the product headlessly / in CI for automation G Agent access | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | partial | 6/10 | Tprobed | |
Explore an interactive API reference with runnable examples G Api quality | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | partial | 5/10 | Tprobed | |
Get AI-generated insights and suggestions from my data inside the product G Agentic features | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | partial | 4/10 | Cclaimed | |
Issue scoped/least-privilege API credentials for an agent G Agent access | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | none | 0/10 | ||
Rely on versioned APIs with a documented deprecation policy G Api quality | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | none | 0/10 | ||
Operate the product with natural-language commands G Agentic features | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | n/a | untested | none yet | |
Set up automations that run autonomously in the background G Agentic features | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | n/a | untested | none yet | |
Use an official CLI G Agent access | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 2 | none | untested | none yet | |
Test against a sandbox environment without touching production data G Api quality | ai-native user | Agenticness — how well agents can access and operate the productAgenticness | 1 | partial | 3/10 | Cclaimed | |
Capture traces of my LLM calls with inputs, outputs, latency, and token usage C Trace capture | developer | Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation | 3 | fullfree | 9/10 | Xcommunity | |
Version prompts and deploy changes to production without shipping code C Prompt workflow | developer | Prompt management — stories about prompt management in this arenaPrompt management | 3 | full | 8/10 | Cclaimed | |
Self-host the core product G | ai-native user | Openness — open source, data portability, and self-hosting storiesOpenness | 3 | partialfree | 6/10 | Xcommunity | |
Define rules that trigger actions automatically on events G | ai-native user | Automation depth — how much of the product can run unattendedAutomation depth | 3 | partial | 5/10 | Cclaimed | |
Export all of my data in open formats and leave G | ai-native user | Openness — open source, data portability, and self-hosting storiesOpenness | 3 | partial | 5/10 | Xcommunity | |
Have an agent query my traces, metrics, and eval results through an API or MCP server to debug my app C Ai observability | ai-native user | Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation | 3 | partial | 5/10 | Tprobed | |
See cost and token usage per request, model, and time period in dashboards C Cost tracking | developer | Cost monitoring — stories about cost monitoring in this arenaCost monitoring | 3 | partialfree | 5/10 | Xcommunity | |
Curate datasets from production traces and run offline evaluations against them C Offline evals | ml engineer | Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets | 3 | partial | 4/10 | Cclaimed | |
Score outputs with configurable LLM-as-a-judge evaluators C Offline evals | ml engineer | Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets | 3 | partial | 4/10 | Cclaimed | |
Compare eval runs side by side to catch regressions between prompt or model versions C Offline evals | ml engineer | Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets | 3 | partial | 3/10 | Cclaimed | |
Prevent my data from being used to train AI models G | ai-native user | Privacy posture — data-handling and privacy storiesPrivacy posture | 3 | n/a | untested | none yet | |
Send and receive traces over OpenTelemetry (OTLP) instead of a proprietary format C Trace capture | developer | Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation | 3 | none | untested | none yet | |
Instrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK C Trace capture | developer | Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation | 2 | full | 8/10 | Xcommunity | |
Trace multi-step agent runs as nested spans grouped into sessions or threads C Trace capture | developer | Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation | 2 | full | 8/10 | Cclaimed | |
Build custom dashboards over latency, error, cost, and eval-score metrics C Monitoring | ml engineer | Alerting dashboards — stories about alerting dashboards in this arenaAlerting dashboards | 2 | partial | 6/10 | Xcommunity | |
Do everything through the API that I can do in the UI G | ai-native user | Openness — open source, data portability, and self-hosting storiesOpenness | 2 | partial | 6/10 | Tprobed | |
Instrument apps in both Python and JS/TS with officially supported SDKs G Sdk coverage | developer | Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation | 2 | partial | 6/10 | Xcommunity | |
Iterate on prompts in a playground against real models and variables C Prompt workflow | developer | Prompt management — stories about prompt management in this arenaPrompt management | 2 | partial | 6/10 | Cclaimed | |
Read the product's source under an open license G | ai-native user | Openness — open source, data portability, and self-hosting storiesOpenness | 2 | disputed | 5/10 | Dcontradicted | |
Run evaluators continuously on live production traffic, not just offline datasets C Online evals | ml engineer | Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets | 2 | partial | 5/10 | Cclaimed | |
Attribute cost and usage to users, sessions, and features via custom metadata C Cost tracking | developer | Cost monitoring — stories about cost monitoring in this arenaCost monitoring | 2 | partial | 4/10 | Cclaimed | |
Bulk-export traces and datasets to blob storage or my data warehouse C Data export | developer | Data access export — stories about data access export in this arenaData access export | 2 | partial | 4/10 | Xcommunity | |
Choose where my data is stored (region/residency) G | ai-native user | Privacy posture — data-handling and privacy storiesPrivacy posture | 2 | partial | 4/10 | Cclaimed | |
Perform bulk operations across many items at once G | ai-native user | Automation depth — how much of the product can run unattendedAutomation depth | 2 | partial | 4/10 | Cclaimed | |
Set alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email C Monitoring | developer | Alerting dashboards — stories about alerting dashboards in this arenaAlerting dashboards | 2 | partial | 4/10 | Cclaimed | |
Opt out of telemetry and usage tracking G | ai-native user | Privacy posture — data-handling and privacy storiesPrivacy posture | 2 | partial | 3/10 | Cclaimed | |
Write custom code-based scorers and metrics for my evaluations C Offline evals | ml engineer | Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets | 2 | partial | 3/10 | Cclaimed | |
Control data retention and deletion G | ai-native user | Privacy posture — data-handling and privacy storiesPrivacy posture | 2 | none | untested | none yet | |
Have an agent create a dataset, trigger an eval run programmatically, and read back the results C Ai eval ops | ai-native user | Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets | 2 | none | untested | none yet | |
Mask or redact sensitive data before it is stored in traces C Data controls | developer | Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation | 2 | none | untested | none yet | |
Route outputs to human annotation queues for review and labeling C Human review | ml engineer | Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets | 2 | none | untested | none yet | |
Run evals in CI and gate deployments on their results C Offline evals | developer | Evals datasets — measuring quality — datasets, eval runs, regression trackingEvals datasets | 2 | none | untested | none yet | |
Schedule recurring jobs or workflows G | ai-native user | Automation depth — how much of the product can run unattendedAutomation depth | 2 | n/a | untested | none yet | |
Version, review, and roll back my automations G | ai-native user | Automation depth — how much of the product can run unattendedAutomation depth | 1 | partial | 6/10 | Cclaimed | |
Capture multimodal payloads (images, audio, files) inside my traces C Trace capture | developer | Tracing instrumentation — instrumenting code and tracing requests end to endTracing instrumentation | 1 | none | untested | none yet |
Opportunities — the stories that would move this product's scores, from its own judged verdictsOpportunitiestop 8 of 36 stories with headroom
What would move Helicone’s scores — derived from its own judged verdicts, biggest headroom first. Each line quotes what the judge found missing; shipping it (or evidencing it publicly) is the fix.
Agenticness — how well agents can access and operate the productConnect an agent via an official MCP server
nonemoves agent-readyimpact 45
Helicone is an LLM observability/gateway platform, and this axis (offering an official MCP server so agents can connect) plausibly applies since it has an ecosystem of integrations, docs, and APIs.
Tracing instrumentation — instrumenting code and tracing requests end to endSend and receive traces over OpenTelemetry (OTLP) instead of a proprietary format
nonemoves PA Scoreimpact 30
No evidence pack items mention OpenTelemetry, OTLP, or any standard tracing protocol support; Helicone's docs describe proprietary logging via SDK integration, sessions, and REST API, not OTLP ingestion/export.
Agenticness — how well agents can access and operate the productUse an official CLI
nonemoves agent-readyimpact 30
No evidence of an official Helicone CLI tool; integration is via SDKs, API keys, gateway, and REST/OpenAPI, but no CLI is mentioned anywhere in docs, GitHub, or community sources.
Agenticness — how well agents can access and operate the productIssue scoped/least-privilege API credentials for an agent
nonemoves agent-readyimpact 30
No evidence Helicone supports issuing scoped or least-privilege API credentials/keys for agents; docs mention bringing your own provider keys or using Helicone's own key, but nothing about granular permission scoping.
Agenticness — how well agents can access and operate the productRely on versioned APIs with a documented deprecation policy
nonemoves API qualityimpact 30
An OpenAPI spec exists (helicone-probe-3) confirming a REST API, but there is no evidence of API versioning scheme (e.g., v1/v2 paths with migration guides) or a documented deprecation policy for endpoints/models; nothing in the docs pack addresses lifecycle or backward-compatibility commitments.
Evals datasets — measuring quality — datasets, eval runs, regression trackingCompare eval runs side by side to catch regressions between prompt or model versions
partialq3/10moves PA Scoreimpact 21
Missing: a documented eval-run comparison UI, dataset-based batch evaluation runs, and any hands-on/community confirmation of side-by-side regression catching.
Evals datasets — measuring quality — datasets, eval runs, regression trackingHave an agent create a dataset, trigger an eval run programmatically, and read back the results
nonemoves PA Scoreimpact 20
Evidence shows observability features (sessions, webhooks, HQL query, REST API for point queries) and a vague mention of 'real-time evaluation' scoring, but there is no documentation of a dataset-creation API, a way to programmatically trigger an eval run, or an API to read back eval results — the core building blocks of this story are absent from the evidence pack.
Evals datasets — measuring quality — datasets, eval runs, regression trackingRun evals in CI and gate deployments on their results
nonemoves PA Scoreimpact 20
Helicone offers observability, webhooks, real-time scoring, caching, and prompt versioning, but there is no evidence of a CI-integrated eval runner, test suite, or deployment gating mechanism tied to eval results.
Showing the top 8 of 36 — every none/partial verdict in the story verdicts table is headroom.
Think a verdict is wrong? Every verdicts-table row has a Flag link — see the methodology.
Coverage map — which docs area, API section, or community source covers which judged storiesCoverage map8 surfaces · 35 covered stories
Where the cited evidence behind each covered verdict came from — the same citations the verdicts table shows, no extra judging.
Features docs18 stories
- Run the product headlessly / in CI for automation
- Subscribe to events via webhooks
- Get AI-generated insights and suggestions from my data inside the product
- Build custom dashboards over latency, error, cost, and eval-score metrics
- Set alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email
- Define rules that trigger actions automatically on events
- Version, review, and roll back my automations
- Attribute cost and usage to users, sessions, and features via custom metadata
- Write custom code-based scorers and metrics for my evaluations
- Compare eval runs side by side to catch regressions between prompt or model versions
- Score outputs with configurable LLM-as-a-judge evaluators
- Curate datasets from production traces and run offline evaluations against them
- Run evaluators continuously on live production traffic, not just offline datasets
- Do everything through the API that I can do in the UI
- Iterate on prompts in a playground against real models and variables
- Version prompts and deploy changes to production without shipping code
- Have an agent query my traces, metrics, and eval results through an API or MCP server to debug my app
- Trace multi-step agent runs as nested spans grouped into sessions or threads
GitHub README17 stories
- Run the product headlessly / in CI for automation
- Drive the product through a documented public API
- Build against official SDKs
- Explore an interactive API reference with runnable examples
- Test against a sandbox environment without touching production data
- Build custom dashboards over latency, error, cost, and eval-score metrics
- Perform bulk operations across many items at once
- Bulk-export traces and datasets to blob storage or my data warehouse
- Compare eval runs side by side to catch regressions between prompt or model versions
- Curate datasets from production traces and run offline evaluations against them
- Do everything through the API that I can do in the UI
- Export all of my data in open formats and leave
- Iterate on prompts in a playground against real models and variables
- Instrument apps in both Python and JS/TS with officially supported SDKs
- Trace multi-step agent runs as nested spans grouped into sessions or threads
- Instrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK
- Capture traces of my LLM calls with inputs, outputs, latency, and token usage
Getting started docs12 stories
- Point an agent at llms.txt or agent-oriented docs
- Run the product headlessly / in CI for automation
- Build against official SDKs
- Test against a sandbox environment without touching production data
- Export all of my data in open formats and leave
- Read the product's source under an open license
- Self-host the core product
- Choose where my data is stored (region/residency)
- Opt out of telemetry and usage tracking
- Instrument apps in both Python and JS/TS with officially supported SDKs
- Instrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK
- Capture traces of my LLM calls with inputs, outputs, latency, and token usage
Rest docs12 stories
- Run the product headlessly / in CI for automation
- Drive the product through a documented public API
- Explore an interactive API reference with runnable examples
- Download a machine-readable API spec (OpenAPI or equivalent)
- Build custom dashboards over latency, error, cost, and eval-score metrics
- Perform bulk operations across many items at once
- See cost and token usage per request, model, and time period in dashboards
- Bulk-export traces and datasets to blob storage or my data warehouse
- Curate datasets from production traces and run offline evaluations against them
- Do everything through the API that I can do in the UI
- Export all of my data in open formats and leave
- Have an agent query my traces, metrics, and eval results through an API or MCP server to debug my app
Hacker News10 stories
- Build against official SDKs
- Build custom dashboards over latency, error, cost, and eval-score metrics
- See cost and token usage per request, model, and time period in dashboards
- Bulk-export traces and datasets to blob storage or my data warehouse
- Export all of my data in open formats and leave
- Read the product's source under an open license
- Self-host the core product
- Instrument apps in both Python and JS/TS with officially supported SDKs
- Instrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK
- Capture traces of my LLM calls with inputs, outputs, latency, and token usage
Pricing docs10 stories
- Drive the product through a documented public API
- Build custom dashboards over latency, error, cost, and eval-score metrics
- Perform bulk operations across many items at once
- See cost and token usage per request, model, and time period in dashboards
- Bulk-export traces and datasets to blob storage or my data warehouse
- Curate datasets from production traces and run offline evaluations against them
- Do everything through the API that I can do in the UI
- Read the product's source under an open license
- Self-host the core product
- Have an agent query my traces, metrics, and eval results through an API or MCP server to debug my app
Swagger docs7 stories
- Point an agent at llms.txt or agent-oriented docs
- Run the product headlessly / in CI for automation
- Drive the product through a documented public API
- Explore an interactive API reference with runnable examples
- Download a machine-readable API spec (OpenAPI or equivalent)
- Do everything through the API that I can do in the UI
- Have an agent query my traces, metrics, and eval results through an API or MCP server to debug my app
Claims vs evidence — vendor claims reconciled against independent verdictsClaims vs evidence
4 of 14 testable claims verified · 2 contradicted → integrity 0/100
26 distinct capability claims found in Helicone’s own claimed-docs/GitHub materials, reconciled against our judge’s independent verdicts.
4
Verified
8
Unverified
2
Contradicted
22
Undersold
Verified (6)
“Log first LLM request via AI Gateway in under 2 minutes”
Capture traces of my LLM calls with inputs, outputs, latency, and token usagefullproof ↗
“Access 100+ LLM models across providers using familiar OpenAI SDK with auto logging, observability, and fallbacks”
Instrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDKfullproof ↗
“Offers multiple deployment methods to fit different infrastructure and scale needs”
“One-line code integration logs requests from OpenAI, Anthropic, LangChain, Gemini, Vercel AI SDK, and more”
Instrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDKfullproof ↗
“Export data to PostHog in one line for custom dashboards”
Build custom dashboards over latency, error, cost, and eval-score metricspartialproof ↗
“Docker Compose deployment option for quick local setups without complex infra”
Unverified (10)
“Group related requests into sessions to trace full agent flow in one unified view”
Trace multi-step agent runs as nested spans grouped into sessions or threadsfullproof ↗
“Test and deploy prompt changes instantly without rebuilding or redeploying the app”
Version prompts and deploy changes to production without shipping codefullproof ↗
“Reference a prompt ID in the AI Gateway to use it instantly with no code changes”
Version prompts and deploy changes to production without shipping codefullproof ↗
“Webhooks send instant notifications when LLM requests complete to automate workflows and scoring”
“Webhooks can be filtered to trigger only when all specified properties match”
Define rules that trigger actions automatically on eventspartialproof ↗
“Alerts monitor error rates and costs on LLM requests to catch issues early”
Set alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or emailpartialproof ↗
“Playground lets you rapidly test and iterate on prompts, sessions, and traces in the UI”
Iterate on prompts in a playground against real models and variablespartialproof ↗
“Track every prompt change, compare versions, and roll back instantly if needed”
“Track every prompt change, compare versions, and roll back instantly if needed”
Version prompts and deploy changes to production without shipping codefullproof ↗
“Real-time evaluation automatically scores LLM responses for quality, safety, and relevance”
Run evaluators continuously on live production traffic, not just offline datasetspartialproof ↗
Contradicted (2)
“Bring your own provider API keys for more control”
Issue scoped/least-privilege API credentials for an agentnoneproof ↗
“Open-source platform offering more provider flexibility and cost-effective scaling”
Read the product's source under an open licensedisputedproof ↗
Undersold (22)
Point an agent at llms.txt or agent-oriented docsfullproof ↗
Run the product headlessly / in CI for automationpartialproof ↗
Drive the product through a documented public APIfullproof ↗
Get AI-generated insights and suggestions from my data inside the productpartialproof ↗
Explore an interactive API reference with runnable examplespartialproof ↗
Download a machine-readable API spec (OpenAPI or equivalent)fullproof ↗
Test against a sandbox environment without touching production datapartialproof ↗
Perform bulk operations across many items at oncepartialproof ↗
Attribute cost and usage to users, sessions, and features via custom metadatapartialproof ↗
See cost and token usage per request, model, and time period in dashboardspartialproof ↗
Bulk-export traces and datasets to blob storage or my data warehousepartialproof ↗
Write custom code-based scorers and metrics for my evaluationspartialproof ↗
Compare eval runs side by side to catch regressions between prompt or model versionspartialproof ↗
Score outputs with configurable LLM-as-a-judge evaluatorspartialproof ↗
Curate datasets from production traces and run offline evaluations against thempartialproof ↗
Do everything through the API that I can do in the UIpartialproof ↗
Export all of my data in open formats and leavepartialproof ↗
Choose where my data is stored (region/residency)partialproof ↗
Have an agent query my traces, metrics, and eval results through an API or MCP server to debug my apppartialproof ↗
Instrument apps in both Python and JS/TS with officially supported SDKspartialproof ↗
Claims outside our story set (9)
Real capability claims found in Helicone’s own materials, but no story in this arena’s taxonomy covers them yet — that’s feedback on the taxonomy, not a mark against the product.
“Switch between 100+ models just by changing the model name”
source ↗“Requests sharing the same path are grouped as the same type of work over time”
source ↗“Caching stores complete responses on Cloudflare's edge network to cut redundant calls, latency, and cost”
source ↗“Create separate cache namespaces for different users or contexts”
source ↗“Supports point queries to fetch individual requests”
source ↗“Supports a query language (HQL) for querying data”
source ↗“Free tier includes 10,000 requests”
source ↗“AI Gateway provides access to 100+ models via one API key with intelligent routing and automatic fallbacks”
source ↗“Supports fine-tuning through partners OpenPipe or Autonomi”
source ↗
Business model
Apache-licensed and self-hostable free; Helicone Cloud has a free tier (10k requests/month), then per-seat Pro with usage-based request overages and custom enterprise plans.
pricing ↗Score trend
How this product’s scores have moved as evidence and verdicts are re-derived — a point per change, not per day.
Flag
⚑ Flag a verdictThink a verdict is wrong? Opens a prefilled GitHub issue — or use the ⚑ next to any verdict above.
For agents
Agent surface uptime llms.txt 100% · openapi.json 100% (30d, checked every 6h since Sep 8 '26)