Braintrust

Connect an agent via an official MCP server G

Agent access

Drive the product through a documented public API G

Agent access

Delegate tasks to a built-in AI assistant inside the product G

Agentic features

Plug MCP servers into this product so it can use their tools G

Agent access

Run the product headlessly / in CI for automation G

Agent access

Use an official CLI G

Agent access

Build against official SDKs G

Agent access

Get AI-generated insights and suggestions from my data inside the product G

Agentic features

Operate the product with natural-language commands G

Agentic features

Point an agent at llms.txt or agent-oriented docs G

Agent access

Set up automations that run autonomously in the background G

Agentic features

Download a machine-readable API spec (OpenAPI or equivalent) G

Api quality

Explore an interactive API reference with runnable examples G

Api quality

Issue scoped/least-privilege API credentials for an agent G

Agent access

Rely on versioned APIs with a documented deprecation policy G

Api quality

Subscribe to events via webhooks G

Agent access

Test against a sandbox environment without touching production data G

Api quality

Compare eval runs side by side to catch regressions between prompt or model versions C

Offline evals

Curate datasets from production traces and run offline evaluations against them C

Offline evals

Have an agent query my traces, metrics, and eval results through an API or MCP server to debug my app C

Ai observability

Capture traces of my LLM calls with inputs, outputs, latency, and token usage C

Trace capture

Score outputs with configurable LLM-as-a-judge evaluators C

Offline evals

Self-host the core product G

Version prompts and deploy changes to production without shipping code C

Prompt workflow

Define rules that trigger actions automatically on events G

Export all of my data in open formats and leave G

See cost and token usage per request, model, and time period in dashboards C

Cost tracking

Prevent my data from being used to train AI models G

Send and receive traces over OpenTelemetry (OTLP) instead of a proprietary format C

Trace capture

Have an agent create a dataset, trigger an eval run programmatically, and read back the results C

Ai eval ops

Iterate on prompts in a playground against real models and variables C

Prompt workflow

Run evals in CI and gate deployments on their results C

Offline evals

Run evaluators continuously on live production traffic, not just offline datasets C

Online evals

Write custom code-based scorers and metrics for my evaluations C

Offline evals

Do everything through the API that I can do in the UI G

Perform bulk operations across many items at once G

Choose where my data is stored (region/residency) G

Instrument apps in both Python and JS/TS with officially supported SDKs G

Sdk coverage

Trace multi-step agent runs as nested spans grouped into sessions or threads C

Trace capture

Build custom dashboards over latency, error, cost, and eval-score metrics C

Monitoring

Bulk-export traces and datasets to blob storage or my data warehouse C

Data export

Route outputs to human annotation queues for review and labeling C

Human review

Schedule recurring jobs or workflows G

Attribute cost and usage to users, sessions, and features via custom metadata C

Cost tracking

Control data retention and deletion G

Instrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK C

Trace capture

Read the product's source under an open license G

Mask or redact sensitive data before it is stored in traces C

Data controls

Opt out of telemetry and usage tracking G

Set alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email C

Monitoring

Version, review, and roll back my automations G

Capture multimodal payloads (images, audio, files) inside my traces C

Trace capture