Galileo

Connect an agent via an official MCP server G

Agent access

Drive the product through a documented public API G

Agent access

Plug MCP servers into this product so it can use their tools G

Agent access

Delegate tasks to a built-in AI assistant inside the product G

Agentic features

Point an agent at llms.txt or agent-oriented docs G

Agent access

Build against official SDKs G

Agent access

Run the product headlessly / in CI for automation G

Agent access

Get AI-generated insights and suggestions from my data inside the product G

Agentic features

Operate the product with natural-language commands G

Agentic features

Set up automations that run autonomously in the background G

Agentic features

Download a machine-readable API spec (OpenAPI or equivalent) G

Api quality

Explore an interactive API reference with runnable examples G

Api quality

Rely on versioned APIs with a documented deprecation policy G

Api quality

Subscribe to events via webhooks G

Agent access

Issue scoped/least-privilege API credentials for an agent G

Agent access

Use an official CLI G

Agent access

Test against a sandbox environment without touching production data G

Api quality

Curate datasets from production traces and run offline evaluations against them C

Offline evals

Score outputs with configurable LLM-as-a-judge evaluators C

Offline evals

Capture traces of my LLM calls with inputs, outputs, latency, and token usage C

Trace capture

Compare eval runs side by side to catch regressions between prompt or model versions C

Offline evals

Have an agent query my traces, metrics, and eval results through an API or MCP server to debug my app C

Ai observability

Send and receive traces over OpenTelemetry (OTLP) instead of a proprietary format C

Trace capture

Define rules that trigger actions automatically on events G

Version prompts and deploy changes to production without shipping code C

Prompt workflow

Export all of my data in open formats and leave G

Prevent my data from being used to train AI models G

See cost and token usage per request, model, and time period in dashboards C

Cost tracking

Self-host the core product G

Run evaluators continuously on live production traffic, not just offline datasets C

Online evals

Trace multi-step agent runs as nested spans grouped into sessions or threads C

Trace capture

Have an agent create a dataset, trigger an eval run programmatically, and read back the results C

Ai eval ops

Write custom code-based scorers and metrics for my evaluations C

Offline evals

Do everything through the API that I can do in the UI G

Iterate on prompts in a playground against real models and variables C

Prompt workflow

Perform bulk operations across many items at once G

Set alerts on error rates, cost spikes, or eval-score drops and get notified in Slack, PagerDuty, or email C

Monitoring

Attribute cost and usage to users, sessions, and features via custom metadata C

Cost tracking

Instrument apps in both Python and JS/TS with officially supported SDKs G

Sdk coverage

Route outputs to human annotation queues for review and labeling C

Human review

Read the product's source under an open license G

Bulk-export traces and datasets to blob storage or my data warehouse C

Data export

Instrument my app through existing integrations for frameworks like LangChain, the OpenAI SDK, or the Vercel AI SDK C

Trace capture

Run evals in CI and gate deployments on their results C

Offline evals

Build custom dashboards over latency, error, cost, and eval-score metrics C

Monitoring

Choose where my data is stored (region/residency) G

Control data retention and deletion G

Mask or redact sensitive data before it is stored in traces C

Data controls

Opt out of telemetry and usage tracking G

Schedule recurring jobs or workflows G

Capture multimodal payloads (images, audio, files) inside my traces C

Trace capture

Version, review, and roll back my automations G