LiteLLM vs Portkey

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round drawn
   LiteLLM✓fullprobed8/10

   A direct probe confirms LiteLLM serves a valid llms.txt file at https://docs.litellm.ai/llms.txt returning HTTP 200 with an agent-readable overview and links, exactly matching the story's request. Missing for 10: no independent/community corroboration of agents actually consuming this file, and the related docs-as-markdown endpoint (.md) returns 404, suggesting the agent-oriented docs surface is only partially built out.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.litellm.ai/llms.txt # https://docs.litellm.ai/ llms.txt - [LiteLLM Overview](https://docs.litellm.…”
   • [probe] “PROBE docs-md: HTTP 404 at https://docs.litellm.ai/docs/.md”

   proof ↗⚑ flag

   Portkey✓fullprobed8/10

   Portkey hosts a working llms.txt file (HTTP 200) with a description of the product, confirming agents can be pointed at it for agent-oriented docs. Missing for 10: no docs.md fallback (404) or OpenAPI spec confirmation, and no independent/community corroboration of agents actually using it.

   • [probe] “PROBE llms.txt: HTTP 200 at https://portkey.ai/llms.txt # Portkey > Portkey is a production stack for Gen AI builders, equipping AI teams w…”
   • [probe] “PROBE docs-md: HTTP 404 at https://portkey.ai/docs.md”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to LiteLLM
   LiteLLM~partialprobed6/10

   LiteLLM is a Python SDK and proxy server with a documented CLI (worker processes, uvicorn/gunicorn/Granian) that can run headlessly in scripts or containers, and community evidence shows it running via Docker in unattended homelab/gateway setups. However there is no explicit CI/CD pipeline example, GitHub Actions integration, or automation-specific documentation in the evidence pack. Missing for 10: explicit CI pipeline examples, automation/scripting docs beyond CLI flags, evidence of non-interactive auth/config for CI environments.

   • [claimed-docs] “The number of worker processes to spin up (uvicorn, gunicorn, or Granian --workers).”
   • [claimed-docs] “The number of worker processes to spin up (uvicorn, gunicorn, or Granian `--workers`).”
   • [probe] “official CLI documented at https://docs.litellm.ai/docs/proxy/cli”
   • [community] “A user described running LiteLLM as a proxy in their homelab via the litellm/litellm docker image for local LLM gateway management, noting t…”
   • [claimed-docs] “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI”

   proof ↗⚑ flag

   Portkey~partialclaimed5/10

   Portkey's gateway can be run headlessly via a CLI command (npx @portkey-ai/gateway) and REST API with a base URL, and it is open source and self-hostable, which supports CI/automation usage. However, there is no explicit documentation of CI pipeline integration, GitHub Actions examples, or headless automation testing workflows. missing for 10: explicit CI/CD examples, headless automation guides, non-interactive scripted usage documentation, independent verification of CI usage.

   • [claimed-docs] “npx @portkey-ai/gateway”
   • [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
   • [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to LiteLLM
   LiteLLM✓fullclaimed8/10

   LiteLLM Proxy has a documented MCP Gateway feature letting users add MCP servers via the UI and use a fixed endpoint to expose their tools, controllable by key/team — directly matching the story. missing for 10: independent/hands-on corroboration of MCP tool usage beyond first-party docs, and detail on tool-calling behavior once connected.

   • [claimed-docs] “LiteLLM Proxy provides an MCP Gateway that allows you to use a fixed endpoint for all MCP tools and control MCP access by Key, Team.”
   • [claimed-docs] “On the LiteLLM UI, Navigate to "MCP Servers" and click "Add New MCP Server".”

   proof ↗⚑ flag

   Portkey~partialclaimed6/10

   Portkey's docs explicitly state it can 'Connect to Remote MCP servers, allowing you to connect external tools and data sources' via the AI Gateway, directly supporting the story. However, evidence lacks detail on setup steps, supported MCP transport, or independent/hands-on confirmation of this working in practice. Missing for 10: detailed configuration docs for adding MCP servers, independent verification of MCP tool invocation, and examples of tool-calling via MCP in a live agent workflow.

   • [claimed-docs] “Connect to Remote MCP severs, allowing you to connect external tools and data sources.”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round to LiteLLM
   LiteLLM✓fullclaimed7/10

   LiteLLM Proxy ships an official MCP Gateway feature exposing a fixed endpoint for all configured MCP tools, with access controlled by Key/Team, and a UI flow to add MCP servers — this is exactly the 'connect an agent via an official MCP server' capability for a gateway product. Missing for 10: independent/hands-on confirmation that external agents (e.g. Claude, Cursor) successfully connect via this gateway, and deeper documentation of the MCP protocol compliance/auth flow beyond the brief doc mentions.

   • [claimed-docs] “LiteLLM Proxy provides an MCP Gateway that allows you to use a fixed endpoint for all MCP tools and control MCP access by Key, Team.”
   • [claimed-docs] “On the LiteLLM UI, Navigate to "MCP Servers" and click "Add New MCP Server".”

   proof ↗⚑ flag

   Portkey—none0/10

   Portkey's docs describe connecting *to* remote MCP servers as a client-side feature (portkey-docs-17), but there is no evidence that Portkey itself exposes an official MCP server endpoint that external agents can connect to. Missing for 10: any documentation of Portkey hosting/serving an MCP endpoint, MCP server setup instructions, or third-party confirmation of agents connecting via Portkey's MCP server.

   • [claimed-docs] “Connect to Remote MCP severs, allowing you to connect external tools and data sources.”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to LiteLLM
   LiteLLM~partialprobed6/10

   LiteLLM ships an official CLI for its proxy (litellm docs-proxy-cli, litellm-probe-4) covering server startup and worker config, but the docs shown focus narrowly on process/worker management rather than a broad AI-native workflow CLI (e.g., chat, model testing, key/budget management from CLI). missing for 10: evidence of CLI commands beyond process startup (e.g., testing prompts, managing virtual keys/budgets, MCP servers via CLI), independent/hands-on confirmation of CLI usability.

   • [claimed-docs] “The number of worker processes to spin up (uvicorn, gunicorn, or Granian --workers).”
   • [claimed-docs] “The number of worker processes to spin up (uvicorn, gunicorn, or Granian `--workers`).”
   • [probe] “official CLI documented at https://docs.litellm.ai/docs/proxy/cli”

   proof ↗⚑ flag

   Portkey~partialclaimed3/10

   The only CLI-like evidence is running the open-source gateway locally via `npx @portkey-ai/gateway`, which self-hosts the gateway rather than offering a full-featured CLI for managing configs, logs, or other Portkey resources. Missing for 10: a documented dedicated CLI tool/command set, CLI-based workflow examples, and independent corroboration of CLI usage.

   • [claimed-docs] “npx @portkey-ai/gateway”
   • [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to Portkey
   LiteLLM✓fullprobed7/10

   LiteLLM's core interface — the completion() function and the OpenAI-compatible proxy REST API — is extensively documented as a public, drop-in API that AI-native users can call directly, corroborated by community Show HN discussion confirming streaming/function-calling parity with the OpenAI SDK. Missing for 10: a discoverable machine-readable OpenAPI/swagger spec (probe found only 404s) and independent hands-on confirmation of API stability/versioning.

   • [claimed-docs] “Call any provider using the same completion() interface, with no API to re-learn for each one”
   • [claimed-docs] “Every response follows the OpenAI Chat Completions format, regardless of provider.”
   • [github] “Drop-in OpenAI compatibility — swap providers without rewriting your code”
   • [claimed-docs] “Call any provider using the same `completion()` interface, with no API to re-learn for each one”
   • [claimed-docs] “Consistent output format regardless of which provider or model you use”
   • [community] “LiteLLM maintainer confirmed proxy supports streaming and function-calling in the same way as the openai-python SDK, and added Ollama integr…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.litellm.ai/openapi.json, https://docs.litellm.ai/swagger.json, https://docs.litellm.ai/…”

   proof ↗⚑ flag

   Portkey✓fullprobed8/10

   Portkey exposes a documented REST API (base URL https://api.portkey.ai/v1) with API-reference docs for its inference API, OpenAI-compatible integration, and llms.txt discovery confirms API-first positioning; gateway is also open-sourced and npx-runnable. Missing for 10: a discoverable OpenAPI/swagger spec (probe found all candidate paths 404) and independent/third-party corroboration of API usage.

   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
   • [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
   • [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://portkey.ai/llms.txt # Portkey > Portkey is a production stack for Gen AI builders, equipping AI teams w…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
   • [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to LiteLLM
   LiteLLM✓fullcommunity8/10

   LiteLLM's proxy virtual keys let admins scope credentials by model access, budget, team, and rate limits (rpm_limit), directly enabling least-privilege API keys for agents. This is well documented across multiple first-party docs (virtual_keys, users, cost_tracking) showing budgets, per-key model restrictions, and team-based limits. Missing for 10: independent/hands-on verification of the scoping granularity in practice, and one community comment (litellm-comm-3) noted vague 'issues with creating tokens for other users' suggesting rough edges.

   • [claimed-docs] “Track Spend, and control model access via virtual keys for the proxy”
   • [claimed-docs] “Personal budgets: Create virtual keys without team_id for individual spending limits”
   • [claimed-docs] “Apply a budget across all calls on the proxy... budget_duration: 30d # (str) frequency of reset”
   • [claimed-docs] “"team_alias": "my-new-team_4", ... "rpm_limit": 99”
   • [claimed-docs] “Track spend for keys, users, and teams across 100+ LLMs.”
   • [community] “'Does anyone know a good alternate project... LiteLLM has been getting worse and trying to get me to upgrade to a paid version. I also had i…”

   proof ↗⚑ flag

   Portkey~partialclaimed7/10

   Portkey lets admins issue API keys scoped with budgets, rate limits, and model allow-lists at org/workspace level, and virtual keys keep provider credentials out of code — this maps well to least-privilege credential issuance. However, the docs frame this around human/org governance rather than explicitly per-agent scoped credentials, and there's no independent corroboration of enforcement in practice. Missing for 10: explicit agent-specific credential scoping workflow, independent/hands-on validation of enforcement.

   • [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
   • [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
   • [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
   • [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
   • [claimed-docs] “Provider credentials stored securely, never exposed in code”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to LiteLLM
   LiteLLM✓fullcommunity8/10

   LiteLLM ships an official Python SDK (the `completion()` interface) that mirrors the OpenAI SDK format across 100+ providers, letting AI-native developers build without learning per-provider APIs; this is corroborated by community Show HN feedback praising it for standardizing APIs and confirming SDK parity for streaming/function-calling. Missing for 10: evidence of official SDKs in other languages (e.g., JS/TS) and independent benchmarking of SDK completeness beyond community praise.

   • [claimed-docs] “Call any provider using the same completion() interface, with no API to re-learn for each one”
   • [claimed-docs] “Every response follows the OpenAI Chat Completions format, regardless of provider.”
   • [github] “Drop-in OpenAI compatibility — swap providers without rewriting your code”
   • [claimed-docs] “Call any provider using the same `completion()` interface, with no API to re-learn for each one”
   • [claimed-docs] “Consistent output format regardless of which provider or model you use”
   • [community] “LiteLLM maintainer confirmed proxy supports streaming and function-calling in the same way as the openai-python SDK, and added Ollama integr…”
   • [community] “Show HN launch of the original LiteLLM library; a user praised it: 'This is amazing. Really needed something like this to standardize all my…”

   proof ↗⚑ flag

   Portkey~partialprobed6/10

   Portkey documents SDK-style integration (base URL swap for OpenAI Python/Node SDKs, REST API reference, open-source gateway installable via npx) supporting AI-native builders, but the evidence pack lacks explicit mention of dedicated first-party SDKs beyond OpenAI-compatible drop-in usage, and OpenAPI spec probes returned 404s. missing for 10: dedicated official multi-language SDK docs (e.g., Python/JS Portkey SDK reference pages), working OpenAPI/spec discovery, independent developer corroboration of SDK usage.

   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
   • [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
   • [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
   • [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
   • [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   LiteLLM—none0/10

   No evidence in the pack mentions webhooks or event subscription mechanisms; LiteLLM's documented features cover logging integrations, cost tracking, and MCP gateway, but nothing about outbound webhook events for subscribers.

   ⚑ flag

   Portkey—none0/10

   No evidence in the pack mentions webhooks or event subscriptions of any kind; Portkey's docs cover gateway routing, caching, retries, budgets, and logs but nothing about outbound webhook notifications for events.

   ⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round to Portkey
   LiteLLM—none0/10

   The evidence pack shows explicit probe failures for an OpenAPI/Swagger spec (litellm-probe-3) and no documented interactive API reference or runnable examples in the docs; only static markdown-style docs and code snippets are cited (litellm-docs-1/2/16/17). Since LiteLLM's proxy is an API gateway, this axis clearly applies, but there's no evidence of an interactive/try-it-out API reference.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.litellm.ai/openapi.json, https://docs.litellm.ai/swagger.json, https://docs.litellm.ai/…”
   • [probe] “PROBE docs-md: HTTP 404 at https://docs.litellm.ai/docs/.md”
   • [claimed-docs] “Call any provider using the same completion() interface, with no API to re-learn for each one”
   • [claimed-docs] “Call any provider using the same `completion()` interface, with no API to re-learn for each one”

   proof ↗⚑ flag

   Portkey~partialprobed4/10

   Portkey documents a REST API reference (base URL, headers, SDK compatibility) at /docs/api-reference, but the evidence pack contains no mention of an interactive explorer or runnable code examples, and probes for a discoverable OpenAPI/Swagger spec all returned 404s. Missing for 10: evidence of an interactive 'try it' console, runnable code snippets, and a published OpenAPI/Swagger schema.

   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
   • [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
   • [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   LiteLLM—none0/10

   The evidence pack includes explicit probes for OpenAPI/swagger endpoints on LiteLLM's docs site, all returning 404, and no other citation shows a downloadable OpenAPI spec (e.g., from the proxy's FastAPI docs). Since LiteLLM's proxy is a gateway with an admin UI and CLI, this axis plausibly applies, but no evidence confirms a machine-readable spec is available.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.litellm.ai/openapi.json, https://docs.litellm.ai/swagger.json, https://docs.litellm.ai/…”
   • [probe] “PROBE docs-md: HTTP 404 at https://docs.litellm.ai/docs/.md”

   proof ↗⚑ flag

   Portkey—none0/10

   Portkey has an API reference (base URL, inference API docs) but no evidence of a downloadable OpenAPI/Swagger spec; a direct probe for common OpenAPI paths returned 404s, and docs.md also 404s.

   • [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
   • [probe] “PROBE docs-md: HTTP 404 at https://portkey.ai/docs.md”
   • [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round drawn
   LiteLLM—none0/10

   No evidence in the pack of a dedicated sandbox/test mode, mock-response capability, or staging environment for validating integrations without touching production data — the docs cover routing, budgets, virtual keys, logging, and reliability, but nothing about isolated test environments. missing for 10: sandbox/test mode, mock LLM responses, staging deployment guidance, docs mentioning safe non-production testing.

   ⚑ flag

   Portkey—none0/10

   Portkey's evidence covers gateway routing, fallbacks, caching, logging, and gradual/canary rollout of models in production, but there is no mention of a distinct sandbox/staging environment or test mode that isolates a user from production data. The axis is fair for a gateway/dev-tool product, but no evidence supports it.

   ⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   LiteLLM—none0/10

   No evidence of API versioning scheme or a documented deprecation policy anywhere in docs; OpenAPI/spec discovery probes returned 404s, suggesting no formal versioned API contract is published.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.litellm.ai/openapi.json, https://docs.litellm.ai/swagger.json, https://docs.litellm.ai/…”
   • [probe] “PROBE docs-md: HTTP 404 at https://docs.litellm.ai/docs/.md”

   proof ↗⚑ flag

   Portkey—none0/10

   No evidence of API versioning scheme (e.g., v1/v2 in URLs beyond a single 'v1' base path) or any documented deprecation policy/changelog; OpenAPI spec probe also returned 404s. Missing for 10: versioning scheme documentation, explicit deprecation policy, changelog/migration guides, and any independent corroboration.

   • [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
   • [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round drawn
   LiteLLM—none0/10

   The evidence pack describes LiteLLM's unified completion interface, routing, fallbacks, cost tracking, and virtual keys, but contains no mention of batch/bulk operations (e.g., batch completions across many prompts, bulk key/user management, or bulk import/export) that would let a user act on many items at once. Since LiteLLM manages many models, keys, and users, this axis is plausible for the product, but no supporting evidence exists.

   ⚑ flag

   Portkey—none0/10

   Portkey's evidence covers gateway routing, fallbacks, caching, logging, and budget/rate-limit controls, but nothing describes a bulk/batch operation capability (e.g., batch inference calls, bulk editing of configs/keys, or bulk log actions) that lets a user act on many items at once. Missing for 10: any documentation of a batch API, bulk config/key management, or bulk log/annotation actions.

   ⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Portkey
   LiteLLM~partialclaimed5/10

   LiteLLM ships several built-in event-triggered automations (automatic fallback to another model/provider on call failure or content-policy violation, budget resets on a schedule, rate limits per key/team) but these are fixed, pre-defined behaviors rather than a general user-definable rule/trigger-action engine (e.g. no webhook/if-this-then-that automation builder is evidenced). Missing for 10: a general-purpose custom rule/condition-action automation system, evidence of user-defined triggers beyond fallback/budget/rate-limit, and any webhook or external-action firing mechanism.

   • [claimed-docs] “If a call fails after num_retries, LiteLLM falls back to another model group, so a failing model or provider automatically fails over to a h…”
   • [claimed-docs] “content_policy_fallbacks: For litellm.ContentPolicyViolationError - LiteLLM maps content policy violation errors across providers”
   • [claimed-docs] “so a failing model or provider automatically fails over to a healthy backup”
   • [claimed-docs] “The request to `model="zephyr-beta"` will fail... litellm proxy will loop through all the model_groups specified in `fallbacks=["gpt-3.5-tur…”
   • [claimed-docs] “Apply a budget across all calls on the proxy... budget_duration: 30d # (str) frequency of reset”
   • [claimed-docs] “"team_alias": "my-new-team_4", ... "rpm_limit": 99”

   proof ↗⚑ flag

   Portkey~partialclaimed6/10

   Portkey ships several built-in rule/event mechanisms — fallback triggers on non-2xx status codes, automatic retries with backoff, circuit-breaker/failure handling per strategy, and budget/rate-limit thresholds that automatically send notifications or block further usage — which are automatic actions triggered by defined conditions. However, these are fixed, product-defined automations (reliability/spend rules) rather than a general user-authorable rule engine where an AI-native user can define arbitrary custom triggers/actions. Missing for 10: general-purpose custom rule/trigger builder, arbitrary event types beyond status codes/spend thresholds, user-defined action targets (e.g. webhooks, custom scripts) beyond built-in notifications/fallback/retry.

   • [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
   • [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
   • [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
   • [claimed-docs] “Configure per-strategy circuit protection and failure handling”
   • [claimed-docs] “When usage reaches this threshold, notifications will be sent to configured recipients”
   • [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
   • [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”

   proof ↗⚑ flag

Caching

1. developerCache responses at the gateway to cut cost and latency on repeated requests

   weight 2 · round to Portkey
   LiteLLM✓fullclaimed8/10

   LiteLLM proxy has a documented caching system that explicitly stores and reuses LLM responses to save costs and reduce latency, directly matching the story. Missing for 10: independent/hands-on benchmarks quantifying latency/cost savings, and detail on cache backend configuration options in the evidence pack.

   • [claimed-docs] “LiteLLM's caching system stores and reuses LLM responses to save costs and reduce latency.”

   proof ↗⚑ flag

   Portkey✓fullclaimed9/10

   Portkey's docs explicitly describe gateway-level caching (simple and semantic) that serves responses up to 20x faster and cheaper, including semantic matching via cosine similarity for near-duplicate queries. This directly matches the story's cost/latency reduction on repeated requests. missing for 10: independent/hands-on benchmark corroboration beyond vendor docs.

   • [claimed-docs] “Cache LLM responses to serve requests up to **20x faster** and cheaper.”
   • [claimed-docs] “Matches requests with **similar meaning** using cosine similarity, not just identical text.”

   proof ↗⚑ flag

Latency

1. platform engineerRun traffic through gateway infrastructure that adds minimal latency overhead to provider calls

   weight 2 · round drawn
   LiteLLM~partialcommunity4/10

   LiteLLM docs describe features that reduce or mitigate latency (response caching, configurable worker processes for uvicorn/gunicorn/Granian) and fast failover, but there is no explicit benchmark or vendor claim quantifying the proxy's own added latency overhead versus direct provider calls. Community evidence includes a report of the proxy being 'incredible flaky and causing outages,' which is a reliability complaint rather than a direct latency measurement, so it doesn't rise to a concrete dispute of a latency claim. missing for 10: explicit latency-overhead benchmarks/numbers, independent performance testing of proxy overhead, direct vendor claim about added latency being minimal.

   • [claimed-docs] “LiteLLM's caching system stores and reuses LLM responses to save costs and reduce latency.”
   • [claimed-docs] “The number of worker processes to spin up (uvicorn, gunicorn, or Granian --workers).”
   • [claimed-docs] “The number of worker processes to spin up (uvicorn, gunicorn, or Granian `--workers`).”
   • [claimed-docs] “If a call fails after num_retries, LiteLLM falls back to another model group, so a failing model or provider automatically fails over to a h…”
   • [community] “'Now I feel lucky that I switched to just using OpenRouter a year ago because LiteLLM was incredible flaky and kept causing outages.'”

   proof ↗⚑ flag

   Portkey~partialclaimed4/10

   Portkey's docs claim caching can serve responses '20x faster' and market the gateway as 'performant,' but there is no direct benchmark or first-party claim quantifying the gateway's own added latency overhead per provider call, nor independent corroboration of low overhead. Missing for 10: explicit latency/overhead benchmarks for the proxy itself, third-party performance testing, and architecture details (e.g., edge/WASM deployment) that would substantiate minimal added latency beyond the cache-hit speedup claim.

   • [claimed-docs] “Cache LLM responses to serve requests up to **20x faster** and cheaper.”
   • [claimed-docs] “It takes 2 mins to integrate and with that, it starts monitoring all of your LLM requests and makes your app resilient, secure, performant, …”
   • [claimed-docs] “offering advanced tools for control, visibility, and security in your Generative AI apps”
   • [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”

   proof ↗⚑ flag

Agent guardrails

1. ai-native userGive an autonomous agent its own key with budget and rate guardrails so it cannot run away on spend

   weight 2 · round drawn
   LiteLLM✓fullclaimed8/10

   LiteLLM's virtual key system explicitly supports per-key budgets (budget_duration, max_budget), rate limits (rpm_limit), and personal/team-scoped keys, directly enabling an autonomous agent to get its own key with spend and rate guardrails. Docs also show spend tracking by key/user/tag to monitor and enforce these limits. Missing for 10: independent/hands-on verification that budget caps hard-stop overspend in practice, and more detail on rate-limit enforcement behavior under load.

   • [claimed-docs] “Personal budgets: Create virtual keys without team_id for individual spending limits”
   • [claimed-docs] “Track Spend, and control model access via virtual keys for the proxy”
   • [claimed-docs] “Apply a budget across all calls on the proxy... budget_duration: 30d # (str) frequency of reset”
   • [claimed-docs] “"team_alias": "my-new-team_4", ... "rpm_limit": 99”
   • [claimed-docs] “Track spend for keys, users, and teams across 100+ LLMs.”
   • [claimed-docs] “pass tags to track spend by tags”
   • [claimed-docs] “Put your full AI stack behind one key. See who is driving spend, cap it before it runs, and send each request to the model that should handl…”

   proof ↗⚑ flag

   Portkey✓fullclaimed8/10

   Portkey docs explicitly support issuing API keys with budget and rate limits (spend thresholds, hourly/daily/per-minute request/token limits) that automatically block further usage when exceeded, directly matching the guardrail need for an autonomous agent's key. Fine-grained control at org/workspace level and per-key configuration is documented. Missing for 10: no explicit mention of a per-agent key issuance workflow or independent/hands-on verification that the cutoff actually stops an agent mid-run.

   • [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
   • [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
   • [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
   • [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
   • [claimed-docs] “**Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC”

   proof ↗⚑ flag

Budgets

1. platform engineerSet hard budgets and spend limits per key, team, or user

   weight 3 · round drawn
   LiteLLM✓fullclaimed9/10

   Docs explicitly cover per-key, per-team, and per-user budgets/spend tracking (virtual keys with cost tracking, personal budgets, team rpm_limit, budget_duration for resets, tag-based spend tracking) plus marketing copy on capping spend before it runs. Missing for 10: independent hands-on verification that hard limits actually block calls once exceeded (only docs, no community confirmation of enforcement behavior).

   • [claimed-docs] “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI”
   • [claimed-docs] “Track spend for keys, users, and teams across 100+ LLMs.”
   • [claimed-docs] “Personal budgets: Create virtual keys without team_id for individual spending limits”
   • [claimed-docs] “Track Spend, and control model access via virtual keys for the proxy”
   • [claimed-docs] “Put your full AI stack behind one key. See who is driving spend, cap it before it runs, and send each request to the model that should handl…”
   • [claimed-docs] “pass tags to track spend by tags”
   • [claimed-docs] “Apply a budget across all calls on the proxy... budget_duration: 30d # (str) frequency of reset”
   • [claimed-docs] “"team_alias": "my-new-team_4", ... "rpm_limit": 99”

   proof ↗⚑ flag

   Portkey✓fullclaimed9/10

   Portkey's docs explicitly describe setting budget limits (max spend or token usage) and rate limits on API keys, with org/workspace-level fine-grained control, automatic enforcement when limits are reached, notification thresholds, and weekly resets — directly matching per-key/team/user budget/spend controls. Missing for 10: no independent/hands-on corroboration beyond first-party docs.

   • [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
   • [claimed-docs] “When usage reaches this threshold, notifications will be sent to configured recipients”
   • [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
   • [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
   • [claimed-docs] “**Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC”
   • [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”

   proof ↗⚑ flag

Spend tracking

1. platform engineerTrack spend per model, key, team, or user across all providers in one place

   weight 3 · round to LiteLLM
   LiteLLM✓fullclaimed9/10

   LiteLLM's docs directly document tracking spend per key, user, team, and by tags across 100+ providers via the proxy's cost tracking feature, plus budgets, virtual keys, and admin UI to view spend — matching the story closely. Missing for 10: independent hands-on verification of the cost dashboard's accuracy/UX and explicit per-model spend breakdown evidence beyond the general 'across 100+ LLMs' claim.

   • [claimed-docs] “Track spend for keys, users, and teams across 100+ LLMs.”
   • [claimed-docs] “Track Spend, and control model access via virtual keys for the proxy”
   • [claimed-docs] “pass tags to track spend by tags”
   • [claimed-docs] “Apply a budget across all calls on the proxy... budget_duration: 30d # (str) frequency of reset”
   • [claimed-docs] “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI”
   • [claimed-docs] “Put your full AI stack behind one key. See who is driving spend, cap it before it runs, and send each request to the model that should handl…”

   proof ↗⚑ flag

   Portkey✓fullclaimed7/10

   Portkey's docs describe budget/rate limits and spend caps on API keys, a unified model catalog with org/workspace-level governance and allow-lists, and logs filterable by Config ID for auditing usage, which together give spend visibility across providers/keys/teams. Missing for 10: explicit per-user spend breakdown dashboards and independent/hands-on confirmation of a unified cost dashboard beyond docs claims.

   • [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
   • [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
   • [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
   • [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
   • [claimed-docs] “Filter logs by `Config ID` to see all requests using that config”
   • [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
   • [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”

   proof ↗⚑ flag

Byok

1. developerBring my own provider API keys and have the gateway use them for my traffic

   weight 2 · round to Portkey
   LiteLLM~partialclaimed6/10

   LiteLLM's core design lets developers call any provider through one interface and issue virtual keys that map to underlying provider credentials, with per-key spend/model access control ('Put your full AI stack behind one key', virtual key docs) — implying BYOK usage. However, the evidence pack lacks an explicit walkthrough of adding a personal provider API key to the proxy config and having it used for a developer's own traffic, and there is no independent confirmation of this specific flow. Missing for 10: explicit BYOK config example, independent/hands-on confirmation of per-developer key routing.

   • [claimed-docs] “Call any provider using the same completion() interface, with no API to re-learn for each one”
   • [claimed-docs] “Track Spend, and control model access via virtual keys for the proxy”
   • [claimed-docs] “Put your full AI stack behind one key. See who is driving spend, cap it before it runs, and send each request to the model that should handl…”
   • [claimed-docs] “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI”

   proof ↗⚑ flag

   Portkey✓fullclaimed8/10

   Portkey's virtual keys and model catalog explicitly let developers add/store their own provider API keys ('Add a provider, then use @provider-slug/model-name', 'Provider credentials stored securely, never exposed in code'), and the gateway routes traffic through these keys with governance/rate-limit controls layered on top. Missing for 10: independent hands-on verification of the BYO-key flow and no explicit mention of per-key traffic isolation/audit beyond docs claims.

   • [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
   • [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”
   • [claimed-docs] “Provider credentials stored securely, never exposed in code”
   • [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
   • [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”

   proof ↗⚑ flag

Programmatic admin

1. ai-native userProvision gateways, keys, and budgets programmatically through an admin API

   weight 3 · round to LiteLLM
   LiteLLM!disputedcontradicted5/10

   Docs describe programmatic key/team/budget management (virtual keys, personal and team budgets, rpm_limit, team_alias via API/config) which supports the story's core claim, but no OpenAPI/swagger spec was found (probe shows 404s) and a community user reported concrete hands-on issues 'creating tokens for other users' via the proxy admin functions, directly contradicting the smoothness implied by the docs. Missing for 10: verified OpenAPI/admin-API reference, evidence of 'gateway' provisioning (multiple proxy instances) via API, and independent confirmation that key/budget creation works reliably at scale.

   • [claimed-docs] “Personal budgets: Create virtual keys without team_id for individual spending limits”
   • [claimed-docs] “Track Spend, and control model access via virtual keys for the proxy”
   • [claimed-docs] “Apply a budget across all calls on the proxy... budget_duration: 30d # (str) frequency of reset”
   • [claimed-docs] “"team_alias": "my-new-team_4", ... "rpm_limit": 99”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.litellm.ai/openapi.json, https://docs.litellm.ai/swagger.json, https://docs.litellm.ai/…”
   • [community] “'Does anyone know a good alternate project... LiteLLM has been getting worse and trying to get me to upgrade to a paid version. I also had i…”

   proof ↗⚑ flag

   Portkey—none0/10

   Evidence shows budget/rate-limit and key management as dashboard-configurable features (docs-13, docs-25, docs-35, docs-38) but never describes a programmatic Admin API for provisioning gateways, keys, or budgets; the only API reference documented is the inference API (docs-15, docs-16, docs-36), and probes for an OpenAPI/admin API spec returned 404 (portkey-probe-3, portkey-probe-2).

   • [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
   • [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
   • [claimed-docs] “**Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC”
   • [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
   • [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
   • [probe] “PROBE docs-md: HTTP 404 at https://portkey.ai/docs.md”

   proof ↗⚑ flag

Virtual keys

1. platform engineerMint gateway-managed keys for teams and apps without exposing raw provider keys

   weight 3 · round to Portkey
   LiteLLM~partialcommunity7/10

   LiteLLM's proxy docs explicitly support minting scoped 'virtual keys' for teams/users/apps with budget and rate limits (litellm-docs-5, litellm-docs-6, litellm-docs-7, litellm-docs-8, litellm-docs-26), which is exactly the gateway-managed-key model that hides raw provider keys behind the proxy. However, one community report describes hands-on 'issues with creating tokens for other users' (litellm-comm-3), and there is no independent corroboration of the key-isolation guarantees or a walkthrough of the actual key-generation API beyond first-party docs. Missing for 10: independent/hands-on confirmation that key scoping reliably prevents provider-key leakage, and resolution of the reported token-creation friction.

   • [claimed-docs] “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI”
   • [claimed-docs] “Track spend for keys, users, and teams across 100+ LLMs.”
   • [claimed-docs] “Personal budgets: Create virtual keys without team_id for individual spending limits”
   • [claimed-docs] “Track Spend, and control model access via virtual keys for the proxy”
   • [claimed-docs] “"team_alias": "my-new-team_4", ... "rpm_limit": 99”
   • [community] “'Does anyone know a good alternate project... LiteLLM has been getting worse and trying to get me to upgrade to a paid version. I also had i…”

   proof ↗⚑ flag

   Portkey✓fullclaimed8/10

   Portkey's virtual keys/model catalog explicitly let platform engineers mint provider-agnostic API keys that map to real provider credentials, which are stored securely and never exposed in code, while budgets, rate limits, and model allow-lists can be enforced per key at org/workspace level for teams and apps. This directly matches minting gateway-managed keys without exposing raw provider keys. Missing for 10: independent/hands-on verification beyond vendor docs, and detail on per-team/app key issuance workflow (e.g., self-service or RBAC-scoped key creation) rather than admin-level config.

   • [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
   • [claimed-docs] “Provider credentials stored securely, never exposed in code”
   • [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
   • [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
   • [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
   • [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
   • [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”

   proof ↗⚑ flag

Integrations

1. developerExport gateway logs and traces to my own observability stack

   weight 1 · round to LiteLLM
   LiteLLM✓fullclaimed8/10

   Docs explicitly confirm proxy logging integrations to external observability stacks (Langfuse, OpenTelemetry, GCS, S3, Azure Blob), plus a unique call_id for tracing requests across systems, directly matching the story of exporting logs/traces to one's own stack. Missing for 10: independent hands-on verification of these integrations working reliably, and more detail on trace format/OpenTelemetry span structure.

   • [claimed-docs] “Log Proxy input, output, and exceptions using: Langfuse, OpenTelemetry, GCS, s3, Azure (Blob) Buckets...”
   • [claimed-docs] “LiteLLM generates a unique call_id for each request. This call_id can be used to track the request across the system.”
   • [claimed-docs] “LiteLLM generates a unique `call_id` for each request. This `call_id` can be used to track the request across the system.”
   • [claimed-docs] “This will prevent the messages and responses from being logged to your logging provider, but request metadata - e.g. spend, will still be tr…”

   proof ↗⚑ flag

   Portkey—none0/10

   Evidence describes Portkey's own logs UI, sharing log URLs, and manual feedback tagging, but there is no mention of exporting logs/traces to third-party observability stacks (e.g., OpenTelemetry, Datadog, Grafana) or any API/webhook for bulk export.

   ⚑ flag

Logs

1. platform engineerInspect logged requests and responses with latency, token counts, and cost attached

   weight 3 · round to LiteLLM
   LiteLLM~partialclaimed6/10

   Docs show LiteLLM logs each request with a unique call_id, tracks spend/cost by key/user/team, and integrates with Langfuse, OpenTelemetry, S3, GCS, etc. for logging proxy input/output/exceptions, all viewable via the admin UI — a reasonable basis for inspecting requests with cost attached. However, the evidence never explicitly confirms that latency or token counts are captured/displayed alongside these logs, and there's no independent/hands-on confirmation of the inspection UI itself. Missing for 10: explicit documentation of per-request latency metrics, explicit token-count logging, and independent verification of the UI/log inspection experience.

   • [claimed-docs] “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI”
   • [claimed-docs] “Track spend for keys, users, and teams across 100+ LLMs.”
   • [claimed-docs] “Log Proxy input, output, and exceptions using: Langfuse, OpenTelemetry, GCS, s3, Azure (Blob) Buckets...”
   • [claimed-docs] “LiteLLM generates a unique call_id for each request. This call_id can be used to track the request across the system.”
   • [claimed-docs] “pass tags to track spend by tags”

   proof ↗⚑ flag

   Portkey~partialclaimed5/10

   Portkey's docs confirm a Logs section showing a chronological list of all requests with unique shareable URLs and manual feedback tagging, supporting log inspection generally, and repeated claims of 'monitoring all LLM requests' and 'visibility' across the gateway. However, the evidence pack never explicitly documents latency, token count, or cost fields being displayed per log entry. Missing for 10: explicit documentation or screenshots showing latency/tokens/cost columns in the logs UI, independent/hands-on confirmation of these fields.

   • [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
   • [claimed-docs] “Each log on Portkey has a unique URL. You can copy the link from the address bar and directly share it with anyone in your org.”
   • [claimed-docs] “you can also add manual feedback on the logs to be analysed and filtered later”
   • [claimed-docs] “It takes 2 mins to integrate and with that, it starts monitoring all of your LLM requests”
   • [claimed-docs] “It takes 2 mins to integrate and with that, it starts monitoring all of your LLM requests and makes your app resilient, secure, performant, …”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to LiteLLM
   LiteLLM~partialprobed6/10

   LiteLLM's proxy exposes virtual keys, budgets, spend tracking, MCP server management, and routing/fallback config via REST API endpoints and CLI, and the admin UI is largely a wrapper over these same proxy APIs, suggesting strong API/UI parity. However, no OpenAPI spec was found at expected locations (404s), and some UI-specific flows like adding MCP servers are only documented via UI clicks rather than confirmed equivalent API calls, and no independent confirmation exists that every UI action has a documented API equivalent. Missing for 10: a published OpenAPI/swagger spec confirming full endpoint coverage, explicit API equivalents for every UI workflow (e.g., MCP server add via API), and independent/hands-on verification that API and UI are truly at parity.

   • [claimed-docs] “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI”
   • [claimed-docs] “Track Spend, and control model access via virtual keys for the proxy”
   • [claimed-docs] “LiteLLM Proxy provides an MCP Gateway that allows you to use a fixed endpoint for all MCP tools and control MCP access by Key, Team.”
   • [claimed-docs] “On the LiteLLM UI, Navigate to "MCP Servers" and click "Add New MCP Server".”
   • [claimed-docs] “The number of worker processes to spin up (uvicorn, gunicorn, or Granian --workers).”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.litellm.ai/openapi.json, https://docs.litellm.ai/swagger.json, https://docs.litellm.ai/…”
   • [probe] “official CLI documented at https://docs.litellm.ai/docs/proxy/cli”

   proof ↗⚑ flag

   Portkey~partialprobed5/10

   Portkey's core AI gateway (chat/completions, fallbacks, caching, retries, load balancing) is clearly exposed via a REST API with a stable base URL, and the open-source gateway can be run locally — showing strong API-first design for request routing. However there is no evidence of API access to the administrative/UI-only features such as budget & rate-limit configuration, log viewing/sharing, feedback tagging, or virtual-key management, and probes show no public OpenAPI spec, so full UI-API parity isn't demonstrated. missing for 10: API endpoints for budget/rate-limit admin, log retrieval/sharing via API, feedback API, and a public OpenAPI spec confirming complete parity.

   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
   • [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
   • [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
   • [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
   • [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
   • [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   LiteLLM—none0/10

   LiteLLM stores spend, logs, virtual-key and config data (self-hosted proxy with admin UI), so an explicit 'export all data in open formats' capability is a fair ask, but no evidence pack item documents any export/download feature, open data format, or data-portability tooling — only logging integrations to third-party sinks (Langfuse, S3, GCS) are mentioned, which is not the same as user-initiated full data export. missing for 10: documented export/download feature, open-format data export (CSV/JSON) of spend/logs/keys, any data-portability or 'leave the platform' guidance.

   • [claimed-docs] “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI”
   • [claimed-docs] “Track spend for keys, users, and teams across 100+ LLMs.”
   • [claimed-docs] “Log Proxy input, output, and exceptions using: Langfuse, OpenTelemetry, GCS, s3, Azure (Blob) Buckets...”

   proof ↗⚑ flag

   Portkey—none0/10

   Portkey's docs cover logging, gateway configuration, and self-hosting the open-source gateway, but there is no mention of exporting stored logs/configs/data in open formats or facilitating a full data export for migration away from the platform. Shareable log URLs (portkey-docs-12) are not a bulk export/portability feature.

   • [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
   • [claimed-docs] “Each log on Portkey has a unique URL. You can copy the link from the address bar and directly share it with anyone in your org.”
   • [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round to Portkey
   LiteLLM~partialcommunity4/10

   The product's source is publicly hosted on GitHub (BerriAI/litellm), confirmed by community references to 'public repos' being vandalized, showing the codebase is openly readable. However, the evidence pack never states or cites an actual license (e.g., MIT/Apache) for the repository, so the 'open license' claim itself is unconfirmed. Missing for 10: explicit license documentation/citation, first-party statement of licensing terms.

   • [github] “Drop-in OpenAI compatibility — swap providers without rewriting your code”
   • [community] “The GitHub account of LiteLLM's Founder/CTO (krrishdholakia) appears to have been fully compromised, with public repos vandalized to say 'te…”

   proof ↗⚑ flag

   Portkey~partialclaimed5/10

   Portkey explicitly states it open-sourced its AI gateway component ('We've open sourced our battle-tested AI gateway to the community... run it locally with a single command'), which supports reading some source code. However, this only covers the gateway, not the full product (observability, admin console, model catalog, etc.), and no explicit license name or repository link is provided in the evidence. Missing for 10: explicit open-source license identification, confirmation that the broader Portkey platform (not just the gateway) is open source, and a linked public repository.

   • [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
   • [claimed-docs] “npx @portkey-ai/gateway”

   proof ↗⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round to LiteLLM
   LiteLLM✓fullcommunity9/10

   LiteLLM explicitly ships a self-hosted LLM Gateway/Proxy (docs-5, docs-23) with configurable worker processes and CLI (docs-13/20), and this is corroborated by hands-on community use running it via Docker in a homelab (litellm-comm-6) and widespread adoption as a self-hosted gateway (litellm-comm-7). missing for 10: no independent benchmark or enterprise-scale self-hosting case study beyond a single homelab anecdote.

   • [claimed-docs] “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI”
   • [claimed-docs] “Self-hosted [LLM Gateway (Proxy)](/docs/simple_proxy) with virtual keys, cost tracking, and an admin UI”
   • [claimed-docs] “The number of worker processes to spin up (uvicorn, gunicorn, or Granian --workers).”
   • [claimed-docs] “The number of worker processes to spin up (uvicorn, gunicorn, or Granian `--workers`).”
   • [community] “A user described running LiteLLM as a proxy in their homelab via the litellm/litellm docker image for local LLM gateway management, noting t…”
   • [community] “'That's a bad supply-chain attack, many folks use litellm as main gateway' — reflecting how widely used LiteLLM is as an LLM gateway.”

   proof ↗⚑ flag

   Portkey~partialclaimed6/10

   Docs confirm the AI Gateway — described as the 'battle-tested' core routing component — is open-sourced and can be run locally via a single npx command, which is genuine self-hosting evidence. However, other core parts of the product (observability/logs UI, admin budget/rate-limit controls, model catalog governance) are described only as hosted SaaS features with no mention of self-hosting. Missing for 10: evidence that the full platform (observability, admin, model catalog) can be self-hosted, not just the gateway component, and any independent confirmation of self-hosting in practice.

   • [claimed-docs] “npx @portkey-ai/gateway”
   • [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
   • [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
   • [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   LiteLLM—none0/10

   LiteLLM is self-hostable (which lets users control physical data location by choosing their own deployment region), but no evidence pack item documents any explicit data residency/region-selection feature, region-pinning configuration, or compliance-region controls offered by the product itself.

   ⚑ flag

   Portkey—none0/10

   No evidence pack items mention data residency, regional storage options, or geographic hosting choices; Portkey's docs focus on gateway routing, caching, retries, budgets, and observability rather than data residency controls.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   LiteLLM—none0/10

   The evidence shows LiteLLM can disable logging of prompts/responses to its own logging providers (litellm-docs-12), but nothing indicates it offers a mechanism to opt out of model-training use by the underlying LLM providers (e.g., passing zero-retention/no-train flags to OpenAI/Anthropic/etc.). As a gateway, this is a plausible capability, but no evidence supports it.

   • [claimed-docs] “This will prevent the messages and responses from being logged to your logging provider, but request metadata - e.g. spend, will still be tr…”

   proof ↗⚑ flag

   Portkey—none0/10

   Evidence pack covers gateway routing, caching, retries, logging, budgeting, etc., but contains no mention of a data-training opt-out, zero-retention policy, or any privacy control preventing use of data for AI model training.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round to LiteLLM
   LiteLLM~partialclaimed5/10

   LiteLLM lets admins disable logging of request/response content to reduce data retention while still tracking metadata (litellm-docs-12), and being self-hosted gives users infrastructure-level control over where data lives. However, there is no documented explicit data-deletion API, retention-period settings, or data lifecycle/export controls in the evidence pack. Missing for 10: explicit deletion/purge API, configurable retention windows, documented data lifecycle policy, independent confirmation these logging controls work as described.

   • [claimed-docs] “This will prevent the messages and responses from being logged to your logging provider, but request metadata - e.g. spend, will still be tr…”
   • [claimed-docs] “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI”

   proof ↗⚑ flag

   Portkey—none0/10

   The evidence pack covers gateway routing, caching, retries, budgets, and logging, but contains no mention of data retention policies, log/data deletion controls, or configurable retention windows.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   LiteLLM—none0/10

   The evidence covers logging integrations (Langfuse, OTel, S3, etc.) and a way to disable message/response logging to those *external* logging providers (litellm-docs-12), but nothing in the pack addresses opting out of LiteLLM's own product telemetry/usage tracking sent back to the vendor. As a self-hosted gateway this axis plausibly applies, but there is no documented telemetry opt-out flag or setting in the evidence pack. Missing for 10: any mention of a telemetry/usage-tracking toggle (e.g., an env var or config flag disabling anonymized usage reporting to the vendor), independent confirmation that no phone-home occurs.

   • [claimed-docs] “This will prevent the messages and responses from being logged to your logging provider, but request metadata - e.g. spend, will still be tr…”
   • [claimed-docs] “Log Proxy input, output, and exceptions using: Langfuse, OpenTelemetry, GCS, s3, Azure (Blob) Buckets...”

   proof ↗⚑ flag

   Portkey—none0/10

   Portkey is essentially a telemetry/observability platform (logs, monitoring of all LLM requests) with no evidence of any opt-out mechanism for its own tracking or usage telemetry; the evidence pack only documents features for logging, caching, retries, and governance, never a privacy/telemetry opt-out control.

   ⚑ flag

Fallbacks

1. platform engineerConfigure automatic fallback to another model or provider when one fails

   weight 3 · round drawn
   LiteLLM✓fullcommunity9/10

   LiteLLM's proxy reliability docs explicitly describe configuring fallbacks so a failing model/provider automatically fails over to a healthy backup, including model-group fallback chains and content-policy-specific fallback handling, corroborated by a maintainer's stated motivation for reliable model switching across providers. Missing for 10: independent hands-on validation of fallback behavior in production beyond docs/maintainer statements.

   • [claimed-docs] “If a call fails after num_retries, LiteLLM falls back to another model group, so a failing model or provider automatically fails over to a h…”
   • [claimed-docs] “so a failing model or provider automatically fails over to a healthy backup”
   • [claimed-docs] “The request to `model="zephyr-beta"` will fail... litellm proxy will loop through all the model_groups specified in `fallbacks=["gpt-3.5-tur…”
   • [claimed-docs] “content_policy_fallbacks: For litellm.ContentPolicyViolationError - LiteLLM maps content policy violation errors across providers”
   • [community] “Show HN launch of LiteLLM Proxy Server: praised as 'super compelling' with maintainers noting motivation was avoiding messy 100-line if/else…”

   proof ↗⚑ flag

   Portkey✓fullclaimed9/10

   Portkey's docs explicitly describe configuring a prioritized list of providers/models with automatic fallback when the primary fails, customizable trigger status codes, plus complementary retries and load balancing for resilience. missing for 10: independent/hands-on corroboration beyond vendor docs.

   • [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
   • [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
   • [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
   • [claimed-docs] “Configure per-strategy circuit protection and failure handling”

   proof ↗⚑ flag

Load balancing

1. platform engineerLoad-balance traffic across providers, deployments, or keys by weight, latency, or cost

   weight 2 · round to Portkey
   LiteLLM~partialclaimed5/10

   Docs confirm load-balancing across multiple deployments and automatic failover/fallback to healthy backups, which covers reliability-based routing, but the evidence pack never cites explicit weight-based, latency-based, or cost-based routing strategies by name (LiteLLM's known routing options aren't documented here). missing for 10: explicit documentation of weight-based routing strategy, latency-based routing strategy, and cost-based routing strategy configuration.

   • [claimed-docs] “Load-balance across multiple deployments (e.g. Azure/OpenAI)”
   • [claimed-docs] “If a call fails after num_retries, LiteLLM falls back to another model group, so a failing model or provider automatically fails over to a h…”
   • [claimed-docs] “so a failing model or provider automatically fails over to a healthy backup”
   • [claimed-docs] “The request to `model="zephyr-beta"` will fail... litellm proxy will loop through all the model_groups specified in `fallbacks=["gpt-3.5-tur…”
   • [claimed-docs] “content_policy_fallbacks: For litellm.ContentPolicyViolationError - LiteLLM maps content policy violation errors across providers”

   proof ↗⚑ flag

   Portkey✓fullclaimed8/10

   Portkey's docs explicitly document load balancing to distribute traffic across multiple LLMs/providers by weight, plus gradual migration (percentage-based rollout) and fallback on failure/latency-driven retries, all configurable via Configs. This directly matches weight-based and failure/latency-based routing across providers/deployments/keys, backed by first-party docs on load-balancing, fallbacks, and virtual keys. Missing for 10: explicit cost-based routing strategy documentation and independent/hands-on verification beyond vendor docs.

   • [claimed-docs] “Distribute traffic across multiple LLMs to prevent any single provider from becoming a bottleneck.”
   • [claimed-docs] “**Gradual Migration** | Test new models with small percentage before full rollout”
   • [claimed-docs] “**Gradual Migration**|Test new models with small percentage before full rollout”
   • [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
   • [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
   • [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
   • [claimed-docs] “Configure per-strategy circuit protection and failure handling”

   proof ↗⚑ flag

Policy routing

1. ai-native userMy agent can switch models mid-task by policy — cost, capability, or availability — through gateway routing rules

   weight 3 · round to Portkey
   LiteLLM✓fullcommunity7/10

   LiteLLM's proxy documents concrete routing policies matching the story: load balancing across deployments, automatic fallback to a healthy backup model/provider on failure, and content-policy-specific fallbacks mapped across providers — all confirmed by community commentary on its origin as a fix for manual multi-provider if/else logic. Missing for 10: explicit cost-triggered routing (switching to cheaper model based on budget threshold) and any hands-on demonstration of live mid-task switching rather than static failover/fallback configuration.

   • [claimed-docs] “Load-balance across multiple deployments (e.g. Azure/OpenAI)”
   • [claimed-docs] “If a call fails after num_retries, LiteLLM falls back to another model group, so a failing model or provider automatically fails over to a h…”
   • [claimed-docs] “so a failing model or provider automatically fails over to a healthy backup”
   • [claimed-docs] “content_policy_fallbacks: For litellm.ContentPolicyViolationError - LiteLLM maps content policy violation errors across providers”
   • [claimed-docs] “The request to `model="zephyr-beta"` will fail... litellm proxy will loop through all the model_groups specified in `fallbacks=["gpt-3.5-tur…”
   • [community] “Show HN launch of LiteLLM Proxy Server: praised as 'super compelling' with maintainers noting motivation was avoiding messy 100-line if/else…”

   proof ↗⚑ flag

   Portkey✓fullclaimed8/10

   Portkey's gateway supports policy-based model switching via conditional fallbacks (status-code triggers), load balancing/weighted routing, gradual migration and canary rollout, plus budget/rate-limit enforcement that can trigger automatic switching for cost/availability reasons - all configurable via routing configs applied at request time. This directly matches mid-task, policy-driven model switching for cost, capability, or availability. Missing for 10: independent/hands-on corroboration of dynamic mid-task capability-based switching (vs. failure-triggered fallback), and explicit example of capability-based routing rules (e.g., routing by task type).

   • [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
   • [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
   • [claimed-docs] “Distribute traffic across multiple LLMs to prevent any single provider from becoming a bottleneck.”
   • [claimed-docs] “**Gradual Migration** | Test new models with small percentage before full rollout”
   • [claimed-docs] “Canary test new models in production”
   • [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
   • [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
   • [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
   • [claimed-docs] “Configure per-strategy circuit protection and failure handling”

   proof ↗⚑ flag

Rate limits

1. platform engineerSmooth provider rate limits by spreading traffic across keys and queuing or throttling requests

   weight 2 · round to Portkey
   LiteLLM✓fullclaimed7/10

   LiteLLM's Router explicitly supports load-balancing across multiple deployments/keys (e.g., multiple Azure/OpenAI keys) and virtual keys carry per-key rpm_limit fields, directly enabling spreading traffic across provider keys to smooth rate limits, with retries/fallbacks for failures. Missing for 10: explicit documentation of a request queuing/throttling mechanism beyond rpm_limit and load balancing, and independent hands-on confirmation that this smooths real provider rate-limit errors under load.

   • [claimed-docs] “Load-balance across multiple deployments (e.g. Azure/OpenAI)”
   • [claimed-docs] “"team_alias": "my-new-team_4", ... "rpm_limit": 99”
   • [claimed-docs] “If a call fails after num_retries, LiteLLM falls back to another model group, so a failing model or provider automatically fails over to a h…”
   • [claimed-docs] “so a failing model or provider automatically fails over to a healthy backup”
   • [claimed-docs] “The request to `model="zephyr-beta"` will fail... litellm proxy will loop through all the model_groups specified in `fallbacks=["gpt-3.5-tur…”

   proof ↗⚑ flag

   Portkey✓fullclaimed8/10

   Portkey documents load balancing across multiple provider keys/models to avoid bottlenecks, automatic retries with exponential backoff or provider retry-after headers, and configurable per-key/org rate limits (hourly/daily/per-minute) alongside budget limits — directly addressing spreading traffic and throttling/queuing around rate limits. Multiple virtual keys per provider further support distributing load to smooth rate limits. Missing for 10: explicit first-party or independent evidence of a request queue (vs. retry/backoff) and hands-on/independent validation of these features working under real rate-limit pressure.

   • [claimed-docs] “Distribute traffic across multiple LLMs to prevent any single provider from becoming a bottleneck.”
   • [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
   • [claimed-docs] “Enable `use_retry_after_headers` to use the provider’s `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”
   • [claimed-docs] “Enable `use_retry_after_headers` to use the provider's `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”
   • [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
   • [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
   • [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
   • [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”

   proof ↗⚑ flag

Retries

1. platform engineerSet automatic retry policies for transient provider errors

   weight 2 · round to Portkey
   LiteLLM✓fullclaimed7/10

   Docs explicitly describe a `num_retries` setting on the proxy, after which LiteLLM falls back to another model/group, directly supporting configurable retry policies for transient failures (docs-4, docs-21, docs-22). Missing for 10: detailed backoff/jitter configuration, granular retry-per-error-type mapping (only content policy fallback is separately documented), and independent hands-on verification of retry behavior under real transient provider errors.

   • [claimed-docs] “If a call fails after num_retries, LiteLLM falls back to another model group, so a failing model or provider automatically fails over to a h…”
   • [claimed-docs] “so a failing model or provider automatically fails over to a healthy backup”
   • [claimed-docs] “The request to `model="zephyr-beta"` will fail... litellm proxy will loop through all the model_groups specified in `fallbacks=["gpt-3.5-tur…”
   • [claimed-docs] “content_policy_fallbacks: For litellm.ContentPolicyViolationError - LiteLLM maps content policy violation errors across providers”

   proof ↗⚑ flag

   Portkey✓fullclaimed8/10

   Portkey's docs explicitly describe automatic retries with exponential backoff and support for provider retry-after headers, directly matching the story of configuring retry policies for transient errors. missing for 10: independent/hands-on corroboration beyond vendor docs, and configuration details on per-error-code or max-retry-count customization.

   • [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
   • [claimed-docs] “Enable `use_retry_after_headers` to use the provider’s `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”
   • [claimed-docs] “Enable `use_retry_after_headers` to use the provider's `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”

   proof ↗⚑ flag

Streaming

1. developerStream token-by-token responses through the gateway from any provider

   weight 3 · round to LiteLLM
   LiteLLM✓fullcommunity7/10

   LiteLLM's unified completion() interface and consistent output format across providers is well documented, and a maintainer explicitly confirmed the proxy 'supports streaming and function-calling in the same way as the openai-python SDK' across 50+ providers, matching the streaming-tools story directly. Missing for 10: a dedicated first-party streaming docs page/code snippet (e.g., stream=True) and independent hands-on confirmation beyond the single maintainer HN comment.

   • [community] “LiteLLM maintainer confirmed proxy supports streaming and function-calling in the same way as the openai-python SDK, and added Ollama integr…”
   • [claimed-docs] “Call any provider using the same completion() interface, with no API to re-learn for each one”
   • [claimed-docs] “Every response follows the OpenAI Chat Completions format, regardless of provider.”
   • [claimed-docs] “Call any provider using the same `completion()` interface, with no API to re-learn for each one”
   • [claimed-docs] “Consistent output format regardless of which provider or model you use”
   • [community] “Show HN launch of LiteLLM Proxy Server: praised as 'super compelling' with maintainers noting motivation was avoiding messy 100-line if/else…”

   proof ↗⚑ flag

   Portkey—none0/10

   The evidence describes Portkey's gateway, fallbacks, retries, load balancing, caching, and OpenAI-compatible base URL integration, but none of the provided docs mention streaming or token-by-token response support. Streaming is a fair and expected capability for an LLM gateway product, so its absence in the evidence pack means this axis applies but is unproven.

   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
   • [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
   • [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”

   proof ↗⚑ flag

Tool calling

1. developerRequest structured JSON-schema outputs across providers

   weight 1 · round drawn
   LiteLLM—none0/10

   The evidence pack covers LiteLLM's unified completion() interface, consistent OpenAI-format responses, retries/fallbacks, and proxy features, but nowhere mentions structured/JSON-schema output support (e.g. response_format, json_schema mode) across providers. This is a fair axis for a multi-provider LLM gateway, but no citation substantiates the specific capability.

   ⚑ flag

   Portkey—none0/10

   The evidence pack covers gateway features like fallbacks, retries, load balancing, caching, virtual keys, and logging, but contains no mention of structured JSON-schema output support, response_format normalization, or schema validation across providers. Since Portkey acts as a universal AI gateway, this capability would be plausible, but no documentation cites it.

   ⚑ flag

2. developerMake tool and function calls across different providers with a consistent schema

   weight 3 · round to LiteLLM
   LiteLLM✓fullcommunity8/10

   LiteLLM documents a unified completion() interface that normalizes tool/function calls and outputs to the OpenAI Chat Completions schema across 100+ providers, and a maintainer confirms function-calling parity with the openai-python SDK. missing for 10: independent hands-on verification of tool-call schema consistency across many specific providers, and no dedicated docs excerpt specifically on function-calling parameter translation edge cases.

   • [claimed-docs] “Call any provider using the same completion() interface, with no API to re-learn for each one”
   • [claimed-docs] “Every response follows the OpenAI Chat Completions format, regardless of provider.”
   • [claimed-docs] “Call any provider using the same `completion()` interface, with no API to re-learn for each one”
   • [claimed-docs] “Consistent output format regardless of which provider or model you use”
   • [github] “Drop-in OpenAI compatibility — swap providers without rewriting your code”
   • [community] “LiteLLM maintainer confirmed proxy supports streaming and function-calling in the same way as the openai-python SDK, and added Ollama integr…”

   proof ↗⚑ flag

   Portkey~partialclaimed5/10

   Portkey's gateway offers a unified OpenAI-SDK-compatible API (base URL + headers) across 250+ models, which implies a consistent request format including function/tool call fields, and it supports connecting external tools via remote MCP servers. However, there is no explicit documentation describing how tool/function-calling schemas are normalized or translated across different providers' native formats. Missing for 10: explicit docs on function/tool-call schema normalization, examples of tool-calling requests across multiple providers, and independent verification that tool calls work consistently across providers.

   • [claimed-docs] “It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in y…”
   • [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
   • [claimed-docs] “Connect to Remote MCP severs, allowing you to connect external tools and data sources.”

   proof ↗⚑ flag

Catalog

1. developerBrowse or query a catalog of available models with pricing and context-window metadata

   weight 2 · round to Portkey
   LiteLLM—none0/10

   While LiteLLM claims support for 100+ LLMs and tracks spend/cost, the evidence pack contains no mention of a browsable/queryable catalog listing models with pricing and context-window metadata (e.g., no model_cost table, /model/info endpoint, or docs page referencing context window sizes). Cost-tracking docs (litellm-docs-6) cover spend tracking, not a metadata catalog.

   ⚑ flag

   Portkey~partialclaimed5/10

   Portkey documents a 'Model Catalog' providing a single pane to view/manage all providers and models with discovery and governance controls, and claims access to 250+ models, but no evidence shows the catalog surfaces pricing or context-window metadata for models. missing for 10: explicit documentation of per-model pricing info, explicit documentation of context-window size metadata, and any UI/query example showing these fields.

   • [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
   • [claimed-docs] “It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in y…”
   • [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”
   • [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”

   proof ↗⚑ flag

Compatibility

1. developerPoint existing OpenAI-compatible code at the gateway by changing only the base URL and key

   weight 3 · round to Portkey
   LiteLLM✓fullcommunity8/10

   LiteLLM's docs and GitHub explicitly advertise drop-in OpenAI compatibility with consistent Chat Completions format across providers, and the self-hosted proxy exposes a single gateway endpoint with virtual keys, matching the exact story of only swapping base URL and key. Community evidence (Show HN, HN threads) corroborates real-world use as a drop-in gateway for existing OpenAI-style code. Missing for 10: independent hands-on verification of literal 'change only base_url+key' migration and no explicit code diff example in the pack.

   • [claimed-docs] “Call any provider using the same completion() interface, with no API to re-learn for each one”
   • [claimed-docs] “Every response follows the OpenAI Chat Completions format, regardless of provider.”
   • [github] “Drop-in OpenAI compatibility — swap providers without rewriting your code”
   • [claimed-docs] “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI”
   • [claimed-docs] “Self-hosted [LLM Gateway (Proxy)](/docs/simple_proxy) with virtual keys, cost tracking, and an admin UI”
   • [community] “Show HN launch of LiteLLM Proxy Server: praised as 'super compelling' with maintainers noting motivation was avoiding messy 100-line if/else…”
   • [community] “LiteLLM maintainer confirmed proxy supports streaming and function-calling in the same way as the openai-python SDK, and added Ollama integr…”

   proof ↗⚑ flag

   Portkey✓fullclaimed9/10

   Docs explicitly state that developers using OpenAI's Python/Node SDKs can integrate Portkey by only changing the base URL and adding Portkey-specific headers, with the base URL documented as https://api.portkey.ai/v1. This is a first-party documented, purpose-built migration path. Missing for 10: independent/hands-on third-party confirmation beyond vendor docs.

   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
   • [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
   • [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”

   proof ↗⚑ flag

One endpoint

1. developerCall many model providers through one consistent API

   weight 3 · round to Portkey
   LiteLLM✓fullcommunity8/10

   LiteLLM's core value proposition—calling 100+ providers through one OpenAI-compatible completion() interface with consistent output format—is well documented (litellm-docs-1, litellm-docs-2, litellm-docs-16, litellm-docs-17, litellm-gh-1) and corroborated by independent Show HN community feedback praising it as a way to standardize disparate AI APIs and quickly add new providers (litellm-comm-12, litellm-comm-13, litellm-comm-14, litellm-comm-15). Some community sentiment flags proxy flakiness/outages, but that concerns operational reliability, not whether the unified-API abstraction itself works. Missing for 10: no independent benchmark verifying identical output format across all supported providers, and no public OpenAPI spec was found for the proxy (litellm-probe-3).

   • [claimed-docs] “Call any provider using the same completion() interface, with no API to re-learn for each one”
   • [claimed-docs] “Every response follows the OpenAI Chat Completions format, regardless of provider.”
   • [claimed-docs] “Call any provider using the same `completion()` interface, with no API to re-learn for each one”
   • [claimed-docs] “Consistent output format regardless of which provider or model you use”
   • [github] “Drop-in OpenAI compatibility — swap providers without rewriting your code”
   • [community] “Show HN launch of LiteLLM Proxy Server: praised as 'super compelling' with maintainers noting motivation was avoiding messy 100-line if/else…”
   • [community] “LiteLLM maintainer confirmed proxy supports streaming and function-calling in the same way as the openai-python SDK, and added Ollama integr…”
   • [community] “Show HN launch of the original LiteLLM library; a user praised it: 'This is amazing. Really needed something like this to standardize all my…”
   • [community] “Maintainer distinguished LiteLLM from a similar library (llm-client-sdk) by stating 'while your library looks really powerful my goal with L…”

   proof ↗⚑ flag

   Portkey✓fullclaimed9/10

   Portkey documents a unified API/gateway (single base URL, single API key) that supports 250+ providers/models via consistent SDK/REST calls, with virtual keys and model catalog abstracting provider differences. Missing for 10: independent third-party benchmark or hands-on confirmation of the claimed 250+ provider coverage beyond vendor docs.

   • [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
   • [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
   • [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
   • [claimed-docs] “It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in y…”
   • [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”
   • [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”

   proof ↗⚑ flag