Model Gateways & Routers Arena
Portkey vs Vercel AI Gateway
Portkey
Portkey, Inc.
Vercel AI Gateway wins · 13–14 (16 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to Vercel AI GatewayPortkey hosts a working llms.txt file (HTTP 200) with a description of the product, confirming agents can be pointed at it for agent-oriented docs. Missing for 10: no docs.md fallback (404) or OpenAPI spec confirmation, and no independent/community corroboration of agents actually using it.
Vercel confirms a live llms.txt (probe-1) plus per-page .md documentation endpoints (probe-2) and an OpenAPI spec (probe-3), all machine-readable for agent consumption, and docs explicitly mention coding-agent setup guidance (docs-8, docs-24). Missing for 10: no independent/community confirmation that agents actually use these llms.txt/docs successfully in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://vercel.com/llms.txt # Vercel > Vercel is a cloud platform for building, deploying, and scaling web appl…”
- [probe] “PROBE docs-md: HTTP 200 at https://vercel.com/docs/ai-gateway.md --- title: AI Gateway product: vercel url: /docs/ai-gateway canonical_url: …”
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
- [claimed-docs] “vercel ai-gateway coding-agents setup”
- [claimed-docs] “AI Gateway routes requests, manages fallbacks and budgets, monitors usage, and connects supported coding agents.”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to Vercel AI GatewayPortkey's gateway can be run headlessly via a CLI command (npx @portkey-ai/gateway) and REST API with a base URL, and it is open source and self-hostable, which supports CI/automation usage. However, there is no explicit documentation of CI pipeline integration, GitHub Actions examples, or headless automation testing workflows. missing for 10: explicit CI/CD examples, headless automation guides, non-interactive scripted usage documentation, independent verification of CI usage.
- [claimed-docs] “npx @portkey-ai/gateway”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
AI Gateway is an API/HTTP gateway with OpenAI-compatible endpoints, unauthenticated model listing, OIDC token auth for deployed apps, and a keyless models endpoint verified live — all of which support pure headless/CI/automation use without any UI dependency. missing for 10: no explicit CI pipeline example (e.g., GitHub Actions) or independent third-party report of CI usage.
- [claimed-docs] “You can use existing OpenAI client libraries, switch to AI Gateway with a URL change, and keep your current tools and workflows without code…”
- [claimed-docs] “For applications deployed on Vercel, OIDC tokens are automatically available as `VERCEL_OIDC_TOKEN`. No secrets to manage, no keys to rotate…”
- [claimed-docs] “This endpoint requires no authentication and returns detailed information including model IDs, context windows, and pricing.”
- [claimed-docs] “For applications deployed on Vercel, OIDC tokens are automatically available as VERCEL_OIDC_TOKEN. No secrets to manage, no keys to rotate. …”
- [claimed-docs] “You can use the AI Gateway's Chat Completions API with existing tools and libraries like the OpenAI client libraries and AI SDK. Point your …”
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
- [probe] “PROBE keyless-endpoint: GET https://ai-gateway.vercel.sh/v1/models returned HTTP 200 application/json with an OpenAI-compatible {"object":"l…”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round to PortkeyPortkey's docs explicitly state it can 'Connect to Remote MCP servers, allowing you to connect external tools and data sources' via the AI Gateway, directly supporting the story. However, evidence lacks detail on setup steps, supported MCP transport, or independent/hands-on confirmation of this working in practice. Missing for 10: detailed configuration docs for adding MCP servers, independent verification of MCP tool invocation, and examples of tool-calling via MCP in a live agent workflow.
- [claimed-docs] “Connect to Remote MCP severs, allowing you to connect external tools and data sources.”
ai-native userConnect an agent via an official MCP server
weight 3 · round drawnPortkeynone0/10Portkey's docs describe connecting *to* remote MCP servers as a client-side feature (portkey-docs-17), but there is no evidence that Portkey itself exposes an official MCP server endpoint that external agents can connect to. Missing for 10: any documentation of Portkey hosting/serving an MCP endpoint, MCP server setup instructions, or third-party confirmation of agents connecting via Portkey's MCP server.
- [claimed-docs] “Connect to Remote MCP severs, allowing you to connect external tools and data sources.”
Vercel AI Gatewaynone0/10The docs mention connecting 'supported coding agents' and generic gateway integration, but there is no evidence of an official MCP (Model Context Protocol) server or endpoint that agents can connect to. Missing for 10: explicit MCP server documentation, MCP endpoint URL, and demonstration of an agent connecting via MCP.
- [claimed-docs] “vercel ai-gateway coding-agents setup”
- [claimed-docs] “AI Gateway routes requests, manages fallbacks and budgets, monitors usage, and connects supported coding agents.”
ai-native userUse an official CLI
weight 2 · round drawnThe only CLI-like evidence is running the open-source gateway locally via `npx @portkey-ai/gateway`, which self-hosts the gateway rather than offering a full-featured CLI for managing configs, logs, or other Portkey resources. Missing for 10: a documented dedicated CLI tool/command set, CLI-based workflow examples, and independent corroboration of CLI usage.
- [claimed-docs] “npx @portkey-ai/gateway”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
The only evidence of a CLI is the fragment 'vercel ai-gateway coding-agents setup', implying an official `vercel` CLI subcommand for configuring the AI Gateway with coding agents, but there is no detailed command reference, install instructions, or broader CLI documentation for AI-native workflows beyond this single mention. Missing for 10: full CLI command reference/docs, examples of CLI usage for model routing/config, independent confirmation of CLI functionality.
- [claimed-docs] “vercel ai-gateway coding-agents setup”
ai-native userDrive the product through a documented public API
weight 3 · round to Vercel AI GatewayPortkey exposes a documented REST API (base URL https://api.portkey.ai/v1) with API-reference docs for its inference API, OpenAI-compatible integration, and llms.txt discovery confirms API-first positioning; gateway is also open-sourced and npx-runnable. Missing for 10: a discoverable OpenAPI/swagger spec (probe found all candidate paths 404) and independent/third-party corroboration of API usage.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
- [probe] “PROBE llms.txt: HTTP 200 at https://portkey.ai/llms.txt # Portkey > Portkey is a production stack for Gen AI builders, equipping AI teams w…”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
AI Gateway exposes a documented, OpenAI-compatible public API (Chat Completions, models endpoint) with a verified OpenAPI spec, unauthenticated models endpoint, and probe confirming a live JSON API response, enabling programmatic/agentic control via standard SDKs or raw HTTP. missing for 10: independent third-party developer writeups explicitly praising the API's completeness/stability beyond Vercel's own docs and probes.
- [claimed-docs] “You can use existing OpenAI client libraries, switch to AI Gateway with a URL change, and keep your current tools and workflows without code…”
- [claimed-docs] “This endpoint requires no authentication and returns detailed information including model IDs, context windows, and pricing.”
- [claimed-docs] “Query the models endpoint directly to get a JSON list of all available models with pricing and capabilities”
- [claimed-docs] “You can use the AI Gateway's Chat Completions API with existing tools and libraries like the OpenAI client libraries and AI SDK. Point your …”
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
- [probe] “PROBE keyless-endpoint: GET https://ai-gateway.vercel.sh/v1/models returned HTTP 200 application/json with an OpenAI-compatible {"object":"l…”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to PortkeyPortkey lets admins issue API keys scoped with budgets, rate limits, and model allow-lists at org/workspace level, and virtual keys keep provider credentials out of code — this maps well to least-privilege credential issuance. However, the docs frame this around human/org governance rather than explicitly per-agent scoped credentials, and there's no independent corroboration of enforcement in practice. Missing for 10: explicit agent-specific credential scoping workflow, independent/hands-on validation of enforcement.
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
- [claimed-docs] “Provider credentials stored securely, never exposed in code”
Docs show budget caps that can be set per API key, per project, or per team member (docs-16), and per-key usage breakdown in Requests logs (docs-12), plus short-lived OIDC tokens for Vercel-deployed apps that avoid static secrets (docs-14/22). This supports issuing distinct, budget-limited keys for an agent, but there is no documentation of fine-grained permission scoping (e.g. restricting a key to specific models/providers/actions) beyond spend limits. missing for 10: explicit least-privilege scoping controls (model/provider/action restrictions per key), role-based permission model, first-party or independent confirmation of true least-privilege agent credentials
- [claimed-docs] “To cap how much your team, a project, an API key, or a team member can spend, set [budgets]”
- [claimed-docs] “The Requests section displays summaries by project, API key, and a detailed log of all requests.”
- [claimed-docs] “For applications deployed on Vercel, OIDC tokens are automatically available as `VERCEL_OIDC_TOKEN`. No secrets to manage, no keys to rotate…”
- [claimed-docs] “For applications deployed on Vercel, OIDC tokens are automatically available as VERCEL_OIDC_TOKEN. No secrets to manage, no keys to rotate. …”
ai-native userBuild against official SDKs
weight 2 · round to Vercel AI GatewayPortkey documents SDK-style integration (base URL swap for OpenAI Python/Node SDKs, REST API reference, open-source gateway installable via npx) supporting AI-native builders, but the evidence pack lacks explicit mention of dedicated first-party SDKs beyond OpenAI-compatible drop-in usage, and OpenAPI spec probes returned 404s. missing for 10: dedicated official multi-language SDK docs (e.g., Python/JS Portkey SDK reference pages), working OpenAPI/spec discovery, independent developer corroboration of SDK usage.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
AI Gateway is designed to be used via existing official SDKs (OpenAI client libraries, Anthropic SDK, Vercel AI SDK) with just a base URL/model-string change, documented in openai-compat and getting-started guides, and confirmed by a live OpenAI-compatible endpoint probe. It also supports agent-building primitives like ToolLoopAgent for agentic workflows. missing for 10: no independent/hands-on developer confirmation beyond vendor docs and one API probe.
- [claimed-docs] “You can use existing OpenAI client libraries, switch to AI Gateway with a URL change, and keep your current tools and workflows without code…”
- [claimed-docs] “Point your existing OpenAI or Anthropic SDK at AI Gateway. Same calls, no rewrites.”
- [claimed-docs] “You can use the AI Gateway's Chat Completions API with existing tools and libraries like the OpenAI client libraries and AI SDK. Point your …”
- [claimed-docs] “Move existing OpenAI, Anthropic, and AI SDK integrations to AI Gateway with a base URL swap.”
- [claimed-docs] “const agent = new ToolLoopAgent({ model: 'anthropic/claude-sonnet-5', tools: { getWeather: tool({”
- [probe] “PROBE keyless-endpoint: GET https://ai-gateway.vercel.sh/v1/models returned HTTP 200 application/json with an OpenAI-compatible {"object":"l…”
ai-native userSubscribe to events via webhooks
weight 2 · round drawnPortkeynone0/10No evidence in the pack mentions webhooks or event subscriptions of any kind; Portkey's docs cover gateway routing, caching, retries, budgets, and logs but nothing about outbound webhook notifications for events.
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round to Vercel AI GatewayPortkey documents a REST API reference (base URL, headers, SDK compatibility) at /docs/api-reference, but the evidence pack contains no mention of an interactive explorer or runnable code examples, and probes for a discoverable OpenAPI/Swagger spec all returned 404s. Missing for 10: evidence of an interactive 'try it' console, runnable code snippets, and a published OpenAPI/Swagger schema.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
There's a documented OpenAPI spec and a live, keyless models endpoint that returns JSON model data, showing some interactive/runnable API surface, but there's no evidence of an interactive API reference UI (like Swagger/Redoc) with embedded runnable examples or a try-it-now console. missing for 10: an actual interactive API explorer/playground page, runnable code snippets embedded in docs, independent confirmation of a docs UI with live request execution.
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
- [probe] “PROBE keyless-endpoint: GET https://ai-gateway.vercel.sh/v1/models returned HTTP 200 application/json with an OpenAI-compatible {"object":"l…”
- [claimed-docs] “This endpoint requires no authentication and returns detailed information including model IDs, context windows, and pricing.”
- [claimed-docs] “Query the models endpoint directly to get a JSON list of all available models with pricing and capabilities”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round to Vercel AI GatewayPortkeynone0/10Portkey has an API reference (base URL, inference API docs) but no evidence of a downloadable OpenAPI/Swagger spec; a direct probe for common OpenAPI paths returned 404s, and docs.md also 404s.
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
- [probe] “PROBE docs-md: HTTP 404 at https://portkey.ai/docs.md”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
A probe confirms a live OpenAPI JSON spec at vercel.com/openapi.json (contains the 'openapi' key) and the docs also expose a keyless models JSON endpoint and OpenAI-compatible API, all of which are machine-readable and programmatically consumable. Missing for 10: explicit confirmation the openapi.json spec covers AI Gateway endpoints specifically (vs. general Vercel API) and independent third-party validation of the spec's completeness.
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
- [probe] “PROBE keyless-endpoint: GET https://ai-gateway.vercel.sh/v1/models returned HTTP 200 application/json with an OpenAI-compatible {"object":"l…”
- [claimed-docs] “This endpoint requires no authentication and returns detailed information including model IDs, context windows, and pricing.”
- [claimed-docs] “Query the models endpoint directly to get a JSON list of all available models with pricing and capabilities”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnPortkeynone0/10No evidence of API versioning scheme (e.g., v1/v2 in URLs beyond a single 'v1' base path) or any documented deprecation policy/changelog; OpenAPI spec probe also returned 404s. Missing for 10: versioning scheme documentation, explicit deprecation policy, changelog/migration guides, and any independent corroboration.
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
Vercel AI Gatewaynone0/10Evidence shows an OpenAI-compatible endpoint with a 'v1' path and an openapi.json spec, but nowhere does the pack document an explicit API versioning scheme or a deprecation policy for endpoints/models. Missing for 10: explicit versioning strategy documentation, deprecation/sunset timelines, migration guidance for breaking changes.
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
- [probe] “PROBE keyless-endpoint: GET https://ai-gateway.vercel.sh/v1/models returned HTTP 200 application/json with an OpenAI-compatible {"object":"l…”
- [claimed-docs] “Query the models endpoint directly to get a JSON list of all available models with pricing and capabilities”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round drawnPortkeynone0/10Portkey's evidence covers gateway routing, fallbacks, caching, logging, and budget/rate-limit controls, but nothing describes a bulk/batch operation capability (e.g., batch inference calls, bulk editing of configs/keys, or bulk log actions) that lets a user act on many items at once. Missing for 10: any documentation of a batch API, bulk config/key management, or bulk log/annotation actions.
Vercel AI Gatewaynone0/10The evidence pack covers routing, failover, BYOK, observability, and embeddings, but there is no mention of any batch/bulk request API or capability to process many items in a single operation. Since a gateway product could plausibly offer batch endpoints (e.g., batch inference or bulk embeddings), the axis applies, but no evidence supports it being delivered.
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to PortkeyPortkey ships several built-in rule/event mechanisms — fallback triggers on non-2xx status codes, automatic retries with backoff, circuit-breaker/failure handling per strategy, and budget/rate-limit thresholds that automatically send notifications or block further usage — which are automatic actions triggered by defined conditions. However, these are fixed, product-defined automations (reliability/spend rules) rather than a general user-authorable rule engine where an AI-native user can define arbitrary custom triggers/actions. Missing for 10: general-purpose custom rule/trigger builder, arbitrary event types beyond status codes/spend thresholds, user-defined action targets (e.g. webhooks, custom scripts) beyond built-in notifications/fallback/retry.
- [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
- [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
- [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
- [claimed-docs] “Configure per-strategy circuit protection and failure handling”
- [claimed-docs] “When usage reaches this threshold, notifications will be sent to configured recipients”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
The Gateway supports narrow built-in automation triggers — failover/retry on provider failure, per-provider timeouts triggering fast failover, and budget caps that block spend once a threshold is hit — which are rule-like actions triggered by events (docs-3, docs-10, docs-13, docs-16). However, there's no general-purpose, user-defined rules/event system (e.g., custom conditions, webhooks, arbitrary trigger-action pairs) for AI-native users to configure; the 'rules' are limited to routing, caching, and spend-cap logic. Missing for 10: a configurable rules engine for arbitrary events/actions, webhook or external-trigger support, and evidence of user-defined conditional automation beyond routing/budget defaults.
- [claimed-docs] “High reliability. Automatically retries requests to other providers if one fails”
- [claimed-docs] “You can set per-provider timeouts to trigger fast failover when a provider is slow to respond.”
- [claimed-docs] “If a query using your credentials fails, AI Gateway will retry the query with its system credentials to improve service availability.”
- [claimed-docs] “To cap how much your team, a project, an API key, or a team member can spend, set [budgets]”
- [claimed-docs] “You can use order, only, and sort in providerOptions.gateway to control which providers handle your requests, in what order, and how they ar…”
Caching performance — stories about caching performance in this arenaCaching performance
Stories about caching performance in this arena
Caching
developerCache responses at the gateway to cut cost and latency on repeated requests
weight 2 · round to PortkeyPortkey's docs explicitly describe gateway-level caching (simple and semantic) that serves responses up to 20x faster and cheaper, including semantic matching via cosine similarity for near-duplicate queries. This directly matches the story's cost/latency reduction on repeated requests. missing for 10: independent/hands-on benchmark corroboration beyond vendor docs.
- [claimed-docs] “Cache LLM responses to serve requests up to **20x faster** and cheaper.”
- [claimed-docs] “Matches requests with **similar meaning** using cosine similarity, not just identical text.”
Docs explicitly describe a `caching: 'auto'` option in providerOptions.gateway that lets the gateway apply provider-appropriate caching automatically, directly supporting cost/latency reduction on repeated requests, and community pricing discussion confirms cache-read pricing exists for at least one model. However, details on cache hit-rate guarantees, TTL, scope, or independent benchmarks of latency/cost savings are absent. Missing for 10: independent hands-on benchmarks showing latency/cost reduction, documentation on cache TTL/invalidation/scope, confirmation caching works uniformly across all providers rather than being provider-dependent.
- [claimed-docs] “You can use `caching: 'auto'` in `providerOptions.gateway` to let AI Gateway automatically apply the appropriate caching strategy based on t…”
- [claimed-docs] “You can use caching: 'auto' in providerOptions.gateway to let AI Gateway automatically apply the appropriate caching strategy based on the p…”
- [community] “Vercel cache read pricing for DeepSeek-v4: $0.01/M for flash, $0.14/M for pro — a 64% discount vs DeepSeek/OpenRouter on flash caching, prom…”
Latency
platform engineerRun traffic through gateway infrastructure that adds minimal latency overhead to provider calls
weight 2 · round drawnPortkey's docs claim caching can serve responses '20x faster' and market the gateway as 'performant,' but there is no direct benchmark or first-party claim quantifying the gateway's own added latency overhead per provider call, nor independent corroboration of low overhead. Missing for 10: explicit latency/overhead benchmarks for the proxy itself, third-party performance testing, and architecture details (e.g., edge/WASM deployment) that would substantiate minimal added latency beyond the cache-hit speedup claim.
- [claimed-docs] “Cache LLM responses to serve requests up to **20x faster** and cheaper.”
- [claimed-docs] “It takes 2 mins to integrate and with that, it starts monitoring all of your LLM requests and makes your app resilient, secure, performant, …”
- [claimed-docs] “offering advanced tools for control, visibility, and security in your Generative AI apps”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
Docs show performance-related controls (per-provider timeouts for fast failover, sort-by-performance metric, automatic caching) that imply attention to latency, but there is no direct benchmark or explicit claim quantifying the gateway's own added latency overhead. Missing for 10: published latency benchmarks or first-party/independent measurements showing the gateway adds minimal overhead versus direct provider calls, and no community corroboration of low added latency.
- [claimed-docs] “You can set per-provider timeouts to trigger fast failover when a provider is slow to respond.”
- [claimed-docs] “You can also use `sort` to rank providers by a performance or cost metric. The gateway sorts providers by the chosen metric and tries them i…”
- [claimed-docs] “You can use `caching: 'auto'` in `providerOptions.gateway` to let AI Gateway automatically apply the appropriate caching strategy based on t…”
- [claimed-docs] “You can use caching: 'auto' in providerOptions.gateway to let AI Gateway automatically apply the appropriate caching strategy based on the p…”
Cost controls — stories about cost controls in this arenaCost controls
Stories about cost controls in this arena
Agent guardrails
ai-native userGive an autonomous agent its own key with budget and rate guardrails so it cannot run away on spend
weight 2 · round to PortkeyPortkey docs explicitly support issuing API keys with budget and rate limits (spend thresholds, hourly/daily/per-minute request/token limits) that automatically block further usage when exceeded, directly matching the guardrail need for an autonomous agent's key. Fine-grained control at org/workspace level and per-key configuration is documented. Missing for 10: no explicit mention of a per-agent key issuance workflow or independent/hands-on verification that the cutoff actually stops an agent mid-run.
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
- [claimed-docs] “**Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC”
Docs confirm you can create per-API-key budgets and view spend/usage broken out by API key (docs-16, docs-12, docs-5), which supports giving an agent its own key with a spend cap. However, there is no explicit documentation of rate-limit guardrails (only spend budgets), and community evidence shows real friction around balance-based enforcement even with BYOK keys, so the 'cannot run away on spend' guarantee is only partially evidenced. Missing for 10: explicit per-key rate-limiting controls, clear docs on key-level rate guardrails (not just budget caps), and independent confirmation that budget enforcement reliably halts an autonomous agent before overspend.
- [claimed-docs] “To cap how much your team, a project, an API key, or a team member can spend, set [budgets]”
- [claimed-docs] “The Requests section displays summaries by project, API key, and a detailed log of all requests.”
- [claimed-docs] “Spend monitoring. Monitor your spending across different providers”
- [community] “If I bring my own Anthropic/OpenAI key and have usage available, why does Vercel's account balance determine whether my request executes? (V…”
Budgets
platform engineerSet hard budgets and spend limits per key, team, or user
weight 3 · round to PortkeyPortkey's docs explicitly describe setting budget limits (max spend or token usage) and rate limits on API keys, with org/workspace-level fine-grained control, automatic enforcement when limits are reached, notification thresholds, and weekly resets — directly matching per-key/team/user budget/spend controls. Missing for 10: no independent/hands-on corroboration beyond first-party docs.
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “When usage reaches this threshold, notifications will be sent to configured recipients”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
- [claimed-docs] “**Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
Docs explicitly state budgets can be set to cap spend for a team, project, API key, or team member, directly matching the story, and spend monitoring/usage dashboards support enforcement. missing for 10: independent/hands-on confirmation that hard budgets actually block spend at the limit (community evidence only discusses BYOK balance blocking, not per-key/team/user budget enforcement specifics) and more detail on granularity/configuration UI.
- [claimed-docs] “To cap how much your team, a project, an API key, or a team member can spend, set [budgets]”
- [claimed-docs] “Spend monitoring. Monitor your spending across different providers”
- [claimed-docs] “The Requests section displays summaries by project, API key, and a detailed log of all requests.”
- [claimed-docs] “The Usage section displays four metrics to help you monitor your AI Gateway activity.”
Spend tracking
platform engineerTrack spend per model, key, team, or user across all providers in one place
weight 3 · round to Vercel AI GatewayPortkey's docs describe budget/rate limits and spend caps on API keys, a unified model catalog with org/workspace-level governance and allow-lists, and logs filterable by Config ID for auditing usage, which together give spend visibility across providers/keys/teams. Missing for 10: explicit per-user spend breakdown dashboards and independent/hands-on confirmation of a unified cost dashboard beyond docs claims.
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
- [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
- [claimed-docs] “Filter logs by `Config ID` to see all requests using that config”
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
Docs show request/usage summaries broken down by project and API key, budgets settable per team/project/API key/team member, and detailed logs filterable by model/provider/status with CSV/JSON export, directly matching the story's cost-tracking dimensions. Missing for 10: explicit per-user (not just per-team-member/API key) breakdowns and independent third-party confirmation of the dashboard's granularity in practice.
- [claimed-docs] “Spend monitoring. Monitor your spending across different providers”
- [claimed-docs] “The Requests section displays summaries by project, API key, and a detailed log of all requests.”
- [claimed-docs] “To cap how much your team, a project, an API key, or a team member can spend, set [budgets]”
- [claimed-docs] “open the Logs page, where you can search by request ID, filter by model, provider, or status code, follow requests live, and open one reques…”
- [claimed-docs] “The Usage section displays four metrics to help you monitor your AI Gateway activity.”
Key management — stories about key management in this arenaKey management
Stories about key management in this arena
Byok
developerBring my own provider API keys and have the gateway use them for my traffic
weight 2 · round to PortkeyPortkey's virtual keys and model catalog explicitly let developers add/store their own provider API keys ('Add a provider, then use @provider-slug/model-name', 'Provider credentials stored securely, never exposed in code'), and the gateway routes traffic through these keys with governance/rate-limit controls layered on top. Missing for 10: independent hands-on verification of the BYO-key flow and no explicit mention of per-key traffic isolation/audit beyond docs claims.
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”
- [claimed-docs] “Provider credentials stored securely, never exposed in code”
- [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
Vercel AI Gatewaydisputedcontradicted5/10Vercel's docs clearly describe BYOK support — pass provider credentials per-request via providerOptions.gateway.byok, with no markup on tokens, and even fallback to system credentials on failure — suggesting bring-your-own-key traffic should run on the developer's own credentials (docs-6, docs-13, docs-19, docs-27). However, a concrete community report describes BYOK requests being blocked by Vercel's account balance reaching $0 even when the user's own provider key has available usage, directly contradicting the expectation that BYOK traffic is unaffected by platform account balance (comm-1). missing for 10: independent confirmation resolving the balance-dependency complaint, clearer documentation on how billing/balance interacts with BYOK requests.
- [claimed-docs] “No markup on tokens. Tokens cost the same as they would from the provider directly, with zero markup, including with [Bring Your Own Key (BY…”
- [claimed-docs] “If a query using your credentials fails, AI Gateway will retry the query with its system credentials to improve service availability.”
- [claimed-docs] “No markup on tokens. Tokens cost the same as they would from the provider directly, with zero markup, including with Bring Your Own Key (BYO…”
- [claimed-docs] “you can pass provider credentials on a per-request basis using the byok option in providerOptions.gateway. This is useful when you need to u…”
- [community] “If I bring my own Anthropic/OpenAI key and have usage available, why does Vercel's account balance determine whether my request executes? (V…”
Programmatic admin
ai-native userProvision gateways, keys, and budgets programmatically through an admin API
weight 3 · round drawnPortkeynone0/10Evidence shows budget/rate-limit and key management as dashboard-configurable features (docs-13, docs-25, docs-35, docs-38) but never describes a programmatic Admin API for provisioning gateways, keys, or budgets; the only API reference documented is the inference API (docs-15, docs-16, docs-36), and probes for an OpenAPI/admin API spec returned 404 (portkey-probe-3, portkey-probe-2).
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “**Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
- [probe] “PROBE docs-md: HTTP 404 at https://portkey.ai/docs.md”
Vercel AI Gatewaynone0/10Docs describe setting budgets, API keys, and monitoring spend through the Vercel dashboard/CLI, and a general Vercel OpenAPI spec exists, but there is no evidence of a documented admin API specifically for programmatically provisioning AI Gateway resources, keys, or budgets. Missing for 10: explicit admin/API endpoints for creating gateways, generating keys, or setting budgets programmatically, and any docs or examples showing this workflow.
- [claimed-docs] “To cap how much your team, a project, an API key, or a team member can spend, set [budgets]”
- [claimed-docs] “The Requests section displays summaries by project, API key, and a detailed log of all requests.”
- [probe] “PROBE openapi: HTTP 200 at https://vercel.com/openapi.json — contains "openapi" key”
Virtual keys
platform engineerMint gateway-managed keys for teams and apps without exposing raw provider keys
weight 3 · round to PortkeyPortkey's virtual keys/model catalog explicitly let platform engineers mint provider-agnostic API keys that map to real provider credentials, which are stored securely and never exposed in code, while budgets, rate limits, and model allow-lists can be enforced per key at org/workspace level for teams and apps. This directly matches minting gateway-managed keys without exposing raw provider keys. Missing for 10: independent/hands-on verification beyond vendor docs, and detail on per-team/app key issuance workflow (e.g., self-service or RBAC-scoped key creation) rather than admin-level config.
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “Provider credentials stored securely, never exposed in code”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
- [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”
Docs show a single Gateway-managed API key (or OIDC token) replaces direct provider keys, with per-API-key usage tracking and budget caps that imply teams/apps can have distinct managed keys, and BYOK/system-credential fallback keeps raw provider secrets out of app code. Missing for 10: explicit dashboard walkthrough for creating/rotating multiple named keys per team or app, and documentation of per-key role/permission scoping.
- [claimed-docs] “One key, hundreds of models. Access models from multiple providers with a single API key”
- [claimed-docs] “The Requests section displays summaries by project, API key, and a detailed log of all requests.”
- [claimed-docs] “To cap how much your team, a project, an API key, or a team member can spend, set [budgets]”
- [claimed-docs] “If a query using your credentials fails, AI Gateway will retry the query with its system credentials to improve service availability.”
- [claimed-docs] “For applications deployed on Vercel, OIDC tokens are automatically available as `VERCEL_OIDC_TOKEN`. No secrets to manage, no keys to rotate…”
- [claimed-docs] “you can pass provider credentials on a per-request basis using the byok option in providerOptions.gateway. This is useful when you need to u…”
Observability — seeing what the system is doing — logs, metrics, traces, alertsObservability
Seeing what the system is doing — logs, metrics, traces, alerts
Integrations
developerExport gateway logs and traces to my own observability stack
weight 1 · round to Vercel AI GatewayPortkeynone0/10Evidence describes Portkey's own logs UI, sharing log URLs, and manual feedback tagging, but there is no mention of exporting logs/traces to third-party observability stacks (e.g., OpenTelemetry, Datadog, Grafana) or any API/webhook for bulk export.
The gateway provides a Logs UI with filtering and CSV/JSON export of request logs, which supports manual export of logs, but there's no evidence of native integration with external observability stacks (e.g., OpenTelemetry export, webhook/streaming to Datadog/Grafana, or trace export in standard formats like OTLP). missing for 10: OpenTelemetry/OTLP trace export, native integrations with third-party observability platforms, automated/streaming log export (vs manual CSV/JSON download), independent confirmation of export usability.
- [claimed-docs] “open the Logs page, where you can search by request ID, filter by model, provider, or status code, follow requests live, and open one reques…”
- [claimed-docs] “The Requests section displays summaries by project, API key, and a detailed log of all requests.”
- [claimed-docs] “The Usage section displays four metrics to help you monitor your AI Gateway activity.”
Logs
platform engineerInspect logged requests and responses with latency, token counts, and cost attached
weight 3 · round to Vercel AI GatewayPortkey's docs confirm a Logs section showing a chronological list of all requests with unique shareable URLs and manual feedback tagging, supporting log inspection generally, and repeated claims of 'monitoring all LLM requests' and 'visibility' across the gateway. However, the evidence pack never explicitly documents latency, token count, or cost fields being displayed per log entry. Missing for 10: explicit documentation or screenshots showing latency/tokens/cost columns in the logs UI, independent/hands-on confirmation of these fields.
- [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
- [claimed-docs] “Each log on Portkey has a unique URL. You can copy the link from the address bar and directly share it with anyone in your org.”
- [claimed-docs] “you can also add manual feedback on the logs to be analysed and filtered later”
- [claimed-docs] “It takes 2 mins to integrate and with that, it starts monitoring all of your LLM requests”
- [claimed-docs] “It takes 2 mins to integrate and with that, it starts monitoring all of your LLM requests and makes your app resilient, secure, performant, …”
Docs describe a Logs page with per-request detail, search by request ID, filtering by model/provider/status, and export as CSV/JSON, plus a Requests section with summaries by project/API key and a Usage section with metrics — covering latency, tokens, cost, and routing detail. Missing for 10: independent/hands-on confirmation that latency and cost are explicitly visible per logged request (only vendor docs cited), and no third-party corroboration of the observability UI.
- [claimed-docs] “The Requests section displays summaries by project, API key, and a detailed log of all requests.”
- [claimed-docs] “open the Logs page, where you can search by request ID, filter by model, provider, or status code, follow requests live, and open one reques…”
- [claimed-docs] “The Usage section displays four metrics to help you monitor your AI Gateway activity.”
- [claimed-docs] “Spend monitoring. Monitor your spending across different providers”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to Vercel AI GatewayPortkey's core AI gateway (chat/completions, fallbacks, caching, retries, load balancing) is clearly exposed via a REST API with a stable base URL, and the open-source gateway can be run locally — showing strong API-first design for request routing. However there is no evidence of API access to the administrative/UI-only features such as budget & rate-limit configuration, log viewing/sharing, feedback tagging, or virtual-key management, and probes show no public OpenAPI spec, so full UI-API parity isn't demonstrated. missing for 10: API endpoints for budget/rate-limit admin, log retrieval/sharing via API, feedback API, and a public OpenAPI spec confirming complete parity.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
The gateway is largely API-first: model access, provider routing, caching, BYOK, and even a keyless public models endpoint are confirmed to work programmatically (docs-1,2,9,15,25,27; probe-4). However, budgets, spend caps, and the detailed Requests/Logs dashboard are described only as UI features (docs-5,12,16,23,30) without documented API endpoints to set budgets or pull logs programmatically. Missing for 10: budgets/spend-cap management API, log/usage export API parity, team/key management API parity.
- [claimed-docs] “One key, hundreds of models. Access models from multiple providers with a single API key”
- [claimed-docs] “Unified API. Switch between providers and models with minimal code changes”
- [claimed-docs] “You can use `order`, `only`, and `sort` in `providerOptions.gateway` to control which providers handle your requests, in what order, and how…”
- [claimed-docs] “This endpoint requires no authentication and returns detailed information including model IDs, context windows, and pricing.”
- [claimed-docs] “Query the models endpoint directly to get a JSON list of all available models with pricing and capabilities”
- [claimed-docs] “you can pass provider credentials on a per-request basis using the byok option in providerOptions.gateway. This is useful when you need to u…”
- [claimed-docs] “Spend monitoring. Monitor your spending across different providers”
- [claimed-docs] “To cap how much your team, a project, an API key, or a team member can spend, set [budgets]”
- [claimed-docs] “open the Logs page, where you can search by request ID, filter by model, provider, or status code, follow requests live, and open one reques…”
- [probe] “PROBE keyless-endpoint: GET https://ai-gateway.vercel.sh/v1/models returned HTTP 200 application/json with an OpenAI-compatible {"object":"l…”
ai-native userExport all of my data in open formats and leave
weight 3 · round to Vercel AI GatewayPortkeynone0/10Portkey's docs cover logging, gateway configuration, and self-hosting the open-source gateway, but there is no mention of exporting stored logs/configs/data in open formats or facilitating a full data export for migration away from the platform. Shareable log URLs (portkey-docs-12) are not a bulk export/portability feature.
- [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
- [claimed-docs] “Each log on Portkey has a unique URL. You can copy the link from the address bar and directly share it with anyone in your org.”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
Vercel documents that the Logs page lets users export request/usage data as CSV or JSON, and its OpenAI-compatible API means you can point clients elsewhere with minimal rewrite, reducing lock-in. However, there's no evidence of a comprehensive account/data export (billing history, configs, keys) beyond request logs, and BYOK behavior tied to Vercel balance (comm-1) suggests some platform dependency remains. Missing for 10: full account/config data export, evidence of exporting all historical usage/billing data, and confirmation that BYOK/API keys are fully portable without gateway dependency.
- [claimed-docs] “open the Logs page, where you can search by request ID, filter by model, provider, or status code, follow requests live, and open one reques…”
- [claimed-docs] “You can use existing OpenAI client libraries, switch to AI Gateway with a URL change, and keep your current tools and workflows without code…”
- [claimed-docs] “Point your existing OpenAI or Anthropic SDK at AI Gateway. Same calls, no rewrites.”
- [claimed-docs] “You can use the AI Gateway's Chat Completions API with existing tools and libraries like the OpenAI client libraries and AI SDK. Point your …”
- [community] “If I bring my own Anthropic/OpenAI key and have usage available, why does Vercel's account balance determine whether my request executes? (V…”
ai-native userRead the product's source under an open license
weight 2 · round to PortkeyPortkey explicitly states it open-sourced its AI gateway component ('We've open sourced our battle-tested AI gateway to the community... run it locally with a single command'), which supports reading some source code. However, this only covers the gateway, not the full product (observability, admin console, model catalog, etc.), and no explicit license name or repository link is provided in the evidence. Missing for 10: explicit open-source license identification, confirmation that the broader Portkey platform (not just the gateway) is open source, and a linked public repository.
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
- [claimed-docs] “npx @portkey-ai/gateway”
ai-native userSelf-host the core product
weight 3 · round to PortkeyDocs confirm the AI Gateway — described as the 'battle-tested' core routing component — is open-sourced and can be run locally via a single npx command, which is genuine self-hosting evidence. However, other core parts of the product (observability/logs UI, admin budget/rate-limit controls, model catalog governance) are described only as hosted SaaS features with no mention of self-hosting. Missing for 10: evidence that the full platform (observability, admin, model catalog) can be self-hosted, not just the gateway component, and any independent confirmation of self-hosting in practice.
- [claimed-docs] “npx @portkey-ai/gateway”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
- [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round drawnPortkeynone0/10No evidence pack items mention data residency, regional storage options, or geographic hosting choices; Portkey's docs focus on gateway routing, caching, retries, budgets, and observability rather than data residency controls.
Vercel AI Gatewaynone0/10No evidence of region/data-residency controls or storage location choices in the AI Gateway docs; the pack covers routing, failover, pricing, and observability but nothing about where data is stored or processed. Missing for 10: any mention of regional data residency options, storage location controls, or compliance certifications addressing data locality.
ai-native userPrevent my data from being used to train AI models
weight 3 · round drawnPortkeynone0/10Evidence pack covers gateway routing, caching, retries, logging, budgeting, etc., but contains no mention of a data-training opt-out, zero-retention policy, or any privacy control preventing use of data for AI model training.
Vercel AI Gatewaynone0/10No evidence in the pack addresses data-training opt-out policies, zero-retention agreements, or any explicit statement that prompts/outputs are excluded from model training; the docs focus on routing, pricing, and observability instead. Missing for 10: any privacy policy or training-opt-out statement, zero-data-retention terms, provider-level data-use guarantees.
ai-native userControl data retention and deletion
weight 2 · round drawnPortkeynone0/10The evidence pack covers gateway routing, caching, retries, budgets, and logging, but contains no mention of data retention policies, log/data deletion controls, or configurable retention windows.
Vercel AI Gatewaynone0/10No evidence in the pack addresses data retention policies, log/data deletion controls, or retention configuration for requests/prompts processed through AI Gateway; observability docs mention logging/exporting requests but nothing about retention duration or deletion mechanisms.
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnPortkeynone0/10Portkey is essentially a telemetry/observability platform (logs, monitoring of all LLM requests) with no evidence of any opt-out mechanism for its own tracking or usage telemetry; the evidence pack only documents features for logging, caching, retries, and governance, never a privacy/telemetry opt-out control.
Vercel AI Gatewaynone0/10No evidence of any telemetry opt-out or privacy/data-collection controls; the gateway's observability features are about logging usage for the account owner, not about disabling tracking. Missing for 10: any documented telemetry opt-out setting, privacy policy on data retention/training use, or way to disable request logging.
Routing resilience — stories about routing resilience in this arenaRouting resilience
Stories about routing resilience in this arena
Fallbacks
platform engineerConfigure automatic fallback to another model or provider when one fails
weight 3 · round drawnPortkey's docs explicitly describe configuring a prioritized list of providers/models with automatic fallback when the primary fails, customizable trigger status codes, plus complementary retries and load balancing for resilience. missing for 10: independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
- [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
- [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
- [claimed-docs] “Configure per-strategy circuit protection and failure handling”
Docs explicitly describe automatic retries to other providers on failure, plus fine-grained control via `order`, `only`, and `sort` in providerOptions.gateway, per-provider timeouts for fast failover, and even BYOK-to-system-credential fallback — directly matching the fallback/routing-resilience story. This is well-documented first-party capability with concrete configuration options, though missing for 10: independent hands-on verification that fallback behaves correctly in production scenarios.
- [claimed-docs] “High reliability. Automatically retries requests to other providers if one fails”
- [claimed-docs] “You can use `order`, `only`, and `sort` in `providerOptions.gateway` to control which providers handle your requests, in what order, and how…”
- [claimed-docs] “You can set per-provider timeouts to trigger fast failover when a provider is slow to respond.”
- [claimed-docs] “If a query using your credentials fails, AI Gateway will retry the query with its system credentials to improve service availability.”
- [claimed-docs] “You can use order, only, and sort in providerOptions.gateway to control which providers handle your requests, in what order, and how they ar…”
- [claimed-docs] “AI Gateway routes requests, manages fallbacks and budgets, monitors usage, and connects supported coding agents.”
Load balancing
platform engineerLoad-balance traffic across providers, deployments, or keys by weight, latency, or cost
weight 2 · round to PortkeyPortkey's docs explicitly document load balancing to distribute traffic across multiple LLMs/providers by weight, plus gradual migration (percentage-based rollout) and fallback on failure/latency-driven retries, all configurable via Configs. This directly matches weight-based and failure/latency-based routing across providers/deployments/keys, backed by first-party docs on load-balancing, fallbacks, and virtual keys. Missing for 10: explicit cost-based routing strategy documentation and independent/hands-on verification beyond vendor docs.
- [claimed-docs] “Distribute traffic across multiple LLMs to prevent any single provider from becoming a bottleneck.”
- [claimed-docs] “**Gradual Migration** | Test new models with small percentage before full rollout”
- [claimed-docs] “**Gradual Migration**|Test new models with small percentage before full rollout”
- [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
- [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “Configure per-strategy circuit protection and failure handling”
Docs confirm ordering/sorting by performance or cost metric (`order`, `only`, `sort`), per-provider timeouts for fast failover, and automatic retries/fallbacks to other providers — covering latency- and cost-based routing and failover. However, explicit weight-based load balancing across providers/deployments/keys is not documented, and there's no evidence of key-level load balancing beyond BYOK fallback to system credentials. missing for 10: explicit weighted load-balancing configuration, per-key routing/balancing control, independent verification of routing behavior under load.
- [claimed-docs] “You can use `order`, `only`, and `sort` in `providerOptions.gateway` to control which providers handle your requests, in what order, and how…”
- [claimed-docs] “You can set per-provider timeouts to trigger fast failover when a provider is slow to respond.”
- [claimed-docs] “You can also use `sort` to rank providers by a performance or cost metric. The gateway sorts providers by the chosen metric and tries them i…”
- [claimed-docs] “High reliability. Automatically retries requests to other providers if one fails”
- [claimed-docs] “If a query using your credentials fails, AI Gateway will retry the query with its system credentials to improve service availability.”
Policy routing
ai-native userMy agent can switch models mid-task by policy — cost, capability, or availability — through gateway routing rules
weight 3 · round to PortkeyPortkey's gateway supports policy-based model switching via conditional fallbacks (status-code triggers), load balancing/weighted routing, gradual migration and canary rollout, plus budget/rate-limit enforcement that can trigger automatic switching for cost/availability reasons - all configurable via routing configs applied at request time. This directly matches mid-task, policy-driven model switching for cost, capability, or availability. Missing for 10: independent/hands-on corroboration of dynamic mid-task capability-based switching (vs. failure-triggered fallback), and explicit example of capability-based routing rules (e.g., routing by task type).
- [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
- [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
- [claimed-docs] “Distribute traffic across multiple LLMs to prevent any single provider from becoming a bottleneck.”
- [claimed-docs] “**Gradual Migration** | Test new models with small percentage before full rollout”
- [claimed-docs] “Canary test new models in production”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
- [claimed-docs] “Configure per-strategy circuit protection and failure handling”
Docs show routing rules via providerOptions.gateway (order, only, sort, timeouts) that let requests fail over across providers by availability/latency, plus cost/spend visibility and budgets, which supports policy-driven switching. However, there's no explicit evidence of dynamic mid-task capability-based switching or agent-driven runtime policy changes beyond static config, and community evidence flags BYOK requests being blocked once account balance hits $0, undermining pure availability-based routing in some cases. missing for 10: explicit support for capability-based (not just cost/availability) routing rules, evidence of an agent dynamically changing models mid-task at runtime, and independent confirmation that failover/ordering works reliably in production.
- [claimed-docs] “You can use `order`, `only`, and `sort` in `providerOptions.gateway` to control which providers handle your requests, in what order, and how…”
- [claimed-docs] “You can set per-provider timeouts to trigger fast failover when a provider is slow to respond.”
- [claimed-docs] “You can also use `sort` to rank providers by a performance or cost metric. The gateway sorts providers by the chosen metric and tries them i…”
- [claimed-docs] “High reliability. Automatically retries requests to other providers if one fails”
- [claimed-docs] “To cap how much your team, a project, an API key, or a team member can spend, set [budgets]”
- [community] “If I bring my own Anthropic/OpenAI key and have usage available, why does Vercel's account balance determine whether my request executes? (V…”
Rate limits
platform engineerSmooth provider rate limits by spreading traffic across keys and queuing or throttling requests
weight 2 · round to PortkeyPortkey documents load balancing across multiple provider keys/models to avoid bottlenecks, automatic retries with exponential backoff or provider retry-after headers, and configurable per-key/org rate limits (hourly/daily/per-minute) alongside budget limits — directly addressing spreading traffic and throttling/queuing around rate limits. Multiple virtual keys per provider further support distributing load to smooth rate limits. Missing for 10: explicit first-party or independent evidence of a request queue (vs. retry/backoff) and hands-on/independent validation of these features working under real rate-limit pressure.
- [claimed-docs] “Distribute traffic across multiple LLMs to prevent any single provider from becoming a bottleneck.”
- [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
- [claimed-docs] “Enable `use_retry_after_headers` to use the provider’s `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”
- [claimed-docs] “Enable `use_retry_after_headers` to use the provider's `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
The gateway supports failover/retry across providers, per-provider timeouts for fast failover, and ordering/sorting of providers (docs-3, docs-9, docs-10, docs-13, docs-20, docs-29), which can mitigate a single provider's rate limits by shifting traffic elsewhere. However, there's no documented mechanism for spreading load across multiple API keys for the same provider, nor any explicit request queuing/throttling feature to smooth bursts against rate limits. Missing for 10: multi-key load balancing, explicit request queuing/throttling controls, and any rate-limit-specific documentation or independent verification.
- [claimed-docs] “High reliability. Automatically retries requests to other providers if one fails”
- [claimed-docs] “You can use `order`, `only`, and `sort` in `providerOptions.gateway` to control which providers handle your requests, in what order, and how…”
- [claimed-docs] “You can set per-provider timeouts to trigger fast failover when a provider is slow to respond.”
- [claimed-docs] “If a query using your credentials fails, AI Gateway will retry the query with its system credentials to improve service availability.”
- [claimed-docs] “You can also use `sort` to rank providers by a performance or cost metric. The gateway sorts providers by the chosen metric and tries them i…”
Retries
platform engineerSet automatic retry policies for transient provider errors
weight 2 · round drawnPortkey's docs explicitly describe automatic retries with exponential backoff and support for provider retry-after headers, directly matching the story of configuring retry policies for transient errors. missing for 10: independent/hands-on corroboration beyond vendor docs, and configuration details on per-error-code or max-retry-count customization.
- [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
- [claimed-docs] “Enable `use_retry_after_headers` to use the provider’s `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”
- [claimed-docs] “Enable `use_retry_after_headers` to use the provider's `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”
Docs explicitly describe automatic failover/retry to other providers on failure (docs-3), configurable provider ordering/fallback with `order`, `only`, `sort` (docs-9, docs-29), per-provider timeouts to trigger fast failover (docs-10), and even automatic retry with system credentials when BYOK fails (docs-13). This directly covers automatic retry policies for transient provider errors. Missing for 10: no independent/hands-on confirmation of retry behavior in production, and no explicit documentation of configurable retry counts/backoff intervals distinct from provider ordering.
- [claimed-docs] “High reliability. Automatically retries requests to other providers if one fails”
- [claimed-docs] “You can use `order`, `only`, and `sort` in `providerOptions.gateway` to control which providers handle your requests, in what order, and how…”
- [claimed-docs] “You can set per-provider timeouts to trigger fast failover when a provider is slow to respond.”
- [claimed-docs] “If a query using your credentials fails, AI Gateway will retry the query with its system credentials to improve service availability.”
- [claimed-docs] “You can also use `sort` to rank providers by a performance or cost metric. The gateway sorts providers by the chosen metric and tries them i…”
Streaming tools — stories about streaming tools in this arenaStreaming tools
Stories about streaming tools in this arena
Streaming
developerStream token-by-token responses through the gateway from any provider
weight 3 · round to Vercel AI GatewayPortkeynone0/10The evidence describes Portkey's gateway, fallbacks, retries, load balancing, caching, and OpenAI-compatible base URL integration, but none of the provided docs mention streaming or token-by-token response support. Streaming is a fair and expected capability for an LLM gateway product, so its absence in the evidence pack means this axis applies but is unproven.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
The gateway advertises full OpenAI/Anthropic SDK compatibility and AI SDK integration (docs-7, docs-18, docs-28, docs-31), and since streaming is a standard capability of those client libraries, it is implicitly supported when pointing existing clients at the gateway's base URL. However, no evidence explicitly documents streaming/token-by-token responses, SSE support, or shows this working across all providers. Missing for 10: explicit docs on streaming endpoints/parameters, hands-on confirmation of token-by-token streaming across multiple providers, and any independent verification.
- [claimed-docs] “You can use existing OpenAI client libraries, switch to AI Gateway with a URL change, and keep your current tools and workflows without code…”
- [claimed-docs] “Point your existing OpenAI or Anthropic SDK at AI Gateway. Same calls, no rewrites.”
- [claimed-docs] “You can use the AI Gateway's Chat Completions API with existing tools and libraries like the OpenAI client libraries and AI SDK. Point your …”
- [claimed-docs] “Move existing OpenAI, Anthropic, and AI SDK integrations to AI Gateway with a base URL swap.”
Tool calling
developerRequest structured JSON-schema outputs across providers
weight 1 · round drawnPortkeynone0/10The evidence pack covers gateway features like fallbacks, retries, load balancing, caching, virtual keys, and logging, but contains no mention of structured JSON-schema output support, response_format normalization, or schema validation across providers. Since Portkey acts as a universal AI gateway, this capability would be plausible, but no documentation cites it.
Vercel AI Gatewaynone0/10The evidence describes unified APIs, OpenAI-compatibility, provider routing, embeddings, and observability, but nowhere mentions structured JSON-schema/response_format output support or how it is normalized across different providers. Missing for 10: any documentation of JSON-schema/structured-output support, cross-provider schema normalization, or examples using response_format/json_schema.
developerMake tool and function calls across different providers with a consistent schema
weight 3 · round to Vercel AI GatewayPortkey's gateway offers a unified OpenAI-SDK-compatible API (base URL + headers) across 250+ models, which implies a consistent request format including function/tool call fields, and it supports connecting external tools via remote MCP servers. However, there is no explicit documentation describing how tool/function-calling schemas are normalized or translated across different providers' native formats. Missing for 10: explicit docs on function/tool-call schema normalization, examples of tool-calling requests across multiple providers, and independent verification that tool calls work consistently across providers.
- [claimed-docs] “It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in y…”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “Connect to Remote MCP severs, allowing you to connect external tools and data sources.”
Vercel AI Gateway supports a unified API/schema (OpenAI-compatible and AI SDK) that lets developers call tools/functions across many providers with consistent code, including an explicit ToolLoopAgent example with tool calling, provider routing/fallback, and no-rewrite compatibility with existing OpenAI/Anthropic clients. missing for 10: independent hands-on verification of cross-provider tool-call schema consistency (e.g., confirming tool-calling behaves identically across Anthropic/OpenAI/other models) and any documented edge cases or limitations in function-calling parity across providers.
- [claimed-docs] “Unified API. Switch between providers and models with minimal code changes”
- [claimed-docs] “You can use existing OpenAI client libraries, switch to AI Gateway with a URL change, and keep your current tools and workflows without code…”
- [claimed-docs] “Point your existing OpenAI or Anthropic SDK at AI Gateway. Same calls, no rewrites.”
- [claimed-docs] “const agent = new ToolLoopAgent({ model: 'anthropic/claude-sonnet-5', tools: { getWeather: tool({”
- [claimed-docs] “You can use the AI Gateway's Chat Completions API with existing tools and libraries like the OpenAI client libraries and AI SDK. Point your …”
- [probe] “PROBE keyless-endpoint: GET https://ai-gateway.vercel.sh/v1/models returned HTTP 200 application/json with an OpenAI-compatible {"object":"l…”
Unified api — stories about unified api in this arenaUnified api
Stories about unified api in this arena
Catalog
developerBrowse or query a catalog of available models with pricing and context-window metadata
weight 2 · round to Vercel AI GatewayPortkey documents a 'Model Catalog' providing a single pane to view/manage all providers and models with discovery and governance controls, and claims access to 250+ models, but no evidence shows the catalog surfaces pricing or context-window metadata for models. missing for 10: explicit documentation of per-model pricing info, explicit documentation of context-window size metadata, and any UI/query example showing these fields.
- [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
- [claimed-docs] “It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in y…”
- [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
Docs explicitly describe a models endpoint returning model IDs, context windows, and pricing with no authentication required, and a probe independently confirms the live unauthenticated /v1/models endpoint returns a full JSON model catalog. This directly satisfies browsing/querying a model catalog with pricing and context-window metadata. Missing for 10: no evidence of a dedicated visual browsing UI/dashboard for the catalog beyond the API and docs page.
- [claimed-docs] “This endpoint requires no authentication and returns detailed information including model IDs, context windows, and pricing.”
- [claimed-docs] “Query the models endpoint directly to get a JSON list of all available models with pricing and capabilities”
- [probe] “PROBE keyless-endpoint: GET https://ai-gateway.vercel.sh/v1/models returned HTTP 200 application/json with an OpenAI-compatible {"object":"l…”
Compatibility
developerPoint existing OpenAI-compatible code at the gateway by changing only the base URL and key
weight 3 · round drawnDocs explicitly state that developers using OpenAI's Python/Node SDKs can integrate Portkey by only changing the base URL and adding Portkey-specific headers, with the base URL documented as https://api.portkey.ai/v1. This is a first-party documented, purpose-built migration path. Missing for 10: independent/hands-on third-party confirmation beyond vendor docs.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
Vercel explicitly documents OpenAI-compatible Chat Completions API where developers just swap the base URL and key with existing OpenAI SDKs, with no code rewrites, and a probe confirms the gateway serves an OpenAI-compatible /v1/models JSON response. Community evidence around BYOK balance behavior is unrelated to base-URL compatibility, so it doesn't dispute this. Missing for 10: independent third-party hands-on confirmation of a full migration beyond Vercel's own docs/probes.
- [claimed-docs] “You can use existing OpenAI client libraries, switch to AI Gateway with a URL change, and keep your current tools and workflows without code…”
- [claimed-docs] “Point your existing OpenAI or Anthropic SDK at AI Gateway. Same calls, no rewrites.”
- [claimed-docs] “You can use the AI Gateway's Chat Completions API with existing tools and libraries like the OpenAI client libraries and AI SDK. Point your …”
- [claimed-docs] “Move existing OpenAI, Anthropic, and AI SDK integrations to AI Gateway with a base URL swap.”
- [probe] “PROBE keyless-endpoint: GET https://ai-gateway.vercel.sh/v1/models returned HTTP 200 application/json with an OpenAI-compatible {"object":"l…”
One endpoint
developerCall many model providers through one consistent API
weight 3 · round drawnPortkey documents a unified API/gateway (single base URL, single API key) that supports 250+ providers/models via consistent SDK/REST calls, with virtual keys and model catalog abstracting provider differences. Missing for 10: independent third-party benchmark or hands-on confirmation of the claimed 250+ provider coverage beyond vendor docs.
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in y…”
- [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
Extensive first-party docs describe a single unified API/key across hundreds of providers, OpenAI-compatible client support, and provider routing/fallback controls, and a live probe confirms the OpenAI-compatible /v1/models endpoint works unauthenticated. There is a minor community-reported caveat about BYOK requests being blocked when Vercel balance hits $0, which slightly limits the 'bring your own key' flow but doesn't contradict the core unified-API story. Missing for 10: independent hands-on developer confirmation of multi-provider switching in production beyond docs and one probe.
- [claimed-docs] “One key, hundreds of models. Access models from multiple providers with a single API key”
- [claimed-docs] “Unified API. Switch between providers and models with minimal code changes”
- [claimed-docs] “You can use existing OpenAI client libraries, switch to AI Gateway with a URL change, and keep your current tools and workflows without code…”
- [claimed-docs] “Point your existing OpenAI or Anthropic SDK at AI Gateway. Same calls, no rewrites.”
- [claimed-docs] “You can use the AI Gateway's Chat Completions API with existing tools and libraries like the OpenAI client libraries and AI SDK. Point your …”
- [probe] “PROBE keyless-endpoint: GET https://ai-gateway.vercel.sh/v1/models returned HTTP 200 application/json with an OpenAI-compatible {"object":"l…”
- [community] “If I bring my own Anthropic/OpenAI key and have usage available, why does Vercel's account balance determine whether my request executes? (V…”
Not comparable on these axes
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · not comparablePortkeynone0/10Portkey's evidence pack covers gateway routing, fallbacks, caching, logging, and budget controls, but there is no mention of AI-generated insights, summaries, or suggestions derived from a user's data within the product itself — logs support manual feedback tagging (portkey-docs-34), not AI-driven analysis. missing for 10: any AI-generated insight/summary feature, anomaly detection or recommendation engine over logs/usage data, in-product AI assistant surfacing suggestions.
- [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
- [claimed-docs] “you can also add manual feedback on the logs to be analysed and filtered later”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
Vercel AI Gatewayn/aVercel AI Gateway is developer infrastructure that routes and manages LLM API calls between an application and model providers; it has no end-user product surface or stored user data of its own from which to surface insights. This capability belongs to applications built on top of the gateway, not the gateway itself, so the axis is a category error here.
ai-native userSet up automations that run autonomously in the background
weight 2 · not comparablePortkeyn/aPortkey is an AI gateway/observability layer for routing, caching, and governing LLM requests, not a workflow/automation platform for running autonomous background tasks; the evidence covers gateway features (fallbacks, retries, caching, load balancing, logs, budgets) with no scheduler or autonomous-agent execution capability. This story targets a different product category and does not fit Portkey's axis.
Vercel AI Gatewayn/aVercel AI Gateway is a model-routing/API gateway product (unified API, failover, spend monitoring, observability) — it is not an automation/orchestration platform with schedulers, triggers, or persistent background execution. Building or running autonomous background automations is outside its product category; that capability would live in a separate agent/orchestration tool that might use the Gateway for model calls.
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · not comparablePortkeyn/aPortkey is an AI gateway/observability platform for routing and managing LLM traffic, not a product with a built-in chat/assistant UI that a user delegates tasks to; this axis is a category error for its product type.
Vercel AI Gatewayn/aVercel AI Gateway is an infrastructure/routing layer for LLM API calls, not a product with a built-in AI assistant UI for users to delegate tasks to; it enables developers to build agents (e.g., ToolLoopAgent) but does not itself ship a first-party assistant persona for end users.
ai-native userOperate the product with natural-language commands
weight 2 · not comparablePortkeynone0/10Portkey's evidence describes an API/SDK-based AI gateway, dashboard configs, logs, and MCP server connectivity for tools/data sources, but nothing shows an interface where a user issues natural-language commands to operate Portkey itself (e.g., a chat-based admin/control plane). The axis is plausible for a platform like this, but no evidence supports it.
Vercel AI Gatewayn/aVercel AI Gateway is developer infrastructure (API routing/failover/billing for LLM providers) that is configured via code, dashboard, and API calls — not an interactive assistant meant to be 'operated' via natural-language commands. Evidence about coding-agent connectivity and ToolLoopAgent describes building agents that use the gateway, not commanding the gateway itself in natural language, so this axis is a category mismatch for this product type.
ai-native userTest against a sandbox environment without touching production data
weight 1 · not comparablePortkeynone0/10Portkey's evidence covers gateway routing, fallbacks, caching, logging, and gradual/canary rollout of models in production, but there is no mention of a distinct sandbox/staging environment or test mode that isolates a user from production data. The axis is fair for a gateway/dev-tool product, but no evidence supports it.
Vercel AI Gatewayn/aAI Gateway is a model-routing/API proxy layer, not a data-plane product with production datasets or sandbox/staging environments to isolate from production data; the story's premise (sandbox vs production data) doesn't map onto its category. No evidence describes environment separation or sandbox testing modes, confirming this axis is a category mismatch rather than a gap.
ai-native userSchedule recurring jobs or workflows
weight 2 · not comparablePortkeyn/aPortkey is an AI gateway/observability platform for routing, caching, and monitoring LLM requests, not a workflow/job scheduling or orchestration product; nothing in the evidence pack relates to scheduling recurring jobs or workflows, making this a category mismatch rather than a missing feature.
Vercel AI Gatewayn/aVercel AI Gateway is a request-routing/proxy layer for LLM API calls (unified API, retries, budgets, observability) — it has no scheduler, cron, or workflow orchestration feature for recurring jobs. This is a category mismatch, not a missing capability of this product type.
ai-native userVersion, review, and roll back my automations
weight 1 · not comparablePortkeynone0/10Portkey's evidence covers gateway configs, logs, retries, caching, and budgets, but nothing describes versioning, review workflows, or rollback of automations/configs — the closest hint is 'Config ID' filtering in logs, which is not version control. Missing for 10: any documentation of config/version history, diff/review process, or rollback mechanism for automations.
- [claimed-docs] “Filter logs by `Config ID` to see all requests using that config”
- [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
- [claimed-docs] “Each log on Portkey has a unique URL. You can copy the link from the address bar and directly share it with anyone in your org.”