Model Gateways & Routers Arena
Portkey vs Kong AI Gateway
Portkey
Portkey, Inc.
Portkey wins · 21–10 (15 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round drawnPortkey hosts a working llms.txt file (HTTP 200) with a description of the product, confirming agents can be pointed at it for agent-oriented docs. Missing for 10: no docs.md fallback (404) or OpenAPI spec confirmation, and no independent/community corroboration of agents actually using it.
A probe confirms developer.konghq.com/llms.txt returns HTTP 200 with a valid llms.txt manifest listing Kong's docs (including AI Gateway), and individual doc pages are also served as .md files (e.g. ai-providers.md, a2a.md), making the docs directly agent-consumable. missing for 10: no explicit vendor statement instructing agents to use llms.txt, and the generic /.md probe for the ai-gateway index returned 404, showing coverage is per-page rather than fully uniform.
- [probe] “PROBE llms.txt: HTTP 200 at https://developer.konghq.com/llms.txt # developer.konghq.com > Kong's developer documentation for Kong Gateway,…”
- [claimed-docs] “The core of AI Gateway is the ability to serve AI Models from various AI Model Providers via a provider-agnostic API.”
- [probe] “PROBE docs-md: HTTP 404 at https://developer.konghq.com/ai-gateway/.md”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to Kong AI GatewayPortkey's gateway can be run headlessly via a CLI command (npx @portkey-ai/gateway) and REST API with a base URL, and it is open source and self-hostable, which supports CI/automation usage. However, there is no explicit documentation of CI pipeline integration, GitHub Actions examples, or headless automation testing workflows. missing for 10: explicit CI/CD examples, headless automation guides, non-interactive scripted usage documentation, independent verification of CI usage.
- [claimed-docs] “npx @portkey-ai/gateway”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
Kong AI Gateway exposes a CLI (kongctl) and Konnect APIs (list-ai-gateways, config conversion) plus a quickstart script, implying it can be driven headlessly and scripted rather than only via UI. However there is no explicit CI or CD pipeline example, declarative config-as-code workflow, or GitOps documentation demonstrating automated headless operation in CI. Missing for 10: explicit CI or CD pipeline documentation, dedicated headless-mode guide, independent CI usage report.
- [claimed-docs] “list-ai-gateways summary: List AI Gateways description: Returns a list of AI Gateways in the organization.”
- [claimed-docs] “walks you through migrating an existing configuration using the kongctl AI Gateway conversion extension”
- [claimed-docs] “You can use the quickstart script to get a demo instance of AI Gateway running almost instantly.”
- [probe] “official CLI documented at https://developer.konghq.com/kongctl/”
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · round drawnPortkey's docs explicitly state it can 'Connect to Remote MCP servers, allowing you to connect external tools and data sources' via the AI Gateway, directly supporting the story. However, evidence lacks detail on setup steps, supported MCP transport, or independent/hands-on confirmation of this working in practice. Missing for 10: detailed configuration docs for adding MCP servers, independent verification of MCP tool invocation, and examples of tool-calling via MCP in a live agent workflow.
- [claimed-docs] “Connect to Remote MCP severs, allowing you to connect external tools and data sources.”
Kong AI Gateway documents an 'AI MCP Proxy' capability that explicitly can front upstream MCP servers (in addition to converting REST APIs into MCP tools), meaning it can plug in external MCP servers so the traffic/tools flow through the gateway. However, the evidence is a single brief concept mention with no detailed configuration walkthrough, no example of registering a specific upstream MCP server, and no independent/hands-on corroboration. Missing for 10: a full docs page on connecting/registering upstream MCP servers, concrete config examples, and third-party verification.
- [claimed-docs] “AI MCP Proxy bridges Kong-managed Services to the Model Context Protocol, converting REST APIs into MCP tools or fronting upstream MCP serve…”
- [claimed-docs] “Turn any API into an MCP server using the AI MCP Server entity. This approach does not require an LLM and provides full control over product…”
- [claimed-docs] “Govern LLM, MCP, and agent-to-agent (A2A) traffic with the same Kong AI Gateway.”
ai-native userConnect an agent via an official MCP server
weight 3 · round to Kong AI GatewayPortkeynone0/10Portkey's docs describe connecting *to* remote MCP servers as a client-side feature (portkey-docs-17), but there is no evidence that Portkey itself exposes an official MCP server endpoint that external agents can connect to. Missing for 10: any documentation of Portkey hosting/serving an MCP endpoint, MCP server setup instructions, or third-party confirmation of agents connecting via Portkey's MCP server.
- [claimed-docs] “Connect to Remote MCP severs, allowing you to connect external tools and data sources.”
Kong AI Gateway ships an official 'AI MCP Server' entity and 'AI MCP Proxy' that converts REST APIs into MCP tools or fronts upstream MCP servers, letting agents connect via a first-party MCP interface, plus governance of MCP traffic (auth, OTel metrics, rate limiting) alongside it. Missing for 10: independent/hands-on verification of an agent successfully connecting through the MCP server and no example configuration walkthrough shown in the pack.
- [claimed-docs] “AI MCP Proxy bridges Kong-managed Services to the Model Context Protocol, converting REST APIs into MCP tools or fronting upstream MCP serve…”
- [claimed-docs] “Turn any API into an MCP server using the AI MCP Server entity. This approach does not require an LLM and provides full control over product…”
- [claimed-docs] “Govern LLM, MCP, and agent-to-agent (A2A) traffic with the same Kong AI Gateway.”
- [claimed-docs] “AI Gateway can export OpenTelemetry (OTLP) metrics for generative AI, MCP, and A2A traffic through an OpenTelemetry AI Policy.”
ai-native userUse an official CLI
weight 2 · round to Kong AI GatewayThe only CLI-like evidence is running the open-source gateway locally via `npx @portkey-ai/gateway`, which self-hosts the gateway rather than offering a full-featured CLI for managing configs, logs, or other Portkey resources. Missing for 10: a documented dedicated CLI tool/command set, CLI-based workflow examples, and independent corroboration of CLI usage.
- [claimed-docs] “npx @portkey-ai/gateway”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
Kong ships an official CLI, kongctl, documented at a dedicated docs URL and explicitly used for AI Gateway tasks (e.g., the 'kongctl AI Gateway conversion extension' for migrating configurations). This confirms a real, first-party CLI exists for AI Gateway workflows. Missing for 10: deeper documentation of kongctl's full command surface for AI-native tasks (agents, MCP, A2A) and independent/hands-on corroboration beyond the docs probe.
- [claimed-docs] “walks you through migrating an existing configuration using the kongctl AI Gateway conversion extension”
- [probe] “official CLI documented at https://developer.konghq.com/kongctl/”
ai-native userDrive the product through a documented public API
weight 3 · round to PortkeyPortkey exposes a documented REST API (base URL https://api.portkey.ai/v1) with API-reference docs for its inference API, OpenAI-compatible integration, and llms.txt discovery confirms API-first positioning; gateway is also open-sourced and npx-runnable. Missing for 10: a discoverable OpenAPI/swagger spec (probe found all candidate paths 404) and independent/third-party corroboration of API usage.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
- [probe] “PROBE llms.txt: HTTP 200 at https://portkey.ai/llms.txt # Portkey > Portkey is a production stack for Gen AI builders, equipping AI teams w…”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
Kong documents a Konnect API for managing AI Gateway resources (e.g., list-ai-gateways endpoint) and a dedicated kongctl CLI, showing programmatic control exists and is documented. However, probes for a discoverable OpenAPI/swagger spec at standard locations all returned 404, so the breadth/openness of the API surface is not independently confirmed. Missing for 10: a publicly discoverable OpenAPI/swagger spec, broader endpoint coverage beyond the single example, and independent/hands-on confirmation of API usage.
- [claimed-docs] “list-ai-gateways summary: List AI Gateways description: Returns a list of AI Gateways in the organization.”
- [probe] “official CLI documented at https://developer.konghq.com/kongctl/”
- [probe] “PROBE openapi: all candidate paths 404 (https://developer.konghq.com/openapi.json, https://developer.konghq.com/swagger.json, https://develo…”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to PortkeyPortkey lets admins issue API keys scoped with budgets, rate limits, and model allow-lists at org/workspace level, and virtual keys keep provider credentials out of code — this maps well to least-privilege credential issuance. However, the docs frame this around human/org governance rather than explicitly per-agent scoped credentials, and there's no independent corroboration of enforcement in practice. Missing for 10: explicit agent-specific credential scoping workflow, independent/hands-on validation of enforcement.
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
- [claimed-docs] “Provider credentials stored securely, never exposed in code”
Kong AI Gateway provides building blocks relevant to credential control for agents — AI Auth Strategy entities can authenticate via enterprise IdPs (Okta/Azure AD/OIDC) 'without managing keys manually,' and AI Vault lets model providers, auth strategies, agents, and MCP servers reference secrets from external backends instead of storing literal keys, plus Azure Managed Identity support for scoped cloud authentication. However, none of this evidence explicitly describes issuing scoped/least-privilege credentials per agent (e.g., fine-grained permission scopes, per-agent token minting) — it's centralized secret referencing and IdP auth rather than a dedicated least-privilege credential-issuance mechanism. missing for 10: explicit documentation of scoped/least-privilege credential issuance per agent, granular permission/scope controls tied to individual agent identities, and independent/hands-on verification of this behavior.
- [claimed-docs] “An AI Vault entity lets you register an external secret backend (AWS Secrets Manager, HashiCorp Vault, environment variables, or others) so …”
- [claimed-docs] “Authenticate enterprise users through an existing identity provider (Okta, Azure AD, Google, or any OIDC-compliant IdP) without managing key…”
- [claimed-docs] “Yes, if AI Gateway is running on Azure, you can configure an AI Model Provider to detect the designated Managed Identity or User-Assigned Id…”
ai-native userBuild against official SDKs
weight 2 · round to PortkeyPortkey documents SDK-style integration (base URL swap for OpenAI Python/Node SDKs, REST API reference, open-source gateway installable via npx) supporting AI-native builders, but the evidence pack lacks explicit mention of dedicated first-party SDKs beyond OpenAI-compatible drop-in usage, and OpenAPI spec probes returned 404s. missing for 10: dedicated official multi-language SDK docs (e.g., Python/JS Portkey SDK reference pages), working OpenAPI/spec discovery, independent developer corroboration of SDK usage.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
Kong AI Gatewaynone0/10The evidence pack contains no mention of official SDKs for building against Kong AI Gateway (only a CLI 'kongctl' and REST API references), and probes explicitly show no OpenAPI/SDK artifacts (404s for openapi.json, swagger.json, etc.). While an API gateway could plausibly offer official client SDKs, none are documented here.
- [probe] “PROBE openapi: all candidate paths 404 (https://developer.konghq.com/openapi.json, https://developer.konghq.com/swagger.json, https://develo…”
- [probe] “official CLI documented at https://developer.konghq.com/kongctl/”
- [claimed-docs] “list-ai-gateways summary: List AI Gateways description: Returns a list of AI Gateways in the organization.”
Agentic features
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · round drawnPortkeynone0/10Portkey's evidence pack covers gateway routing, fallbacks, caching, logging, and budget controls, but there is no mention of AI-generated insights, summaries, or suggestions derived from a user's data within the product itself — logs support manual feedback tagging (portkey-docs-34), not AI-driven analysis. missing for 10: any AI-generated insight/summary feature, anomaly detection or recommendation engine over logs/usage data, in-product AI assistant surfacing suggestions.
- [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
- [claimed-docs] “you can also add manual feedback on the logs to be analysed and filtered later”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
Kong AI Gatewaynone0/10Kong AI Gateway provides usage analytics, logs, and cost/latency metrics (docs-3, docs-9, docs-10), but these are raw operational metrics/dashboards, not AI-generated insights or suggestions derived from the user's own data. No evidence shows the product itself using AI to analyze a customer's traffic/data and proactively surface recommendations or insights.
- [claimed-docs] “See usage analytics for requests, tokens, errors, and latency”
- [claimed-docs] “You can use these metrics to: Track LLM request latency and upstream provider processing time * Monitor token consumption ... * Calculate AI…”
- [claimed-docs] “AI Gateway logs capture detailed information about the request and response payloads, token usage, model details, latency, and cost metrics.”
ai-native userOperate the product with natural-language commands
weight 2 · round drawnPortkeynone0/10Portkey's evidence describes an API/SDK-based AI gateway, dashboard configs, logs, and MCP server connectivity for tools/data sources, but nothing shows an interface where a user issues natural-language commands to operate Portkey itself (e.g., a chat-based admin/control plane). The axis is plausible for a platform like this, but no evidence supports it.
Kong AI Gatewaynone0/10Kong AI Gateway's documentation covers proxying LLM/CLI/A2A/MCP traffic, observability, and cost tracking, but there is no evidence that the gateway itself can be configured or operated via natural-language commands (its control plane relies on kongctl CLI and declarative config, not NL commands).
- [probe] “official CLI documented at https://developer.konghq.com/kongctl/”
- [claimed-docs] “walks you through migrating an existing configuration using the kongctl AI Gateway conversion extension”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round to PortkeyPortkey documents a REST API reference (base URL, headers, SDK compatibility) at /docs/api-reference, but the evidence pack contains no mention of an interactive explorer or runnable code examples, and probes for a discoverable OpenAPI/Swagger spec all returned 404s. Missing for 10: evidence of an interactive 'try it' console, runnable code snippets, and a published OpenAPI/Swagger schema.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
Kong AI Gatewaynone0/10Evidence shows Kong publishes API reference content (e.g., list-ai-gateways endpoint docs) but there is no indication of an interactive reference with runnable/try-it-out examples; probes for OpenAPI/swagger specs on the docs site all returned 404, suggesting no such interactive tooling is exposed.
- [claimed-docs] “list-ai-gateways summary: List AI Gateways description: Returns a list of AI Gateways in the organization.”
- [probe] “PROBE openapi: all candidate paths 404 (https://developer.konghq.com/openapi.json, https://developer.konghq.com/swagger.json, https://develo…”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round drawnPortkeynone0/10Portkey has an API reference (base URL, inference API docs) but no evidence of a downloadable OpenAPI/Swagger spec; a direct probe for common OpenAPI paths returned 404s, and docs.md also 404s.
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
- [probe] “PROBE docs-md: HTTP 404 at https://portkey.ai/docs.md”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
Kong AI Gatewaynone0/10Evidence includes API reference pages (e.g. Konnect AI Gateway API docs) but no downloadable OpenAPI/Swagger spec file is evidenced; a direct probe for standard OpenAPI/swagger.json locations returned 404 at all candidate paths, indicating no machine-readable spec is exposed there.
- [probe] “PROBE openapi: all candidate paths 404 (https://developer.konghq.com/openapi.json, https://developer.konghq.com/swagger.json, https://develo…”
- [claimed-docs] “list-ai-gateways summary: List AI Gateways description: Returns a list of AI Gateways in the organization.”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round drawnPortkeynone0/10Portkey's evidence covers gateway routing, fallbacks, caching, logging, and gradual/canary rollout of models in production, but there is no mention of a distinct sandbox/staging environment or test mode that isolates a user from production data. The axis is fair for a gateway/dev-tool product, but no evidence supports it.
Kong AI Gatewaynone0/10The evidence pack only shows a quickstart script for spinning up a demo instance and general AI Gateway features (proxying, observability, policies) but no dedicated sandbox/staging environment or test-mode capability that isolates from production data. No mention of environment separation, mock upstreams, or a documented sandbox mode.
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnPortkeynone0/10No evidence of API versioning scheme (e.g., v1/v2 in URLs beyond a single 'v1' base path) or any documented deprecation policy/changelog; OpenAPI spec probe also returned 404s. Missing for 10: versioning scheme documentation, explicit deprecation policy, changelog/migration guides, and any independent corroboration.
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
Kong AI Gatewaynone0/10The evidence shows an API reference exists (e.g., 'v1' Konnect AI Gateway API) but there is no documentation of a versioning scheme or deprecation policy for the AI Gateway APIs, and OpenAPI spec probes returned 404s.
- [claimed-docs] “list-ai-gateways summary: List AI Gateways description: Returns a list of AI Gateways in the organization.”
- [probe] “PROBE openapi: all candidate paths 404 (https://developer.konghq.com/openapi.json, https://developer.konghq.com/swagger.json, https://develo…”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round drawnPortkeynone0/10Portkey's evidence covers gateway routing, fallbacks, caching, logging, and budget/rate-limit controls, but nothing describes a bulk/batch operation capability (e.g., batch inference calls, bulk editing of configs/keys, or bulk log actions) that lets a user act on many items at once. Missing for 10: any documentation of a batch API, bulk config/key management, or bulk log/annotation actions.
Kong AI Gatewaynone0/10The evidence covers proxying, observability, security, and MCP/A2A support but never describes any bulk or batch operation capability (e.g., batch creating/updating routes, plugins, or AI entities across many items at once). The kongctl CLI and Konnect API are mentioned but only for single-entity or config migration tasks, not bulk item processing.
- [claimed-docs] “list-ai-gateways summary: List AI Gateways description: Returns a list of AI Gateways in the organization.”
- [claimed-docs] “walks you through migrating an existing configuration using the kongctl AI Gateway conversion extension”
- [probe] “official CLI documented at https://developer.konghq.com/kongctl/”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to PortkeyPortkey ships several built-in rule/event mechanisms — fallback triggers on non-2xx status codes, automatic retries with backoff, circuit-breaker/failure handling per strategy, and budget/rate-limit thresholds that automatically send notifications or block further usage — which are automatic actions triggered by defined conditions. However, these are fixed, product-defined automations (reliability/spend rules) rather than a general user-authorable rule engine where an AI-native user can define arbitrary custom triggers/actions. Missing for 10: general-purpose custom rule/trigger builder, arbitrary event types beyond status codes/spend thresholds, user-defined action targets (e.g. webhooks, custom scripts) beyond built-in notifications/fallback/retry.
- [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
- [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
- [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
- [claimed-docs] “Configure per-strategy circuit protection and failure handling”
- [claimed-docs] “When usage reaches this threshold, notifications will be sent to configured recipients”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
Kong AI Gateway supports policy-based automatic actions triggered by request/response events—e.g., Prompt Guards and content-safety policies acting on chat/completion requests, AI Rate Limiting Advanced enforcing limits automatically, AWS Guardrails validating requests/responses before forwarding, and semantic caching/routing acting on similarity triggers—which functions as rule-based automation on traffic events. However, this is scoped to gateway traffic policies rather than a general-purpose event/trigger automation framework for arbitrary conditions and actions. Missing for 10: evidence of a generic conditional rule engine (if-this-then-that style), user-defined custom triggers beyond built-in policies, and any workflow/orchestration automation beyond request-level policy enforcement.
- [claimed-docs] “AI Gateway supports content safety features across providers and also includes our Prompt Guards that act on any llm/v1/chat or llm/v1/compl…”
- [claimed-docs] “AI Rate Limiting Advanced - Plugin | Kong Docs”
- [claimed-docs] “Apply safety and DLP policies to prevent toxic content and remove personally identifiable information.”
- [claimed-docs] “AI Gateway can perform intelligent request routing, caching, and content filtering using semantic similarity queries.”
- [claimed-docs] “Use AWS Guardrails to validate requests and/or responses before forwarding them between clients and upstream LLMs.”
ai-native userVersion, review, and roll back my automations
weight 1 · round drawnPortkeynone0/10Portkey's evidence covers gateway configs, logs, retries, caching, and budgets, but nothing describes versioning, review workflows, or rollback of automations/configs — the closest hint is 'Config ID' filtering in logs, which is not version control. Missing for 10: any documentation of config/version history, diff/review process, or rollback mechanism for automations.
- [claimed-docs] “Filter logs by `Config ID` to see all requests using that config”
- [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
- [claimed-docs] “Each log on Portkey has a unique URL. You can copy the link from the address bar and directly share it with anyone in your org.”
Kong AI Gatewaynone0/10The evidence pack covers AI Gateway's traffic proxying, observability, security, and provider-agnostic API features, but contains no mention of versioning, reviewing, or rolling back gateway configurations or 'automations' (e.g., no changelog, approval workflow, or rollback mechanism is documented). The only tangential item is a config migration extension (kongctl conversion tool), which addresses format conversion, not version control or rollback. missing for 10: version history/changelog for AI Gateway configs, review/approval workflow for changes, rollback mechanism for policies or entities.
- [claimed-docs] “walks you through migrating an existing configuration using the kongctl AI Gateway conversion extension”
Caching performance — stories about caching performance in this arenaCaching performance
Stories about caching performance in this arena
Caching
developerCache responses at the gateway to cut cost and latency on repeated requests
weight 2 · round to PortkeyPortkey's docs explicitly describe gateway-level caching (simple and semantic) that serves responses up to 20x faster and cheaper, including semantic matching via cosine similarity for near-duplicate queries. This directly matches the story's cost/latency reduction on repeated requests. missing for 10: independent/hands-on benchmark corroboration beyond vendor docs.
- [claimed-docs] “Cache LLM responses to serve requests up to **20x faster** and cheaper.”
- [claimed-docs] “Matches requests with **similar meaning** using cosine similarity, not just identical text.”
Kong ships a dedicated AI Semantic Cache plugin explicitly designed to cache LLM responses at the gateway, and separate docs confirm gateway-level semantic caching to reduce redundant calls and improve latency/cost. This is first-party documentation with a named product feature, though no independent benchmark or hands-on cost/latency measurement is provided. Missing for 10: independent/hands-on validation of actual cost/latency savings, and exact-match (non-semantic) cache configuration details.
- [claimed-docs] “AI Semantic Cache - Plugin | Kong Docs”
- [claimed-docs] “AI Gateway can perform intelligent request routing, caching, and content filtering using semantic similarity queries.”
Latency
platform engineerRun traffic through gateway infrastructure that adds minimal latency overhead to provider calls
weight 2 · round to Kong AI GatewayPortkey's docs claim caching can serve responses '20x faster' and market the gateway as 'performant,' but there is no direct benchmark or first-party claim quantifying the gateway's own added latency overhead per provider call, nor independent corroboration of low overhead. Missing for 10: explicit latency/overhead benchmarks for the proxy itself, third-party performance testing, and architecture details (e.g., edge/WASM deployment) that would substantiate minimal added latency beyond the cache-hit speedup claim.
- [claimed-docs] “Cache LLM responses to serve requests up to **20x faster** and cheaper.”
- [claimed-docs] “It takes 2 mins to integrate and with that, it starts monitoring all of your LLM requests and makes your app resilient, secure, performant, …”
- [claimed-docs] “offering advanced tools for control, visibility, and security in your Generative AI apps”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
Kong documents latency-related tooling (usage analytics and OTel metrics that track LLM request latency and upstream processing time, semantic caching, load balancing across models) which implies attention to performance, but there is no explicit claim or benchmark quantifying the gateway's own added latency overhead as 'minimal'. missing for 10: quantified latency-overhead benchmarks, independent performance tests, explicit vendor claim of low proxy overhead.
- [claimed-docs] “You can use these metrics to: Track LLM request latency and upstream provider processing time * Monitor token consumption ... * Calculate AI…”
- [claimed-docs] “AI Semantic Cache - Plugin | Kong Docs”
- [claimed-docs] “AI Gateway provides load balancing capabilities to distribute requests across multiple LLM models.”
- [claimed-docs] “See usage analytics for requests, tokens, errors, and latency”
Cost controls — stories about cost controls in this arenaCost controls
Stories about cost controls in this arena
Agent guardrails
ai-native userGive an autonomous agent its own key with budget and rate guardrails so it cannot run away on spend
weight 2 · round to PortkeyPortkey docs explicitly support issuing API keys with budget and rate limits (spend thresholds, hourly/daily/per-minute request/token limits) that automatically block further usage when exceeded, directly matching the guardrail need for an autonomous agent's key. Fine-grained control at org/workspace level and per-key configuration is documented. Missing for 10: no explicit mention of a per-agent key issuance workflow or independent/hands-on verification that the cutoff actually stops an agent mid-run.
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
- [claimed-docs] “**Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC”
Kong AI Gateway supports per-consumer/agent authentication (AI Auth Strategy, AI Vault for keys) plus AI Rate Limiting Advanced and model cost calculation/tracking that matches provider billing, which together let an operator scope an agent's key and cap its request rate and see accrued spend. However, evidence shows cost *calculation and observability* (docs-9, docs-10, docs-18) rather than an explicit hard budget cap/kill-switch that automatically cuts off an agent once a spend threshold is reached, so the 'cannot run away on spend' guarantee is only partially evidenced. Missing for 10: explicit documented budget-threshold enforcement/auto-cutoff tied to a specific agent key, and a worked example of issuing a scoped key to an autonomous agent with combined rate+spend limits.
- [claimed-docs] “AI Rate Limiting Advanced - Plugin | Kong Docs”
- [claimed-docs] “AI Gateway calculates that figure for every request to a large language model (LLM) provider, matching what the provider actually bills.”
- [claimed-docs] “An AI Vault entity lets you register an external secret backend (AWS Secrets Manager, HashiCorp Vault, environment variables, or others) so …”
- [claimed-docs] “Authenticate enterprise users through an existing identity provider (Okta, Azure AD, Google, or any OIDC-compliant IdP) without managing key…”
- [claimed-docs] “You can use these metrics to: Track LLM request latency and upstream provider processing time * Monitor token consumption ... * Calculate AI…”
Budgets
platform engineerSet hard budgets and spend limits per key, team, or user
weight 3 · round to PortkeyPortkey's docs explicitly describe setting budget limits (max spend or token usage) and rate limits on API keys, with org/workspace-level fine-grained control, automatic enforcement when limits are reached, notification thresholds, and weekly resets — directly matching per-key/team/user budget/spend controls. Missing for 10: no independent/hands-on corroboration beyond first-party docs.
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “When usage reaches this threshold, notifications will be sent to configured recipients”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
- [claimed-docs] “**Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
Kong AI Gateway provides cost calculation, tracking across teams, and rate limiting plugins (AI Rate Limiting Advanced) which can enforce request/token limits per consumer, but evidence shows rate limiting and cost tracking as separate capabilities rather than a unified hard 'budget/spend limit' enforcement tied directly to cost thresholds per key/team/user. missing for 10: explicit documentation of setting a dollar-based spend cap or budget that auto-blocks requests once exceeded, per-key/team/user budget configuration UI or API, and independent confirmation of budget enforcement in production.
- [claimed-docs] “AI Rate Limiting Advanced - Plugin | Kong Docs”
- [claimed-docs] “AI Gateway calculates that figure for every request to a large language model (LLM) provider, matching what the provider actually bills.”
- [claimed-docs] “AI Gateway can proxy requests from AI command-line tools to LLM providers. This gives you centralized control over AI traffic: log all reque…”
- [claimed-docs] “You can use these metrics to: Track LLM request latency and upstream provider processing time * Monitor token consumption ... * Calculate AI…”
Spend tracking
platform engineerTrack spend per model, key, team, or user across all providers in one place
weight 3 · round to PortkeyPortkey's docs describe budget/rate limits and spend caps on API keys, a unified model catalog with org/workspace-level governance and allow-lists, and logs filterable by Config ID for auditing usage, which together give spend visibility across providers/keys/teams. Missing for 10: explicit per-user spend breakdown dashboards and independent/hands-on confirmation of a unified cost dashboard beyond docs claims.
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
- [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
- [claimed-docs] “Filter logs by `Config ID` to see all requests using that config”
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
Kong AI Gateway computes exact per-request LLM costs matching provider billing, logs cost metrics with token/model details, exports cost-related OTel metrics, and documentation explicitly states cost tracking 'across teams'; since all providers route through the same gateway, aggregation across providers is plausible. However, evidence never shows a consolidated dashboard or breakdown specifically by model, API key, or individual user — only 'teams' is explicitly named as a tracked dimension. Missing for 10: explicit per-user and per-API-key cost attribution/dashboard, and a documented single-pane cross-provider spend view (vs. raw logs/metrics that would need external aggregation).
- [claimed-docs] “AI Gateway calculates that figure for every request to a large language model (LLM) provider, matching what the provider actually bills.”
- [claimed-docs] “AI Gateway logs capture detailed information about the request and response payloads, token usage, model details, latency, and cost metrics.”
- [claimed-docs] “You can use these metrics to: Track LLM request latency and upstream provider processing time * Monitor token consumption ... * Calculate AI…”
- [claimed-docs] “AI Gateway can proxy requests from AI command-line tools to LLM providers. This gives you centralized control over AI traffic: log all reque…”
- [claimed-docs] “AI Gateway can export OpenTelemetry (OTLP) metrics for generative AI, MCP, and A2A traffic through an OpenTelemetry AI Policy.”
Key management — stories about key management in this arenaKey management
Stories about key management in this arena
Byok
developerBring my own provider API keys and have the gateway use them for my traffic
weight 2 · round to PortkeyPortkey's virtual keys and model catalog explicitly let developers add/store their own provider API keys ('Add a provider, then use @provider-slug/model-name', 'Provider credentials stored securely, never exposed in code'), and the gateway routes traffic through these keys with governance/rate-limit controls layered on top. Missing for 10: independent hands-on verification of the BYO-key flow and no explicit mention of per-key traffic isolation/audit beyond docs claims.
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”
- [claimed-docs] “Provider credentials stored securely, never exposed in code”
- [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
Docs show AI Model Providers are configured with credentials, and an AI Vault entity lets AI Model Providers reference externally-stored secrets (AWS Secrets Manager, HashiCorp Vault, env vars) rather than storing keys as literal values, and Azure-specific managed identity auth is also supported — implying developers can supply and manage their own provider keys/secrets. However, no explicit doc snippet shows the exact 'paste your own API key per provider' config example or per-request key override behavior. Missing for 10: an explicit example of configuring a provider with a literal/bring-your-own API key, and confirmation of per-consumer or per-request key overrides.
- [claimed-docs] “The core of AI Gateway is the ability to serve AI Models from various AI Model Providers via a provider-agnostic API.”
- [claimed-docs] “Yes, if AI Gateway is running on Azure, you can configure an AI Model Provider to detect the designated Managed Identity or User-Assigned Id…”
- [claimed-docs] “An AI Vault entity lets you register an external secret backend (AWS Secrets Manager, HashiCorp Vault, environment variables, or others) so …”
- [claimed-docs] “Authenticate enterprise users through an existing identity provider (Okta, Azure AD, Google, or any OIDC-compliant IdP) without managing key…”
Programmatic admin
ai-native userProvision gateways, keys, and budgets programmatically through an admin API
weight 3 · round to Kong AI GatewayPortkeynone0/10Evidence shows budget/rate-limit and key management as dashboard-configurable features (docs-13, docs-25, docs-35, docs-38) but never describes a programmatic Admin API for provisioning gateways, keys, or budgets; the only API reference documented is the inference API (docs-15, docs-16, docs-36), and probes for an OpenAPI/admin API spec returned 404 (portkey-probe-3, portkey-probe-2).
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “**Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
- [probe] “PROBE docs-md: HTTP 404 at https://portkey.ai/docs.md”
Kong exposes a Konnect API for listing/managing AI Gateways (list-ai-gateways) and entity-based configuration (AI Auth Strategy, AI Vault) that implies programmatic provisioning of gateways and credentials, plus a kongctl CLI for gateway configuration/migration. However, there is no explicit documentation of an API or CLI for creating/managing spend budgets or quotas, and no end-to-end example of provisioning gateway+keys+budget via API. Missing for 10: explicit budget/quota provisioning API, a full API reference for creating AI Auth Strategies/keys programmatically, and independent confirmation of the admin API's completeness for this workflow.
- [claimed-docs] “list-ai-gateways summary: List AI Gateways description: Returns a list of AI Gateways in the organization.”
- [claimed-docs] “An AI Vault entity lets you register an external secret backend (AWS Secrets Manager, HashiCorp Vault, environment variables, or others) so …”
- [claimed-docs] “Authenticate enterprise users through an existing identity provider (Okta, Azure AD, Google, or any OIDC-compliant IdP) without managing key…”
- [claimed-docs] “walks you through migrating an existing configuration using the kongctl AI Gateway conversion extension”
- [claimed-docs] “AI Gateway calculates that figure for every request to a large language model (LLM) provider, matching what the provider actually bills.”
Virtual keys
platform engineerMint gateway-managed keys for teams and apps without exposing raw provider keys
weight 3 · round to PortkeyPortkey's virtual keys/model catalog explicitly let platform engineers mint provider-agnostic API keys that map to real provider credentials, which are stored securely and never exposed in code, while budgets, rate limits, and model allow-lists can be enforced per key at org/workspace level for teams and apps. This directly matches minting gateway-managed keys without exposing raw provider keys. Missing for 10: independent/hands-on verification beyond vendor docs, and detail on per-team/app key issuance workflow (e.g., self-service or RBAC-scoped key creation) rather than admin-level config.
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “Provider credentials stored securely, never exposed in code”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
- [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”
Kong AI Gateway's AI Vault lets provider keys be stored in external secret backends and referenced by AI Model Providers/Agents instead of literal values, and AI Auth Strategy lets client apps authenticate via an IdP or Kong-managed auth without handling provider keys directly — both point toward the platform-engineer being able to shield raw provider keys from consumers. However, the evidence never explicitly describes a workflow for minting distinct gateway-managed keys per team/app, rotating them, or a consumer/key-auth flow tied to AI traffic. Missing for 10: explicit documentation of issuing/rotating per-team or per-app gateway-managed API keys, and any admin UI/API walkthrough for that key-minting process.
- [claimed-docs] “An AI Vault entity lets you register an external secret backend (AWS Secrets Manager, HashiCorp Vault, environment variables, or others) so …”
- [claimed-docs] “Authenticate enterprise users through an existing identity provider (Okta, Azure AD, Google, or any OIDC-compliant IdP) without managing key…”
- [claimed-docs] “Reference an AI Auth Strategy on your AI Agent entities to authenticate A2A clients, and attach the Request Size Limiting Policy to limit re…”
Observability — seeing what the system is doing — logs, metrics, traces, alertsObservability
Seeing what the system is doing — logs, metrics, traces, alerts
Integrations
developerExport gateway logs and traces to my own observability stack
weight 1 · round to Kong AI GatewayPortkeynone0/10Evidence describes Portkey's own logs UI, sharing log URLs, and manual feedback tagging, but there is no mention of exporting logs/traces to third-party observability stacks (e.g., OpenTelemetry, Datadog, Grafana) or any API/webhook for bulk export.
Kong AI Gateway documents exporting OpenTelemetry (OTLP) metrics for generative AI, MCP, and A2A traffic via an OpenTelemetry AI Policy, plus detailed audit logs capturing request/response payloads, token usage, latency, and cost — enabling integration with a developer's own observability stack. Missing for 10: explicit trace export (spans/tracing) documentation and independent/hands-on confirmation of exporting to third-party backends like Datadog or Grafana.
- [claimed-docs] “AI Gateway can export OpenTelemetry (OTLP) metrics for generative AI, MCP, and A2A traffic through an OpenTelemetry AI Policy.”
- [claimed-docs] “You can use these metrics to: Track LLM request latency and upstream provider processing time * Monitor token consumption ... * Calculate AI…”
- [claimed-docs] “AI Gateway logs capture detailed information about the request and response payloads, token usage, model details, latency, and cost metrics.”
Logs
platform engineerInspect logged requests and responses with latency, token counts, and cost attached
weight 3 · round to Kong AI GatewayPortkey's docs confirm a Logs section showing a chronological list of all requests with unique shareable URLs and manual feedback tagging, supporting log inspection generally, and repeated claims of 'monitoring all LLM requests' and 'visibility' across the gateway. However, the evidence pack never explicitly documents latency, token count, or cost fields being displayed per log entry. Missing for 10: explicit documentation or screenshots showing latency/tokens/cost columns in the logs UI, independent/hands-on confirmation of these fields.
- [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
- [claimed-docs] “Each log on Portkey has a unique URL. You can copy the link from the address bar and directly share it with anyone in your org.”
- [claimed-docs] “you can also add manual feedback on the logs to be analysed and filtered later”
- [claimed-docs] “It takes 2 mins to integrate and with that, it starts monitoring all of your LLM requests”
- [claimed-docs] “It takes 2 mins to integrate and with that, it starts monitoring all of your LLM requests and makes your app resilient, secure, performant, …”
Kong AI Gateway docs explicitly describe audit logs capturing request/response payloads, token usage, model details, latency, and cost metrics, plus usage analytics dashboards and OTel metrics for latency, tokens, and cost. missing for 10: independent/hands-on corroboration of the logging UI in practice and detail on log retention/query interface.
- [claimed-docs] “See usage analytics for requests, tokens, errors, and latency”
- [claimed-docs] “You can use these metrics to: Track LLM request latency and upstream provider processing time * Monitor token consumption ... * Calculate AI…”
- [claimed-docs] “AI Gateway logs capture detailed information about the request and response payloads, token usage, model details, latency, and cost metrics.”
- [claimed-docs] “AI Gateway calculates that figure for every request to a large language model (LLM) provider, matching what the provider actually bills.”
- [claimed-docs] “AI Gateway can export OpenTelemetry (OTLP) metrics for generative AI, MCP, and A2A traffic through an OpenTelemetry AI Policy.”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round drawnPortkey's core AI gateway (chat/completions, fallbacks, caching, retries, load balancing) is clearly exposed via a REST API with a stable base URL, and the open-source gateway can be run locally — showing strong API-first design for request routing. However there is no evidence of API access to the administrative/UI-only features such as budget & rate-limit configuration, log viewing/sharing, feedback tagging, or virtual-key management, and probes show no public OpenAPI spec, so full UI-API parity isn't demonstrated. missing for 10: API endpoints for budget/rate-limit admin, log retrieval/sharing via API, feedback API, and a public OpenAPI spec confirming complete parity.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
- [probe] “PROBE openapi: all candidate paths 404 (https://portkey.ai/openapi.json, https://portkey.ai/swagger.json, https://portkey.ai/api/openapi.jso…”
Kong exposes AI Gateway configuration via the Konnect API (e.g. list-ai-gateways endpoint) and the kongctl CLI, and docs explicitly note that 'the same capabilities' configurable via plugins can be set on Services/Routes rather than only the UI, implying strong API-driven configuration. However there is no explicit statement of full UI/API parity, and probes for a discoverable OpenAPI spec returned 404s, leaving completeness unverified. Missing for 10: an explicit parity statement or comprehensive OpenAPI spec confirming every UI action (e.g. dashboards, analytics views) is also API-accessible, and independent confirmation of no UI-only features.
- [claimed-docs] “list-ai-gateways summary: List AI Gateways description: Returns a list of AI Gateways in the organization.”
- [claimed-docs] “This page shows you how to configure the same capabilities with AI plugins on Services and Routes.”
- [probe] “official CLI documented at https://developer.konghq.com/kongctl/”
- [probe] “PROBE openapi: all candidate paths 404 (https://developer.konghq.com/openapi.json, https://developer.konghq.com/swagger.json, https://develo…”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnPortkeynone0/10Portkey's docs cover logging, gateway configuration, and self-hosting the open-source gateway, but there is no mention of exporting stored logs/configs/data in open formats or facilitating a full data export for migration away from the platform. Shareable log URLs (portkey-docs-12) are not a bulk export/portability feature.
- [claimed-docs] “The Logs section presents a chronological list of all the requests processed through Portkey.”
- [claimed-docs] “Each log on Portkey has a unique URL. You can copy the link from the address bar and directly share it with anyone in your org.”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
Kong AI Gatewaynone0/10Kong AI Gateway/Konnect is a SaaS-hosted control plane with configuration, logs, and analytics data, so data portability/export-and-leave is a fair axis to ask, but the evidence pack contains no mention of a bulk data export feature, open-format export of configs/logs/analytics, or a documented migration-out path in open standards. Evidence only covers observability (metrics, logs) and a CLI conversion extension for migrating configs between versions, not exporting all user data to leave the platform.
- [claimed-docs] “walks you through migrating an existing configuration using the kongctl AI Gateway conversion extension”
- [claimed-docs] “You can use these metrics to: Track LLM request latency and upstream provider processing time * Monitor token consumption ... * Calculate AI…”
- [claimed-docs] “AI Gateway logs capture detailed information about the request and response payloads, token usage, model details, latency, and cost metrics.”
ai-native userRead the product's source under an open license
weight 2 · round to PortkeyPortkey explicitly states it open-sourced its AI gateway component ('We've open sourced our battle-tested AI gateway to the community... run it locally with a single command'), which supports reading some source code. However, this only covers the gateway, not the full product (observability, admin console, model catalog, etc.), and no explicit license name or repository link is provided in the evidence. Missing for 10: explicit open-source license identification, confirmation that the broader Portkey platform (not just the gateway) is open source, and a linked public repository.
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
- [claimed-docs] “npx @portkey-ai/gateway”
Kong AI Gatewaynone0/10Kong AI Gateway is a commercial/enterprise product built on Kong Gateway; the evidence pack contains only product documentation, API references, and probes for docs/CLI—no mention of source code availability or an open-source license for AI Gateway itself. Kong Gateway OSS is a separate product; nothing here confirms AI Gateway's source is published under an open license.
ai-native userSelf-host the core product
weight 3 · round to PortkeyDocs confirm the AI Gateway — described as the 'battle-tested' core routing component — is open-sourced and can be run locally via a single npx command, which is genuine self-hosting evidence. However, other core parts of the product (observability/logs UI, admin budget/rate-limit controls, model catalog governance) are described only as hosted SaaS features with no mention of self-hosting. Missing for 10: evidence that the full platform (observability, admin, model catalog) can be self-hosted, not just the gateway component, and any independent confirmation of self-hosting in practice.
- [claimed-docs] “npx @portkey-ai/gateway”
- [claimed-docs] “We've open sourced our battle-tested AI gateway to the community. You can run it locally with a single command:”
- [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
Docs reference configuring AI Gateway capabilities 'on-prem' via plugins on Services and Routes, and a quickstart script to spin up a demo instance, implying self-hosted deployment is possible. However, there's no detailed self-hosting guide, licensing/open-source clarification, or infrastructure requirements documented in the evidence pack. Missing for 10: explicit self-hosting/installation documentation, open-source license details, infrastructure/deployment guides beyond a demo quickstart.
- [claimed-docs] “This page shows you how to configure the same capabilities with AI plugins on Services and Routes.”
- [claimed-docs] “You can use the quickstart script to get a demo instance of AI Gateway running almost instantly.”
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round to Kong AI GatewayPortkeynone0/10No evidence pack items mention data residency, regional storage options, or geographic hosting choices; Portkey's docs focus on gateway routing, caching, retries, budgets, and observability rather than data residency controls.
Kong AI Gateway documentation mentions an on-prem/self-hosted configuration option (configure-on-prem.md), which implies some ability to control where the gateway (and thus data in transit) runs, but there is no explicit documentation of region selection, data residency guarantees, or storage location controls for logs/analytics/cost data. missing for 10: explicit region/residency configuration options, documented data storage location controls, compliance statements about data residency.
- [claimed-docs] “This page shows you how to configure the same capabilities with AI plugins on Services and Routes.”
- [claimed-docs] “AI Gateway logs capture detailed information about the request and response payloads, token usage, model details, latency, and cost metrics.”
ai-native userPrevent my data from being used to train AI models
weight 3 · round drawnPortkeynone0/10Evidence pack covers gateway routing, caching, retries, logging, budgeting, etc., but contains no mention of a data-training opt-out, zero-retention policy, or any privacy control preventing use of data for AI model training.
Kong AI Gatewaynone0/10Kong AI Gateway is an enterprise routing/governance layer for LLM traffic, so control over how provider data is used is a fair question, but the evidence pack contains no mention of training-data opt-out, zero-retention guarantees, or contractual terms preventing model providers from using proxied data for training — only general DLP/PII redaction and content-safety features are documented, which don't address this specific claim.
ai-native userControl data retention and deletion
weight 2 · round drawnPortkeynone0/10The evidence pack covers gateway routing, caching, retries, budgets, and logging, but contains no mention of data retention policies, log/data deletion controls, or configurable retention windows.
Kong AI Gatewaynone0/10Evidence shows Kong AI Gateway logs request/response payloads, token usage, and costs (docs-10), and offers DLP/PII removal (docs-20) and semantic caching (docs-11), but none of the evidence describes configurable data retention periods or deletion/purge controls for logs, caches, or audit trails. Missing for 10: explicit retention policy settings, log/cache deletion or expiry controls, data subject deletion workflow.
- [claimed-docs] “AI Gateway logs capture detailed information about the request and response payloads, token usage, model details, latency, and cost metrics.”
- [claimed-docs] “Apply safety and DLP policies to prevent toxic content and remove personally identifiable information.”
- [claimed-docs] “AI Semantic Cache - Plugin | Kong Docs”
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnPortkeynone0/10Portkey is essentially a telemetry/observability platform (logs, monitoring of all LLM requests) with no evidence of any opt-out mechanism for its own tracking or usage telemetry; the evidence pack only documents features for logging, caching, retries, and governance, never a privacy/telemetry opt-out control.
Routing resilience — stories about routing resilience in this arenaRouting resilience
Stories about routing resilience in this arena
Fallbacks
platform engineerConfigure automatic fallback to another model or provider when one fails
weight 3 · round to PortkeyPortkey's docs explicitly describe configuring a prioritized list of providers/models with automatic fallback when the primary fails, customizable trigger status codes, plus complementary retries and load balancing for resilience. missing for 10: independent/hands-on corroboration beyond vendor docs.
- [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
- [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
- [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
- [claimed-docs] “Configure per-strategy circuit protection and failure handling”
Kong AI Gateway documents load balancing across multiple LLM models/providers, which implies routing resilience, but the evidence never explicitly describes an automatic fallback mechanism triggered by provider failure (e.g., retry-on-error, circuit breaking, or failover behavior). Missing for 10: explicit fallback-on-failure documentation, failure detection/health-check mechanism, and any hands-on confirmation that fallback actually occurs when a provider errors out.
- [claimed-docs] “AI Gateway provides load balancing capabilities to distribute requests across multiple LLM models.”
- [claimed-docs] “The core of AI Gateway is the ability to serve AI Models from various AI Model Providers via a provider-agnostic API.”
Load balancing
platform engineerLoad-balance traffic across providers, deployments, or keys by weight, latency, or cost
weight 2 · round to PortkeyPortkey's docs explicitly document load balancing to distribute traffic across multiple LLMs/providers by weight, plus gradual migration (percentage-based rollout) and fallback on failure/latency-driven retries, all configurable via Configs. This directly matches weight-based and failure/latency-based routing across providers/deployments/keys, backed by first-party docs on load-balancing, fallbacks, and virtual keys. Missing for 10: explicit cost-based routing strategy documentation and independent/hands-on verification beyond vendor docs.
- [claimed-docs] “Distribute traffic across multiple LLMs to prevent any single provider from becoming a bottleneck.”
- [claimed-docs] “**Gradual Migration** | Test new models with small percentage before full rollout”
- [claimed-docs] “**Gradual Migration**|Test new models with small percentage before full rollout”
- [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
- [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “Configure per-strategy circuit protection and failure handling”
Kong AI Gateway explicitly documents load-balancing across multiple LLM models/providers (docs-17), and it also tracks and calculates per-request LLM costs (docs-18) and latency metrics (docs-9), which are the raw signals needed for cost- or latency-aware routing. However, the evidence pack never confirms the specific load-balancing algorithms (e.g., weighted, latency-based, cost-based) are selectable options, so it's unclear whether these criteria can actually be used as balancing strategies. Missing for 10: explicit documentation of weight-based, latency-based, and cost-based load-balancing algorithms/configuration options, and independent confirmation these modes work as described.
- [claimed-docs] “AI Gateway provides load balancing capabilities to distribute requests across multiple LLM models.”
- [claimed-docs] “AI Gateway calculates that figure for every request to a large language model (LLM) provider, matching what the provider actually bills.”
- [claimed-docs] “You can use these metrics to: Track LLM request latency and upstream provider processing time * Monitor token consumption ... * Calculate AI…”
Policy routing
ai-native userMy agent can switch models mid-task by policy — cost, capability, or availability — through gateway routing rules
weight 3 · round to PortkeyPortkey's gateway supports policy-based model switching via conditional fallbacks (status-code triggers), load balancing/weighted routing, gradual migration and canary rollout, plus budget/rate-limit enforcement that can trigger automatic switching for cost/availability reasons - all configurable via routing configs applied at request time. This directly matches mid-task, policy-driven model switching for cost, capability, or availability. Missing for 10: independent/hands-on corroboration of dynamic mid-task capability-based switching (vs. failure-triggered fallback), and explicit example of capability-based routing rules (e.g., routing by task type).
- [claimed-docs] “Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.”
- [claimed-docs] “By default, fallback triggers on any **non-2xx** status code. Customize with `on_status_codes`”
- [claimed-docs] “Distribute traffic across multiple LLMs to prevent any single provider from becoming a bottleneck.”
- [claimed-docs] “**Gradual Migration** | Test new models with small percentage before full rollout”
- [claimed-docs] “Canary test new models in production”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “Budget limits allow you to set maximum LLM spending or token usage thresholds on your API keys, automatically preventing further usage when …”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
- [claimed-docs] “Configure per-strategy circuit protection and failure handling”
Kong AI Gateway documents load balancing across multiple LLM models (covering availability-based routing) and per-request cost calculation, which together imply some policy-driven model selection, but there is no explicit documentation of switching models based on capability or triggering mid-task, dynamic policy switches tied to cost or availability thresholds. Missing for 10: explicit cost/capability-based routing policy documentation, mid-task dynamic model switching evidence, and independent/hands-on corroboration of this routing-resilience behavior.
- [claimed-docs] “AI Gateway provides load balancing capabilities to distribute requests across multiple LLM models.”
- [claimed-docs] “AI Gateway calculates that figure for every request to a large language model (LLM) provider, matching what the provider actually bills.”
- [claimed-docs] “AI Gateway can perform intelligent request routing, caching, and content filtering using semantic similarity queries.”
Rate limits
platform engineerSmooth provider rate limits by spreading traffic across keys and queuing or throttling requests
weight 2 · round to PortkeyPortkey documents load balancing across multiple provider keys/models to avoid bottlenecks, automatic retries with exponential backoff or provider retry-after headers, and configurable per-key/org rate limits (hourly/daily/per-minute) alongside budget limits — directly addressing spreading traffic and throttling/queuing around rate limits. Multiple virtual keys per provider further support distributing load to smooth rate limits. Missing for 10: explicit first-party or independent evidence of a request queue (vs. retry/backoff) and hands-on/independent validation of these features working under real rate-limit pressure.
- [claimed-docs] “Distribute traffic across multiple LLMs to prevent any single provider from becoming a bottleneck.”
- [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
- [claimed-docs] “Enable `use_retry_after_headers` to use the provider’s `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”
- [claimed-docs] “Enable `use_retry_after_headers` to use the provider's `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”
- [claimed-docs] “Set hourly, daily, or per minute rate limits on requests or tokens sent”
- [claimed-docs] “Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization”
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
Kong AI Gateway documents load balancing across multiple LLM models/providers and a dedicated 'AI Rate Limiting Advanced' plugin for enforcing rate limits, which together support spreading traffic and throttling. However, there is no explicit documentation of spreading traffic across multiple API keys for the same provider or of request queuing behavior when limits are hit. Missing for 10: explicit key-rotation/multi-key load balancing, request queuing mechanics, and any hands-on/independent verification of these features working together.
- [claimed-docs] “AI Gateway provides load balancing capabilities to distribute requests across multiple LLM models.”
- [claimed-docs] “AI Rate Limiting Advanced - Plugin | Kong Docs”
- [claimed-docs] “AI Gateway calculates that figure for every request to a large language model (LLM) provider, matching what the provider actually bills.”
Retries
platform engineerSet automatic retry policies for transient provider errors
weight 2 · round to PortkeyPortkey's docs explicitly describe automatic retries with exponential backoff and support for provider retry-after headers, directly matching the story of configuring retry policies for transient errors. missing for 10: independent/hands-on corroboration beyond vendor docs, and configuration details on per-error-code or max-retry-count customization.
- [claimed-docs] “Automatically retry failed LLM requests with exponential backoff.”
- [claimed-docs] “Enable `use_retry_after_headers` to use the provider’s `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”
- [claimed-docs] “Enable `use_retry_after_headers` to use the provider's `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponent…”
Kong AI Gateway documents load balancing across multiple LLM models/providers (docs-17), which implies some failover capability, but the evidence never explicitly describes configurable automatic retry policies for transient provider errors (e.g., retry counts, backoff, error-code triggers). Missing for 10: explicit retry-policy documentation, retry configuration options, and evidence of behavior on transient errors specifically.
- [claimed-docs] “AI Gateway provides load balancing capabilities to distribute requests across multiple LLM models.”
- [claimed-docs] “The core of AI Gateway is the ability to serve AI Models from various AI Model Providers via a provider-agnostic API.”
Streaming tools — stories about streaming tools in this arenaStreaming tools
Stories about streaming tools in this arena
Streaming
developerStream token-by-token responses through the gateway from any provider
weight 3 · round to Kong AI GatewayPortkeynone0/10The evidence describes Portkey's gateway, fallbacks, retries, load balancing, caching, and OpenAI-compatible base URL integration, but none of the provided docs mention streaming or token-by-token response support. Streaming is a fair and expected capability for an LLM gateway product, so its absence in the evidence pack means this axis applies but is unproven.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
Kong AI Gateway has a dedicated streaming.md doc confirming token-by-token streaming from models is supported and rendered in real time, layered on top of its provider-agnostic API supporting multiple LLM providers. Missing for 10: independent/hands-on confirmation of streaming behavior across every provider and any details on streaming with proxy-added policies (e.g., guardrails) applied mid-stream.
- [claimed-docs] “most models can stream each word (or sets of words and tokens) back to the client. This allows the chat response to be rendered in real time…”
- [claimed-docs] “The core of AI Gateway is the ability to serve AI Models from various AI Model Providers via a provider-agnostic API.”
Tool calling
developerRequest structured JSON-schema outputs across providers
weight 1 · round drawnPortkeynone0/10The evidence pack covers gateway features like fallbacks, retries, load balancing, caching, virtual keys, and logging, but contains no mention of structured JSON-schema output support, response_format normalization, or schema validation across providers. Since Portkey acts as a universal AI gateway, this capability would be plausible, but no documentation cites it.
Kong AI Gatewaynone0/10The evidence pack covers provider-agnostic proxying, streaming, load balancing, cost tracking, and safety policies, but no document mentions structured/JSON-schema output enforcement or normalization across providers. This is a fair capability to expect from an AI gateway, but no evidence supports it.
developerMake tool and function calls across different providers with a consistent schema
weight 3 · round to Kong AI GatewayPortkey's gateway offers a unified OpenAI-SDK-compatible API (base URL + headers) across 250+ models, which implies a consistent request format including function/tool call fields, and it supports connecting external tools via remote MCP servers. However, there is no explicit documentation describing how tool/function-calling schemas are normalized or translated across different providers' native formats. Missing for 10: explicit docs on function/tool-call schema normalization, examples of tool-calling requests across multiple providers, and independent verification that tool calls work consistently across providers.
- [claimed-docs] “It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in y…”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “Connect to Remote MCP severs, allowing you to connect external tools and data sources.”
Kong AI Gateway documents a provider-agnostic API for LLM traffic and an AI MCP Proxy/Server that converts REST APIs into MCP tools, suggesting a consistent tool-calling interface across providers and models. However, there is no explicit documentation showing normalization of function/tool-call schemas (e.g., translating OpenAI-style function calls to Anthropic tool-use format) for the same request across different providers. Missing for 10: explicit cross-provider function-call schema translation examples, hands-on/independent verification of consistent tool-call behavior across providers.
- [claimed-docs] “The core of AI Gateway is the ability to serve AI Models from various AI Model Providers via a provider-agnostic API.”
- [claimed-docs] “AI MCP Proxy bridges Kong-managed Services to the Model Context Protocol, converting REST APIs into MCP tools or fronting upstream MCP serve…”
- [claimed-docs] “Turn any API into an MCP server using the AI MCP Server entity. This approach does not require an LLM and provides full control over product…”
- [claimed-docs] “Govern LLM, MCP, and agent-to-agent (A2A) traffic with the same Kong AI Gateway.”
Unified api — stories about unified api in this arenaUnified api
Stories about unified api in this arena
Catalog
developerBrowse or query a catalog of available models with pricing and context-window metadata
weight 2 · round to PortkeyPortkey documents a 'Model Catalog' providing a single pane to view/manage all providers and models with discovery and governance controls, and claims access to 250+ models, but no evidence shows the catalog surfaces pricing or context-window metadata for models. missing for 10: explicit documentation of per-model pricing info, explicit documentation of context-window size metadata, and any UI/query example showing these fields.
- [claimed-docs] “A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage …”
- [claimed-docs] “It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in y…”
- [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”
- [claimed-docs] “Fine-grained control: budgets, rate limits, and model allow-lists at both org and workspace level.”
Kong AI Gatewaynone0/10Evidence shows Kong AI Gateway supports multi-provider routing and calculates per-request costs, but there is no documented catalog/endpoint for browsing or querying available models with pricing and context-window metadata — the closest evidence (model-cost-management, list-ai-gateways API) covers cost calculation per request and listing gateway instances, not a model metadata catalog.
- [claimed-docs] “AI Gateway calculates that figure for every request to a large language model (LLM) provider, matching what the provider actually bills.”
- [claimed-docs] “list-ai-gateways summary: List AI Gateways description: Returns a list of AI Gateways in the organization.”
- [claimed-docs] “The core of AI Gateway is the ability to serve AI Models from various AI Model Providers via a provider-agnostic API.”
Compatibility
developerPoint existing OpenAI-compatible code at the gateway by changing only the base URL and key
weight 3 · round to PortkeyDocs explicitly state that developers using OpenAI's Python/Node SDKs can integrate Portkey by only changing the base URL and adding Portkey-specific headers, with the base URL documented as https://api.portkey.ai/v1. This is a first-party documented, purpose-built migration path. Missing for 10: independent/hands-on third-party confirmation beyond vendor docs.
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey's features into your existing setup.”
Kong AI Gateway is documented as a provider-agnostic API layer that proxies LLM requests (chat/completions endpoints) with streaming support, which implies OpenAI-style compatibility, but no evidence explicitly confirms that existing OpenAI SDK code works unmodified by only swapping base URL and API key. missing for 10: explicit doc/example showing an OpenAI SDK client redirected via base_url with no code changes, and confirmation that request/response schemas exactly mirror OpenAI's format.
- [claimed-docs] “The core of AI Gateway is the ability to serve AI Models from various AI Model Providers via a provider-agnostic API.”
- [claimed-docs] “most models can stream each word (or sets of words and tokens) back to the client. This allows the chat response to be rendered in real time…”
- [claimed-docs] “This page shows you how to configure the same capabilities with AI plugins on Services and Routes.”
One endpoint
developerCall many model providers through one consistent API
weight 3 · round to PortkeyPortkey documents a unified API/gateway (single base URL, single API key) that supports 250+ providers/models via consistent SDK/REST calls, with virtual keys and model catalog abstracting provider differences. Missing for 10: independent third-party benchmark or hands-on confirmation of the claimed 250+ provider coverage beyond vendor docs.
- [claimed-docs] “The core concept remains the same - one Portkey API key gives you access to multiple providers and models.”
- [claimed-docs] “By changing the base URL and adding Portkey-specific headers, you can quickly integrate Portkey’s features into your existing setup.”
- [claimed-docs] “The base URL for all REST API requests is `https://api.portkey.ai/v1`”
- [claimed-docs] “It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in y…”
- [claimed-docs] “Add a provider, then use `@provider-slug/model-name` in your code.”
- [claimed-docs] “Ideal for: if you’re currently utilizing OpenAI’s Python or Node.js SDKs. By changing the base URL and adding Portkey-specific headers, you …”
Kong AI Gateway explicitly provides a provider-agnostic API to route to many LLM providers, with load balancing and unified logging/observability across providers, matching the core unified-API story. Missing for 10: independent/hands-on developer corroboration of consistent request/response schema across providers, and no explicit list of exact supported providers in evidence.
- [claimed-docs] “The core of AI Gateway is the ability to serve AI Models from various AI Model Providers via a provider-agnostic API.”
- [claimed-docs] “AI Gateway provides load balancing capabilities to distribute requests across multiple LLM models.”
- [claimed-docs] “You can use these metrics to: Track LLM request latency and upstream provider processing time * Monitor token consumption ... * Calculate AI…”
- [claimed-docs] “AI Gateway calculates that figure for every request to a large language model (LLM) provider, matching what the provider actually bills.”
Not comparable on these axes
ai-native userSubscribe to events via webhooks
weight 2 · not comparablePortkeynone0/10No evidence in the pack mentions webhooks or event subscriptions of any kind; Portkey's docs cover gateway routing, caching, retries, budgets, and logs but nothing about outbound webhook notifications for events.
Kong AI Gatewayn/aKong AI Gateway is an API/AI traffic gateway product, not an event-driven system for end users to subscribe to via webhooks; its role is proxying, governing, and observing LLM/MCP/A2A traffic, not offering webhook-based event subscription as a user-facing feature. No evidence in the pack even mentions webhooks or event subscription, and this axis is a category mismatch for a gateway of this kind.
ai-native userSet up automations that run autonomously in the background
weight 2 · not comparablePortkeyn/aPortkey is an AI gateway/observability layer for routing, caching, and governing LLM requests, not a workflow/automation platform for running autonomous background tasks; the evidence covers gateway features (fallbacks, retries, caching, load balancing, logs, budgets) with no scheduler or autonomous-agent execution capability. This story targets a different product category and does not fit Portkey's axis.
Kong AI Gatewayn/aKong AI Gateway is a proxy/control-plane for governing LLM, MCP, and A2A traffic (rate limiting, observability, auth, routing) — it does not itself host or execute autonomous background automations/agents; that is a different product category (agent orchestration/runtime platforms).
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · not comparablePortkeyn/aPortkey is an AI gateway/observability platform for routing and managing LLM traffic, not a product with a built-in chat/assistant UI that a user delegates tasks to; this axis is a category error for its product type.
ai-native userSchedule recurring jobs or workflows
weight 2 · not comparablePortkeyn/aPortkey is an AI gateway/observability platform for routing, caching, and monitoring LLM requests, not a workflow/job scheduling or orchestration product; nothing in the evidence pack relates to scheduling recurring jobs or workflows, making this a category mismatch rather than a missing feature.
Kong AI Gatewayn/aKong AI Gateway is an API/traffic gateway product for proxying, governing, and observing AI/LLM/A2A/MCP traffic; it is not a workflow/job scheduling or automation-orchestration product. Scheduling recurring jobs or workflows is outside its category — no evidence pack item relates to cron-like scheduling or workflow orchestration, confirming this is a wrong-axis question rather than a missing feature.