Groq vs Baseten

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round drawn
   Groq✓fullprobed9/10

   Direct probe evidence confirms llms.txt is live (HTTP 200) at console.groq.com/llms.txt, and Groq also serves per-page .md docs (docs/overview.md) explicitly designed for agent consumption, indicating agent-oriented docs beyond just llms.txt. Missing for 10: no independent/community confirmation of an agent actually using llms.txt in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://console.groq.com/llms.txt # https://console.groq.com llms.txt - [JigsawStack 🧩](https://console.groq.c…”
   • [probe] “PROBE docs-md: HTTP 200 at https://console.groq.com/docs/overview.md --- description: Fast LLM inference, OpenAI-compatible. Simple to integ…”

   proof ↗⚑ flag

   Baseten✓fullprobed9/10

   Baseten hosts a verified llms.txt (HTTP 200, indexing docs) and an agent-setup page directing agents to install skills/MCP servers to search the docs, confirmed live via a working MCP endpoint. missing for 10: independent third-party confirmation of an agent successfully using llms.txt end-to-end beyond Baseten's own probes.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.baseten.co/llms.txt # Baseten - [Baseten overview](https://docs.baseten.co/overview.md): Run hoste…”
   • [probe] “PROBE docs-md: HTTP 200 at https://docs.baseten.co/overview.md > ## Documentation Index > Fetch the complete documentation index at: https:/…”
   • [claimed-docs] “Install the Baseten skill and MCP servers so your coding agent can manage your Baseten workspace and search these docs.”
   • [probe] “official MCP server documented at https://docs.baseten.co/agent-setup”
   • [probe] “PROBE mcp-endpoint (2026-09-04): POST initialize to https://docs.baseten.co/mcp answered HTTP 200 with a JSON-RPC/MCP response (event: messa…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Groq
   Groq✓fullprobed8/10

   Groq is a pure API/inference service accessed via OpenAI-compatible REST endpoints and SDKs (API key + base_url swap), with batch processing for large-scale asynchronous automation and confirmed live, key-gated JSON endpoints — all of which are inherently scriptable/headless with no GUI dependency, ideal for CI pipelines. Missing for 10: explicit CI/CD examples (e.g., GitHub Actions) or docs specifically framing headless/automation use cases beyond batch API references.

   • [claimed-docs] “pass your Groq API key to the `api_key` parameter and change the `base_url` to `https://api.groq.com/openai/v1`”
   • [claimed-docs] “Batch processing lets you run thousands of API requests at scale by submitting your workload as an asynchronous batch of requests to Groq wi…”
   • [claimed-docs] “Batch processing lets you run thousands of API requests at scale by submitting your workload as an asynchronous batch of requests to Groq wi…”
   • [github] “Request parameters that correspond to file uploads can be passed in many different forms”
   • [probe] “PROBE models-endpoint (2026-09-04): GET https://api.groq.com/openai/v1/models without a key returned HTTP 401 ({"error":{"message":"Invalid …”
   • [claimed-docs] “To enable streaming, set the parameter stream=True. The completion function will then return an iterator of completion deltas”

   proof ↗⚑ flag

   Baseten✓fullclaimed7/10

   Baseten provides a CLI (truss push, Baseten CLI/Switch), config.yaml-driven deployment, OpenAI-compatible REST APIs with async/webhook/polling support, and API-key-based auth/rate limiting, all of which support scripted, non-interactive CI/automation workflows. missing for 10: explicit CI/CD pipeline examples (e.g. GitHub Actions integration), independent hands-on confirmation of headless CLI use in automation, and documented non-interactive auth flow for CI secrets.

   • [github] “You write a `config.yaml` that specifies the model, the hardware, and the engine, then `uvx truss push` builds a TensorRT-optimized containe…”
   • [github] “Deploying a model to Baseten via Truss turns a Hugging Face model into a production-ready API endpoint. You write a `config.yaml` that speci…”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents, or any …”
   • [claimed-docs] “Call hosted models through an OpenAI-compatible API, deploy your own models on dedicated infrastructure”
   • [claimed-docs] “x-ratelimit-remaining-requests: Reports how many requests remain before you reach the request rate limit.”
   • [claimed-docs] “You can also set a workspace budget and query usage by API key or model.”
   • [claimed-docs] “a `config.yaml` can define the model, hardware, and inference engine without custom serving code”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round to Groq
   Groq✓fullprobed8/10

   Groq's Responses/Chat API supports remote MCP: you provide an MCP server URL and Groq handles tool orchestration without implementing tool logic yourself, documented explicitly in first-party docs and confirmed via a probe of the dedicated MCP docs page. Missing for 10: independent hands-on user reports specifically validating MCP tool use (community evidence covers speed/reliability but not MCP), and no details on multi-server or auth edge cases.

   • [claimed-docs] “point to an MCP server URL and the Groq API will start using its tools without you having to implement any tool logic yourself”
   • [claimed-docs] “you simply point to an MCP server URL and the Groq API will start using its tools without you having to implement any tool logic yourself”
   • [claimed-docs] “Groq's Responses API supports remote tool use via MCP servers via HTTPS where Groq handles all orchestration... You don't implement anything…”
   • [probe] “official MCP server documented at https://console.groq.com/docs/mcp”

   proof ↗⚑ flag

   Baseten—none0/10

   Baseten's evidence only shows it exposing its own MCP server so external coding agents (Claude Code, Codex, Pi) can call Baseten's workspace tools — the reverse relationship. There is no evidence that Baseten itself can consume/plug in third-party MCP servers to gain access to their tools during inference or agent workflows.

   • [claimed-docs] “Install the Baseten skill and MCP servers so your coding agent can manage your Baseten workspace and search these docs.”
   • [claimed-docs] “Connect Claude Code, Codex CLI, or Pi with Baseten Switch.”
   • [claimed-docs] “Use [Baseten Switch](/reference/cli/baseten-switch) to route requests from Claude Code or Codex CLI to Model APIs.”
   • [probe] “PROBE mcp-endpoint (2026-09-04): POST initialize to https://docs.baseten.co/mcp answered HTTP 200 with a JSON-RPC/MCP response (event: messa…”
   • [probe] “official MCP server documented at https://docs.baseten.co/agent-setup”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round to Baseten
   Groq~partialprobed4/10

   There is a probe hit indicating an official MCP-server doc page exists (console.groq.com/docs/mcp), but the evidence pack contains no actual content, setup instructions, or independent confirmation of this server — the detailed MCP docs quoted (groq-docs-6/17/28) describe Groq's API acting as an MCP *client* consuming remote MCP servers, not Groq serving as an MCP server for other agents to connect to. Missing for 10: quoted documentation of the official MCP server's endpoint/connection instructions, evidence of an agent successfully connecting to Groq via that server, and independent/community confirmation it works as advertised.

   • [probe] “official MCP server documented at https://console.groq.com/docs/mcp”
   • [claimed-docs] “point to an MCP server URL and the Groq API will start using its tools without you having to implement any tool logic yourself”
   • [claimed-docs] “Groq's Responses API supports remote tool use via MCP servers via HTTPS where Groq handles all orchestration... You don't implement anything…”

   proof ↗⚑ flag

   Baseten✓fullprobed8/10

   Baseten documents and hosts an official MCP server (docs.baseten.co/agent-setup, docs.baseten.co/mcp) that lets coding agents manage the Baseten workspace and search docs, and a live probe confirms the endpoint responds to MCP initialize with valid JSON-RPC capabilities. missing for 10: independent/community hands-on validation of the MCP server beyond the vendor's own probe, and more detail on the full tool surface exposed.

   • [claimed-docs] “Install the Baseten skill and MCP servers so your coding agent can manage your Baseten workspace and search these docs.”
   • [probe] “official MCP server documented at https://docs.baseten.co/agent-setup”
   • [probe] “PROBE mcp-endpoint (2026-09-04): POST initialize to https://docs.baseten.co/mcp answered HTTP 200 with a JSON-RPC/MCP response (event: messa…”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to Baseten
   Groq—none0/10

   No evidence of an official Groq CLI tool; documentation covers SDKs (Python/TypeScript), REST API, and MCP integration but no CLI is mentioned anywhere in the evidence pack.

   ⚑ flag

   Baseten✓fullclaimed7/10

   Baseten ships official CLIs: Truss (`uvx truss push` to build/deploy models from a config.yaml, per baseten-gh-2/6) and Baseten Switch (a CLI referenced at /reference/cli/baseten-switch for routing coding-agent traffic, baseten-docs-34), plus one-command checkpoint deployment (baseten-docs-29). missing for 10: independent/hands-on verification of CLI behavior, and a full CLI reference/command list beyond scattered doc mentions.

   • [github] “You write a `config.yaml` that specifies the model, the hardware, and the engine, then `uvx truss push` builds a TensorRT-optimized containe…”
   • [github] “Deploying a model to Baseten via Truss turns a Hugging Face model into a production-ready API endpoint. You write a `config.yaml` that speci…”
   • [claimed-docs] “Deploy any synced checkpoint with one CLI command”
   • [claimed-docs] “Use [Baseten Switch](/reference/cli/baseten-switch) to route requests from Claude Code or Codex CLI to Model APIs.”
   • [claimed-docs] “Connect Claude Code, Codex CLI, or Pi with Baseten Switch.”

   proof ↗⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to Groq
   Groq✓fullprobed9/10

   Groq ships an extensively documented, OpenAI-compatible REST API (chat completions, streaming, tool use, structured outputs, batch, service tiers, LoRA, Responses API) with a live confirmed endpoint (401 on missing key, not 404) and a public status page. Missing for 10: a discoverable formal OpenAPI/Swagger spec (all candidate paths 404).

   • [claimed-docs] “Fast LLM inference, OpenAI-compatible. Simple to integrate, easy to scale. Start building in minutes.”
   • [claimed-docs] “pass your Groq API key to the `api_key` parameter and change the `base_url` to `https://api.groq.com/openai/v1`”
   • [claimed-docs] “To start using Groq with OpenAI's client libraries, pass your Groq API key to the api_key parameter and change the base_url to https://api.g…”
   • [claimed-docs] “Generating text with Groq's Chat Completions API enables you to have natural, conversational interactions with Groq's large language models.”
   • [claimed-docs] “To use tools, the model must be provided with tool definitions. These tool definitions are in JSON schema format and are passed to the model…”
   • [claimed-docs] “Structured Outputs is a feature that ensures your model responses conform to your provided JSON Schema”
   • [claimed-docs] “Groq's Responses API is fully compatible with OpenAI's Responses API, making it easy to integrate advanced conversational AI capabilities in…”
   • [probe] “PROBE llms.txt: HTTP 200 at https://console.groq.com/llms.txt # https://console.groq.com llms.txt - [JigsawStack 🧩](https://console.groq.c…”
   • [probe] “PROBE docs-md: HTTP 200 at https://console.groq.com/docs/overview.md --- description: Fast LLM inference, OpenAI-compatible. Simple to integ…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://console.groq.com/openapi.json, https://console.groq.com/swagger.json, https://console.groq.c…”
   • [probe] “PROBE models-endpoint (2026-09-04): GET https://api.groq.com/openai/v1/models without a key returned HTTP 401 ({"error":{"message":"Invalid …”
   • [probe] “PROBE status-page (2026-09-04): https://groqstatus.com returns HTTP 200 and renders a public service-status page (page body includes "operat…”

   proof ↗⚑ flag

   Baseten✓fullprobed8/10

   Baseten exposes an OpenAI/Anthropic-compatible inference API (docs-21, docs-30), a documented CLI/Truss deployment workflow, training APIs (docs-32, docs-36), and usage/rate-limit endpoints, all backed by extensive public docs and a live, probe-verified /v1/models endpoint and MCP server. Missing for 10: a discoverable OpenAPI/swagger spec (probe-3 shows 404s across candidate paths) and independent third-party API usage reports beyond community anecdotes.

   • [claimed-docs] “They support the OpenAI Chat Completions API and the Anthropic Messages API in beta, so you can use familiar client SDKs.”
   • [claimed-docs] “Call hosted models through an OpenAI-compatible API, deploy your own models on dedicated infrastructure”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents”
   • [claimed-docs] “Fine-tuning or running RL on a supported base model: Loops provisions a dedicated trainer and paired sampler, and each training step is an A…”
   • [claimed-docs] “Model APIs bill by token and enforce request and token rate limits. You can also set a workspace budget and query usage by API key or model.”
   • [probe] “PROBE models-endpoint (2026-09-04): GET https://inference.baseten.co/v1/models without a key returned HTTP 401 (No Authorization header prov…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.baseten.co/openapi.json, https://docs.baseten.co/swagger.json, https://docs.baseten.co/…”
   • [github] “You write a `config.yaml` that specifies the model, the hardware, and the engine, then `uvx truss push` builds a TensorRT-optimized containe…”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   Groq—none0/10

   Evidence covers billing spend limits and general API key auth (single api_key parameter) but shows no support for scoped/least-privilege credentials such as role-based keys, granular permission scopes, or per-agent restricted tokens; only one flat API key model is documented.

   • [claimed-docs] “pass your Groq API key to the `api_key` parameter and change the `base_url` to `https://api.groq.com/openai/v1`”
   • [claimed-docs] “To start using Groq with OpenAI's client libraries, pass your Groq API key to the api_key parameter and change the base_url to https://api.g…”
   • [claimed-docs] “Spend Limits: Set automated spending limits and receive budget alerts”
   • [claimed-docs] “Spend Limits:** Set automated spending limits and receive budget alerts”

   proof ↗⚑ flag

   Baseten—none0/10

   Baseten documents API keys, usage monitoring by API key, and workspace budgets, but no evidence describes scoped/least-privilege credential issuance (e.g., role-based permissions, restricted-scope keys, or per-agent credential minting). Monitoring usage by key is not the same as issuing scoped credentials.

   • [claimed-docs] “Model APIs bill by token and enforce request and token rate limits. You can also set a workspace budget and query usage by API key or model.”
   • [claimed-docs] “To monitor token and request consumption by API key or model, see Usage.”
   • [claimed-docs] “You can also set a workspace budget and query usage by API key or model.”
   • [claimed-docs] “To raise a Basic account's limits, request email verification. You can also use that form to move to Pro or Enterprise.”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Groq
   Groq✓fullclaimed8/10

   Groq maintains an official TypeScript SDK (groq-typescript) with concrete usage examples (audio transcription, file uploads) and extensive first-party docs covering chat completions, streaming, tool use, structured outputs, and OpenAI-compatible client usage, all of which are core to building against official SDKs. Missing for 10: explicit evidence of an official Python SDK repo (only OpenAI-compatible base_url swap is shown for Python) and independent/community validation specifically of SDK reliability rather than general API performance.

   • [github] “Request parameters that correspond to file uploads can be passed in many different forms”
   • [github] “await client.audio.transcriptions.create({ model: 'whisper-large-v3-turbo', file: fs.createReadStream('/path/to/file'), });”
   • [github] “If you have access to Node fs we recommend using fs.createReadStream()... Or if you have the web File API you can pass a File instance”
   • [claimed-docs] “pass your Groq API key to the `api_key` parameter and change the `base_url` to `https://api.groq.com/openai/v1`”
   • [claimed-docs] “To start using Groq with OpenAI's client libraries, pass your Groq API key to the api_key parameter and change the base_url to https://api.g…”
   • [claimed-docs] “To enable streaming, set the parameter stream=True. The completion function will then return an iterator of completion deltas”
   • [claimed-docs] “Tool use (or function calling) is what transforms a language model from a conversational interface into an autonomous agent capable of takin…”
   • [claimed-docs] “Structured Outputs is a feature that ensures your model responses conform to your provided JSON Schema”

   proof ↗⚑ flag

   Baseten✓fullclaimed7/10

   Baseten ships the official open-source Truss SDK/CLI for packaging and deploying models (github.com/basetenlabs/truss) and also exposes OpenAI-compatible and Anthropic Messages-compatible endpoints so developers can use those official client SDKs (and LangChain) directly against Baseten (baseten-docs-21, baseten-docs-33, baseten-gh-1/2/5/6). This gives AI-native users multiple first-party and compatible SDK paths for building agentic workflows. Missing for 10: a dedicated first-party 'Baseten Python client' distinct from Truss, and independent/hands-on corroboration of SDK reliability beyond docs and repo README.

   • [github] “Write once, run anywhere: Package model code, weights, and dependencies with a model server that behaves the same in development and product…”
   • [github] “You write a `config.yaml` that specifies the model, the hardware, and the engine, then `uvx truss push` builds a TensorRT-optimized containe…”
   • [github] “Truss lets you serve models with the Baseten Inference Stack as well as deploy models from any open-source framework: vLLM, SGLang, TensorRT…”
   • [github] “Deploying a model to Baseten via Truss turns a Hugging Face model into a production-ready API endpoint. You write a `config.yaml` that speci…”
   • [claimed-docs] “They support the OpenAI Chat Completions API and the Anthropic Messages API in beta, so you can use familiar client SDKs.”
   • [claimed-docs] “Because Baseten exposes an OpenAI-compatible endpoint, you can use LangChain's `ChatOpenAI` with `with_structured_output` by pointing `base_…”
   • [claimed-docs] “Call hosted models through an OpenAI-compatible API, deploy your own models on dedicated infrastructure”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round to Baseten
   Groq—none0/10

   No evidence anywhere in the pack of a webhooks/event-subscription feature for Groq's API—only synchronous/streaming inference, batch, tool-use/MCP, and admin/billing docs are covered.

   ⚑ flag

   Baseten~partialclaimed4/10

   Docs mention that async inference results can be delivered via webhook callback, showing Baseten supports webhook-based notification for job completion, but there is no dedicated webhook subscription system (event types, signing, retry, subscription management) described anywhere in the evidence. Missing for 10: a general event-subscription/webhook API or docs page (event catalog, signature verification, delivery guarantees), coverage beyond async-inference completion notifications, and any independent confirmation of webhook reliability.

   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents, or any …”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling”

   proof ↗⚑ flag

Agentic features

1. ai-native userSet up automations that run autonomously in the background

   weight 2 · round drawn
   Groq—none0/10

   Groq's evidence covers only inference primitives (chat completions, tool use, MCP tool integration, batch processing, structured outputs) — none of which describe a scheduling/trigger system for autonomous background automations; batch jobs are asynchronous request batches, not persistent background agents. Missing for 10: any documented scheduler, cron/trigger mechanism, persistent background agent runtime, or workflow orchestration feature that lets a user 'set up' an automation to run unattended.

   • [claimed-docs] “Batch processing lets you run thousands of API requests at scale by submitting your workload as an asynchronous batch of requests to Groq wi…”
   • [claimed-docs] “Tool use (or function calling) is what transforms a language model from a conversational interface into an autonomous agent capable of takin…”
   • [claimed-docs] “point to an MCP server URL and the Groq API will start using its tools without you having to implement any tool logic yourself”

   proof ↗⚑ flag

   Baseten—none0/10

   Baseten's evidence covers async inference (deferred single requests via webhook/polling), model deployment, and training, but there is no evidence of scheduling, triggers, or autonomous multi-step automations running in the background — async inference is single-request deferral, not an automation/workflow engine.

   ⚑ flag

2. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round drawn
   Groq—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Baseten—none0/10

   Baseten's evidence shows MCP servers and a Baseten 'skill' that let external coding agents (Claude Code, Codex, etc.) manage a Baseten workspace — this is the reverse of a built-in in-product assistant; nothing in the evidence describes a first-party AI assistant living inside the Baseten UI/dashboard that a user can delegate platform tasks to.

   • [claimed-docs] “Install the Baseten skill and MCP servers so your coding agent can manage your Baseten workspace and search these docs.”
   • [claimed-docs] “Connect Claude Code, Codex CLI, or Pi with Baseten Switch.”
   • [claimed-docs] “Use [Baseten Switch](/reference/cli/baseten-switch) to route requests from Claude Code or Codex CLI to Model APIs.”
   • [probe] “official MCP server documented at https://docs.baseten.co/agent-setup”
   • [probe] “PROBE mcp-endpoint (2026-09-04): POST initialize to https://docs.baseten.co/mcp answered HTTP 200 with a JSON-RPC/MCP response (event: messa…”

   proof ↗⚑ flag

3. ai-native userOperate the product with natural-language commands

   weight 2 · round drawn
   Groq~partialclaimed6/10

   Groq's Chat Completions/Responses API is inherently natural-language driven, and its tool-use and remote-MCP features let a model take autonomous actions from NL instructions, which supports agentic operation on top of the API. However, this is evidence of building NL-driven agents *with* Groq's models, not of an NL command interface for operating the Groq platform/product itself (e.g., managing keys, billing, deployments via chat). Missing for 10: an NL/chat interface for controlling the Groq platform itself, independent verification of agentic tool-use reliability in production.

   • [claimed-docs] “Generating text with Groq's Chat Completions API enables you to have natural, conversational interactions with Groq's large language models.”
   • [claimed-docs] “Tool use (or function calling) is what transforms a language model from a conversational interface into an autonomous agent capable of takin…”
   • [claimed-docs] “To use tools, the model must be provided with tool definitions. These tool definitions are in JSON schema format and are passed to the model…”
   • [claimed-docs] “point to an MCP server URL and the Groq API will start using its tools without you having to implement any tool logic yourself”
   • [claimed-docs] “Groq's Responses API supports remote tool use via MCP servers via HTTPS where Groq handles all orchestration... You don't implement anything…”

   proof ↗⚑ flag

   Baseten~partialprobed6/10

   Baseten ships an official skill and MCP server so a coding agent can 'manage your Baseten workspace and search these docs' via natural language, and a probe confirms a live, reachable MCP endpoint responding to JSON-RPC calls — this is genuine natural-language operability of the product through an agent intermediary. However the docs give only a high-level description of what workspace actions are exposed, with no detailed command catalog or independent hands-on verification of breadth/reliability. Missing for 10: a documented list of specific natural-language-operable actions (deploy, scale, monitor, etc.), and independent/hands-on evidence confirming the MCP tools work reliably end-to-end.

   • [claimed-docs] “Install the Baseten skill and MCP servers so your coding agent can manage your Baseten workspace and search these docs.”
   • [probe] “official MCP server documented at https://docs.baseten.co/agent-setup”
   • [probe] “PROBE mcp-endpoint (2026-09-04): POST initialize to https://docs.baseten.co/mcp answered HTTP 200 with a JSON-RPC/MCP response (event: messa…”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round to Groq
   Groq~partialprobed4/10

   Groq's docs (console.groq.com/docs) are extensive and code-example-rich (curl/Python/JS snippets for chat, tool-use, structured outputs, batch, MCP, etc.), functioning as a de facto API reference, but there's no evidence of an interactive, in-browser 'runnable' playground or embedded code execution — probes show no discoverable OpenAPI/swagger spec (404s) and no mention of a try-it-now console feature. Missing for 10: an actual interactive/executable API reference (e.g., Swagger UI, embedded runnable code sandbox), confirmation of an OpenAPI spec, and independent user confirmation of using such an interactive reference.

   • [claimed-docs] “Fast LLM inference, OpenAI-compatible. Simple to integrate, easy to scale. Start building in minutes.”
   • [claimed-docs] “pass your Groq API key to the `api_key` parameter and change the `base_url` to `https://api.groq.com/openai/v1`”
   • [claimed-docs] “To enable streaming, set the parameter stream=True. The completion function will then return an iterator of completion deltas”
   • [probe] “PROBE docs-md: HTTP 200 at https://console.groq.com/docs/overview.md --- description: Fast LLM inference, OpenAI-compatible. Simple to integ…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://console.groq.com/openapi.json, https://console.groq.com/swagger.json, https://console.groq.c…”

   proof ↗⚑ flag

   Baseten—none0/10

   Evidence shows extensive prose documentation (structured outputs, function calling, streaming, pricing) but no interactive API reference or runnable-example playground; a direct probe for an OpenAPI/Swagger spec (which typically powers interactive references) returned 404 on all candidate paths.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.baseten.co/openapi.json, https://docs.baseten.co/swagger.json, https://docs.baseten.co/…”
   • [claimed-docs] “Structured outputs let you generate text that conforms to specific JSON schemas, providing reliable data extraction and controlled text gene…”
   • [claimed-docs] “Function calling* (also called *tool calling*) lets a model choose a tool and produce its arguments from a user request.”
   • [claimed-docs] “Return model output token by token as it is generated.”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   Groq—none0/10

   Groq's API is OpenAI-compatible and well documented, but there is no evidence of a downloadable OpenAPI/Swagger spec; a direct probe of common spec paths (openapi.json, swagger.json, etc.) all returned 404, and no docs page links to a machine-readable spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://console.groq.com/openapi.json, https://console.groq.com/swagger.json, https://console.groq.c…”

   proof ↗⚑ flag

   Baseten—none0/10

   A direct probe for a machine-readable API spec (openapi.json, swagger.json, and related paths) returned 404 on all candidates, and no docs page claims to publish an OpenAPI/Swagger spec — only that the API is OpenAI/Anthropic-compatible in shape, which is not the same as Baseten publishing its own downloadable spec.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.baseten.co/openapi.json, https://docs.baseten.co/swagger.json, https://docs.baseten.co/…”
   • [claimed-docs] “They support the OpenAI Chat Completions API and the Anthropic Messages API in beta, so you can use familiar client SDKs.”
   • [claimed-docs] “Call hosted models through an OpenAI-compatible API, deploy your own models on dedicated infrastructure”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round to Baseten
   Groq—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Baseten~partialclaimed4/10

   Docs mention that deployments 'support stable environments for development, staging, and production' (baseten-docs-20) and that Truss models 'behave the same in development and production' (baseten-gh-1), implying separate non-prod environments exist. However there is no detailed documentation of how sandbox/staging is isolated from production data, no guidance on seeding test data, and no independent confirmation of this workflow. Missing for 10: explicit sandbox/staging setup guide, data-isolation guarantees, and hands-on/community corroboration of safe non-prod testing.

   • [claimed-docs] “They also support stable environments for development, staging, and production.”
   • [github] “Write once, run anywhere: Package model code, weights, and dependencies with a model server that behaves the same in development and product…”

   proof ↗⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round to Baseten
   Groq—none0/10

   No evidence pack item documents API versioning conventions or a formal deprecation policy for Groq's endpoints or models; docs cover OpenAI-compatibility, features, and pricing but not lifecycle/versioning guarantees.

   ⚑ flag

   Baseten~partialprobed3/10

   Baseten documents a deprecation policy for Model APIs, instructing users to migrate to dedicated deployments when weights are deprecated (baseten-docs-12), showing some commitment to managing API lifecycle. However, there is no evidence of explicit API versioning (e.g., version numbers, changelogs) and probes for an OpenAPI/swagger spec returned 404s, suggesting no formal versioned API contract is published. Missing for 10: explicit API version scheme, published changelog, machine-readable API spec, and independent confirmation of deprecation timelines.

   • [claimed-docs] “Migrate to a dedicated deployment with the deprecated model weights. Contact us for assistance.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.baseten.co/openapi.json, https://docs.baseten.co/swagger.json, https://docs.baseten.co/…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Groq
   Groq✓fullclaimed7/10

   Groq's Batch API explicitly supports submitting thousands of API requests asynchronously at 50% lower cost with a 24h-7day processing window, directly satisfying bulk operations across many items, and higher-throughput Flex processing further supports large-scale workloads. Missing for 10: independent hands-on verification of batch job completion/reliability at scale and details on per-batch item limits or error handling.

   • [claimed-docs] “Batch processing lets you run thousands of API requests at scale by submitting your workload as an asynchronous batch of requests to Groq wi…”
   • [claimed-docs] “Batch processing lets you run thousands of API requests at scale by submitting your workload as an asynchronous batch of requests to Groq wi…”
   • [claimed-docs] “Flex processing is available for all models to paid customers only with 10x higher rate limits compared to on-demand processing.”
   • [claimed-docs] “Flex processing is available for all models to paid customers only with 10x higher rate limits compared to on-demand processing. Pricing mat…”

   proof ↗⚑ flag

   Baseten~partialclaimed5/10

   Baseten's async inference is explicitly described as suited to 'batch work, long documents' where a request ID is returned and completion happens later via webhook or polling, which supports processing many items without holding open connections. However, there is no documented batch/bulk API endpoint for submitting many items in a single call, nor bulk management operations (e.g., bulk model/deployment updates) across the platform. Missing for 10: explicit multi-item batch submission API, bulk resource management (models/deployments/keys) in one operation, and independent evidence of throughput at scale for bulk workloads.

   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents, or any …”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling”

   proof ↗⚑ flag

Batch

1. ml-engineerSubmit asynchronous batch inference jobs at a documented discount versus real-time pricing

   weight 2 · round to Groq
   Groq✓fullclaimed8/10

   Groq's official docs explicitly describe an asynchronous batch API delivering 50% lower cost than standard on-demand pricing, with a 24-hour to 7-day processing window and no impact on standard rate limits (groq-docs-7, groq-docs-29). This directly matches the story's requirement of documented discounted async batch inference. Missing for 10: independent/hands-on confirmation of the batch discount actually being applied in practice, and more detail on job submission/monitoring workflow beyond the docs summary.

   • [claimed-docs] “Batch processing lets you run thousands of API requests at scale by submitting your workload as an asynchronous batch of requests to Groq wi…”
   • [claimed-docs] “Batch processing lets you run thousands of API requests at scale by submitting your workload as an asynchronous batch of requests to Groq wi…”

   proof ↗⚑ flag

   Baseten~partialclaimed4/10

   Baseten documents asynchronous inference (request ID returned quickly, completion via webhook/polling) explicitly for batch work and long documents, so the submission mechanism is real. However, the only documented pricing discount is for KV-cached input tokens, not for async/batch requests specifically — there is no evidence of a documented discount rate for async vs real-time inference pricing. missing for 10: a documented async/batch discount rate or pricing tier distinct from real-time pricing, first-party pricing page showing the discount percentage.

   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents, or any …”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling”
   • [claimed-docs] “Cached input tokens are prompt tokens served from the KV cache at a discounted rate. Caching is automatic and requires no request flags.”
   • [claimed-docs] “Model APIs bill by token and enforce request and token rate limits. You can also set a workspace budget and query usage by API key or model.”

   proof ↗⚑ flag

Dedicated

1. ml-engineerDeploy a model on dedicated GPU capacity with autoscaling so my traffic is isolated from the shared serverless pool

   weight 2 · round to Baseten
   Groq—none0/10

   Evidence shows Groq's service tiers (on-demand, flex, auto) and batch processing, all within the shared serverless API, but no documentation of dedicated GPU capacity or customer-isolated infrastructure with autoscaling; one community remark about 'call us for quotes' for newer models hints at an enterprise track but gives no detail on dedicated capacity or autoscaling.

   • [claimed-docs] “Groq offers multiple service tiers so you can tune for latency, throughput, and reliability.”
   • [claimed-docs] “Flex processing is available for all models to paid customers only with 10x higher rate limits compared to on-demand processing.”
   • [claimed-docs] “auto: Pass this if you dont want to think about tiers and you want to leverage the best tier available to you at any given moment.”
   • [community] “As soon as i saw they switched to 'call us for quotes' for the new models, i knew they are over.”

   proof ↗⚑ flag

   Baseten~partialclaimed5/10

   Baseten clearly supports deploying models on dedicated GPUs/dedicated infrastructure (baseten-docs-2, baseten-docs-17, baseten-docs-30) with production-stable environments (baseten-docs-20) and resilient multi-cluster deployment (baseten-docs-15), which is distinct from the shared Model APIs pool (baseten-docs-1). However, no evidence explicitly describes autoscaling behavior for these dedicated deployments (scale-to-zero, concurrency-based scaling, etc.), so the isolation-plus-autoscaling combination is only partially substantiated. Missing for 10: explicit autoscaling documentation for dedicated deployments, and independent/hands-on confirmation of traffic isolation from the shared pool.

   • [claimed-docs] “Deploy an open-source, fine-tuned, or custom model on dedicated GPUs.”
   • [claimed-docs] “Deploy an open-source, fine-tuned, or custom model on dedicated infrastructure.”
   • [claimed-docs] “Call hosted models through an OpenAI-compatible API, deploy your own models on dedicated infrastructure”
   • [claimed-docs] “They also support stable environments for development, staging, and production.”
   • [claimed-docs] “Deployments run active-active across clusters and clouds. If a region or provider loses capacity, MCM reroutes and reprovisions workloads.”
   • [claimed-docs] “Use Model APIs to call supported language models without deploying them.”

   proof ↗⚑ flag

Fine tune

1. ml-engineerFine-tune a supported base model on my own data and serve the result on the same platform

   weight 2 · round to Baseten
   Groq—none0/10

   Groq's docs show it supports serving pre-made LoRA adapters for inference (groq-docs-11, groq-docs-20, groq-docs-32), but there is no evidence of any fine-tuning/training capability on the platform itself — adapters must be trained elsewhere and uploaded. No mention of a fine-tuning API, training jobs, or dataset ingestion for model customization.

   • [claimed-docs] “Upload your existing LoRA adapters to run specialized inference while maintaining the performance and efficiency of Groq's infrastructure.”
   • [claimed-docs] “With LoRA inference on Groq, you can: Run inference with your pre-made LoRA adapters”
   • [claimed-docs] “Groq provides inference services for pre-made Low-Rank Adaptation (LoRA) adapters... Upload your existing LoRA adapters to run specialized i…”

   proof ↗⚑ flag

   Baseten✓fullclaimed8/10

   Baseten documents Training Jobs/Loops for fine-tuning supported base models (Axolotl/TRL/custom loops), automatic checkpoint syncing, and one-CLI-command deployment of any synced checkpoint to production inference on the same platform, matching the story end-to-end. missing for 10: independent/hands-on third-party confirmation of the fine-tune-to-serve workflow (community evidence only covers unrelated Chains/embedding topics) and explicit list of which base models are 'supported' for fine-tuning.

   • [claimed-docs] “Fine-tune with Loops or run your own training code with Training Jobs.”
   • [claimed-docs] “Baseten trains models on managed GPUs and deploys the resulting checkpoints to production inference on the same platform.”
   • [claimed-docs] “Baseten provisions GPUs through MCM, runs your training container, and syncs checkpoints to storage as the job progresses.”
   • [claimed-docs] “Your Axolotl config, TRL script, or custom loop runs unchanged in a container. Baseten provisions the GPUs, syncs checkpoints as your job sa…”
   • [claimed-docs] “Deploy any synced checkpoint with one CLI command”
   • [claimed-docs] “Fine-tuning or running RL on a supported base model: Loops provisions a dedicated trainer and paired sampler, and each training step is an A…”
   • [claimed-docs] “Loops provisions a dedicated trainer and paired sampler, and each training step is an API call from a Python loop you write.”

   proof ↗⚑ flag

2. ml-engineerUpload and serve my own custom model weights or LoRA adapters

   weight 2 · round to Baseten
   Groq~partialclaimed6/10

   Groq's docs explicitly describe LoRA adapter upload and serving ('Upload your existing LoRA adapters to run specialized inference' — groq-docs-11/20/32), directly matching part of the story. However, evidence only covers LoRA adapters on top of existing base models, not uploading fully custom model weights, and there's no independent/hands-on confirmation of this LoRA workflow actually working in practice. Missing for 10: evidence of custom full-weight model upload/serving, and independent corroboration that LoRA upload works reliably in production.

   • [claimed-docs] “Upload your existing LoRA adapters to run specialized inference while maintaining the performance and efficiency of Groq's infrastructure.”
   • [claimed-docs] “With LoRA inference on Groq, you can: Run inference with your pre-made LoRA adapters”
   • [claimed-docs] “Groq provides inference services for pre-made Low-Rank Adaptation (LoRA) adapters... Upload your existing LoRA adapters to run specialized i…”

   proof ↗⚑ flag

   Baseten✓fullclaimed8/10

   Baseten documents deploying custom/fine-tuned/open-source models via Truss (config.yaml + weights, supports transformers/diffusers/vLLM/etc.), and its Training product syncs LoRA/checkpoint artifacts that can be deployed as production endpoints with one CLI command, directly covering custom weights and LoRA adapter serving. Missing for 10: explicit LoRA-adapter-specific hot-swapping/multi-adapter serving docs and independent hands-on confirmation beyond vendor docs/GitHub README.

   • [claimed-docs] “Deploy an open-source, fine-tuned, or custom model on dedicated GPUs.”
   • [claimed-docs] “Deploy an open-source, fine-tuned, or custom model on dedicated infrastructure.”
   • [github] “Write once, run anywhere: Package model code, weights, and dependencies with a model server that behaves the same in development and product…”
   • [github] “You write a `config.yaml` that specifies the model, the hardware, and the engine, then `uvx truss push` builds a TensorRT-optimized containe…”
   • [github] “Support for all Python frameworks: From `transformers` and `diffusers` to PyTorch and TensorFlow to vLLM, SGLang, and TensorRT-LLM, Truss su…”
   • [claimed-docs] “Your Axolotl config, TRL script, or custom loop runs unchanged in a container. Baseten provisions the GPUs, syncs checkpoints as your job sa…”
   • [claimed-docs] “Deploy any synced checkpoint with one CLI command”
   • [claimed-docs] “Baseten trains models on managed GPUs and deploys the resulting checkpoints to production inference on the same platform.”

   proof ↗⚑ flag

Catalog

1. ml-engineerGet newly released open-weight models on the platform quickly after their public release

   weight 2 · round to Groq
   Groq~partialcommunity5/10

   Groq's docs list several open-weight models (e.g. GPT-OSS 120B, Llama, Mixtral) and community reports mention using Kimi K2 'fast' shortly after its release, suggesting some open-weight models are added promptly, but there is no explicit documentation of a release-to-availability SLA or a systematic track record of rapid onboarding. Missing for 10: explicit timeline commitments or a changelog showing new open-weight model releases added within days, and independent verification beyond a single anecdote.

   • [claimed-docs] “GPT-OSS 120B is OpenAI's flagship open-weight language model with 120 billion parameters, built in browser search and code execution, and re…”
   • [community] “I wanted to use Kimi K2 fast for coding and Groq was the only fast provider at the time... Definitely recommend cerebras tho now that groq's…”
   • [claimed-docs] “Upload your existing LoRA adapters to run specialized inference while maintaining the performance and efficiency of Groq's infrastructure.”

   proof ↗⚑ flag

   Baseten—none0/10

   The evidence describes Baseten's general capabilities (Model APIs, Truss deployment, training) but contains no claims or evidence about how quickly newly released open-weight models are added to the hosted Model APIs catalog after their public release. Missing for 10: any documentation or track record on model-catalog update cadence, day-1 support claims, or examples of specific new model releases being onboarded quickly.

   ⚑ flag

2. developerRely on a documented deprecation policy with advance notice before a hosted model is removed

   weight 1 · round to Baseten
   Groq—none0/10

   No evidence pack item documents a deprecation policy, sunset timeline, or advance-notice commitment for hosted models being removed; community comments even reference abrupt shifts (e.g., new models moved to 'call us for quotes') without formal notice process. This is a fair question for a model-hosting API but no supporting documentation exists.

   • [community] “As soon as i saw they switched to 'call us for quotes' for the new models, i knew they are over.”

   proof ↗⚑ flag

   Baseten~partialclaimed4/10

   Baseten has a dedicated deprecation docs page instructing users to migrate to a dedicated deployment with deprecated model weights, showing some documented deprecation handling exists, but the evidence pack shows no explicit advance-notice period, timeline, or notification process. Missing for 10: concrete advance-notice window/SLA, notification channel details, and independent confirmation that notice is actually given before removal.

   • [claimed-docs] “Migrate to a dedicated deployment with the deprecated model weights. Contact us for assistance.”

   proof ↗⚑ flag

3. ai-native userHave an agent enumerate the live model catalog programmatically via a documented GET /v1/models-style endpoint

   weight 2 · round drawn
   Groq✓fullprobed8/10

   Groq exposes an OpenAI-compatible /v1/models endpoint (confirmed live via probe returning JSON error structure, not 404), and docs establish OpenAI SDK/client compatibility with base_url https://api.groq.com/openai/v1, which per OpenAI spec includes the models list endpoint; this is documented and independently verified via a real GET request. missing for 10: no explicit Groq-authored documentation page specifically describing the /v1/models endpoint response schema, and no successful authenticated enumeration example shown (only the 401 unauthenticated probe).

   • [probe] “PROBE models-endpoint (2026-09-04): GET https://api.groq.com/openai/v1/models without a key returned HTTP 401 ({"error":{"message":"Invalid …”
   • [claimed-docs] “pass your Groq API key to the `api_key` parameter and change the `base_url` to `https://api.groq.com/openai/v1`”
   • [claimed-docs] “To start using Groq with OpenAI's client libraries, pass your Groq API key to the api_key parameter and change the base_url to https://api.g…”
   • [claimed-docs] “Fast LLM inference, OpenAI-compatible. Simple to integrate, easy to scale. Start building in minutes.”

   proof ↗⚑ flag

   Baseten✓fullprobed8/10

   A live probe confirms the OpenAI-style GET /v1/models endpoint exists at https://inference.baseten.co/v1/models and returns proper JSON auth-error responses when unauthenticated, consistent with Baseten's documented OpenAI-compatible Chat Completions API (baseten-docs-21, baseten-docs-30). This lets an agent enumerate the model catalog programmatically once authenticated. Missing for 10: explicit first-party documentation of the /v1/models endpoint's response schema/fields and an authenticated hands-on example showing the returned catalog contents.

   • [probe] “PROBE models-endpoint (2026-09-04): GET https://inference.baseten.co/v1/models without a key returned HTTP 401 (No Authorization header prov…”
   • [claimed-docs] “They support the OpenAI Chat Completions API and the Anthropic Messages API in beta, so you can use familiar client SDKs.”
   • [claimed-docs] “Call hosted models through an OpenAI-compatible API, deploy your own models on dedicated infrastructure”

   proof ↗⚑ flag

4. developerChoose among a broad catalog of current open-weight model families (Llama, Qwen, DeepSeek, GPT-OSS and peers) on shared serverless endpoints

   weight 3 · round to Groq
   Groq~partialprobed6/10

   Groq's docs confirm OpenAI-compatible serverless endpoints hosting multiple model families, explicitly naming GPT-OSS 120B and (via community reports) Llama and Mixtral models, but the evidence pack never mentions Qwen or DeepSeek being hosted, so the 'broad catalog' claim is only partially substantiated. missing for 10: explicit documentation of Qwen and DeepSeek model availability, and a public model list/catalog page confirming breadth beyond GPT-OSS/Llama/Mixtral.

   • [claimed-docs] “GPT-OSS 120B is OpenAI's flagship open-weight language model with 120 billion parameters, built in browser search and code execution, and re…”
   • [claimed-docs] “To start using Groq with OpenAI's client libraries, pass your Groq API key to the api_key parameter and change the base_url to https://api.g…”
   • [community] “Incredible tool. The Mixtral 8x7B model running on their hardware did 491.40 T/s for me…”
   • [community] “I'm achieving consistent 450+ tokens/sec for Mixtral 8x7b 32k and ~200 tps for Llama 2 70B-4k.”
   • [probe] “PROBE models-endpoint (2026-09-04): GET https://api.groq.com/openai/v1/models without a key returned HTTP 401 ({"error":{"message":"Invalid …”

   proof ↗⚑ flag

   Baseten~partialprobed4/10

   Baseten's docs confirm a serverless 'Model APIs' offering for calling supported language models without deploying them, with OpenAI/Anthropic-compatible endpoints, implying a hosted catalog exists — but no evidence enumerates specific open-weight families like Llama, Qwen, DeepSeek, or GPT-OSS, and the live models endpoint requires an API key to inspect. missing for 10: explicit list of supported model families, breadth/currency of catalog, independent confirmation of specific models available on shared serverless endpoints.

   • [claimed-docs] “Use Model APIs to call supported language models without deploying them.”
   • [claimed-docs] “They support the OpenAI Chat Completions API and the Anthropic Messages API in beta, so you can use familiar client SDKs.”
   • [claimed-docs] “Call hosted models through an OpenAI-compatible API, deploy your own models on dedicated infrastructure”
   • [probe] “PROBE models-endpoint (2026-09-04): GET https://inference.baseten.co/v1/models without a key returned HTTP 401 (No Authorization header prov…”

   proof ↗⚑ flag

Modalities

1. developerGenerate embeddings (and rerank results) for retrieval pipelines without a second vendor

   weight 1 · round to Baseten
   Groq—none0/10

   No evidence of an embeddings endpoint or reranking model/API in Groq's documentation; the evidence pack covers chat completions, tool use, transcription, LoRA, batch, and service tiers but never mentions embeddings or rerank capabilities.

   ⚑ flag

   Baseten~partialcommunity5/10

   Baseten's general-purpose model deployment (Truss, custom model hosting, transformers/diffusers/vLLM support) can host embedding or reranking models on dedicated infrastructure, and a community report confirms real-world 'Baseten.co embedding workloads' being run on the platform. However, there is no first-party documentation of a dedicated embeddings or rerank API/product page, no mention of specific embedding model catalog entries, or built-in reranking features analogous to competitors' turnkey embedding endpoints. missing for 10: dedicated embeddings/rerank API docs, named embedding/rerank models in the catalog, first-party guidance on building RAG retrieval pipelines end-to-end.

   • [community] “Developer notes that with Baseten.co embedding workloads, the client (not server) becomes the bottleneck due to Python's GIL, prompting them…”
   • [claimed-docs] “Deploy an open-source, fine-tuned, or custom model on dedicated GPUs.”
   • [claimed-docs] “Deploy an open-source, fine-tuned, or custom model on dedicated infrastructure.”
   • [github] “Support for all Python frameworks: From `transformers` and `diffusers` to PyTorch and TensorFlow to vLLM, SGLang, and TensorRT-LLM, Truss su…”
   • [github] “Truss lets you serve models with the Baseten Inference Stack as well as deploy models from any open-source framework: vLLM, SGLang, TensorRT…”

   proof ↗⚑ flag

2. developerCall vision, audio, or image-generation models beyond text chat on the same platform

   weight 1 · round to Groq
   Groq~partialclaimed6/10

   Groq's docs confirm audio transcription (Whisper models via groq-gh-2) and vision input support through the Responses API (text and image inputs, groq-docs-13), extending beyond pure text chat. However, there is no evidence of image-generation models or text-to-speech/audio-generation capability on the platform. missing for 10: image-generation model support, audio-generation/TTS support, independent hands-on confirmation of multimodal calls.

   • [github] “await client.audio.transcriptions.create({ model: 'whisper-large-v3-turbo', file: fs.createReadStream('/path/to/file'), });”
   • [claimed-docs] “The Responses API supports both text and image inputs while producing text outputs, stateful conversations, and function calling to connect …”
   • [claimed-docs] “Groq's Responses API is fully compatible with OpenAI's Responses API, making it easy to integrate advanced conversational AI capabilities in…”

   proof ↗⚑ flag

   Baseten~partialclaimed5/10

   Baseten's Truss framework explicitly supports 'diffusers' (used for image-generation models) and any open-source framework, and the platform lets you deploy 'open-source, fine-tuned, or custom' models on dedicated GPUs, implying vision/audio/image models can be hosted alongside text. However, the hosted 'Model APIs' product is explicitly scoped to 'supported language models' (baseten-docs-1), and no evidence names specific vision, audio, or image-generation models or endpoints being offered as first-class hosted APIs — multimodal support relies on self-deploying via Truss rather than a turnkey catalog like text chat models get. missing for 10: explicit vision/audio/image-generation model catalog or API examples, first-party documentation naming specific multimodal models supported out-of-the-box.

   • [claimed-docs] “Use Model APIs to call supported language models without deploying them.”
   • [claimed-docs] “Deploy an open-source, fine-tuned, or custom model on dedicated infrastructure.”
   • [github] “Support for all Python frameworks: From `transformers` and `diffusers` to PyTorch and TensorFlow to vLLM, SGLang, and TensorRT-LLM, Truss su…”
   • [github] “Truss lets you serve models with the Baseten Inference Stack as well as deploy models from any open-source framework: vLLM, SGLang, TensorRT…”
   • [claimed-docs] “Use a Python model class or a custom Docker server when you need custom preprocessing, postprocessing, dependencies, or server behavior.”

   proof ↗⚑ flag

Compat

1. ai-native userPlug the provider into coding agents and agent frameworks through documented, first-party integration guides

   weight 2 · round to Baseten
   Groq~partialprobed7/10

   Groq's docs clearly document OpenAI-compatible base_url swapping (groq-docs-2/25), tool-use/function-calling (groq-docs-5/27), a Responses API that is 'fully compatible with OpenAI's Responses API' (groq-docs-23), and first-party remote MCP server support where 'Groq handles all orchestration' (groq-docs-6/17/28, groq-probe-4) — all of which make it straightforward to plug into agent frameworks that speak the OpenAI protocol or MCP. However, there are no named first-party guides for specific coding-agent/agent-framework integrations (e.g., LangChain, Cursor, AutoGPT) beyond the generic OpenAI-compat pointer, and community reports flag real-world reliability quirks (groq-comm-12/13/14) that add friction beyond the docs' 'simple to integrate' framing. Missing for 10: named integration guides for specific agent frameworks/coding agents, and independent hands-on confirmation of smooth agent-framework plug-in.

   • [claimed-docs] “pass your Groq API key to the `api_key` parameter and change the `base_url` to `https://api.groq.com/openai/v1`”
   • [claimed-docs] “To start using Groq with OpenAI's client libraries, pass your Groq API key to the api_key parameter and change the base_url to https://api.g…”
   • [claimed-docs] “Tool use (or function calling) is what transforms a language model from a conversational interface into an autonomous agent capable of takin…”
   • [claimed-docs] “To use tools, the model must be provided with tool definitions. These tool definitions are in JSON schema format and are passed to the model…”
   • [claimed-docs] “point to an MCP server URL and the Groq API will start using its tools without you having to implement any tool logic yourself”
   • [claimed-docs] “you simply point to an MCP server URL and the Groq API will start using its tools without you having to implement any tool logic yourself”
   • [claimed-docs] “Groq's Responses API supports remote tool use via MCP servers via HTTPS where Groq handles all orchestration... You don't implement anything…”
   • [claimed-docs] “Groq's Responses API is fully compatible with OpenAI's Responses API, making it easy to integrate advanced conversational AI capabilities in…”
   • [probe] “official MCP server documented at https://console.groq.com/docs/mcp”
   • [community] “My company had a really terrible experience trying to use Groq, and I would NOT recommend anyone use their service if you need reliability. …”
   • [community] “I don't really get the value proposition of groq as a user, the performance is really poor for the token price.”

   proof ↗⚑ flag

   Baseten✓fullprobed9/10

   Baseten has dedicated first-party docs for connecting coding agents (Claude Code, Codex CLI, Pi) via Baseten Switch, plus a documented MCP server/skill for agent workspace management, and OpenAI/Anthropic-compatible endpoints enabling framework integrations like LangChain. This is corroborated by a live probe confirming the MCP endpoint responds correctly. Missing for 10: independent hands-on validation of the coding-agent connection flow beyond docs/probes.

   • [claimed-docs] “To point a coding agent at Model APIs, see [Coding agents](/inference/model-apis/coding-agents).”
   • [claimed-docs] “Install the Baseten skill and MCP servers so your coding agent can manage your Baseten workspace and search these docs.”
   • [claimed-docs] “Switch can also install Pi's direct Baseten provider, compare Baseten spend with estimated costs from Anthropic or OpenAI, and route request…”
   • [claimed-docs] “Connect Claude Code, Codex CLI, or Pi with Baseten Switch.”
   • [claimed-docs] “Use [Baseten Switch](/reference/cli/baseten-switch) to route requests from Claude Code or Codex CLI to Model APIs.”
   • [claimed-docs] “Because Baseten exposes an OpenAI-compatible endpoint, you can use LangChain's `ChatOpenAI` with `with_structured_output` by pointing `base_…”
   • [probe] “official MCP server documented at https://docs.baseten.co/agent-setup”
   • [probe] “PROBE mcp-endpoint (2026-09-04): POST initialize to https://docs.baseten.co/mcp answered HTTP 200 with a JSON-RPC/MCP response (event: messa…”

   proof ↗⚑ flag

2. ai-native userHave an agent switch to or away from this provider mid-workflow because it speaks the standard chat-completions API without provider-specific code changes

   weight 3 · round drawn
   Groq✓fullprobed8/10

   Groq's docs explicitly instruct developers to reuse OpenAI client libraries by just swapping api_key and base_url to https://api.groq.com/openai/v1, and this is corroborated by a live probe showing the OpenAI-style /v1/models endpoint responding in the expected JSON error format. Chat completions, streaming, tool-use, and structured outputs are all documented against this same OpenAI-compatible surface, supporting drop-in provider switching mid-workflow. Missing for 10: independent/hands-on confirmation from a real agent framework that switching providers requires zero code changes, and clarity on any OpenAI parameters/features that are NOT supported (full parity claim not verified end-to-end).

   • [claimed-docs] “pass your Groq API key to the `api_key` parameter and change the `base_url` to `https://api.groq.com/openai/v1`”
   • [claimed-docs] “To start using Groq with OpenAI's client libraries, pass your Groq API key to the api_key parameter and change the base_url to https://api.g…”
   • [claimed-docs] “Fast LLM inference, OpenAI-compatible. Simple to integrate, easy to scale. Start building in minutes.”
   • [probe] “PROBE models-endpoint (2026-09-04): GET https://api.groq.com/openai/v1/models without a key returned HTTP 401 ({"error":{"message":"Invalid …”
   • [claimed-docs] “Generating text with Groq's Chat Completions API enables you to have natural, conversational interactions with Groq's large language models.”

   proof ↗⚑ flag

   Baseten✓fullclaimed8/10

   Baseten documents OpenAI Chat Completions API compatibility for Model APIs, explicitly enabling drop-in use with familiar client SDKs (e.g., LangChain's ChatOpenAI via base_url swap), and even documents a Switch tool that can route requests between Baseten and other providers like Anthropic/OpenAI mid-workflow. This directly supports agent-driven provider switching without provider-specific code changes. Missing for 10: independent/hands-on confirmation of seamless mid-workflow switching beyond vendor docs, and no explicit statement of full drop-in parity with OpenAI's complete API surface.

   • [claimed-docs] “They support the OpenAI Chat Completions API and the Anthropic Messages API in beta, so you can use familiar client SDKs.”
   • [claimed-docs] “Call hosted models through an OpenAI-compatible API, deploy your own models on dedicated infrastructure”
   • [claimed-docs] “Because Baseten exposes an OpenAI-compatible endpoint, you can use LangChain's `ChatOpenAI` with `with_structured_output` by pointing `base_…”
   • [claimed-docs] “Switch can also install Pi's direct Baseten provider, compare Baseten spend with estimated costs from Anthropic or OpenAI, and route request…”
   • [claimed-docs] “Use [Baseten Switch](/reference/cli/baseten-switch) to route requests from Claude Code or Codex CLI to Model APIs.”

   proof ↗⚑ flag

3. developerPoint an existing OpenAI SDK client at the provider by changing only the base URL and API key

   weight 3 · round drawn
   Groq✓fullprobed9/10

   Groq's docs explicitly instruct developers to point OpenAI SDK clients at Groq by only changing api_key and base_url to https://api.groq.com/openai/v1, and a live probe confirms the OpenAI-style /v1/models endpoint is functional. This is corroborated by consistent documentation across chat completions, streaming, tool use, and structured outputs all using OpenAI SDK conventions. Missing for 10: independent third-party confirmation of a drop-in swap working flawlessly in production (community evidence focuses on speed/reliability rather than compatibility specifics).

   • [claimed-docs] “pass your Groq API key to the `api_key` parameter and change the `base_url` to `https://api.groq.com/openai/v1`”
   • [claimed-docs] “To start using Groq with OpenAI's client libraries, pass your Groq API key to the api_key parameter and change the base_url to https://api.g…”
   • [probe] “PROBE models-endpoint (2026-09-04): GET https://api.groq.com/openai/v1/models without a key returned HTTP 401 ({"error":{"message":"Invalid …”
   • [claimed-docs] “Fast LLM inference, OpenAI-compatible. Simple to integrate, easy to scale. Start building in minutes.”

   proof ↗⚑ flag

   Baseten✓fullprobed9/10

   Baseten explicitly documents an OpenAI Chat Completions-compatible endpoint for its Model APIs, and shows real-world usage where only base_url and API key need to change (e.g. LangChain's ChatOpenAI pointed at Baseten). A live probe confirms an OpenAI-style /v1/models endpoint responding at inference.baseten.co. Missing for 10: no independent third-party hands-on report explicitly confirming a drop-in swap with the official OpenAI SDK client itself (only LangChain example and docs claims).

   • [claimed-docs] “They support the OpenAI Chat Completions API and the Anthropic Messages API in beta, so you can use familiar client SDKs.”
   • [claimed-docs] “Call hosted models through an OpenAI-compatible API, deploy your own models on dedicated infrastructure”
   • [claimed-docs] “Because Baseten exposes an OpenAI-compatible endpoint, you can use LangChain's `ChatOpenAI` with `with_structured_output` by pointing `base_…”
   • [probe] “PROBE models-endpoint (2026-09-04): GET https://inference.baseten.co/v1/models without a key returned HTTP 401 (No Authorization header prov…”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round drawn
   Groq~partialprobed6/10

   Groq's API surface covers essentially all the model-facing capabilities shown in its console/docs — chat completions, streaming, tool use, structured outputs, batch processing, service tiers, LoRA inference, and MCP remote tool use are all documented as API features (groq-docs-2,5,6,7,8,11,16,28). However, account/billing controls like spend limits (groq-docs-12/24) appear to be console-only with no documented API equivalent, and the probe found no public OpenAPI spec (all candidate paths 404, groq-probe-3), making it unclear whether the full API surface is discoverable/self-describing the way the UI is. Missing for 10: API-based management of billing/spend limits, a published OpenAPI/schema confirming full UI-API parity, and independent confirmation that every console feature (e.g., playground-only settings) has an API equivalent.

   • [claimed-docs] “pass your Groq API key to the `api_key` parameter and change the `base_url` to `https://api.groq.com/openai/v1`”
   • [claimed-docs] “Tool use (or function calling) is what transforms a language model from a conversational interface into an autonomous agent capable of takin…”
   • [claimed-docs] “point to an MCP server URL and the Groq API will start using its tools without you having to implement any tool logic yourself”
   • [claimed-docs] “Batch processing lets you run thousands of API requests at scale by submitting your workload as an asynchronous batch of requests to Groq wi…”
   • [claimed-docs] “Upload your existing LoRA adapters to run specialized inference while maintaining the performance and efficiency of Groq's infrastructure.”
   • [claimed-docs] “Structured Outputs is a feature that ensures your model responses conform to your provided JSON Schema”
   • [claimed-docs] “Groq's Responses API supports remote tool use via MCP servers via HTTPS where Groq handles all orchestration... You don't implement anything…”
   • [claimed-docs] “Spend Limits: Set automated spending limits and receive budget alerts”
   • [probe] “PROBE openapi: all candidate paths 404 (https://console.groq.com/openapi.json, https://console.groq.com/swagger.json, https://console.groq.c…”

   proof ↗⚑ flag

   Baseten~partialprobed6/10

   Baseten's core workflows — deploying models (Truss/config.yaml/CLI), running inference (OpenAI/Anthropic-compatible endpoints, streaming, async, structured outputs, function calling), and training (Loops API calls, Training Jobs CLI deploy) — are all API/CLI-driven, and usage/budget can be queried by API key, suggesting strong API-UI parity for the platform's main functions. However, there is no explicit documentation asserting full UI/API parity, no discoverable OpenAPI/reference spec (probe found only 404s), and no evidence that workspace/account-management UI features (team, billing settings) are exposed via API. Missing for 10: an explicit parity statement or full API reference, confirmation of workspace/admin functions being API-accessible, and independent verification of parity beyond inference/training/deploy workflows.

   • [github] “You write a `config.yaml` that specifies the model, the hardware, and the engine, then `uvx truss push` builds a TensorRT-optimized containe…”
   • [claimed-docs] “Your Axolotl config, TRL script, or custom loop runs unchanged in a container. Baseten provisions the GPUs, syncs checkpoints as your job sa…”
   • [claimed-docs] “Deploy any synced checkpoint with one CLI command”
   • [claimed-docs] “Loops provisions a dedicated trainer and paired sampler, and each training step is an API call from a Python loop you write.”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents”
   • [claimed-docs] “Structured outputs let you generate text that conforms to specific JSON schemas, providing reliable data extraction and controlled text gene…”
   • [claimed-docs] “Function calling* (also called *tool calling*) lets a model choose a tool and produce its arguments from a user request.”
   • [claimed-docs] “You can also set a workspace budget and query usage by API key or model.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.baseten.co/openapi.json, https://docs.baseten.co/swagger.json, https://docs.baseten.co/…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to Baseten
   Groq—none0/10

   Groq's docs cover API usage, models, tool-use, batch processing and billing, but nothing addresses exporting account data, conversation history, or batch outputs in open/portable formats, nor any account-closure data dump. Even though Groq's Responses API is 'stateful,' there is no documented export or data-portability mechanism.

   ⚑ flag

   Baseten~partialclaimed4/10

   Baseten's tooling leans open: models are packaged via the open-source Truss format (config.yaml, no proprietary container lock-in), and training checkpoints are synced to external storage as the job runs, so a user's model weights and training artifacts aren't trapped in a proprietary format. However there is no explicit documentation of exporting workspace data, logs, usage/billing history, or deployment configs in bulk, and no dedicated 'account export' or data-portability feature. missing for 10: explicit full-account/data export tooling, documentation of exporting logs/usage/config metadata, and any user testimonial of migrating away with all data intact.

   • [github] “Write once, run anywhere: Package model code, weights, and dependencies with a model server that behaves the same in development and product…”
   • [github] “You write a `config.yaml` that specifies the model, the hardware, and the engine, then `uvx truss push` builds a TensorRT-optimized containe…”
   • [claimed-docs] “Your Axolotl config, TRL script, or custom loop runs unchanged in a container. Baseten provisions the GPUs, syncs checkpoints as your job sa…”
   • [claimed-docs] “Deploy any synced checkpoint with one CLI command”
   • [claimed-docs] “Baseten provisions GPUs through MCM, runs your training container, and syncs checkpoints to storage as the job progresses.”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round to Baseten
   Groq—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Baseten~partialclaimed3/10

   Baseten's core platform (inference service, dashboard, orchestration/MCM) is proprietary SaaS with no evidence of an open license; only the Truss packaging library is open-source on GitHub, which is a peripheral client tool rather than the product itself. missing for 10: no open license for the actual Baseten platform/service source, no repository for the core inference or orchestration engine, only a tangential SDK is open.

   • [github] “Write once, run anywhere: Package model code, weights, and dependencies with a model server that behaves the same in development and product…”
   • [github] “You write a `config.yaml` that specifies the model, the hardware, and the engine, then `uvx truss push` builds a TensorRT-optimized containe…”
   • [github] “Fast developer loop: Iterate with live reload, skip Docker and Kubernetes configuration, and use a batteries-included serving environment.”
   • [github] “Support for all Python frameworks: From `transformers` and `diffusers` to PyTorch and TensorFlow to vLLM, SGLang, and TensorRT-LLM, Truss su…”
   • [github] “Truss lets you serve models with the Baseten Inference Stack as well as deploy models from any open-source framework: vLLM, SGLang, TensorRT…”
   • [github] “Deploying a model to Baseten via Truss turns a Hugging Face model into a production-ready API endpoint. You write a `config.yaml` that speci…”

   proof ↗⚑ flag

Limits

1. developerRead documented rate limits and how they scale across usage tiers before I hit them in production

   weight 2 · round to Baseten
   Groq~partialclaimed5/10

   Groq's docs describe multiple service tiers (on-demand, flex, auto) and note Flex Processing gives '10x higher rate limits' for paid customers, plus batch processing having 'no impact to your standard rate limits' and spend-limit alerts — showing tiered scaling exists and is documented at a high level. However, no evidence pack citation shows a dedicated rate-limits page with concrete numeric request/token-per-minute limits per tier (free vs paid vs flex), so a developer can't get exact production-planning numbers from what's cited. Missing for 10: a documented numeric rate-limit table per usage tier, and independent confirmation that these documented limits match real-world behavior.

   • [claimed-docs] “Groq offers multiple service tiers so you can tune for latency, throughput, and reliability.”
   • [claimed-docs] “Flex processing is available for all models to paid customers only with 10x higher rate limits compared to on-demand processing.”
   • [claimed-docs] “Flex Processing is a service tier optimized for high-throughput workloads that prioritizes fast inference and can handle occasional request …”
   • [claimed-docs] “auto: Pass this if you dont want to think about tiers and you want to leverage the best tier available to you at any given moment.”
   • [claimed-docs] “Batch processing lets you run thousands of API requests at scale by submitting your workload as an asynchronous batch of requests to Groq wi…”
   • [claimed-docs] “Flex processing is available for all models to paid customers only with 10x higher rate limits compared to on-demand processing. Pricing mat…”
   • [claimed-docs] “Groq offers multiple service tiers so you can tune for latency, throughput, and reliability. You can distinguish these by providing the serv…”
   • [claimed-docs] “Spend Limits:** Set automated spending limits and receive budget alerts”

   proof ↗⚑ flag

   Baseten~partialclaimed6/10

   Baseten has a dedicated pricing-and-limits doc covering request/token rate limits, rate-limit response headers, tier upgrades (Basic→Pro/Enterprise via email verification), workspace budgets, and usage monitoring by API key/model. However, it lacks a detailed table of exact numeric limits per tier and doesn't cover rate limits for dedicated-deployment (non-Model-APIs) inference. missing for 10: explicit numeric limit tables per tier, independent/community corroboration of documented limits, coverage of dedicated deployment rate limits (not just Model APIs).

   • [claimed-docs] “Model APIs bill by token and enforce request and token rate limits. You can also set a workspace budget and query usage by API key or model.”
   • [claimed-docs] “To monitor token and request consumption by API key or model, see Usage.”
   • [claimed-docs] “To raise a Basic account's limits, request email verification. You can also use that form to move to Pro or Enterprise.”
   • [claimed-docs] “x-ratelimit-remaining-requests: Reports how many requests remain before you reach the request rate limit.”
   • [claimed-docs] “You can also set a workspace budget and query usage by API key or model.”

   proof ↗⚑ flag

Pricing

1. founderSet spending caps or budget alerts so a runaway workload cannot generate an unbounded bill

   weight 1 · round to Groq
   Groq✓fullclaimed8/10

   Groq's billing docs explicitly state users can set automated spending limits and receive budget alerts, directly matching the story. Missing for 10: no independent/hands-on corroboration of the spend-limit UI actually working, and no detail on granularity (per-project vs account-wide) or alert channels.

   • [claimed-docs] “Spend Limits: Set automated spending limits and receive budget alerts”
   • [claimed-docs] “Spend Limits:** Set automated spending limits and receive budget alerts”

   proof ↗⚑ flag

   Baseten~partialclaimed5/10

   Baseten docs mention setting a workspace budget and monitoring usage by API key/model, plus rate limits (request/token) that cap throughput, which supports pricing-limits awareness (baseten-docs-10, -23, -37, -38). However, there's no evidence of automated spending caps that hard-stop billing or proactive budget alerts/notifications — the workspace budget appears to be a settable figure without documented enforcement or alerting mechanics. Missing for 10: explicit alerting/notification system, enforcement behavior when budget is exceeded, and independent confirmation that budgets actually cap spend rather than just track it.

   • [claimed-docs] “Model APIs bill by token and enforce request and token rate limits. You can also set a workspace budget and query usage by API key or model.”
   • [claimed-docs] “To monitor token and request consumption by API key or model, see Usage.”
   • [claimed-docs] “x-ratelimit-remaining-requests: Reports how many requests remain before you reach the request rate limit.”
   • [claimed-docs] “You can also set a workspace budget and query usage by API key or model.”

   proof ↗⚑ flag

2. founderSee public per-token prices for every hosted model without talking to sales

   weight 3 · round to Baseten
   Groq!disputedcontradicted4/10

   Groq's docs discuss billing mechanics (spend limits, flex-processing pricing parity, batch discount) implying a public pricing model, but no evidence pack item shows an actual public per-token price sheet for every hosted model, and a community report explicitly states Groq switched to 'call us for quotes' for newer models, contradicting the claim of fully public pricing without sales contact. Missing for 10: an actual public pricing page listing per-token rates for each model, and confirmation that newer/premium models are not gated behind sales quotes.

   • [claimed-docs] “Spend Limits: Set automated spending limits and receive budget alerts”
   • [claimed-docs] “Spend Limits:** Set automated spending limits and receive budget alerts”
   • [claimed-docs] “Flex processing is available for all models to paid customers only with 10x higher rate limits compared to on-demand processing. Pricing mat…”
   • [community] “As soon as i saw they switched to 'call us for quotes' for the new models, i knew they are over.”
   • [community] “Very impressive! I am even more impressed by the API pricing though - 0.27/1M tokens seems like an order of magnitude cheaper than the GPT-3…”

   proof ↗⚑ flag

   Baseten~partialprobed3/10

   Docs confirm Model APIs are billed per token and that usage/rate limits can be queried without contacting sales, but no evidence shows an actual public price list with per-model per-token rates; the models endpoint even requires an API key to enumerate the catalog. missing for 10: a public pricing/rate-card page listing token prices per model, independent confirmation that pricing is visible without account creation or sales contact.

   • [claimed-docs] “Model APIs bill by token and enforce request and token rate limits. You can also set a workspace budget and query usage by API key or model.”
   • [claimed-docs] “To raise a Basic account's limits, request email verification. You can also use that form to move to Pro or Enterprise.”
   • [probe] “PROBE models-endpoint (2026-09-04): GET https://inference.baseten.co/v1/models without a key returned HTTP 401 (No Authorization header prov…”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   Groq—none0/10

   No evidence in the pack mentions data residency, region selection, or geographic storage controls anywhere in Groq's docs, GitHub, or community discussion.

   ⚑ flag

   Baseten—none0/10

   No evidence in the pack mentions region selection, data residency options, or multi-region storage controls for user data; deployments are described as active-active across clusters/clouds for reliability (baseten-docs-15), not as a user-selectable residency control. Missing for 10: any mention of region/residency selection, data storage location controls, or compliance-driven data locality options.

   • [claimed-docs] “Deployments run active-active across clusters and clouds. If a region or provider loses capacity, MCM reroutes and reprovisions workloads.”

   proof ↗⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Groq—none0/10

   No evidence pack item addresses data usage/training policies, opt-out controls, or privacy commitments regarding whether user data is used to train Groq's models.

   ⚑ flag

   Baseten—none0/10

   No evidence pack content addresses data usage for training, opt-out controls, or any privacy/data-retention policy commitments; the axis is applicable (Baseten processes customer data/prompts and could plausibly offer such guarantees) but nothing in the docs, GitHub, or community evidence confirms it.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Groq—none0/10

   No evidence pack items address data retention policies, deletion controls, or user-facing privacy/data management settings for Groq's API; the docs focus on inference features, tooling, pricing, and rate limits with nothing on retention/deletion.

   ⚑ flag

   Baseten—none0/10

   The evidence pack covers deployment, training, inference APIs, pricing, and coding-agent integration, but no documentation addresses data retention policies, deletion of inputs/outputs, or user control over stored data/logs. This is a fair axis for an AI inference/hosting platform (buyers routinely ask about data handling), so it applies, but there is no evidence it is delivered.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   Groq—none0/10

   No evidence pack item addresses telemetry, usage tracking, opt-out settings, or privacy controls for Groq's API or console; the docs cover inference features, pricing, and infra but nothing about data collection preferences.

   ⚑ flag

   Baseten—none0/10

   No evidence in the pack addresses telemetry/usage-tracking opt-out settings for Baseten's platform, CLI, or SDK; the evidence only covers inference features, training, and pricing/usage monitoring for billing purposes, not privacy telemetry controls.

   ⚑ flag

Reliability

1. founderGet a stated availability SLA on paid or enterprise tiers

   weight 1 · round drawn
   Groq—none0/10

   Evidence shows service tiers, flex processing, and a public status page, but nowhere is there a stated uptime/response-time SLA commitment for paid or enterprise customers; enterprise tiers are only referenced via 'call us for quotes' community comments with no SLA terms disclosed.

   • [claimed-docs] “Groq offers multiple service tiers so you can tune for latency, throughput, and reliability.”
   • [claimed-docs] “Groq offers multiple service tiers so you can tune for latency, throughput, and reliability. You can distinguish these by providing the serv…”
   • [probe] “PROBE status-page (2026-09-04): https://groqstatus.com returns HTTP 200 and renders a public service-status page (page body includes "operat…”
   • [community] “As soon as i saw they switched to 'call us for quotes' for the new models, i knew they are over.”

   proof ↗⚑ flag

   Baseten—none0/10

   Baseten documents Pro/Enterprise tiers, active-active multi-cloud deployments, and a public status page, but no evidence pack item states an actual SLA (uptime percentage, response-time guarantee, or contractual commitment) for any paid or enterprise tier.

   • [claimed-docs] “To raise a Basic account's limits, request email verification. You can also use that form to move to Pro or Enterprise.”
   • [claimed-docs] “Deployments run active-active across clusters and clouds. If a region or provider loses capacity, MCM reroutes and reprovisions workloads.”
   • [probe] “PROBE status-page (2026-09-04): https://status.baseten.co returns HTTP 200 and renders a public service-status page (page body includes "ope…”

   proof ↗⚑ flag

2. founderCheck a public status page with incident history before betting production traffic on the platform

   weight 2 · round to Baseten
   Groq~partialprobed5/10

   A probe confirms a live public status page at groqstatus.com that returns HTTP 200 and shows 'operational' status, indicating Groq does operate a status page a founder could check before committing production traffic. However, the evidence doesn't confirm the page includes detailed incident history, uptime history, or granular component-level status, and there's no first-party documentation describing the status page's features. missing for 10: confirmation of incident history/log on the status page, documented SLA or uptime commitments, and independent corroboration beyond a single automated probe.

   • [probe] “PROBE status-page (2026-09-04): https://groqstatus.com returns HTTP 200 and renders a public service-status page (page body includes "operat…”

   proof ↗⚑ flag

   Baseten~partialprobed6/10

   A live probe confirms Baseten operates a public status page at status.baseten.co that returns HTTP 200 and shows current operational status, which is the core reliability-status signal a founder would check before committing production traffic. However, the evidence pack does not confirm the page includes a visible incident history/archive log, only that it renders current status. missing for 10: explicit confirmation that the status page displays past incident history/timeline, not just live status.

   • [probe] “PROBE status-page (2026-09-04): https://status.baseten.co returns HTTP 200 and renders a public service-status page (page body includes "ope…”

   proof ↗⚑ flag

Benchmarks

1. ml-engineerSee published tokens-per-second or latency numbers, benchmarks, or load-testing guides backing the provider's speed claims

   weight 2 · round to Groq
   Groq!disputedcontradicted5/10

   There is no first-party published benchmark or load-testing page in the evidence (docs cover service tiers, flex processing, batch, streaming, but no concrete TPS/latency numbers or methodology). Independent hands-on community reports do supply concrete numbers (491 T/s and 450+ TPS for Mixtral, deterministic low-latency architecture claims) supporting the speed claim, but later community threads directly contradict this, calling performance 'really poor for the token price' and reliability 'worse than traditional hosts' with frequent errors. Missing for 10: an official Groq benchmark/load-testing page with methodology, and resolution of the conflicting recent community performance complaints.

   • [community] “Incredible tool. The Mixtral 8x7B model running on their hardware did 491.40 T/s for me…”
   • [community] “I'm achieving consistent 450+ tokens/sec for Mixtral 8x7b 32k and ~200 tps for Llama 2 70B-4k.”
   • [community] “Groq AMA: 'Unlike with graphics processors, which really need data parallelism to get good throughput, our LPU architecture allows us to del…”
   • [community] “Groq staff: our system is deterministic, no need for waiting or queuing anywhere, and we can have very low latency interconnect between card…”
   • [community] “There's a trail of complaints going back years now, and they rounded out the bottom of Kimi's verification program. Groq hosted models were/…”
   • [community] “I don't really get the value proposition of groq as a user, the performance is really poor for the token price.”
   • [claimed-docs] “Groq offers multiple service tiers so you can tune for latency, throughput, and reliability.”
   • [claimed-docs] “Flex processing is available for all models to paid customers only with 10x higher rate limits compared to on-demand processing.”

   proof ↗⚑ flag

   Baseten—none0/10

   The evidence pack covers Baseten's product features (Model APIs, Truss, training, coding-agent integrations) but contains no published benchmark numbers, tokens-per-second figures, latency claims, or load-testing guides; the only performance-adjacent comment is a third-party observation about client-side GIL bottlenecks, which is unrelated to Baseten's own speed claims.

   ⚑ flag

Serving

1. developerServe latency-sensitive workloads with fast time-to-first-token and high-throughput generation

   weight 3 · round to Baseten
   Groq!disputedcontradicted6/10

   Early hands-on community reports strongly corroborate Groq's speed claims (491 T/s and ~450 t/s on Mixtral, deterministic low-latency architecture, streaming support docs), and Groq documents service tiers/flex-processing explicitly for latency/throughput tuning. However, more recent independent reports directly contradict this, describing Groq's hosted model performance as 'really poor for the token price' and 'always worse than traditional hosts,' plus reliability complaints ('so many random errors'), which concretely undercuts the fast/high-throughput positioning for current models. Missing for 10: recent (not 2024-era) throughput benchmarks, first-party TTFT numbers, and resolution of the conflicting recent user reports on performance-for-price.

   • [community] “Incredible tool. The Mixtral 8x7B model running on their hardware did 491.40 T/s for me…”
   • [community] “I'm achieving consistent 450+ tokens/sec for Mixtral 8x7b 32k and ~200 tps for Llama 2 70B-4k.”
   • [community] “Groq AMA: 'Unlike with graphics processors, which really need data parallelism to get good throughput, our LPU architecture allows us to del…”
   • [community] “Groq staff: our system is deterministic, no need for waiting or queuing anywhere, and we can have very low latency interconnect between card…”
   • [claimed-docs] “Groq offers multiple service tiers so you can tune for latency, throughput, and reliability.”
   • [claimed-docs] “Flex Processing is a service tier optimized for high-throughput workloads that prioritizes fast inference and can handle occasional request …”
   • [claimed-docs] “To enable streaming, set the parameter stream=True. The completion function will then return an iterator of completion deltas”
   • [community] “My company had a really terrible experience trying to use Groq, and I would NOT recommend anyone use their service if you need reliability. …”
   • [community] “There's a trail of complaints going back years now, and they rounded out the bottom of Kimi's verification program. Groq hosted models were/…”
   • [community] “I don't really get the value proposition of groq as a user, the performance is really poor for the token price.”

   proof ↗⚑ flag

   Baseten✓fullcommunity8/10

   Baseten's docs and GitHub materials show explicit architecture for latency-sensitive, high-throughput serving: token-by-token streaming (baseten-docs-8, baseten-docs-26), dedicated GPU deployment with TensorRT-LLM/vLLM/SGLang optimized builds via Truss (baseten-gh-2, baseten-gh-4, baseten-gh-6), automatic KV-cache discounting for repeated prompt tokens (baseten-docs-11), and active-active multi-cluster routing to avoid capacity-related slowdowns (baseten-docs-15). Async inference is also offered for non-latency-critical batch work, showing the platform differentiates paths for latency vs throughput use cases (baseten-docs-9, baseten-docs-27). missing for 10: published quantitative TTFT/throughput benchmarks or independent third-party latency comparisons; one community note (baseten-comm-3) suggests client-side bottlenecks can arise in high-throughput client code, an important but non-contradicting caveat.

   • [claimed-docs] “Return model output token by token as it is generated.”
   • [claimed-docs] “Streaming sends tokens as they are generated over server-sent events, which suits long generations and UIs where partial output beats a blan…”
   • [claimed-docs] “Cached input tokens are prompt tokens served from the KV cache at a discounted rate. Caching is automatic and requires no request flags.”
   • [claimed-docs] “Deployments run active-active across clusters and clouds. If a region or provider loses capacity, MCM reroutes and reprovisions workloads.”
   • [github] “You write a `config.yaml` that specifies the model, the hardware, and the engine, then `uvx truss push` builds a TensorRT-optimized containe…”
   • [github] “Support for all Python frameworks: From `transformers` and `diffusers` to PyTorch and TensorFlow to vLLM, SGLang, and TensorRT-LLM, Truss su…”
   • [github] “Deploying a model to Baseten via Truss turns a Hugging Face model into a production-ready API endpoint. You write a `config.yaml` that speci…”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents”
   • [claimed-docs] “Async inference returns a request ID quickly and completes later through webhook or polling, which suits batch work, long documents, or any …”
   • [community] “Developer notes that with Baseten.co embedding workloads, the client (not server) becomes the bottleneck due to Python's GIL, prompting them…”

   proof ↗⚑ flag

2. ml-engineerBenefit from prompt/prefix caching that reduces latency or cost on repeated context

   weight 1 · round to Baseten
   Groq—none0/10

   No evidence pack item mentions prompt/prefix caching, context caching, or any discounted repeated-context pricing on Groq's API. Docs cover streaming, batch processing, service tiers, and flex processing, but none describe caching repeated prompt prefixes to cut latency or cost, so this applicable axis is unaddressed.

   ⚑ flag

   Baseten✓fullclaimed7/10

   Docs explicitly describe automatic prompt caching: cached input tokens are served from KV cache at a discounted rate with no request flags needed, directly addressing latency/cost reduction on repeated context. missing for 10: no quantified latency/cost benchmarks, no independent/community corroboration of caching behavior in practice.

   • [claimed-docs] “Cached input tokens are prompt tokens served from the KV cache at a discounted rate. Caching is automatic and requires no request flags.”
   • [claimed-docs] “Model APIs bill by token and enforce request and token rate limits. You can also set a workspace budget and query usage by API key or model.”

   proof ↗⚑ flag

3. developerStream completions token by token over SSE for responsive user experiences

   weight 3 · round to Groq
   Groq✓fullcommunity9/10

   Docs explicitly document setting stream=True for token-by-token delta streaming via the OpenAI-compatible API, aligned with real-time display use cases, and independent community testimonials corroborate very high token throughput consistent with responsive streaming UX. Missing for 10: explicit confirmation of SSE transport format (docs describe iterator/deltas but don't spell out 'SSE' verbatim) and no independent hands-on benchmark specifically of streaming latency/SSE behavior.

   • [claimed-docs] “To enable streaming, set the parameter stream=True. The completion function will then return an iterator of completion deltas”
   • [claimed-docs] “To enable streaming, set the parameter `stream=True`.”
   • [claimed-docs] “you can stream the model's response in real-time. This allows your application to display the response as it's being generated”
   • [claimed-docs] “To enable streaming, set the parameter stream=True. The completion function will then return an iterator of completion deltas rather than a …”
   • [community] “Incredible tool. The Mixtral 8x7B model running on their hardware did 491.40 T/s for me…”
   • [community] “I'm achieving consistent 450+ tokens/sec for Mixtral 8x7b 32k and ~200 tps for Llama 2 70B-4k.”
   • [community] “Very impressed with the speed. This is one of the most impressive tech demos I've ever seen in my life... surreal to see the thing spitting …”

   proof ↗⚑ flag

   Baseten✓fullclaimed8/10

   Baseten's docs explicitly describe SSE token-by-token streaming ('Return model output token by token as it is generated' / 'Streaming sends tokens as they are generated over server-sent events') as a first-class inference mode alongside async, with OpenAI/Anthropic-compatible client SDK support for easy integration. Missing for 10: no independent/hands-on benchmark or community confirmation of streaming latency/responsiveness in practice.

   • [claimed-docs] “Return model output token by token as it is generated.”
   • [claimed-docs] “Streaming sends tokens as they are generated over server-sent events, which suits long generations and UIs where partial output beats a blan…”
   • [claimed-docs] “They support the OpenAI Chat Completions API and the Anthropic Messages API in beta, so you can use familiar client SDKs.”

   proof ↗⚑ flag

Structured

1. developerEnforce structured outputs against a JSON schema (or grammar) so model responses parse reliably

   weight 3 · round drawn
   Groq✓fullclaimed8/10

   Groq's docs explicitly describe Structured Outputs with JSON Schema and a strict:true mode that uses constrained decoding to guarantee schema-conforming output, plus tool-use docs confirming JSON schema-based function calling. This is first-party documented and consistent with OpenAI-compatible API design. Missing for 10: independent/hands-on verification that strict mode reliably parses in production and no community corroboration of structured-output reliability specifically.

   • [claimed-docs] “With strict: true, the model uses constrained decoding to guarantee that the output will always match your schema exactly.”
   • [claimed-docs] “Structured Outputs is a feature that ensures your model responses conform to your provided JSON Schema”
   • [claimed-docs] “With `strict: true`, the model uses constrained decoding to guarantee that the output will always match your schema exactly”
   • [claimed-docs] “To use tools, the model must be provided with tool definitions. These tool definitions are in JSON schema format and are passed to the model…”

   proof ↗⚑ flag

   Baseten✓fullclaimed8/10

   Baseten has dedicated first-party docs on structured outputs enforcing JSON schemas for reliable data extraction, plus separate function/tool calling support and OpenAI-compatible client integration (e.g., LangChain with_structured_output) confirming reliable parsing workflows. Missing for 10: independent/hands-on validation of schema-enforcement reliability and no mention of grammar-based (e.g., regex/CFG) constraints beyond JSON schema.

   • [claimed-docs] “Structured outputs let you generate text that conforms to specific JSON schemas, providing reliable data extraction and controlled text gene…”
   • [claimed-docs] “Function calling* (also called *tool calling*) lets a model choose a tool and produce its arguments from a user request.”
   • [claimed-docs] “Function calling (also called tool calling) lets a model choose a tool and produce its arguments from a user request.”
   • [claimed-docs] “Because Baseten exposes an OpenAI-compatible endpoint, you can use LangChain's `ChatOpenAI` with `with_structured_output` by pointing `base_…”
   • [claimed-docs] “They support the OpenAI Chat Completions API and the Anthropic Messages API in beta, so you can use familiar client SDKs.”

   proof ↗⚑ flag

Tools

1. ai-native userRely on faithful function/tool calling — including parallel and multi-step tool use — so agent loops run on open models without breaking

   weight 3 · round to Groq
   Groq~partialcommunity5/10

   Groq's docs clearly document OpenAI-compatible function/tool calling (JSON-schema tool definitions, strict-mode structured outputs, and now Remote MCP where 'Groq handles all orchestration') which supports agentic tool use, but none of the evidence explicitly confirms parallel tool calls or multi-step tool-use loops working reliably. Community reports also raise reliability concerns ('so many random errors, so many silly quirks') that create some doubt about robustness for agent loops, though not a concrete tool-calling failure. missing for 10: explicit documentation/examples of parallel tool calling, multi-step agentic loop reliability testing, and independent hands-on verification that tool calls remain faithful across steps.

   • [claimed-docs] “Tool use (or function calling) is what transforms a language model from a conversational interface into an autonomous agent capable of takin…”
   • [claimed-docs] “To use tools, the model must be provided with tool definitions. These tool definitions are in JSON schema format and are passed to the model…”
   • [claimed-docs] “With strict: true, the model uses constrained decoding to guarantee that the output will always match your schema exactly.”
   • [claimed-docs] “point to an MCP server URL and the Groq API will start using its tools without you having to implement any tool logic yourself”
   • [claimed-docs] “Groq's Responses API supports remote tool use via MCP servers via HTTPS where Groq handles all orchestration... You don't implement anything…”
   • [community] “My company had a really terrible experience trying to use Groq, and I would NOT recommend anyone use their service if you need reliability. …”
   • [community] “There's a trail of complaints going back years now, and they rounded out the bottom of Kimi's verification program. Groq hosted models were/…”

   proof ↗⚑ flag

   Baseten~partialclaimed4/10

   Baseten docs confirm function/tool-calling support ("lets a model choose a tool and produce its arguments from a user request") and structured outputs for reliable JSON generation, but the pack never mentions parallel tool calls, multi-step agent loops, or any reliability testing on open models. Missing for 10: explicit parallel tool-call support docs, multi-step agent loop examples, and independent/hands-on evidence of reliability across open models.

   • [claimed-docs] “Function calling* (also called *tool calling*) lets a model choose a tool and produce its arguments from a user request.”
   • [claimed-docs] “Function calling (also called tool calling) lets a model choose a tool and produce its arguments from a user request.”
   • [claimed-docs] “Structured outputs let you generate text that conforms to specific JSON schemas, providing reliable data extraction and controlled text gene…”
   • [claimed-docs] “Because Baseten exposes an OpenAI-compatible endpoint, you can use LangChain's `ChatOpenAI` with `with_structured_output` by pointing `base_…”

   proof ↗⚑ flag