Local LLM Runtimes — procurement report
ProductArena · rankings as of 2026-09-04 · evidence as of 2026-09-04 · 7 products · 92 judged requirements · 644 judged cells
Methodology: Every product is judged against a shared taxonomy of user stories using cited evidence — hands-on probes > repository code > independent community sources > vendor claims — never opinion. Full writeup: https://ultrametric.ai/productarena/methodology
Leaderboard
| # | Product | PA Score | Coverage score | Applicable cells | Confidence |
|---|---|---|---|---|---|
| 1 | LocalAI | 31.7 | 29.1 | 90/92 | C |
| 2 | vLLM | 28.9 | 36.2 | 66/92 | C |
| 3 | LM Studio | 27.3 | 26.3 | 85/92 | C |
| 4 | Ollama | 22.2 | 22.7 | 83/92 | B |
| 5 | llamafile | 22.1 | 22.5 | 79/92 | C |
| 6 | llama.cpp | 16.6 | 29.1 | 84/92 | C |
| 7 | Jan | 15.7 | 14.7 | 89/92 | C |
PA Score = agent-readiness blend (see methodology). Coverage score = weighted share of judged requirements met. Confidence = how much of the score rests on tested vs claimed evidence (A–D).
Uncertainty note
This arena is currently a close race: LocalAI (31.7) vs vLLM (28.9), a gap of 2.8 PA Score. The ordering was re-checked with extra judge samples: 36 decisive cells were triple-judged and 8 came back unstable. Treat the #1/#2 ordering as contested — shortlist both.
Buyer checklist (RFP)
The arena's 92 judged user stories as requirements, grouped by theme. Priorities mirror the story weights our scoring uses (3 = must-have, 2 = should-have, 1 = nice-to-have). Interactive version with per-requirement verdicts for the top products: /arena/local-llm-runtimes/checklist
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
- ai-native userPlug MCP servers into this product so it can use their toolsmust-have
- ai-native userConnect an agent via an official MCP servermust-have
- ai-native userDrive the product through a documented public APImust-have
- ai-native userConnect a coding agent to this product as a working backendmust-have
- ai-native userDelegate tasks to a built-in AI assistant inside the productmust-have
- ai-native userPoint an agent at llms.txt or agent-oriented docsshould-have
- ai-native userRun the product headlessly / in CI for automationshould-have
- ai-native userUse an official CLIshould-have
- ai-native userIssue scoped/least-privilege API credentials for an agentshould-have
- ai-native userBuild against official SDKsshould-have
- ai-native userSubscribe to events via webhooksshould-have
- ai-native userGet AI-generated insights and suggestions from my data inside the productshould-have
- ai-native userSet up automations that run autonomously in the backgroundshould-have
- ai-native userOperate the product with natural-language commandsshould-have
- ai-native userExplore an interactive API reference with runnable examplesshould-have
- ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)should-have
- ai-native userRely on versioned APIs with a documented deprecation policyshould-have
- ai-native userTest against a sandbox environment without touching production datanice-to-have
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
- ai-native userDefine rules that trigger actions automatically on eventsmust-have
- ai-native userPerform bulk operations across many items at onceshould-have
- ai-native userSchedule recurring jobs or workflowsshould-have
- ai-native userVersion, review, and roll back my automationsnice-to-have
Ecosystem — integrations, plugins, and third-party ecosystem storiesEcosystem
Integrations, plugins, and third-party ecosystem stories
- power-userRun inference entirely on my own machine so my data and prompts never leave my devicemust-have
- developerBuild the runtime from source with minimal external dependenciesshould-have
- developerRun the runtime inside a container for reproducible deploymentshould-have
- developerInstall using prebuilt binaries or packages instead of compiling from sourceshould-have
- developerCall the runtime from official client libraries in languages like Python or JavaScriptshould-have
- power-userWhether commercial or enterprise use requires a paid license or subscription beyond the free community editionshould-have
- developerHow quickly the project ships patches for critical bugs and security vulnerabilities based on its public release historyshould-have
- developerWhether downloaded model files and caches can be reused by other runtimes without re-downloading or re-converting themshould-have
- developerInstall the runtime quickly using a standard package managernice-to-have
- developerContribute code and become a recognized collaborator through the project's open-source processnice-to-have
Model support — which models run and how well — coverage, formats, update cadenceModel support
Which models run and how well — coverage, formats, update cadence
- developerRun hundreds of different model architectures including LLMs, MoE, multi-modal, and embedding modelsmust-have
- power-userDownload and run open models directly from Hugging Facemust-have
- developerServe embedding models for retrieval and search applicationsshould-have
- power-userCreate specialized custom assistants configured for specific tasksshould-have
- power-userConnect to cloud AI providers alongside local models within the same interfaceshould-have
- power-userThe pricing tiers, rate limits, and usage caps that apply when offloading inference to the vendor's hosted cloud tiershould-have
- power-userRun vision-language models that understand images alongside textshould-have
- power-userOffload very large models to a hosted cloud tier without downloading them when my local hardware is insufficientnice-to-have
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
- ai-native userExport all of my data in open formats and leavemust-have
- ai-native userSelf-host the core productmust-have
- ai-native userDo everything through the API that I can do in the UIshould-have
- ai-native userRead the product's source under an open licenseshould-have
Performance hardware — raw speed and hardware efficiency — throughput, latency, resource usePerformance hardware
Raw speed and hardware efficiency — throughput, latency, resource use
- power-userRun models larger than my available VRAM using combined CPU+GPU offloadmust-have
- power-userRun models on NVIDIA, AMD, or other GPU vendors using vendor-specific acceleration kernelsmust-have
- power-userGet accelerated inference on Apple Silicon via native ARM and Metal optimizationsmust-have
- power-userAchieve high serving throughput via continuous batching and chunked prefillmust-have
- developerDistribute inference across multiple GPUs using tensor, pipeline, or data parallelismshould-have
- power-userAccelerate inference on AMD GPUs via a Vulkan backend without needing a full ROCm installshould-have
- power-userControl how context memory is allocated when running multiple model instances concurrentlyshould-have
- power-userLeverage advanced x86 CPU instruction sets like AVX, AVX2, AVX512, and AMX for faster inferenceshould-have
- power-userGet a fast cold start from a lightweight runtime binary instead of waiting seconds before inference beginsshould-have
- developerRely on paged memory management for attention key/value cache to maximize concurrent request capacity without memory fragmentationshould-have
- power-userThe runtime reserves dedicated capacity so throughput holds steady when multiple agents or sessions issue requests concurrentlyshould-have
- power-userSpeed up repeated-prompt workloads using prefix cachingshould-have
- power-userAccelerate generation speed using speculative decoding techniquesshould-have
- developerDisaggregate prefill and decode phases for optimized large-scale servingnice-to-have
- developerRun inference on specialized accelerators like TPUs or Gaudi through plugin supportnice-to-have
- power-userWhy GPU acceleration failed and silently fell back to CPU through clear diagnostic outputnice-to-have
- developerRun inference on diverse CPU architectures beyond x86 and ARM, such as PowerPCnice-to-have
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
- ai-native userPrevent my data from being used to train AI modelsmust-have
- ai-native userChoose where my data is stored (region/residency)should-have
- ai-native userControl data retention and deletionshould-have
- ai-native userOpt out of telemetry and usage trackingshould-have
Quantization formats — stories about quantization formats in this arenaQuantization formats
Stories about quantization formats in this arena
- power-userLoad and run models packaged in the GGUF formatmust-have
- power-userReduce memory footprint using integer quantization ranging from very low-bit to 8-bit precisionmust-have
- developerEfficiently serve multiple LoRA adapters on top of a base modelshould-have
- developerWhether upgrading the runtime can break compatibility with previously downloaded quantized model filesshould-have
- developerLoad models quantized in formats like FP8, INT4, GPTQ, or AWQshould-have
Serving api — serving models over an API — endpoints, compatibility, reliabilityServing api
Serving models over an API — endpoints, compatibility, reliability
- developerLaunch a local OpenAI-compatible API server for any loaded modelmust-have
- developerStream generated tokens back to my application as they are producedmust-have
- developerThe documented maximum concurrent requests or connections the local server can handle before throughput degradesmust-have
- developerRun the runtime headlessly with no GUI for use in servers or CI pipelinesshould-have
- developerConstrain model output to structured formats like JSON using grammarsshould-have
- developerUse native tool-calling and reasoning-parser support in my requestsshould-have
- power-userLoad and switch between multiple models without restarting the servershould-have
- power-userServe models over my local network for access from other devicesshould-have
- power-userOverride low-level engine settings like memory locking or mmap behavior instead of being limited to opinionated defaultsshould-have
- developerCall the server through an Anthropic-compatible messages endpointnice-to-have
- developerAssign a custom identifier to a loaded model for consistent reference in API callsnice-to-have
Ux tooling — the working surface itself — layout, ergonomics, quality-of-life toolingUx tooling
The working surface itself — layout, ergonomics, quality-of-life tooling
- power-userChat with local models using a built-in graphical chat interfacemust-have
- ai-native userRely on an AI assistant to recommend which local model best fits my hardware and task before I download itshould-have
- developerStart an interactive chat session with a model directly from the terminalshould-have
- developerSearch, download, and manage models from a command-line interfaceshould-have
- developerLaunch popular third-party coding agent CLIs pre-configured to use my local models with a single commandshould-have
- ai-native userChat with my own documents entirely offline using automatic retrieval-augmented generationshould-have
- power-userManage my downloaded models, saved prompts, and per-model configurations in one placeshould-have
- developerLoad a model with custom GPU offload and context length settings from the command linenice-to-have
- developerStart and stop the local model server from the command linenice-to-have
- ai-native userHave an AI agent draft and edit documents in an integrated workspace with changes saved automaticallynice-to-have
- ai-native userDictate speech that gets transcribed in real time by an on-device modelnice-to-have
Appendix: recorded probes
No replayable probe recordings exist for this arena yet. Probe-tier evidence (hands-on checks) still backs verdicts where cited — see each product page for the evidence trail.
Cite as: ProductArena by Ultrametric Inc, Local LLM Runtimes arena, rankings as of 2026-09-04 — https://ultrametric.ai/productarena/arena/local-llm-runtimes
License: © 2026 Ultrametric Inc. Brief quotation of individual verdicts, scores, or evidence excerpts is permitted with attribution to "ProductArena by Ultrametric Inc (ultrametric.ai/productarena)", as is use of the data to evaluate, contest, or contribute corrections. Bulk copying, redistribution, or use to build competing datasets requires prior written permission (see DATA-LICENSE in the repository).
No liability: rankings, verdicts, and scores are research outputs derived from the cited evidence at a point in time, provided "as is", without warranties. Ultrametric Inc accepts no responsibility for procurement, purchasing, or other decisions made in reliance on them — verify against the cited evidence before acting (https://ultrametric.ai/productarena/terms).