llama.cpp

Connect a coding agent to this product as a working backend C

Agent access

Drive the product through a documented public API G

Agent access

Delegate tasks to a built-in AI assistant inside the product G

Agentic features

Connect an agent via an official MCP server G

Agent access

Plug MCP servers into this product so it can use their tools G

Agent access

Use an official CLI G

Agent access

Run the product headlessly / in CI for automation G

Agent access

Build against official SDKs G

Agent access

Download a machine-readable API spec (OpenAPI or equivalent) G

Api quality

Operate the product with natural-language commands G

Agentic features

Point an agent at llms.txt or agent-oriented docs G

Agent access

Rely on versioned APIs with a documented deprecation policy G

Api quality

Explore an interactive API reference with runnable examples G

Api quality

Get AI-generated insights and suggestions from my data inside the product G

Agentic features

Issue scoped/least-privilege API credentials for an agent G

Agent access

Set up automations that run autonomously in the background G

Agentic features

Subscribe to events via webhooks G

Agent access

Test against a sandbox environment without touching production data G

Api quality

Get accelerated inference on Apple Silicon via native ARM and Metal optimizations C

Platform acceleration

Reduce memory footprint using integer quantization ranging from very low-bit to 8-bit precision C

Quantization levels

Run inference entirely on my own machine so my data and prompts never leave my device C

Privacy control

Self-host the core product G

Download and run open models directly from Hugging Face C

Model hub download

Load and run models packaged in the GGUF format C

File formats

Run models larger than my available VRAM using combined CPU+GPU offload C

Gpu acceleration

Run models on NVIDIA, AMD, or other GPU vendors using vendor-specific acceleration kernels C

Gpu acceleration

Chat with local models using a built-in graphical chat interface C

Chat interface

Prevent my data from being used to train AI models G

Launch a local OpenAI-compatible API server for any loaded model C

Api compatibility

Run hundreds of different model architectures including LLMs, MoE, multi-modal, and embedding models C

Architecture coverage

Export all of my data in open formats and leave G

Stream generated tokens back to my application as they are produced C

Generation controls

Achieve high serving throughput via continuous batching and chunked prefill C

Throughput optimization

Define rules that trigger actions automatically on events G

The documented maximum concurrent requests or connections the local server can handle before throughput degrades C

Scale limits

Build the runtime from source with minimal external dependencies C

Build and install

Leverage advanced x86 CPU instruction sets like AVX, AVX2, AVX512, and AMX for faster inference C

Platform acceleration

Run the runtime headlessly with no GUI for use in servers or CI pipelines C

Deployment modes

Run the runtime inside a container for reproducible deployment C

Build and install

Run vision-language models that understand images alongside text C

Multi modal support

Start an interactive chat session with a model directly from the terminal C

Cli tooling

Constrain model output to structured formats like JSON using grammars C

Generation controls

Get a fast cold start from a lightweight runtime binary instead of waiting seconds before inference begins C

Startup footprint

Read the product's source under an open license G

Choose where my data is stored (region/residency) G

Do everything through the API that I can do in the UI G

Install using prebuilt binaries or packages instead of compiling from source C

Build and install

Search, download, and manage models from a command-line interface C

Cli tooling

Serve models over my local network for access from other devices C

Remote serving

Control data retention and deletion G

Create specialized custom assistants configured for specific tasks C

Custom assistants

Accelerate inference on AMD GPUs via a Vulkan backend without needing a full ROCm install C

Gpu acceleration

Call the runtime from official client libraries in languages like Python or JavaScript C

Language bindings

Distribute inference across multiple GPUs using tensor, pipeline, or data parallelism C

Distributed serving

Load and switch between multiple models without restarting the server C

Model lifecycle

Load models quantized in formats like FP8, INT4, GPTQ, or AWQ C

Quantization levels

Manage my downloaded models, saved prompts, and per-model configurations in one place C

Local model management

Accelerate generation speed using speculative decoding techniques C

Throughput optimization

Chat with my own documents entirely offline using automatic retrieval-augmented generation C

Document intelligence

Connect to cloud AI providers alongside local models within the same interface C

Hybrid cloud local

Control how context memory is allocated when running multiple model instances concurrently C

Memory management

Efficiently serve multiple LoRA adapters on top of a base model C

Adapters

How quickly the project ships patches for critical bugs and security vulnerabilities based on its public release history C

Maintenance health

Launch popular third-party coding agent CLIs pre-configured to use my local models with a single command P

Cli tooling

Opt out of telemetry and usage tracking G

Override low-level engine settings like memory locking or mmap behavior instead of being limited to opinionated defaults C

Server configuration

Perform bulk operations across many items at once G

Rely on an AI assistant to recommend which local model best fits my hardware and task before I download it C

Ai assisted setup

Rely on paged memory management for attention key/value cache to maximize concurrent request capacity without memory fragmentation P

Throughput optimization

Schedule recurring jobs or workflows G

Serve embedding models for retrieval and search applications C

Architecture coverage

Speed up repeated-prompt workloads using prefix caching C

Throughput optimization

The pricing tiers, rate limits, and usage caps that apply when offloading inference to the vendor's hosted cloud tier G

Hybrid cloud local

The runtime reserves dedicated capacity so throughput holds steady when multiple agents or sessions issue requests concurrently C

Throughput optimization

Use native tool-calling and reasoning-parser support in my requests C

Generation controls

Whether commercial or enterprise use requires a paid license or subscription beyond the free community edition G

Licensing and cost

Whether downloaded model files and caches can be reused by other runtimes without re-downloading or re-converting them C

Model portability

Whether upgrading the runtime can break compatibility with previously downloaded quantized model files C

File formats

Load a model with custom GPU offload and context length settings from the command line C

Cli tooling

Start and stop the local model server from the command line C

Cli tooling

Contribute code and become a recognized collaborator through the project's open-source process G

Community contribution

Install the runtime quickly using a standard package manager C

Build and install

Call the server through an Anthropic-compatible messages endpoint C

Api compatibility

Offload very large models to a hosted cloud tier without downloading them when my local hardware is insufficient C

Hybrid cloud local

Run inference on specialized accelerators like TPUs or Gaudi through plugin support C

Gpu acceleration

Assign a custom identifier to a loaded model for consistent reference in API calls C

Model lifecycle

Dictate speech that gets transcribed in real time by an on-device model C

Document intelligence

Disaggregate prefill and decode phases for optimized large-scale serving C

Distributed serving

Have an AI agent draft and edit documents in an integrated workspace with changes saved automatically C

Document intelligence

Run inference on diverse CPU architectures beyond x86 and ARM, such as PowerPC P

Platform acceleration

Version, review, and roll back my automations G

Why GPU acceleration failed and silently fell back to CPU through clear diagnostic output C

Gpu acceleration