Ollama

Connect a coding agent to this product as a working backend C

Agent access

Drive the product through a documented public API G

Agent access

Connect an agent via an official MCP server G

Agent access

Delegate tasks to a built-in AI assistant inside the product G

Agentic features

Plug MCP servers into this product so it can use their tools G

Agent access

Point an agent at llms.txt or agent-oriented docs G

Agent access

Use an official CLI G

Agent access

Build against official SDKs G

Agent access

Run the product headlessly / in CI for automation G

Agent access

Operate the product with natural-language commands G

Agentic features

Download a machine-readable API spec (OpenAPI or equivalent) G

Api quality

Explore an interactive API reference with runnable examples G

Api quality

Rely on versioned APIs with a documented deprecation policy G

Api quality

Subscribe to events via webhooks G

Agent access

Get AI-generated insights and suggestions from my data inside the product G

Agentic features

Issue scoped/least-privilege API credentials for an agent G

Agent access

Set up automations that run autonomously in the background G

Agentic features

Test against a sandbox environment without touching production data G

Api quality

Run inference entirely on my own machine so my data and prompts never leave my device C

Privacy control

Self-host the core product G

Prevent my data from being used to train AI models G

Chat with local models using a built-in graphical chat interface C

Chat interface

Launch a local OpenAI-compatible API server for any loaded model C

Api compatibility

Download and run open models directly from Hugging Face C

Model hub download

Run hundreds of different model architectures including LLMs, MoE, multi-modal, and embedding models C

Architecture coverage

Run models on NVIDIA, AMD, or other GPU vendors using vendor-specific acceleration kernels C

Gpu acceleration

Get accelerated inference on Apple Silicon via native ARM and Metal optimizations C

Platform acceleration

Reduce memory footprint using integer quantization ranging from very low-bit to 8-bit precision C

Quantization levels

Export all of my data in open formats and leave G

Run models larger than my available VRAM using combined CPU+GPU offload C

Gpu acceleration

Stream generated tokens back to my application as they are produced C

Generation controls

Achieve high serving throughput via continuous batching and chunked prefill C

Throughput optimization

Define rules that trigger actions automatically on events G

Load and run models packaged in the GGUF format C

File formats

The documented maximum concurrent requests or connections the local server can handle before throughput degrades C

Scale limits

Call the runtime from official client libraries in languages like Python or JavaScript C

Language bindings

Load and switch between multiple models without restarting the server C

Model lifecycle

Search, download, and manage models from a command-line interface C

Cli tooling

Start an interactive chat session with a model directly from the terminal C

Cli tooling

Launch popular third-party coding agent CLIs pre-configured to use my local models with a single command P

Cli tooling

Connect to cloud AI providers alongside local models within the same interface C

Hybrid cloud local

Run the runtime headlessly with no GUI for use in servers or CI pipelines C

Deployment modes

Do everything through the API that I can do in the UI G

Install using prebuilt binaries or packages instead of compiling from source C

Build and install

Opt out of telemetry and usage tracking G

Read the product's source under an open license G

Run vision-language models that understand images alongside text C

Multi modal support

Choose where my data is stored (region/residency) G

Control data retention and deletion G

Serve models over my local network for access from other devices C

Remote serving

The runtime reserves dedicated capacity so throughput holds steady when multiple agents or sessions issue requests concurrently C

Throughput optimization

Manage my downloaded models, saved prompts, and per-model configurations in one place C

Local model management

Run the runtime inside a container for reproducible deployment C

Build and install

Accelerate inference on AMD GPUs via a Vulkan backend without needing a full ROCm install C

Gpu acceleration

Control how context memory is allocated when running multiple model instances concurrently C

Memory management

Get a fast cold start from a lightweight runtime binary instead of waiting seconds before inference begins C

Startup footprint

Load models quantized in formats like FP8, INT4, GPTQ, or AWQ C

Quantization levels

Override low-level engine settings like memory locking or mmap behavior instead of being limited to opinionated defaults C

Server configuration

Use native tool-calling and reasoning-parser support in my requests C

Generation controls

Whether downloaded model files and caches can be reused by other runtimes without re-downloading or re-converting them C

Model portability

Accelerate generation speed using speculative decoding techniques C

Throughput optimization

Build the runtime from source with minimal external dependencies C

Build and install

Chat with my own documents entirely offline using automatic retrieval-augmented generation C

Document intelligence

Constrain model output to structured formats like JSON using grammars C

Generation controls

Create specialized custom assistants configured for specific tasks C

Custom assistants

Distribute inference across multiple GPUs using tensor, pipeline, or data parallelism C

Distributed serving

Efficiently serve multiple LoRA adapters on top of a base model C

Adapters

How quickly the project ships patches for critical bugs and security vulnerabilities based on its public release history C

Maintenance health

Leverage advanced x86 CPU instruction sets like AVX, AVX2, AVX512, and AMX for faster inference C

Platform acceleration

Perform bulk operations across many items at once G

Rely on an AI assistant to recommend which local model best fits my hardware and task before I download it C

Ai assisted setup

Rely on paged memory management for attention key/value cache to maximize concurrent request capacity without memory fragmentation P

Throughput optimization

Schedule recurring jobs or workflows G

Serve embedding models for retrieval and search applications C

Architecture coverage

Speed up repeated-prompt workloads using prefix caching C

Throughput optimization

The pricing tiers, rate limits, and usage caps that apply when offloading inference to the vendor's hosted cloud tier G

Hybrid cloud local

Whether commercial or enterprise use requires a paid license or subscription beyond the free community edition G

Licensing and cost

Whether upgrading the runtime can break compatibility with previously downloaded quantized model files C

File formats

Offload very large models to a hosted cloud tier without downloading them when my local hardware is insufficient C

Hybrid cloud local

Why GPU acceleration failed and silently fell back to CPU through clear diagnostic output C

Gpu acceleration

Start and stop the local model server from the command line C

Cli tooling

Assign a custom identifier to a loaded model for consistent reference in API calls C

Model lifecycle

Call the server through an Anthropic-compatible messages endpoint C

Api compatibility

Load a model with custom GPU offload and context length settings from the command line C

Cli tooling

Run inference on specialized accelerators like TPUs or Gaudi through plugin support C

Gpu acceleration

Contribute code and become a recognized collaborator through the project's open-source process G

Community contribution

Dictate speech that gets transcribed in real time by an on-device model C

Document intelligence

Disaggregate prefill and decode phases for optimized large-scale serving C

Distributed serving

Have an AI agent draft and edit documents in an integrated workspace with changes saved automatically C

Document intelligence

Install the runtime quickly using a standard package manager C

Build and install

Run inference on diverse CPU architectures beyond x86 and ARM, such as PowerPC P

Platform acceleration

Version, review, and roll back my automations G