Skip to content
ProductArena

How llamafile’s scores are calculated

The full audit trail, recomputed from the verdict data at build time through the same code that produced the leaderboard: verdict × quality × story weight per cell, cells sum to dimension scores, dimensions blend into the PA Score. Every number on the product page is reproducible from this page alone; for why the formula looks like this, see the methodology.

verdict factors: full ×1.0 · partial ×0.6 · disputed ×0.3 · none ×0.0 · n/a excluded from both sides · cell points = weight × quality × factor · cell max = weight × 10

PA Score22/100

Agent-ready 30.1 × 0.30 = 9.03

API quality 0.0 × 0.20 = 0.00

Openness 57.4 × 0.20 = 11.48

Built-in AI 10.3 × 0.15 = 1.55

Automation 0.0 × 0.15 = 0.00

(9.03 + 0.00 + 11.48 + 1.55 + 0.00) ÷ (0.30 + 0.20 + 0.20 + 0.15 + 0.15) = 22.05 ÷ 1.00 = 22.1

Scores are stored to 1 decimal; the product page’s pills round to whole numbers for display. Each dimension below shows the stories, verdicts, and cited evidence behind its number.

Agent-ready30.1/100×0.30 of the PA blend

Outside-in: can YOUR agent reach and drive this product — API, MCP, CLI, headless runs, agent docs.

Point an agent at llms.txt or agent-oriented docsweight 2

2 (weight) × 4 (quality) × 0.6 (partial) = 4.8 of 20 max

  • [probe] https://docs.mozilla.ai/llms.txt“PROBE llms.txt: HTTP 200 at https://docs.mozilla.ai/llms.txt # Mozilla.ai Docs ## any-llm - [Introduction](https://docs.mozilla.ai/index.md) - [Quickstart](https://docs.mozilla.ai”
  • [probe] https://docs.mozilla.ai/llamafile.md“PROBE docs-md: HTTP 200 at https://docs.mozilla.ai/llamafile.md # Page Not Found The URL `llamafile` does not exist. This page may have been moved, renamed, or deleted. ## Suggested”
  • [claimed-docs] https://docs.mozilla.ai/llamafile“llamafile lets you distribute and run LLMs with a single file.”

Run the product headlessly / in CI for automationweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile“If you add the `--cli` argument to a llamafile, you will run a CLI version of the model that answers to whatever you provide as a prompt”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/cli_arguments“llamafile --server --help ... HTTP server, API, Web UI, slot, and server sandbox options.”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/creating_llamafiles“A llamafile bundles the llamafile executable, model weights, and a set of default arguments into a single self-contained file using the APE (Actually Portable Executable) format”
  • [community] https://news.ycombinator.com/item?id=39887263“I have tried out Llamafile and I think it is bloody great. The simplicity of it is commendable. One issue I hope they overcome for Windows however, is being able to run the executable when it's more than 4GB (a Windows limitation).”
  • [community] https://hn.algolia.com/api/v1/items/45753850“there is anyway a nuance for Window systems which is the size limit for a Windows executable which is 4Gb maximum. As LLM models are tend to be quite large this limit is reached pretty fast.”

Plug MCP servers into this product so it can use their toolsweight 3

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Connect an agent via an official MCP serverweight 3

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Use an official CLIweight 2

2 (weight) × 7 (quality) × 1.0 (full) = 14.0 of 20 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile“If you add the `--cli` argument to a llamafile, you will run a CLI version of the model that answers to whatever you provide as a prompt”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/cli_arguments“llamafile --server --help ... HTTP server, API, Web UI, slot, and server sandbox options.”
  • [probe] https://docs.mozilla.ai/llamafile/reference/cli_arguments“official CLI documented at https://docs.mozilla.ai/llamafile/reference/cli_arguments”
  • [community] https://news.ycombinator.com/item?id=39887263“I have tried out Llamafile and I think it is bloody great. The simplicity of it is commendable. One issue I hope they overcome for Windows however, is being able to run the executable when it's more than 4GB (a Windows limitation).”
  • [community] https://news.ycombinator.com/item?id=39887263“I use my llamafile nearly every day.”

Drive the product through a documented public APIweight 3

3 (weight) × 5 (quality) × 0.6 (partial) = 9.0 of 30 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/cli_arguments“llamafile --server --help ... HTTP server, API, Web UI, slot, and server sandbox options.”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/getting-started/quickstart“you can also chat with it using [llama.cpp](https://github.com/ggml-org/llama.cpp)'s Web UI: just open a browser window and connect to http://localhost:8080/”
  • [probe] https://docs.mozilla.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.mozilla.ai/openapi.json, https://docs.mozilla.ai/swagger.json, https://docs.mozilla.ai/api/openapi.json, https://docs.mozilla.ai/.well-known/openapi.json)”

Issue scoped/least-privilege API credentials for an agentweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Build against official SDKsweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/cli_arguments“llamafile --server --help ... HTTP server, API, Web UI, slot, and server sandbox options.”
  • [probe] https://docs.mozilla.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.mozilla.ai/openapi.json, https://docs.mozilla.ai/swagger.json, https://docs.mozilla.ai/api/openapi.json, https://docs.mozilla.ai/.well-known/openapi.json)”
  • [probe] https://docs.mozilla.ai/llamafile.md“PROBE docs-md: HTTP 200 at https://docs.mozilla.ai/llamafile.md # Page Not Found The URL `llamafile` does not exist. This page may have been moved, renamed, or deleted. ## Suggested”

Subscribe to events via webhooksweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Connect a coding agent to this product as a working backendweight 3

3 (weight) × 4 (quality) × 0.6 (partial) = 7.2 of 30 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/cli_arguments“llamafile --server --help ... HTTP server, API, Web UI, slot, and server sandbox options.”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/getting-started/quickstart“you can also chat with it using [llama.cpp](https://github.com/ggml-org/llama.cpp)'s Web UI: just open a browser window and connect to http://localhost:8080/”
  • [probe] https://docs.mozilla.ai/llamafile/reference/cli_arguments“official CLI documented at https://docs.mozilla.ai/llamafile/reference/cli_arguments”

Agent-ready = 42.2 ÷ 140 × 100 = 30.1

API quality0.0/100×0.20 of the PA blend

The programmable surface once an agent is there — machine-readable spec, interactive docs, sandbox, versioning discipline.

Explore an interactive API reference with runnable examplesweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/cli_arguments“llamafile --server --help ... HTTP server, API, Web UI, slot, and server sandbox options.”
  • [probe] https://docs.mozilla.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.mozilla.ai/openapi.json, https://docs.mozilla.ai/swagger.json, https://docs.mozilla.ai/api/openapi.json, https://docs.mozilla.ai/.well-known/openapi.json)”
  • [probe] https://docs.mozilla.ai/llamafile.md“PROBE docs-md: HTTP 200 at https://docs.mozilla.ai/llamafile.md # Page Not Found The URL `llamafile` does not exist. This page may have been moved, renamed, or deleted. ## Suggested”

Download a machine-readable API spec (OpenAPI or equivalent)weight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [probe] https://docs.mozilla.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.mozilla.ai/openapi.json, https://docs.mozilla.ai/swagger.json, https://docs.mozilla.ai/api/openapi.json, https://docs.mozilla.ai/.well-known/openapi.json)”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/cli_arguments“llamafile --server --help ... HTTP server, API, Web UI, slot, and server sandbox options.”

Test against a sandbox environment without touching production dataweight 1

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Rely on versioned APIs with a documented deprecation policyweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/cli_arguments“llamafile --server --help ... HTTP server, API, Web UI, slot, and server sandbox options.”
  • [probe] https://docs.mozilla.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.mozilla.ai/openapi.json, https://docs.mozilla.ai/swagger.json, https://docs.mozilla.ai/api/openapi.json, https://docs.mozilla.ai/.well-known/openapi.json)”

API quality = 0.0 ÷ 60 × 100 = 0.0

Openness57.4/100×0.20 of the PA blend

Can you leave, inspect, or self-host — data export, open source, portability.

Do everything through the API that I can do in the UIweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/getting-started/quickstart“you can also upload an image by using the `/upload` command and specifying the path to the image”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/getting-started/quickstart“you can also chat with it using [llama.cpp](https://github.com/ggml-org/llama.cpp)'s Web UI: just open a browser window and connect to http://localhost:8080/”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile“If you add the `--cli` argument to a llamafile, you will run a CLI version of the model that answers to whatever you provide as a prompt”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/cli_arguments“llamafile --server --help ... HTTP server, API, Web UI, slot, and server sandbox options.”
  • [probe] https://docs.mozilla.ai/openapi.json“PROBE openapi: all candidate paths 404 (https://docs.mozilla.ai/openapi.json, https://docs.mozilla.ai/swagger.json, https://docs.mozilla.ai/api/openapi.json, https://docs.mozilla.ai/.well-known/openapi.json)”
  • [probe] https://docs.mozilla.ai/llamafile/reference/cli_arguments“official CLI documented at https://docs.mozilla.ai/llamafile/reference/cli_arguments”

Export all of my data in open formats and leaveweight 3

n/a — not applicable to this product: excluded from numerator and denominator

  • [claimed-docs] https://www.mozilla.ai/open-tools/llamafile“Models run entirely on your device. No cloud, no data sharing, no external dependencies. Works fully offline for privacy-first AI workflows.”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/creating_llamafiles“A llamafile bundles the llamafile executable, model weights, and a set of default arguments into a single self-contained file using the APE (Actually Portable Executable) format”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/security“No outbound network. `anet` allows `accept()` but not `connect()`, so the only networking the server can do is answer connections it received.”

Read the product's source under an open licenseweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

  • [github] https://github.com/mozilla-ai/llamafile“llamafile also includes whisperfile, a single-file speech-to-text tool built on whisper.cpp and the same Cosmopolitan packaging. It supports transcription and translation of audio files”

Self-host the core productweight 3

3 (weight) × 9 (quality) × 1.0 (full) = 27.0 of 30 max

  • [claimed-docs] https://www.mozilla.ai/open-tools/llamafile“Models run entirely on your device. No cloud, no data sharing, no external dependencies. Works fully offline for privacy-first AI workflows.”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/creating_llamafiles“A llamafile bundles the llamafile executable, model weights, and a set of default arguments into a single self-contained file using the APE (Actually Portable Executable) format”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/security“No outbound network. `anet` allows `accept()` but not `connect()`, so the only networking the server can do is answer connections it received.”
  • [community] https://hn.algolia.com/api/v1/items/38464057“This is pretty darn crazy. One file runs on 6 operating systems, with GPU support.”
  • [community] https://hn.algolia.com/api/v1/items/38464057“great! worked easily on desktop Linux, first try. It appears to execute with zero network connection... thx to Mozilla and Justin Tunney for this very easy, local experiment today!”
  • [community] https://hn.algolia.com/api/v1/items/38464057“Can confirm that this runs on an ancient i3 NUC under Ubuntu 20.04. It emits a token every five or six seconds, which is 'ask a question then go get coffee' speed. Still, very cool.”
  • [community] https://news.ycombinator.com/item?id=39887263“I use my llamafile nearly every day.”

Openness = 40.2 ÷ 70 × 100 = 57.4

Built-in AI10.3/100×0.15 of the PA blend

Inside-out: how agentic the product itself is for its users — built-in assistants, autonomous features.

Get AI-generated insights and suggestions from my data inside the productweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/getting-started/quickstart“you can also upload an image by using the `/upload` command and specifying the path to the image”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/getting-started/quickstart“you can also chat with it using [llama.cpp](https://github.com/ggml-org/llama.cpp)'s Web UI: just open a browser window and connect to http://localhost:8080/”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile“If you add the `--cli` argument to a llamafile, you will run a CLI version of the model that answers to whatever you provide as a prompt”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile“Here's how you can use llamafile to describe a jpg/png/gif/bmp image with a multimodal model (Qwen3.5, Ministral3, llava1.6 are all good candidates)”

Set up automations that run autonomously in the backgroundweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Delegate tasks to a built-in AI assistant inside the productweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/getting-started/quickstart“you can also upload an image by using the `/upload` command and specifying the path to the image”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/getting-started/quickstart“you can also chat with it using [llama.cpp](https://github.com/ggml-org/llama.cpp)'s Web UI: just open a browser window and connect to http://localhost:8080/”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile“If you add the `--cli` argument to a llamafile, you will run a CLI version of the model that answers to whatever you provide as a prompt”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/reference/cli_arguments“llamafile --server --help ... HTTP server, API, Web UI, slot, and server sandbox options.”

Operate the product with natural-language commandsweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/getting-started/quickstart“you can also upload an image by using the `/upload` command and specifying the path to the image”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/getting-started/quickstart“you can also chat with it using [llama.cpp](https://github.com/ggml-org/llama.cpp)'s Web UI: just open a browser window and connect to http://localhost:8080/”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile“If you add the `--cli` argument to a llamafile, you will run a CLI version of the model that answers to whatever you provide as a prompt”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile“Here's how you can use llamafile to describe a jpg/png/gif/bmp image with a multimodal model (Qwen3.5, Ministral3, llava1.6 are all good candidates)”
  • [community] https://news.ycombinator.com/item?id=39887263“I use my llamafile nearly every day.”
  • [community] https://hn.algolia.com/api/v1/items/45753850“Cosmocc and Cosmopolitan are remarkable technical achievements and llamafile made me discover them. The llamafile UX (CLI interface and web server with chat) is great... However I fail to see use cases where I would build a solution built on a llamafile.”

Built-in AI = 7.2 ÷ 70 × 100 = 10.3

Automation0.0/100×0.15 of the PA blend

Depth of automation primitives — rules, scheduling, bulk operations, webhooks.

Perform bulk operations across many items at onceweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile“If you add the `--cli` argument to a llamafile, you will run a CLI version of the model that answers to whatever you provide as a prompt”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile“Here's how you can use llamafile to describe a jpg/png/gif/bmp image with a multimodal model (Qwen3.5, Ministral3, llava1.6 are all good candidates)”
  • [claimed-docs] https://docs.mozilla.ai/llamafile/getting-started/quickstart“you can also upload an image by using the `/upload` command and specifying the path to the image”

Define rules that trigger actions automatically on eventsweight 3

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Schedule recurring jobs or workflowsweight 2

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Version, review, and roll back my automationsweight 1

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Automation = 0.0 ÷ 20 × 100 = 0.0