Skip to content
ProductArena

How Zulip’s scores are calculated

The full audit trail, recomputed from the verdict data at build time through the same code that produced the leaderboard: verdict × quality × story weight per cell, cells sum to dimension scores, dimensions blend into the PA Score. Every number on the product page is reproducible from this page alone; for why the formula looks like this, see the methodology.

verdict factors: full ×1.0 · partial ×0.6 · disputed ×0.3 · none ×0.0 · n/a excluded from both sides · cell points = weight × quality × factor · cell max = weight × 10

PA Score32/100

Agent-ready 33.9 × 0.30 = 10.17

API quality 34.3 × 0.20 = 6.86

Openness 61.0 × 0.20 = 12.20

Built-in AI 6.7 × 0.15 = 1.00

Automation 10.3 × 0.15 = 1.55

(10.17 + 6.86 + 12.20 + 1.00 + 1.55) ÷ (0.30 + 0.20 + 0.20 + 0.15 + 0.15) = 31.78 ÷ 1.00 = 31.8

Scores are stored to 1 decimal; the product page’s pills round to whole numbers for display. Each dimension below shows the stories, verdicts, and cited evidence behind its number.

Agent-ready33.9/100×0.30 of the PA blend

Outside-in: can YOUR agent reach and drive this product — API, MCP, CLI, headless runs, agent docs.

Point an agent at llms.txt or agent-oriented docsweight 2

2 (weight) × 3 (quality) × 0.6 (partial) = 3.6 of 20 max

  • [probe] https://zulip.com/llms.txt“PROBE llms.txt: HTTP 404 at https://zulip.com/llms.txt”
  • [probe] https://zulip.com/api/.md“PROBE docs-md: HTTP 404 at https://zulip.com/api/.md”
  • [probe] https://zulip.com/openapi.json“PROBE openapi: all candidate paths 404 (https://zulip.com/openapi.json, https://zulip.com/swagger.json, https://zulip.com/api/openapi.json, https://zulip.com/.well-known/openapi.json)”
  • [probe] https://raw.githubusercontent.com/zulip/zulip/main/zerver/openapi/zulip.yaml“PROBE openapi-spec: fetched zerver/openapi/zulip.yaml from the Zulip repo → HTTP 200, ~1.5 MB spec beginning 'openapi: 3.0.1' — the complete REST API ships as a machine-readable OpenAPI document (hands-on)”

Run the product headlessly / in CI for automationweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [probe] https://chat.zulip.org/api/v1/server_settings“PROBE api-live: GET https://chat.zulip.org/api/v1/server_settings → HTTP 200 {"result":"success",...,"zulip_version":"12.0-919-gb1bd7d7d3e"} — documented endpoint answers keylessly on a public Zulip server (hands-on)”
  • [probe] https://raw.githubusercontent.com/zulip/zulip/main/zerver/openapi/zulip.yaml“PROBE openapi-spec: fetched zerver/openapi/zulip.yaml from the Zulip repo → HTTP 200, ~1.5 MB spec beginning 'openapi: 3.0.1' — the complete REST API ships as a machine-readable OpenAPI document (hands-on)”
  • [probe] https://zulip.com/api/installation-instructions“PROBE cli-runtime: pip install zulip (v0.9.1) → official zulip-send CLI installs from PyPI and runs headlessly (--help prints usage for sending messages from scripts) (hands-on, keyless)”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip's API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot”
  • [claimed-docs] https://zulip.com/help/running-bots“Eventually, you’ll probably want to run it in a production environment where it’ll stay up, by deploying it on a server using the Zulip Botserver.”

Plug MCP servers into this product so it can use their toolsweight 3

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Connect an agent via an official MCP serverweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

  • [claimed-docs] https://zulip.com/help/running-bots“Zulip's API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip’s API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot, or run an existing Zulip bot.”
  • [probe] https://raw.githubusercontent.com/zulip/zulip/main/zerver/openapi/zulip.yaml“PROBE openapi-spec: fetched zerver/openapi/zulip.yaml from the Zulip repo → HTTP 200, ~1.5 MB spec beginning 'openapi: 3.0.1' — the complete REST API ships as a machine-readable OpenAPI document (hands-on)”

Use an official CLIweight 2

2 (weight) × 6 (quality) × 0.6 (partial) = 7.2 of 20 max

  • [probe] https://zulip.com/api/installation-instructions“PROBE cli-runtime: pip install zulip (v0.9.1) → official zulip-send CLI installs from PyPI and runs headlessly (--help prints usage for sending messages from scripts) (hands-on, keyless)”

Drive the product through a documented public APIweight 3

3 (weight) × 9 (quality) × 1.0 (full) = 27.0 of 30 max

  • [claimed-docs] https://zulip.com/api/“Send a message”
  • [probe] https://chat.zulip.org/api/v1/server_settings“PROBE api-live: GET https://chat.zulip.org/api/v1/server_settings → HTTP 200 {"result":"success",...,"zulip_version":"12.0-919-gb1bd7d7d3e"} — documented endpoint answers keylessly on a public Zulip server (hands-on)”
  • [probe] https://raw.githubusercontent.com/zulip/zulip/main/zerver/openapi/zulip.yaml“PROBE openapi-spec: fetched zerver/openapi/zulip.yaml from the Zulip repo → HTTP 200, ~1.5 MB spec beginning 'openapi: 3.0.1' — the complete REST API ships as a machine-readable OpenAPI document (hands-on)”
  • [probe] https://zulip.com/api/installation-instructions“PROBE cli-runtime: pip install zulip (v0.9.1) → official zulip-send CLI installs from PyPI and runs headlessly (--help prints usage for sending messages from scripts) (hands-on, keyless)”

Issue scoped/least-privilege API credentials for an agentweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://zulip.com/help/running-bots“Zulip's API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip’s API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot, or run an existing Zulip bot.”
  • [claimed-docs] https://zulip.com/help/running-bots“Eventually, you’ll probably want to run it in a production environment where it’ll stay up, by deploying it on a server using the Zulip Botserver.”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip’s API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot”

Build against official SDKsweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [probe] https://chat.zulip.org/api/v1/server_settings“PROBE api-live: GET https://chat.zulip.org/api/v1/server_settings → HTTP 200 {"result":"success",...,"zulip_version":"12.0-919-gb1bd7d7d3e"} — documented endpoint answers keylessly on a public Zulip server (hands-on)”
  • [probe] https://raw.githubusercontent.com/zulip/zulip/main/zerver/openapi/zulip.yaml“PROBE openapi-spec: fetched zerver/openapi/zulip.yaml from the Zulip repo → HTTP 200, ~1.5 MB spec beginning 'openapi: 3.0.1' — the complete REST API ships as a machine-readable OpenAPI document (hands-on)”
  • [probe] https://zulip.com/api/installation-instructions“PROBE cli-runtime: pip install zulip (v0.9.1) → official zulip-send CLI installs from PyPI and runs headlessly (--help prints usage for sending messages from scripts) (hands-on, keyless)”
  • [claimed-docs] https://zulip.com/api/“Send a message”

Subscribe to events via webhooksweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://zulip.com/integrations/“Incoming webhooks”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip's API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip’s API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot, or run an existing Zulip bot.”

Agent-ready = 61.0 ÷ 180 × 100 = 33.9

API quality34.3/100×0.20 of the PA blend

The programmable surface once an agent is there — machine-readable spec, interactive docs, sandbox, versioning discipline.

Explore an interactive API reference with runnable examplesweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

  • [claimed-docs] https://zulip.com/api/“Send a message”
  • [probe] https://raw.githubusercontent.com/zulip/zulip/main/zerver/openapi/zulip.yaml“PROBE openapi-spec: fetched zerver/openapi/zulip.yaml from the Zulip repo → HTTP 200, ~1.5 MB spec beginning 'openapi: 3.0.1' — the complete REST API ships as a machine-readable OpenAPI document (hands-on)”
  • [probe] https://chat.zulip.org/api/v1/server_settings“PROBE api-live: GET https://chat.zulip.org/api/v1/server_settings → HTTP 200 {"result":"success",...,"zulip_version":"12.0-919-gb1bd7d7d3e"} — documented endpoint answers keylessly on a public Zulip server (hands-on)”

Download a machine-readable API spec (OpenAPI or equivalent)weight 2

2 (weight) × 9 (quality) × 1.0 (full) = 18.0 of 20 max

  • [probe] https://raw.githubusercontent.com/zulip/zulip/main/zerver/openapi/zulip.yaml“PROBE openapi-spec: fetched zerver/openapi/zulip.yaml from the Zulip repo → HTTP 200, ~1.5 MB spec beginning 'openapi: 3.0.1' — the complete REST API ships as a machine-readable OpenAPI document (hands-on)”
  • [claimed-docs] https://zulip.com/api/“Send a message”
  • [probe] https://zulip.com/openapi.json“PROBE openapi: all candidate paths 404 (https://zulip.com/openapi.json, https://zulip.com/swagger.json, https://zulip.com/api/openapi.json, https://zulip.com/.well-known/openapi.json)”

Test against a sandbox environment without touching production dataweight 1

1 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 10 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Rely on versioned APIs with a documented deprecation policyweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

API quality = 24.0 ÷ 70 × 100 = 34.3

Openness61.0/100×0.20 of the PA blend

Can you leave, inspect, or self-host — data export, open source, portability.

Do everything through the API that I can do in the UIweight 2

2 (weight) × 7 (quality) × 0.6 (partial) = 8.4 of 20 max

  • [probe] https://raw.githubusercontent.com/zulip/zulip/main/zerver/openapi/zulip.yaml“PROBE openapi-spec: fetched zerver/openapi/zulip.yaml from the Zulip repo → HTTP 200, ~1.5 MB spec beginning 'openapi: 3.0.1' — the complete REST API ships as a machine-readable OpenAPI document (hands-on)”
  • [probe] https://chat.zulip.org/api/v1/server_settings“PROBE api-live: GET https://chat.zulip.org/api/v1/server_settings → HTTP 200 {"result":"success",...,"zulip_version":"12.0-919-gb1bd7d7d3e"} — documented endpoint answers keylessly on a public Zulip server (hands-on)”
  • [probe] https://zulip.com/api/installation-instructions“PROBE cli-runtime: pip install zulip (v0.9.1) → official zulip-send CLI installs from PyPI and runs headlessly (--help prints usage for sending messages from scripts) (hands-on, keyless)”
  • [claimed-docs] https://zulip.com/api/“Send a message”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip’s API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot, or run an existing Zulip bot.”

Export all of my data in open formats and leaveweight 3

3 (weight) × 8 (quality) × 1.0 (full) = 24.0 of 30 max

  • [claimed-docs] https://zulip.com/help/export-your-organization“Zulip has high quality export tools that can be used to migrate between the hosted Zulip Cloud service and your own servers.”
  • [claimed-docs] https://zulip.com/plans/“You can move freely between Zulip Cloud hosting and your own servers with our high quality export and import tools.”
  • [claimed-docs] https://zulip.com/help/export-your-organization“Public and private data (with consent): Everything in the export of public data, plus all the private channel messages and direct messages that members who have allowed administrators to export their private data can access.”
  • [claimed-docs] https://zulip.com/self-hosting/“When you self-host Zulip, you get the same software as our Zulip Cloud customers.”
  • [community] https://news.ycombinator.com/item?id=27149123“Zulip is my favorite modern chat program because it's actually 100% FOSS, unlike Mattermost and RocketChat, which are only 'open core'.”

Read the product's source under an open licenseweight 2

2 (weight) × 8 (quality) × 1.0 (full) = 16.0 of 20 max

  • [github] https://github.com/zulip/zulip“Contributing code. Check out our guide for new contributors to get started.”
  • [community] https://news.ycombinator.com/item?id=27149123“Zulip is my favorite modern chat program because it's actually 100% FOSS, unlike Mattermost and RocketChat, which are only 'open core'.”

Self-host the core productweight 3

3 (weight) × 7 (quality) × 0.6 (partial) = 12.6 of 30 max

  • [claimed-docs] https://zulip.com/self-hosting/“When you self-host Zulip, you get the same software as our Zulip Cloud customers.”
  • [claimed-docs] https://zulip.readthedocs.io/en/stable/production/install.html“install a self-hosted Zulip server on a system of your choice”
  • [claimed-docs] https://zulip.readthedocs.io/en/stable/production/install.html“To import data from Slack, Mattermost, Rocket.Chat, Zulip Cloud, or another Zulip server, follow the linked instructions.”
  • [claimed-docs] https://zulip.readthedocs.io/en/stable/production/install.html“Then, run the installer, providing your email address and the public hostname that users will be able to access your server with”
  • [github] https://github.com/zulip/zulip“Contributing code. Check out our guide for new contributors to get started.”
  • [community] https://news.ycombinator.com/item?id=17622987“On the downside, if you are self-hosting, Zulip requires a dedicated server (or VM). There are some efforts to make it run in a container but nothing that works really well.”
  • [community] https://news.ycombinator.com/item?id=16863675“I love Zulip... what Zulip lacks right now is polish. Server installation for example is a mess. You basically need a dedicated Debian machine to install it... search function barely works, lacks archiving topics.”
  • [community] https://news.ycombinator.com/item?id=16863675“The install procedure looks like a bad joke. So if I'm using RHEL, Fedora, Arch or any other Linux distribution, tough luck. If they're supporting only Ubuntu, they could have at least created some deb packages.”
  • [community] https://news.ycombinator.com/item?id=10279961“Something has gone horribly wrong when a chat server can barely run on 2G... recommended 4GB of RAM for production use.”

Openness = 61.0 ÷ 100 × 100 = 61.0

Built-in AI6.7/100×0.15 of the PA blend

Inside-out: how agentic the product itself is for its users — built-in assistants, autonomous features.

Get AI-generated insights and suggestions from my data inside the productweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Set up automations that run autonomously in the backgroundweight 2

2 (weight) × 5 (quality) × 0.6 (partial) = 6.0 of 20 max

  • [claimed-docs] https://zulip.com/help/running-bots“Zulip's API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip’s API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot, or run an existing Zulip bot.”
  • [claimed-docs] https://zulip.com/help/running-bots“Eventually, you’ll probably want to run it in a production environment where it’ll stay up, by deploying it on a server using the Zulip Botserver.”
  • [probe] https://raw.githubusercontent.com/zulip/zulip/main/zerver/openapi/zulip.yaml“PROBE openapi-spec: fetched zerver/openapi/zulip.yaml from the Zulip repo → HTTP 200, ~1.5 MB spec beginning 'openapi: 3.0.1' — the complete REST API ships as a machine-readable OpenAPI document (hands-on)”
  • [probe] https://zulip.com/api/installation-instructions“PROBE cli-runtime: pip install zulip (v0.9.1) → official zulip-send CLI installs from PyPI and runs headlessly (--help prints usage for sending messages from scripts) (hands-on, keyless)”

Delegate tasks to a built-in AI assistant inside the productweight 3

3 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 30 max

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Operate the product with natural-language commandsweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://zulip.com/help/search-for-messages“channel:design has:image new logo: Search for messages in #design that include an image and contain the keywords new and logo.”
  • [claimed-docs] https://zulip.com/help/search-for-messages“Zulip also offers a wide array of search filters, which can be used on their own, or in combination with keywords.”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip's API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip’s API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot, or run an existing Zulip bot.”

Built-in AI = 6.0 ÷ 90 × 100 = 6.7

Automation10.3/100×0.15 of the PA blend

Depth of automation primitives — rules, scheduling, bulk operations, webhooks.

Perform bulk operations across many items at onceweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://zulip.com/api/“Send a message”
  • [probe] https://raw.githubusercontent.com/zulip/zulip/main/zerver/openapi/zulip.yaml“PROBE openapi-spec: fetched zerver/openapi/zulip.yaml from the Zulip repo → HTTP 200, ~1.5 MB spec beginning 'openapi: 3.0.1' — the complete REST API ships as a machine-readable OpenAPI document (hands-on)”
  • [probe] https://zulip.com/api/installation-instructions“PROBE cli-runtime: pip install zulip (v0.9.1) → official zulip-send CLI installs from PyPI and runs headlessly (--help prints usage for sending messages from scripts) (hands-on, keyless)”

Define rules that trigger actions automatically on eventsweight 3

3 (weight) × 4 (quality) × 0.6 (partial) = 7.2 of 30 max

  • [claimed-docs] https://zulip.com/help/running-bots“Zulip's API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot”
  • [claimed-docs] https://zulip.com/integrations/“Over 130 native integrations. And hundreds more through Zapier and IFTTT.”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip’s API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot, or run an existing Zulip bot.”
  • [claimed-docs] https://zulip.com/integrations/“Incoming webhooks”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip’s API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot”

Schedule recurring jobs or workflowsweight 2

2 (weight) × 0 (quality) × 0.0 (none) = 0.0 of 20 max

  • [claimed-docs] https://zulip.com/help/running-bots“Zulip's API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot”
  • [claimed-docs] https://zulip.com/help/running-bots“Zulip’s API has a powerful framework for interactive bots that react to messages in Zulip. You can write and run a custom bot, or run an existing Zulip bot.”
  • [claimed-docs] https://zulip.com/help/running-bots“Eventually, you’ll probably want to run it in a production environment where it’ll stay up, by deploying it on a server using the Zulip Botserver.”
  • [claimed-docs] https://zulip.com/api/“Send a message”

Version, review, and roll back my automationsweight 1

n/a — not applicable to this product: excluded from numerator and denominator

no evidence cited — the verdict rests on absence of evidence, re-checked on refresh

Automation = 7.2 ÷ 70 × 100 = 10.3