Code Hosting arenaBuyer checklist
Every requirement we judge code hosting products against, as a ready-to-send RFP checklist — with each item's priority, why it matters, and how the top-ranked products score on it today.
72 requirements · 11 themes · verdicts for 4 products · updated 2026-09-16 · priorities mirror the story weights our scoring uses (methodology)
Show the markdown export
# Code Hosting — buyer checklist (RFP) Derived from ProductArena's evidence-graded user-story taxonomy for Code Hosting: 72 judged requirements. Priorities mirror story weights (3 = must-have, 2 = should-have, 1 = nice-to-have). ## Agenticness - [ ] **[must-have]** Plug MCP servers into this product so it can use their tools - [ ] **[must-have]** Connect an agent via an official MCP server - [ ] **[must-have]** Drive the product through a documented public API - [ ] **[must-have]** Delegate tasks to a built-in AI assistant inside the product - [ ] **[should-have]** Point an agent at llms.txt or agent-oriented docs - [ ] **[should-have]** Run the product headlessly / in CI for automation - [ ] **[should-have]** Use an official CLI - [ ] **[should-have]** Issue scoped/least-privilege API credentials for an agent - [ ] **[should-have]** Build against official SDKs - [ ] **[should-have]** Subscribe to events via webhooks - [ ] **[should-have]** Get AI-generated insights and suggestions from my data inside the product - [ ] **[should-have]** Set up automations that run autonomously in the background - [ ] **[should-have]** Operate the product with natural-language commands - [ ] **[should-have]** Explore an interactive API reference with runnable examples - [ ] **[should-have]** Download a machine-readable API spec (OpenAPI or equivalent) - [ ] **[should-have]** Rely on versioned APIs with a documented deprecation policy - [ ] **[nice-to-have]** Test against a sandbox environment without touching production data ## Automation depth - [ ] **[must-have]** Define rules that trigger actions automatically on events - [ ] **[should-have]** Perform bulk operations across many items at once - [ ] **[should-have]** Schedule recurring jobs or workflows - [ ] **[nice-to-have]** Version, review, and roll back my automations ## Ci cd - [ ] **[must-have]** Define CI/CD pipelines as code using a YAML syntax - [ ] **[must-have]** Run CI/CD jobs on runners hosted by the platform without managing infrastructure - [ ] **[should-have]** Automate application deployments as part of my CI/CD pipeline - [ ] **[should-have]** Launch a ready-to-code cloud-based development environment in seconds - [ ] **[should-have]** Reuse community-built CI/CD actions or plugins in my pipelines - [ ] **[should-have]** Get centralized visibility into CI/CD workflow runs across the organization - [ ] **[should-have]** Run CI/CD jobs on my own self-hosted or private runners - [ ] **[nice-to-have]** Map incidents to deployments and require change approval before releases ## Ecosystem integrations - [ ] **[should-have]** Extend my stack with third-party apps, actions, and AI models from a marketplace - [ ] **[nice-to-have]** Sync repository notifications and activity with chat tools - [ ] **[nice-to-have]** Connect the platform to external CI/CD tools instead of using its native pipelines - [ ] **[nice-to-have]** Install third-party security scanning integrations directly from a security tab ## Governance - [ ] **[should-have]** Enforce coding standards and CI/CD policies across all teams and repositories - [ ] **[nice-to-have]** Make a single annual commitment and flexibly allocate spend across seat-based and usage-based licensing - [ ] **[nice-to-have]** Access built-in resources to learn the Git version control system - [ ] **[nice-to-have]** Require automated change approval workflows before code reaches production ## Openness - [ ] **[must-have]** Export all of my data in open formats and leave - [ ] **[must-have]** Self-host the core product - [ ] **[should-have]** Do everything through the API that I can do in the UI - [ ] **[should-have]** Read the product's source under an open license ## Privacy posture - [ ] **[must-have]** Prevent my data from being used to train AI models - [ ] **[should-have]** Choose where my data is stored (region/residency) - [ ] **[should-have]** Control data retention and deletion - [ ] **[should-have]** Opt out of telemetry and usage tracking ## Project planning - [ ] **[must-have]** Track requirements, features, and bugs using issues with milestones, time tracking, and dependencies - [ ] **[should-have]** Automatically update linked issue tracker statuses by referencing issue keys in my commits - [ ] **[should-have]** Organize work from high-level roadmaps to everyday tasks using project boards - [ ] **[nice-to-have]** Manage external issue tracker items directly within the code hosting UI - [ ] **[nice-to-have]** Manage projects and assign tasks from my mobile device ## Repos collaboration - [ ] **[must-have]** Review code changes online and respond to inline review comments - [ ] **[must-have]** Create and manage Git repositories and browse their commit history - [ ] **[must-have]** Propose, discuss, and merge code changes using a pull or merge request workflow - [ ] **[should-have]** Have an AI reviewer analyze code changes and suggest fixes during review - [ ] **[should-have]** Assign the initial code review of a pull request to an AI reviewer for faster turnaround - [ ] **[should-have]** Review large diffs and view third-party code quality reports directly in the review screen - [ ] **[should-have]** Enforce standardized, customizable merge checks before code can be merged - [ ] **[should-have]** Publish and consume software packages in multiple package formats from the same platform - [ ] **[should-have]** Control granular access permissions at the workspace, project, repository, and branch level - [ ] **[should-have]** Search across repositories to find code, files, and commits ## Security - [ ] **[must-have]** Secure my account with two-factor authentication - [ ] **[must-have]** Automatically receive pull requests that update vulnerable dependencies - [ ] **[must-have]** Consolidate static analysis, dependency, secret, and dynamic security scanning into one platform - [ ] **[should-have]** Restrict repository access to specific IP address ranges - [ ] **[should-have]** Generate and manage personal access tokens for authenticating to the platform - [ ] **[should-have]** Set up SSH keys to authenticate and connect securely to the platform - [ ] **[should-have]** Have AI automatically generate and apply fixes for detected security vulnerabilities - [ ] **[should-have]** Detect and remediate leaked secrets across the organization's repositories ## Self hosting - [ ] **[must-have]** Install a self-managed instance of the platform without laborious configuration - [ ] **[should-have]** Run my self-hosted instance on my choice of operating system and architecture, including containers - [ ] **[should-have]** Scale my self-hosted installation using reference architectures supporting tens of thousands of users - [ ] **[should-have]** Use migration tooling to move my code and users from another platform or from self-hosted to cloud --- Source: https://ultrametric.ai/productarena/arena/code-hosting (evidence-graded verdicts for 4 products) · methodology: https://ultrametric.ai/productarena/methodology
Chips show the top 4 ranked products' current verdict on each requirement — ✓ full · ~ partial · ! disputed · — none · n/a not applicable.
Agenticness — how well agents can access and operate the productAgenticness· 17 items
How well agents can access and operate the product
Automation depth — how much of the product can run unattendedAutomation depth· 4 items
How much of the product can run unattended
Ci cd — continuous integration and delivery — pipelines, runners, cachingCi cd· 8 items
Continuous integration and delivery — pipelines, runners, caching
Ecosystem integrations — the surrounding ecosystem — integrations, marketplaces, community packagesEcosystem integrations· 4 items
The surrounding ecosystem — integrations, marketplaces, community packages
Governance — stories about governance in this arenaGovernance· 4 items
Stories about governance in this arena
Openness — open source, data portability, and self-hosting storiesOpenness· 4 items
Open source, data portability, and self-hosting stories
Privacy posture — data-handling and privacy storiesPrivacy posture· 4 items
Data-handling and privacy stories
Project planning — stories about project planning in this arenaProject planning· 5 items
Stories about project planning in this arena
Repos collaboration — working on repos together — pull requests, reviews, permissionsRepos collaboration· 10 items
Working on repos together — pull requests, reviews, permissions
Security — security posture and hardening storiesSecurity· 8 items
Security posture and hardening stories
Self hosting — running it yourself — self-hosted servers, open-source clientsSelf hosting· 4 items
Running it yourself — self-hosted servers, open-source clients
Full evidence behind every verdict lives on the arena page and each product page — chips above deep-link straight to the judged story.