Code Hosting — procurement report
ProductArena · rankings as of 2026-09-16 · evidence as of 2026-09-16 · 4 products · 72 judged requirements · 288 judged cells
Methodology: Every product is judged against a shared taxonomy of user stories using cited evidence — hands-on probes > repository code > independent community sources > vendor claims — never opinion. Full writeup: https://ultrametric.ai/productarena/methodology
Leaderboard
| # | Product | PA Score | Coverage score | Applicable cells | Confidence |
|---|---|---|---|---|---|
| 1 | GitLab | 55.5 | 40.7 | 72/72 | C |
| 2 | GitHub | 43.5 | 38.6 | 72/72 | C |
| 3 | Gitea | 28.4 | 30.1 | 69/72 | C |
| 4 | Bitbucket | 14.0 | 26.8 | 71/72 | C |
PA Score = agent-readiness blend (see methodology). Coverage score = weighted share of judged requirements met. Confidence = how much of the score rests on tested vs claimed evidence (A–D).
Uncertainty note
The current #1/#2 gap in this arena is not close enough to qualify for the multi-judge uncertainty pass — no extra caveat applies beyond the per-product confidence grades above.
Buyer checklist (RFP)
The arena's 72 judged user stories as requirements, grouped by theme. Priorities mirror the story weights our scoring uses (3 = must-have, 2 = should-have, 1 = nice-to-have). Interactive version with per-requirement verdicts for the top products: /arena/code-hosting/checklist
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
- ai-native userPlug MCP servers into this product so it can use their toolsmust-have
- ai-native userConnect an agent via an official MCP servermust-have
- ai-native userDrive the product through a documented public APImust-have
- ai-native userDelegate tasks to a built-in AI assistant inside the productmust-have
- ai-native userPoint an agent at llms.txt or agent-oriented docsshould-have
- ai-native userRun the product headlessly / in CI for automationshould-have
- ai-native userUse an official CLIshould-have
- ai-native userIssue scoped/least-privilege API credentials for an agentshould-have
- ai-native userBuild against official SDKsshould-have
- ai-native userSubscribe to events via webhooksshould-have
- ai-native userGet AI-generated insights and suggestions from my data inside the productshould-have
- ai-native userSet up automations that run autonomously in the backgroundshould-have
- ai-native userOperate the product with natural-language commandsshould-have
- ai-native userExplore an interactive API reference with runnable examplesshould-have
- ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)should-have
- ai-native userRely on versioned APIs with a documented deprecation policyshould-have
- ai-native userTest against a sandbox environment without touching production datanice-to-have
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
- ai-native userDefine rules that trigger actions automatically on eventsmust-have
- ai-native userPerform bulk operations across many items at onceshould-have
- ai-native userSchedule recurring jobs or workflowsshould-have
- ai-native userVersion, review, and roll back my automationsnice-to-have
Ci cd — continuous integration and delivery — pipelines, runners, cachingCi cd
Continuous integration and delivery — pipelines, runners, caching
- developerDefine CI/CD pipelines as code using a YAML syntaxmust-have
- devops-leadRun CI/CD jobs on runners hosted by the platform without managing infrastructuremust-have
- developerAutomate application deployments as part of my CI/CD pipelineshould-have
- developerLaunch a ready-to-code cloud-based development environment in secondsshould-have
- developerReuse community-built CI/CD actions or plugins in my pipelinesshould-have
- devops-leadGet centralized visibility into CI/CD workflow runs across the organizationshould-have
- devops-leadRun CI/CD jobs on my own self-hosted or private runnersshould-have
- devops-leadMap incidents to deployments and require change approval before releasesnice-to-have
Ecosystem integrations — the surrounding ecosystem — integrations, marketplaces, community packagesEcosystem integrations
The surrounding ecosystem — integrations, marketplaces, community packages
- developerExtend my stack with third-party apps, actions, and AI models from a marketplaceshould-have
- developerSync repository notifications and activity with chat toolsnice-to-have
- devops-leadConnect the platform to external CI/CD tools instead of using its native pipelinesnice-to-have
- devops-leadInstall third-party security scanning integrations directly from a security tabnice-to-have
Governance — stories about governance in this arenaGovernance
Stories about governance in this arena
- devops-leadEnforce coding standards and CI/CD policies across all teams and repositoriesshould-have
- devops-leadMake a single annual commitment and flexibly allocate spend across seat-based and usage-based licensingnice-to-have
- developerAccess built-in resources to learn the Git version control systemnice-to-have
- devops-leadRequire automated change approval workflows before code reaches productionnice-to-have
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
- ai-native userExport all of my data in open formats and leavemust-have
- ai-native userSelf-host the core productmust-have
- ai-native userDo everything through the API that I can do in the UIshould-have
- ai-native userRead the product's source under an open licenseshould-have
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
- ai-native userPrevent my data from being used to train AI modelsmust-have
- ai-native userChoose where my data is stored (region/residency)should-have
- ai-native userControl data retention and deletionshould-have
- ai-native userOpt out of telemetry and usage trackingshould-have
Project planning — stories about project planning in this arenaProject planning
Stories about project planning in this arena
- developerTrack requirements, features, and bugs using issues with milestones, time tracking, and dependenciesmust-have
- developerAutomatically update linked issue tracker statuses by referencing issue keys in my commitsshould-have
- developerOrganize work from high-level roadmaps to everyday tasks using project boardsshould-have
- developerManage external issue tracker items directly within the code hosting UInice-to-have
- developerManage projects and assign tasks from my mobile devicenice-to-have
Repos collaboration — working on repos together — pull requests, reviews, permissionsRepos collaboration
Working on repos together — pull requests, reviews, permissions
- developerReview code changes online and respond to inline review commentsmust-have
- developerCreate and manage Git repositories and browse their commit historymust-have
- developerPropose, discuss, and merge code changes using a pull or merge request workflowmust-have
- ai-native userHave an AI reviewer analyze code changes and suggest fixes during reviewshould-have
- ai-native userAssign the initial code review of a pull request to an AI reviewer for faster turnaroundshould-have
- developerReview large diffs and view third-party code quality reports directly in the review screenshould-have
- devops-leadEnforce standardized, customizable merge checks before code can be mergedshould-have
- open-source-maintainerPublish and consume software packages in multiple package formats from the same platformshould-have
- devops-leadControl granular access permissions at the workspace, project, repository, and branch levelshould-have
- developerSearch across repositories to find code, files, and commitsshould-have
Security — security posture and hardening storiesSecurity
Security posture and hardening stories
- developerSecure my account with two-factor authenticationmust-have
- developerAutomatically receive pull requests that update vulnerable dependenciesmust-have
- devops-leadConsolidate static analysis, dependency, secret, and dynamic security scanning into one platformmust-have
- devops-leadRestrict repository access to specific IP address rangesshould-have
- developerGenerate and manage personal access tokens for authenticating to the platformshould-have
- developerSet up SSH keys to authenticate and connect securely to the platformshould-have
- ai-native userHave AI automatically generate and apply fixes for detected security vulnerabilitiesshould-have
- devops-leadDetect and remediate leaked secrets across the organization's repositoriesshould-have
Self hosting — running it yourself — self-hosted servers, open-source clientsSelf hosting
Running it yourself — self-hosted servers, open-source clients
- devops-leadInstall a self-managed instance of the platform without laborious configurationmust-have
- devops-leadRun my self-hosted instance on my choice of operating system and architecture, including containersshould-have
- devops-leadScale my self-hosted installation using reference architectures supporting tens of thousands of usersshould-have
- devops-leadUse migration tooling to move my code and users from another platform or from self-hosted to cloudshould-have
Appendix: recorded probes
No replayable probe recordings exist for this arena yet. Probe-tier evidence (hands-on checks) still backs verdicts where cited — see each product page for the evidence trail.
Cite as: ProductArena by Ultrametric Inc, Code Hosting arena, rankings as of 2026-09-16 — https://ultrametric.ai/productarena/arena/code-hosting
License: © 2026 Ultrametric Inc. Brief quotation of individual verdicts, scores, or evidence excerpts is permitted with attribution to "ProductArena by Ultrametric Inc (ultrametric.ai/productarena)", as is use of the data to evaluate, contest, or contribute corrections. Bulk copying, redistribution, or use to build competing datasets requires prior written permission (see DATA-LICENSE in the repository).
No liability: rankings, verdicts, and scores are research outputs derived from the cited evidence at a point in time, provided "as is", without warranties. Ultrametric Inc accepts no responsibility for procurement, purchasing, or other decisions made in reliance on them — verify against the cited evidence before acting (https://ultrametric.ai/productarena/terms).