Security Scanners — procurement report
ProductArena · rankings as of 2026-09-16 · evidence as of 2026-09-16 · 6 products · 55 judged requirements · 330 judged cells
Methodology: Every product is judged against a shared taxonomy of user stories using cited evidence — hands-on probes > repository code > independent community sources > vendor claims — never opinion. Full writeup: https://ultrametric.ai/productarena/methodology
Leaderboard
| # | Product | PA Score | Coverage score | Applicable cells | Confidence |
|---|---|---|---|---|---|
| 1 | Gecko Security | 34.6 | 27.6 | 53/55 | B |
| 2 | Semgrep | 30.2 | 38.4 | 54/55 | A |
| 3 | Gitleaks | 30.1 | 35.6 | 32/55 | A |
| 4 | Trivy | 27.0 | 25.6 | 48/55 | B |
| 5 | Snyk | 24.2 | 28.9 | 55/55 | B |
| 6 | TruffleHog | 22.6 | 29.6 | 38/55 | B |
PA Score = agent-readiness blend (see methodology). Coverage score = weighted share of judged requirements met. Confidence = how much of the score rests on tested vs claimed evidence (A–D).
Uncertainty note
The current #1/#2 gap in this arena is not close enough to qualify for the multi-judge uncertainty pass — no extra caveat applies beyond the per-product confidence grades above.
Buyer checklist (RFP)
The arena's 55 judged user stories as requirements, grouped by theme. Priorities mirror the story weights our scoring uses (3 = must-have, 2 = should-have, 1 = nice-to-have). Interactive version with per-requirement verdicts for the top products: /arena/security-scanners/checklist
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
- ai-native userPlug MCP servers into this product so it can use their toolsmust-have
- ai-native userConnect an agent via an official MCP servermust-have
- ai-native userDrive the product through a documented public APImust-have
- ai-native userDelegate tasks to a built-in AI assistant inside the productmust-have
- ai-native userPoint an agent at llms.txt or agent-oriented docsshould-have
- ai-native userRun the product headlessly / in CI for automationshould-have
- ai-native userUse an official CLIshould-have
- ai-native userIssue scoped/least-privilege API credentials for an agentshould-have
- ai-native userBuild against official SDKsshould-have
- ai-native userSubscribe to events via webhooksshould-have
- ai-native userGet AI-generated insights and suggestions from my data inside the productshould-have
- ai-native userSet up automations that run autonomously in the backgroundshould-have
- ai-native userOperate the product with natural-language commandsshould-have
- ai-native userExplore an interactive API reference with runnable examplesshould-have
- ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)should-have
- ai-native userRely on versioned APIs with a documented deprecation policyshould-have
- ai-native userTest against a sandbox environment without touching production datanice-to-have
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
- ai-native userDefine rules that trigger actions automatically on eventsmust-have
- ai-native userPerform bulk operations across many items at onceshould-have
- ai-native userSchedule recurring jobs or workflowsshould-have
- ai-native userVersion, review, and roll back my automationsnice-to-have
Ci precommit — stories about ci precommit in this arenaCi precommit
Stories about ci precommit in this arena
- ai-native userHave an agent run a scan headlessly via the CLI, parse machine-readable results, and file issues for real findingsmust-have
- devops-leadRun scans in CI on every pull request and fail the build on new findings via exit codesmust-have
- developerRun the scanner as a pre-commit hook so secrets and issues never enter my git historymust-have
- ai-native userWire the scanner into my coding agent so AI-generated code is scanned for issues before it landsshould-have
- developerSee scan findings surfaced as pull-request comments or checks in GitHub or GitLabshould-have
Container scanning — stories about container scanning in this arenaContainer scanning
Stories about container scanning in this arena
- devops-leadScan container images for OS-package and application-layer vulnerabilities before deploying themmust-have
- devops-leadScan Terraform, Kubernetes, and other IaC files for security misconfigurationsshould-have
Coverage accuracy — stories about coverage accuracy in this arenaCoverage accuracy
Stories about coverage accuracy in this arena
- developerScan projects across all the major languages and package ecosystems my organization usesshould-have
- developerEmit scan results in machine-readable formats like JSON or SARIF for downstream toolingshould-have
Dependency sca — stories about dependency sca in this arenaDependency sca
Stories about dependency sca in this arena
- developerScan my dependency manifests and lockfiles for packages with known vulnerabilitiesmust-have
- developerGet concrete upgrade paths or automated fix pull requests that remediate vulnerable dependenciesshould-have
- security-engineerPrioritize dependency alerts by whether the vulnerable code is actually reachable from my codeshould-have
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
- ai-native userExport all of my data in open formats and leavemust-have
- ai-native userSelf-host the core productmust-have
- ai-native userDo everything through the API that I can do in the UIshould-have
- ai-native userRead the product's source under an open licenseshould-have
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
- ai-native userPrevent my data from being used to train AI modelsmust-have
- ai-native userChoose where my data is stored (region/residency)should-have
- ai-native userControl data retention and deletionshould-have
- ai-native userOpt out of telemetry and usage trackingshould-have
Sast rules — stories about sast rules in this arenaSast rules
Stories about sast rules in this arena
- developerRun static analysis that finds vulnerability classes like injection and XSS in my source codemust-have
- ai-native userHave AI draft remediation pull requests for security findings that I review and mergeshould-have
- security-engineerDetect taint-style vulnerabilities that flow across files and function boundariesshould-have
- security-engineerWrite custom static-analysis rules in a simple syntax and share them across repositoriesshould-have
- developerApply suggested or automatic code fixes for findings instead of only reading reportsshould-have
Secret detection — stories about secret detection in this arenaSecret detection
Stories about secret detection in this arena
- developerScan a repository and its full git history for hardcoded secrets and credentialsmust-have
- security-engineerDefine custom detection rules for my organization's internal token and secret formatsshould-have
- security-engineerScan non-git sources like S3 buckets, Docker images, and CI logs for leaked secretsshould-have
- security-engineerHave detected secrets verified against the issuing service so I know which ones are liveshould-have
Supply chain sbom — stories about supply chain sbom in this arenaSupply chain sbom
Stories about supply chain sbom in this arena
- security-engineerGenerate a software bill of materials (SPDX or CycloneDX) for my project or imageshould-have
- devops-leadDetect the licenses of my dependencies and enforce a license policy in scansnice-to-have
Triage noise — stories about triage noise in this arenaTriage noise
Stories about triage noise in this arena
- ai-native userUse AI to triage findings — explaining them, deduplicating, and flagging likely false positivesmust-have
- security-engineerBaseline or ignore existing and false-positive findings so only new issues alertmust-have
- developerFilter and gate on findings by severity or confidence so low-value noise does not block buildsshould-have
Appendix: recorded probes
Hands-on probe recordings — transcripts/videos a human can replay, the strongest evidence tier. Watch them at https://ultrametric.ai/productarena/proofs
- Gecko Security
curl -s https://gecko.security/docs/llms.txt | head -6terminal · recorded 2026-09-14 · exit 0 - Gecko Security
curl -si -X POST https://app.gecko.security/api/mcp -H 'Content-Type: application/json' -d '<jsonrpc initialize>'terminal · recorded 2026-09-14 · exit 0 - Gecko Security
curl -s https://app.gecko.security/api/v1/openapi.json | head -c 200 # OpenAPI 3.1, keylessterminal · recorded 2026-09-14 · exit 0 - Gitleaks
gitleaks versionterminal · recorded 2026-09-04 · exit 0 - Semgrep
semgrep --versionterminal · recorded 2026-09-04 · exit 0 - Snyk
snyk --versionterminal · recorded 2026-09-04 · exit 0 - Snyk
echo '<jsonrpc initialize>' | snyk mcp -t stdioterminal · recorded 2026-09-04 · exit 0 - Trivy
trivy --versionterminal · recorded 2026-09-04 · exit 0 - Trivy
echo '<jsonrpc initialize>' | trivy mcp --transport stdioterminal · recorded 2026-09-04 · exit 0 - TruffleHog
trufflehog --versionterminal · recorded 2026-09-04 · exit 0
Cite as: ProductArena by Ultrametric Inc, Security Scanners arena, rankings as of 2026-09-16 — https://ultrametric.ai/productarena/arena/security-scanners
License: © 2026 Ultrametric Inc. Brief quotation of individual verdicts, scores, or evidence excerpts is permitted with attribution to "ProductArena by Ultrametric Inc (ultrametric.ai/productarena)", as is use of the data to evaluate, contest, or contribute corrections. Bulk copying, redistribution, or use to build competing datasets requires prior written permission (see DATA-LICENSE in the repository).
No liability: rankings, verdicts, and scores are research outputs derived from the cited evidence at a point in time, provided "as is", without warranties. Ultrametric Inc accepts no responsibility for procurement, purchasing, or other decisions made in reliance on them — verify against the cited evidence before acting (https://ultrametric.ai/productarena/terms).