Semgrep vs Gecko Security

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Gecko Security
   Semgrep✓fullprobed8/10

   Semgrep hosts a working llms.txt at docs.semgrep.dev/llms.txt (HTTP 200, confirmed by probe) plus per-page .md agent-friendly docs (e.g. semgrep-docs-49, semgrep-docs-50), showing an agent can be pointed directly at machine-readable docs. missing for 10: independent/community confirmation that agents actually consume this llms.txt successfully, and no evidence of a broader agent-oriented docs index beyond the single llms.txt file.

   • [probe] “PROBE llms.txt: HTTP 200 at https://docs.semgrep.dev/llms.txt # Semgrep ## Home - [Semgrep Docs](https://docs.semgrep.dev/index.md): Get s…”
   • [claimed-docs] “scan your first project, which can be any codebase, repository, or folder within a monorepo, for security issues, and view your findings”
   • [claimed-docs] “You can run local Semgrep CLI scans with the Semgrep command-line tool.”
   • [probe] “official CLI documented at https://docs.semgrep.dev/getting-started/cli”

   proof ↗⚑ flag

   Gecko Security✓fullprobed9/10

   Direct probes confirm Gecko serves both llms.txt (HTTP 200) and a docs.md aggregate at the docs root, plus an OpenAPI spec and documented MCP/AI-tool integration for agents to consume — exactly the agent-oriented docs surface the story asks for. Missing for 10: no independent/community confirmation that an agent actually consumed llms.txt successfully in practice.

   • [probe] “PROBE llms.txt: HTTP 200 at https://gecko.security/llms.txt # Gecko Security > Gecko Security is an AI security engineer that finds exploit…”
   • [probe] “PROBE docs-md: HTTP 200 at https://gecko.security/docs.md --- title: "Gecko Security - Gecko Security" description: "An AI security engineer…”
   • [probe] “PROBE openapi: HTTP 200 at https://gecko.security/openapi.json — contains "openapi" key”
   • [claimed-docs] “Connect it to an AI tool and the tool can list your repositories and scans, pull full vulnerability evidence (proof of concept, call chain, …”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to Semgrep
   Semgrep✓fullcommunity9/10

   Semgrep has a well-documented CLI (`semgrep ci`, `semgrep scan`) designed for headless/CI use, with dedicated CI provider integrations, pre-commit hooks, SARIF export, no-login/dry-run modes, and an API for programmatic access to findings—clearly supporting automated, non-interactive pipelines. Independent evidence (Zulip, Microsoft SARIF+LLM triage pipeline) confirms real-world CI/automation usage at scale. Missing for 10: no first-party benchmark of CI runtime performance at scale beyond anecdotal community complaints about slowness.

   • [claimed-docs] “navigate to the root of your project, and run your first scan: semgrep ci”
   • [claimed-docs] “This guide walks you through creating a Semgrep job in the following CI providers, which are explicitly supported in Semgrep AppSec Platform”
   • [claimed-docs] “your development cycle benefits from the automated scanning of repositories at various events, such as: Push events; Pull requests or merge …”
   • [claimed-docs] “Semgrep is integrated into CI environments by creating a **job** that is run by the CI provider. After a scan, findings are sent to Semgrep …”
   • [claimed-docs] “The pre-commit framework can run semgrep when you commit changes. This is helpful in preventing secrets and security issues from leaking int…”
   • [claimed-docs] “The `--dry-run` flag ensures that your scans are not uploaded to the Semgrep web app.”
   • [claimed-docs] “You do **not** need to be logged in to run a scan.”
   • [claimed-docs] “To export the results to a SARIF file: semgrep ci --sarif --sarif-output=semgrep.sarif”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”
   • [community] “Built a tool using Semgrep's SARIF output plus prompted LLMs to help prioritize triage; used at Microsoft for the past year to help surface …”
   • [community] “We've used Semgrep for Zulip's Python codebase for months and really appreciate the semantic checks, especially for security lint rules - it…”

   proof ↗⚑ flag

   Gecko Security✓fullprobed7/10

   Gecko exposes a v1 API to trigger scans and read/triage results, dedicated CI/CD-labeled API keys, PR-check integration with configurable fail-on-severity thresholds that post pass/fail status back to GitHub/GitLab, and webhooks for automation pipelines — all enabling headless operation in CI. Missing for 10: independent/hands-on verification that CI pipelines actually work end-to-end, and no explicit example CI configs (e.g., GitHub Actions YAML) are shown.

   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [claimed-docs] “Give the key a descriptive name, such as `CI/CD Pipeline` or `Production`.”
   • [claimed-docs] “Gecko posts a pass/fail check back to GitHub or GitLab so reviewers see the security status inline.”
   • [claimed-docs] “**Fail-on severity**: the severity threshold that fails the check (for example, fail on High and above).”
   • [claimed-docs] “Gecko can push events to your systems as scans run and findings change: page a channel when a scan fails, open a workflow when a critical la…”
   • [probe] “PROBE openapi: HTTP 200 at https://gecko.security/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

3. ai-native userPlug MCP servers into this product so it can use their tools

   weight 3 · round drawn
   Semgrep—none0/10

   Evidence shows Semgrep exposes itself as an MCP server (semgrep-probe-3) so other agents can call its scanning tools, but there is no evidence that Semgrep itself can consume/plug in external MCP servers to use their tools — no client-side MCP integration is documented anywhere in the docs, CLI reference, or platform API pages.

   • [probe] “official MCP server documented at https://docs.semgrep.dev/mcp”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep AppSec Platform, and list p…”

   proof ↗⚑ flag

   Gecko Security—none0/10

   All evidence describes Gecko exposing its own data via an MCP server so external AI tools/agents can call into Gecko (docs-9, probe-4) — the reverse relationship. There is no evidence that Gecko itself can act as an MCP client and plug in external MCP servers to use their tools.

   • [claimed-docs] “Connect it to an AI tool and the tool can list your repositories and scans, pull full vulnerability evidence (proof of concept, call chain, …”
   • [probe] “official MCP server documented at https://gecko.security/docs/ai-tools/overview”

   proof ↗⚑ flag

4. ai-native userConnect an agent via an official MCP server

   weight 3 · round to Gecko Security
   Semgrep✓fullprobed6/10

   Semgrep is not itself an agent, so a first-party MCP server for connecting external agents is a fair axis, and evidence confirms one exists at docs.semgrep.dev/mcp. However, the evidence pack only shows the URL's existence via a probe with no detail on the tools/capabilities exposed or independent corroboration. Missing for 10: detailed documentation of what MCP tools/resources are exposed, setup instructions, and independent/hands-on confirmation of usage.

   • [probe] “official MCP server documented at https://docs.semgrep.dev/mcp”

   proof ↗⚑ flag

   Gecko Security✓fullprobed7/10

   Gecko documents an official MCP/AI-tools integration (docs-9, probe-4) letting a connected AI agent list repositories/scans, pull full vulnerability evidence, triage findings, and file tickets, and the docs.md/llms.txt endpoints confirm agent-facing documentation exists. Missing for 10: independent or hands-on confirmation that the MCP server works as described, and details on setup/auth flow for connecting an external agent.

   • [claimed-docs] “Connect it to an AI tool and the tool can list your repositories and scans, pull full vulnerability evidence (proof of concept, call chain, …”
   • [probe] “official MCP server documented at https://gecko.security/docs/ai-tools/overview”
   • [probe] “PROBE docs-md: HTTP 200 at https://gecko.security/docs.md --- title: "Gecko Security - Gecko Security" description: "An AI security engineer…”

   proof ↗⚑ flag

5. ai-native userUse an official CLI

   weight 2 · round to Semgrep
   Semgrep✓fullprobed8/10

   Semgrep ships a well-documented, installable official CLI (pipx/uv install, `semgrep scan`/`semgrep ci`, SARIF export, autofix, dry-run, no-login mode) that is fully scriptable and thus usable by AI agents, and this is corroborated by community usage reports and a dedicated probe confirming the CLI docs page. Missing for 10: explicit documentation of machine-readable (e.g., JSON) output tailored for programmatic/agent consumption and independent evidence of AI agents specifically driving the CLI.

   • [claimed-docs] “pipx install semgrep # preferred: install through uv (https://docs.astral.sh/uv/) uv tool install semgrep”
   • [claimed-docs] “semgrep scan - This is the recommended command for scanning local codebases or scanning a project when you don't have a Semgrep account”
   • [claimed-docs] “navigate to the root of your project, and run your first scan: semgrep ci”
   • [claimed-docs] “The `--dry-run` flag ensures that your scans are not uploaded to the Semgrep web app.”
   • [claimed-docs] “You do **not** need to be logged in to run a scan.”
   • [claimed-docs] “To export the results to a SARIF file: semgrep ci --sarif --sarif-output=semgrep.sarif”
   • [probe] “official CLI documented at https://docs.semgrep.dev/getting-started/cli”
   • [community] “We've used Semgrep for Zulip's Python codebase for months and really appreciate the semantic checks, especially for security lint rules - it…”

   proof ↗⚑ flag

   Gecko Security—none0/10

   No evidence of an official CLI; Gecko offers a GitHub App, REST API, webhooks, and an MCP server for AI tools, but nothing in the docs, changelog, or probes mentions a CLI tool for AI-native workflows.

   ⚑ flag

6. ai-native userDrive the product through a documented public API

   weight 3 · round to Gecko Security
   Semgrep~partialprobed6/10

   Semgrep AppSec Platform documents a public API for listing deployments, findings, and projects, and the CLI itself is scriptable for automation, but there is no OpenAPI/swagger spec discoverable (probe returned 404s on all candidate paths) and no independent/community corroboration of programmatic API usage. missing for 10: discoverable OpenAPI/schema documentation, broader API surface details (write operations, rate limits), and independent evidence of third-party integrations built against the API.

   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep AppSec Platform, and list p…”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.semgrep.dev/openapi.json, https://docs.semgrep.dev/swagger.json, https://docs.semgrep.d…”

   proof ↗⚑ flag

   Gecko Security✓fullprobed8/10

   Gecko documents a public v1 API (trigger scans, read scan/repo/vulnerability data, triage findings, manage schedules, rotate tokens, webhooks) with a live OpenAPI spec confirmed by probe, plus API key management and per-request logging. missing for 10: independent third-party developer confirmation of API usage/reliability beyond docs and probes, and more detail on API rate limits/versioning stability.

   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [claimed-docs] “API keys are now visible to the whole team with owner attribution and per-request usage logging”
   • [claimed-docs] “Gecko can push events to your systems as scans run and findings change: page a channel when a scan fails, open a workflow when a critical la…”
   • [claimed-docs] “Give the key a descriptive name, such as `CI/CD Pipeline` or `Production`.”
   • [probe] “PROBE openapi: HTTP 200 at https://gecko.security/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

7. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round to Gecko Security
   Semgrep—none0/10

   Evidence shows Semgrep AppSec Platform has an API for listing deployments/findings/projects, but there is no mention of scoped or least-privilege API tokens/credentials designed for agent use, nor any granular permission/role system for API keys.

   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep AppSec Platform, and list p…”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”

   proof ↗⚑ flag

   Gecko Security!disputedcontradicted4/10

   Gecko documents API keys with named scopes, GitHub App 'scoped, auto-rotating access' and GitLab token rotation (docs-11, docs-25, docs-26, docs-12), which nominally support least-privilege credential issuance for connected tools/agents. However, community hands-on reports concretely contradict the least-privilege claim: the GitHub OAuth flow requests broad 'act on my behalf' and profile/email write permissions rather than fine-grained scopes, prompting explicit user pushback (comm-1, comm-7). Missing for 10: documented per-agent scope configuration, evidence that API keys/tokens can be restricted to specific actions/resources, and resolution of the OAuth over-permissioning complaint.

   • [claimed-docs] “The App grants Gecko scoped, auto-rotating access, with no personal access token to manage.”
   • [claimed-docs] “Give the key a descriptive name, such as `CI/CD Pipeline` or `Production`.”
   • [claimed-docs] “API keys are now visible to the whole team with owner attribution and per-request usage logging”
   • [claimed-docs] “PATCH /integrations/gitlab swaps in a fresh token, built for secret managers and rotation scripts.”
   • [community] “OAuth flow asks for permission to write GitHub email/profile settings and 'act on my behalf' - concerning for a vuln scan tool, not just fin…”
   • [community] “User objects to GitHub OAuth requesting 'act on your behalf' permission, equating it to full admin access across all repos and orgs; asks Ge…”

   proof ↗⚑ flag

8. ai-native userBuild against official SDKs

   weight 2 · round to Gecko Security
   Semgrep—none0/10

   The evidence shows Semgrep offers a CLI, a REST API for the AppSec Platform, and an MCP server, but nowhere is an official client SDK (e.g., Python/JS/Go library for programmatic integration) documented or referenced.

   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep AppSec Platform, and list p…”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.semgrep.dev/openapi.json, https://docs.semgrep.dev/swagger.json, https://docs.semgrep.d…”
   • [probe] “official CLI documented at https://docs.semgrep.dev/getting-started/cli”

   proof ↗⚑ flag

   Gecko Security~partialprobed5/10

   Gecko documents a v1 REST API with OpenAPI spec, API keys, and webhooks that developers can build against, but there is no evidence of official client SDK libraries (e.g., Python/JS packages) beyond the raw API/OpenAPI reference. missing for 10: dedicated SDK libraries/language clients, independent developer corroboration of building against them.

   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [claimed-docs] “API keys are now visible to the whole team with owner attribution and per-request usage logging”
   • [probe] “PROBE openapi: HTTP 200 at https://gecko.security/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

9. ai-native userSubscribe to events via webhooks

   weight 2 · round to Gecko Security
   Semgrep—none0/10

   Evidence shows Semgrep provides a REST API for querying findings/deployments and CI-integration on push/PR events, but nothing describes a webhook subscription mechanism for events; no docs mention webhook endpoints or event-driven notifications to external systems.

   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep AppSec Platform, and list p…”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”
   • [claimed-docs] “your development cycle benefits from the automated scanning of repositories at various events, such as: Push events; Pull requests or merge …”

   proof ↗⚑ flag

   Gecko Security✓fullclaimed8/10

   Docs explicitly describe webhook subscriptions for scan/finding lifecycle events (scan fail, critical finding) and the API reference lists webhook events as a first-class capability alongside the API. This directly matches the story's ask for event subscription via webhooks. Missing for 10: independent/hands-on confirmation that webhook delivery works reliably and no detail on payload schema/retry semantics.

   • [claimed-docs] “Gecko can push events to your systems as scans run and findings change: page a channel when a scan fails, open a workflow when a critical la…”
   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Semgrep
   Semgrep~partialcommunity6/10

   Semgrep Code's 'AI-Powered Detection' and 'Semgrep Assistant (AI)' features are documented as generating insights (identifying complex business-logic flaws like IDORs, reducing false-positive noise by ~20%, combining AI reasoning with rule-based analysis for triage/remediation), which matches the story of in-product AI-generated insights. However, independent corroboration is thin — the one relevant community datapoint (semgrep-comm-14) describes a third party building their own external LLM triage layer on top of Semgrep's SARIF output rather than validating Semgrep Assistant itself, and another (semgrep-comm-13) notes an LLM hallucinating semgrep syntax when asked to help write rules, which is tangential. missing for 10: independent/hands-on validation of Semgrep Assistant's own AI suggestions, more detail on scope/accuracy of AI-Powered Detection, and confirmation these AI features are broadly available rather than gated/beta.

   • [claimed-docs] “With AI-Powered Detection, you can automatically identify complex business logic flaws, such as IDORs and broken authorization.”
   • [github] “Contextual post-processing of findings with Semgrep Assistant (AI) to further reduce noise by ~20%”
   • [claimed-docs] “Combine AI reasoning with rule-based analysis for detection, triage, and remediation”
   • [claimed-docs] “Semgrep Code is transparent; you can configure the rules it runs and inspect its syntax to understand how the finding was detected.”
   • [community] “Built a tool using Semgrep's SARIF output plus prompted LLMs to help prioritize triage; used at Microsoft for the past year to help surface …”

   proof ↗⚑ flag

   Gecko Security!disputedcontradicted5/10

   Gecko generates AI insights (severity, confidence score, call chain, PoC, suggested patch, AI-written repo wiki, endpoint map, PR review summaries) directly from scanned code data, per docs-2/17/18/19. However, hands-on community reports show these AI-generated findings can be inaccurate — 100%-confidence 'vulnerabilities' that weren't real, missed real issues a competing tool caught, and a trivial/irrelevant finding on another scan — directly contradicting the reliability of the insights. Missing for 10: independent validation of insight accuracy at scale, resolution of the false-positive/false-negative reports.

   • [claimed-docs] “Each finding includes a severity (CVSS 4.0), a confidence score, the full source-to-sink call chain, a proof of concept, and a suggested pat…”
   • [claimed-docs] “Repository wiki: An AI-written map of your app: architecture, routing, and security model.”
   • [claimed-docs] “Endpoint map: The HTTP attack surface Gecko discovered in your code.”
   • [claimed-docs] “PR reviews & fixes: On pull requests, a security review summary and one-click fixes.”
   • [community] “Tool gave 100% confidence for two vulnerabilities (severity 9.4 and 6.5) that turned out not to be real.”
   • [community] “Ran Gecko on a C project after Semgrep AI flagged a minor real issue; Gecko found nothing. After removing the fix and recreating the project…”
   • [community] “Uploaded a project (sydbox) to scan; took 10 minutes and found only one vulnerability, an 'RCE: IRC Message Command Execution Bypass' in an …”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Gecko Security
   Semgrep~partialclaimed5/10

   Semgrep supports automation via CI triggers on push/PR events, pre-commit hooks, and bulk 'Managed Scans' that run without manual invocation, and it has an AI Assistant/AI-Powered Detection layer for triage — providing real background automation. However, this is rule-based scan automation, not an autonomous agent that independently plans/executes multi-step actions; there's no evidence of scheduled/cron-like agentic orchestration beyond scan-on-event triggers. Missing for 10: evidence of autonomous multi-step agentic decision-making, scheduling/orchestration beyond CI-triggered scans, and independent confirmation that AI-driven remediation runs unattended.

   • [claimed-docs] “your development cycle benefits from the automated scanning of repositories at various events, such as: Push events; Pull requests or merge …”
   • [claimed-docs] “your development cycle benefits from the automated scanning of repositories at various events, such as: Push events, Pull requests or merge …”
   • [claimed-docs] “consider Semgrep Managed Scans, which enables you to bulk onboard and scan your repositories without requiring changes to your CI”
   • [claimed-docs] “The pre-commit framework can run semgrep when you commit changes. This is helpful in preventing secrets and security issues from leaking int…”
   • [claimed-docs] “The pre-commit framework can run `semgrep` when you commit changes. This is helpful in preventing secrets and security issues from leaking i…”
   • [claimed-docs] “With AI-Powered Detection, you can automatically identify complex business logic flaws, such as IDORs and broken authorization.”
   • [github] “Contextual post-processing of findings with Semgrep Assistant (AI) to further reduce noise by ~20%”
   • [claimed-docs] “Combine AI reasoning with rule-based analysis for detection, triage, and remediation”

   proof ↗⚑ flag

   Gecko Security✓fullclaimed7/10

   Gecko documents autonomous background workflows: scheduled scans via the API ('manage scan schedules'), automatic PR-triggered scans with pass/fail checks, auto-fix PR generation and re-verification, ticket creation, and webhook events pushed to external systems as scans run — all without manual intervention once configured. This directly matches the 'automations that run autonomously in the background' story for a security-scanning product. missing for 10: independent/hands-on confirmation that scheduled scans and auto-fix loops run reliably unattended over time, and more detail on configuring recurring schedules beyond a brief API mention.

   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [claimed-docs] “Gecko can push events to your systems as scans run and findings change: page a channel when a scan fails, open a workflow when a critical la…”
   • [claimed-docs] “Gecko posts a pass/fail check back to GitHub or GitLab so reviewers see the security status inline.”
   • [claimed-docs] “Click Request fix to open a gecko/* pull request with the patch applied, or apply the patch yourself.”
   • [claimed-docs] “Gecko rechecks the finding and marks it Fix verified once the vulnerability is gone.”
   • [claimed-docs] “create a ticket in Jira, Linear, ClickUp, or Shortcut”
   • [claimed-docs] “Commit to an existing PR: For findings from a PR scan, Gecko can commit the fix directly to the PR's head branch so the fix rides along with…”

   proof ↗⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round drawn
   Semgrep~partialclaimed5/10

   Semgrep ships 'Semgrep Assistant', a built-in AI feature that performs AI-powered detection of complex logic flaws, contextual post-processing to reduce noise, and combines AI reasoning with rule-based analysis for detection/triage/remediation, which functions as a task-delegable AI assistant within the product. However, this is scoped to security triage/detection only, not a general-purpose conversational assistant, and there's no independent hands-on account confirming quality of delegation. Missing for 10: evidence of a general chat/agent interface for arbitrary task delegation, independent/hands-on validation of Assistant's outputs, and detail on how users interact with/invoke it beyond marketing copy.

   • [claimed-docs] “With AI-Powered Detection, you can automatically identify complex business logic flaws, such as IDORs and broken authorization.”
   • [github] “Contextual post-processing of findings with Semgrep Assistant (AI) to further reduce noise by ~20%”
   • [claimed-docs] “Scan and fix AI-generated code the moment it's written”
   • [claimed-docs] “Combine AI reasoning with rule-based analysis for detection, triage, and remediation”

   proof ↗⚑ flag

   Gecko Security~partialprobed5/10

   Gecko is framed as an 'AI security engineer' that autonomously scans, triages, patches, and files tickets, so users can delegate security tasks (request fix, ticket creation, PR fixes) to its built-in AI (gecko-security-docs-3, gecko-security-docs-6, gecko-security-docs-15, gecko-security-probe-1). However, this is a fixed pipeline of security actions rather than a general-purpose in-product assistant you can converse with or direct on arbitrary tasks; community reports also raise doubts about the AI's accuracy in these delegated tasks (gecko-security-comm-3, gecko-security-comm-5). Missing for 10: an explicit conversational/task-delegation assistant UI, evidence of open-ended task delegation beyond the scan/fix/ticket workflow, and independent confirmation the delegated AI reliably completes tasks.

   • [claimed-docs] “Click Request fix to open a gecko/* pull request with the patch applied, or apply the patch yourself.”
   • [claimed-docs] “create a ticket in Jira, Linear, ClickUp, or Shortcut”
   • [claimed-docs] “Click **Request fix** to open a `gecko/*` pull request with the patch applied, or apply the patch yourself.”
   • [probe] “PROBE llms.txt: HTTP 200 at https://gecko.security/llms.txt # Gecko Security > Gecko Security is an AI security engineer that finds exploit…”
   • [community] “Tool gave 100% confidence for two vulnerabilities (severity 9.4 and 6.5) that turned out not to be real.”
   • [community] “Uploaded a project (sydbox) to scan; took 10 minutes and found only one vulnerability, an 'RCE: IRC Message Command Execution Bypass' in an …”

   proof ↗⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round to Gecko Security
   Semgrep~partialprobed3/10

   Semgrep is fundamentally a CLI/rule-based SAST tool; there's no documented natural-language command interface for the core product. The only relevant hook is a documented MCP server (semgrep-probe-3) which would let an AI agent invoke Semgrep via natural language, plus marketing mentions of 'AI reasoning' combined with rule-based detection (semgrep-docs-47/48), but no detail on what commands/actions are exposed or how they map from NL input. missing for 10: documented list of MCP tools/actions, evidence of actual natural-language command usage, hands-on confirmation that AI-native NL control works end-to-end.

   • [probe] “official MCP server documented at https://docs.semgrep.dev/mcp”
   • [claimed-docs] “Scan and fix AI-generated code the moment it's written”
   • [claimed-docs] “Combine AI reasoning with rule-based analysis for detection, triage, and remediation”
   • [github] “Contextual post-processing of findings with Semgrep Assistant (AI) to further reduce noise by ~20%”

   proof ↗⚑ flag

   Gecko Security~partialprobed6/10

   Gecko documents an official MCP/AI-tools integration letting a connected AI assistant list repos/scans, pull vulnerability evidence, triage findings, and file tickets via natural language, plus a full v1 API and OpenAPI spec (docs-9, docs-10, probe-3, probe-4). However, there's no hands-on or independent evidence confirming the natural-language workflow actually works well end-to-end — community reports focus on scan accuracy issues rather than the NL/agent interface itself. Missing for 10: independent/hands-on verification of the MCP natural-language workflow, and detail on command coverage/limitations.

   • [claimed-docs] “Connect it to an AI tool and the tool can list your repositories and scans, pull full vulnerability evidence (proof of concept, call chain, …”
   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [probe] “PROBE openapi: HTTP 200 at https://gecko.security/openapi.json — contains "openapi" key”
   • [probe] “official MCP server documented at https://gecko.security/docs/ai-tools/overview”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Semgrep—none0/10

   Semgrep documents an AppSec Platform API (list deployments, findings, projects) but there is no evidence of an interactive API reference with runnable/try-it examples; the openapi.json/swagger probes returned 404 on all candidate paths, indicating no interactive spec-based docs.

   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep AppSec Platform, and list p…”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.semgrep.dev/openapi.json, https://docs.semgrep.dev/swagger.json, https://docs.semgrep.d…”

   proof ↗⚑ flag

   Gecko Security—none0/10

   While Gecko exposes an OpenAPI spec (gecko-security-probe-3) and documents API capabilities (gecko-security-docs-10), there is no evidence of an interactive API reference UI with runnable/try-it examples for developers or AI agents to explore.

   • [probe] “PROBE openapi: HTTP 200 at https://gecko.security/openapi.json — contains "openapi" key”
   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round to Gecko Security
   Semgrep—none0/10

   Semgrep documents a REST API for the AppSec Platform (semgrep-docs-17/59) but a direct probe for machine-readable OpenAPI/Swagger specs at standard paths returned 404 everywhere (semgrep-probe-2), and no evidence of a published API spec exists elsewhere in the pack.

   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.semgrep.dev/openapi.json, https://docs.semgrep.dev/swagger.json, https://docs.semgrep.d…”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep AppSec Platform, and list p…”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”

   proof ↗⚑ flag

   Gecko Security✓fullprobed9/10

   A probe confirms a live OpenAPI spec at gecko.security/openapi.json (HTTP 200, contains 'openapi' key), and docs describe a full v1 API for scans, vulnerabilities, webhooks, and integrations. Missing for 10: no independent/community confirmation of the spec's completeness or usage beyond the vendor's own probe.

   • [probe] “PROBE openapi: HTTP 200 at https://gecko.security/openapi.json — contains "openapi" key”
   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”

   proof ↗⚑ flag

3. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round to Gecko Security
   Semgrep—none0/10

   Semgrep documents an AppSec Platform API (semgrep-docs-17/59) but no evidence pack item mentions API versioning scheme or a documented deprecation policy; an OpenAPI spec probe returned 404s (semgrep-probe-2), suggesting no formal API contract is published. missing for 10: versioning scheme documentation, deprecation policy/notice process, changelog or migration guides for API changes.

   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep AppSec Platform, and list p…”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.semgrep.dev/openapi.json, https://docs.semgrep.dev/swagger.json, https://docs.semgrep.d…”

   proof ↗⚑ flag

   Gecko Security~partialprobed3/10

   Gecko documents a versioned v1 API with an OpenAPI spec and a release/channel/version resolution endpoint (digest-pinned image refs), showing some notion of API versioning, but there is no documented deprecation policy, version sunset timeline, or migration guidance for API consumers. missing for 10: explicit deprecation policy, version support lifecycle/EOL dates, migration guides between API versions.

   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [claimed-docs] “resolves a version to its release... the response includes the digest-pinned image_ref to pull”
   • [claimed-docs] “GET /releases/{channel}/{version} resolves a version to its release.”
   • [probe] “PROBE openapi: HTTP 200 at https://gecko.security/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round to Semgrep
   Semgrep~partialclaimed6/10

   Semgrep supports bulk-style operations such as scanning an entire codebase/monorepo in one command, cross-file/interfile analysis across many files, and 'Managed Scans' explicitly designed to 'bulk onboard and scan your repositories without requiring changes to your CI.' The AppSec Platform API also lets you programmatically list deployments/projects/findings across an organization. However, there is no documented bulk-action tooling (e.g., bulk triage, bulk rule application, batch API calls) beyond onboarding/scanning, so the story is only partially covered. Missing for 10: explicit bulk API endpoints for mass finding triage/remediation, evidence of batch operations beyond scanning/onboarding, and independent confirmation of large-scale bulk workflows in practice.

   • [claimed-docs] “consider Semgrep Managed Scans, which enables you to bulk onboard and scan your repositories without requiring changes to your CI”
   • [claimed-docs] “Learn how to set up Semgrep, scan your first project, which can be any codebase, repository, or folder within a monorepo, for security issue…”
   • [claimed-docs] “scan your first project, which can be any codebase, repository, or folder within a monorepo, for security issues, and view your findings”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep AppSec Platform, and list p…”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”
   • [claimed-docs] “To turn on cross-file analysis, which allows you to detect vulnerabilities across files and folders”

   proof ↗⚑ flag

   Gecko Security~partialclaimed5/10

   The v1 API and AI-tool integration let a user programmatically list repositories/scans, pull vulnerability data, triage findings, and trigger scans across many items, which supports scripted bulk-style automation, but there is no documented bulk-fix, bulk-triage, or multi-select UI action for operating on many findings/repos simultaneously. missing for 10: a dedicated bulk-action endpoint or UI (e.g., bulk fix/triage/dismiss across many findings at once), independent evidence of bulk workflows in practice.

   • [claimed-docs] “Connect it to an AI tool and the tool can list your repositories and scans, pull full vulnerability evidence (proof of concept, call chain, …”
   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [claimed-docs] “Gecko can push events to your systems as scans run and findings change: page a channel when a scan fails, open a workflow when a critical la…”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round to Semgrep
   Semgrep✓fullclaimed8/10

   Semgrep rules can be configured to run automatically on events (push, PR/MR, pre-commit) and trigger downstream actions such as PR comments, alerts/notifications, ticket creation, and autofix application — all documented as first-class features (semgrep-docs-25/28/16/46/22/30/38/53/57). This directly matches the story of rules triggering automated actions on events. missing for 10: independent/hands-on evidence confirming these automated trigger-action pipelines (e.g., ticket creation, alerting) work reliably in production beyond vendor docs.

   • [claimed-docs] “your development cycle benefits from the automated scanning of repositories at various events, such as: Push events; Pull requests or merge …”
   • [claimed-docs] “your development cycle benefits from the automated scanning of repositories at various events, such as: Push events, Pull requests or merge …”
   • [claimed-docs] “The pre-commit framework can run semgrep when you commit changes. This is helpful in preventing secrets and security issues from leaking int…”
   • [claimed-docs] “The pre-commit framework can run `semgrep` when you commit changes. This is helpful in preventing secrets and security issues from leaking i…”
   • [claimed-docs] “Triage findings * Send alerts and notifications or create tickets to track findings identified by Semgrep Code”
   • [claimed-docs] “Triage findings Send alerts and notifications or create tickets to track findings identified by Semgrep Code”
   • [claimed-docs] “You can apply the Rule-defined fix directly to the file using the `--autofix` flag.”
   • [claimed-docs] “You can apply the Rule-defined fix directly to the file using the --autofix flag.”
   • [claimed-docs] “You can use Semgrep and its GitHub integration to automate PR comments that you frequently make in code reviews.”

   proof ↗⚑ flag

   Gecko Security~partialclaimed5/10

   Gecko supports some event-driven automation—webhooks that fire on scan completion/finding changes, configurable fail-on-severity thresholds for PR checks, and automatic ticket/ PR creation on findings—but these are fixed, product-defined triggers rather than a user-configurable rules engine where arbitrary conditions map to arbitrary actions. missing for 10: a general-purpose rule/condition builder, support for chaining multiple custom actions per event, and evidence of user-defined logic beyond severity thresholds.

   • [claimed-docs] “Gecko can push events to your systems as scans run and findings change: page a channel when a scan fails, open a workflow when a critical la…”
   • [claimed-docs] “**Fail-on severity**: the severity threshold that fails the check (for example, fail on High and above).”
   • [claimed-docs] “create a ticket in Jira, Linear, ClickUp, or Shortcut”
   • [claimed-docs] “Click Request fix to open a gecko/* pull request with the patch applied, or apply the patch yourself.”

   proof ↗⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round to Gecko Security
   Semgrep~partialclaimed3/10

   Semgrep supports automated scanning triggered by CI events (push, PR/MR) and offers Managed Scans for bulk repo scanning without CI changes, which implies some recurring automation, but there is no explicit documentation of a scheduling/cron mechanism or workflow orchestration for recurring jobs as an AI-native automation feature. missing for 10: explicit scheduling/cron support, documented recurring job configuration, and evidence of AI-native workflow orchestration beyond CI trigger events.

   • [claimed-docs] “consider Semgrep Managed Scans, which enables you to bulk onboard and scan your repositories without requiring changes to your CI”
   • [claimed-docs] “your development cycle benefits from the automated scanning of repositories at various events, such as: Push events; Pull requests or merge …”
   • [claimed-docs] “your development cycle benefits from the automated scanning of repositories at various events, such as: Push events, Pull requests or merge …”
   • [claimed-docs] “This guide walks you through creating a Semgrep job in the following CI providers, which are explicitly supported in Semgrep AppSec Platform”

   proof ↗⚑ flag

   Gecko Security~partialclaimed4/10

   The v1 API reference states it lets you 'manage scan schedules,' implying recurring scan jobs can be configured programmatically, but there is no dedicated doc page, UI walkthrough, or example showing how to set cadence, or evidence this extends to broader 'workflows' beyond scanning. missing for 10: dedicated scheduling documentation/UI, example of setting recurring cadence, evidence of scheduling beyond scans (e.g., recurring triage/ticket workflows), independent confirmation it works as claimed.

   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”

   proof ↗⚑ flag

4. ai-native userVersion, review, and roll back my automations

   weight 1 · round drawn
   Semgrep—none0/10

   Semgrep's evidence covers rule creation, testing, private-rule publishing, and CI integration, but nothing describes version history, review workflows, or rollback of rules/policies within Semgrep itself (rules are just YAML files presumably managed via external git, not a documented Semgrep versioning feature). Missing for 10: any documentation of rule/version history, approval/review workflow for rule changes, or a rollback mechanism in the Semgrep AppSec Platform.

   • [claimed-docs] “Users with Semgrep Code's Team or Enterprise tier can publish rules to the Semgrep Registry as private rules that are not visible to those o…”
   • [claimed-docs] “Semgrep provides a testing mechanism for your rules. You can write code and provide annotations to let Semgrep know where you are or aren't …”
   • [claimed-docs] “you can write custom rules to determine what Semgrep detects in your repositories”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko Security is a vulnerability-scanning/security tool, not an automation-building or workflow tool; the evidence shows PR creation, fix verification, and ticket filing but nothing about versioning, reviewing, or rolling back 'automations' (i.e., the scanning/triage workflows themselves). No changelog history, version control, or rollback mechanism for the automations it runs is documented.

   • [claimed-docs] “Click Request fix to open a gecko/* pull request with the patch applied, or apply the patch yourself.”
   • [claimed-docs] “Gecko rechecks the finding and marks it Fix verified once the vulnerability is gone.”
   • [claimed-docs] “create a ticket in Jira, Linear, ClickUp, or Shortcut”

   proof ↗⚑ flag

Agent scanning

1. ai-native userHave an agent run a scan headlessly via the CLI, parse machine-readable results, and file issues for real findings

   weight 3 · round to Semgrep
   Semgrep~partialcommunity7/10

   Semgrep's CLI clearly supports headless, unauthenticated scanning (`semgrep ci`/`semgrep scan`, no login required, --dry-run) and machine-readable output via SARIF export, which an agent could parse. Community evidence (semgrep-comm-14) shows this pattern used in practice — SARIF output plus LLM triage to surface real findings for human review at Microsoft. Docs also mention 'send alerts/notifications or create tickets to track findings' for Semgrep Code, but this is vague and tied to the platform tier rather than a demonstrated CLI-to-issue-tracker pipeline. Missing for 10: explicit documented JSON/SARIF-to-ticket automation workflow, first-party GitHub/Jira issue-filing integration for CLI findings, and independent hands-on confirmation that filed issues correspond only to 'real' (non-false-positive) findings given community complaints about false positives (semgrep-comm-11).

   • [claimed-docs] “navigate to the root of your project, and run your first scan: semgrep ci”
   • [claimed-docs] “semgrep scan - This is the recommended command for scanning local codebases or scanning a project when you don't have a Semgrep account”
   • [claimed-docs] “The `--dry-run` flag ensures that your scans are not uploaded to the Semgrep web app.”
   • [claimed-docs] “You do **not** need to be logged in to run a scan.”
   • [claimed-docs] “To export the results to a SARIF file:”
   • [claimed-docs] “To export the results to a SARIF file: semgrep ci --sarif --sarif-output=semgrep.sarif”
   • [claimed-docs] “Triage findings * Send alerts and notifications or create tickets to track findings identified by Semgrep Code”
   • [claimed-docs] “Triage findings Send alerts and notifications or create tickets to track findings identified by Semgrep Code”
   • [community] “Built a tool using Semgrep's SARIF output plus prompted LLMs to help prioritize triage; used at Microsoft for the past year to help surface …”

   proof ↗⚑ flag

   Gecko Security~partialcommunity4/10

   Gecko exposes a v1 API and webhooks that can trigger scans, pull machine-readable vulnerability data, triage findings, and file tickets in Jira/Linear/ClickUp/Shortcut, and an MCP/AI-tools connector for agents to do the same — satisfying the 'headless, parse, file issues' intent via API rather than a literal CLI. However, no dedicated CLI binary/command is documented anywhere in the evidence, and community reports raise concrete concerns about false positives (100% confidence findings that weren't real, missed real issues) undermining 'file issues for real findings'. Missing for 10: an actual CLI tool/subcommand documented, and independent confirmation that AI-driven ticket filing acts only on validated/real findings rather than the false positives community testers observed.

   • [claimed-docs] “Connect it to an AI tool and the tool can list your repositories and scans, pull full vulnerability evidence (proof of concept, call chain, …”
   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [claimed-docs] “create a ticket in Jira, Linear, ClickUp, or Shortcut”
   • [claimed-docs] “Gecko can push events to your systems as scans run and findings change: page a channel when a scan fails, open a workflow when a critical la…”
   • [community] “Tool gave 100% confidence for two vulnerabilities (severity 9.4 and 6.5) that turned out not to be real.”
   • [community] “Uploaded a project (sydbox) to scan; took 10 minutes and found only one vulnerability, an 'RCE: IRC Message Command Execution Bypass' in an …”

   proof ↗⚑ flag

2. ai-native userWire the scanner into my coding agent so AI-generated code is scanned for issues before it lands

   weight 2 · round to Semgrep
   Semgrep✓fullprobed7/10

   Semgrep ships an official MCP server (docs.semgrep.dev/mcp) letting coding agents call the scanner directly, plus first-party messaging on scanning/fixing AI-generated code the moment it's written and combining AI reasoning with rule-based detection, and a pre-commit hook to catch issues before they land in Git history. Missing for 10: independent/hands-on validation that the MCP integration reliably works inside real agent workflows, and detail on latency or false-positive behavior when used inline with AI code generation.

   • [probe] “official MCP server documented at https://docs.semgrep.dev/mcp”
   • [claimed-docs] “Scan and fix AI-generated code the moment it's written”
   • [claimed-docs] “Combine AI reasoning with rule-based analysis for detection, triage, and remediation”
   • [claimed-docs] “The pre-commit framework can run semgrep when you commit changes. This is helpful in preventing secrets and security issues from leaking int…”
   • [claimed-docs] “The pre-commit framework can run `semgrep` when you commit changes. This is helpful in preventing secrets and security issues from leaking i…”

   proof ↗⚑ flag

   Gecko Security~partialprobed6/10

   Gecko documents an MCP/AI-tool integration that lets an agent list repos/scans, pull vulnerability evidence, triage findings, and file tickets, plus PR/CI checks that fail builds above a severity threshold — together enabling a coding-agent workflow to scan AI-generated code before merge. However, independent hands-on reports raise real accuracy concerns (false positives at 100% confidence, missed real bugs, UI/scan bugs), which undercuts confidence that the scanning is reliable enough for a pre-merge gate. Missing for 10: independent corroboration that the MCP/agent wiring works smoothly in practice, and evidence addressing the accuracy complaints specifically in an agent-driven pre-commit/pre-merge context.

   • [claimed-docs] “Connect it to an AI tool and the tool can list your repositories and scans, pull full vulnerability evidence (proof of concept, call chain, …”
   • [probe] “official MCP server documented at https://gecko.security/docs/ai-tools/overview”
   • [claimed-docs] “Gecko posts a pass/fail check back to GitHub or GitLab so reviewers see the security status inline.”
   • [claimed-docs] “**Fail-on severity**: the severity threshold that fails the check (for example, fail on High and above).”
   • [community] “Tool gave 100% confidence for two vulnerabilities (severity 9.4 and 6.5) that turned out not to be real.”
   • [community] “Ran Gecko on a C project after Semgrep AI flagged a minor real issue; Gecko found nothing. After removing the fix and recreating the project…”
   • [community] “Uploaded a project (sydbox) to scan; took 10 minutes and found only one vulnerability, an 'RCE: IRC Message Command Execution Bypass' in an …”

   proof ↗⚑ flag

Ci gates

1. devops-leadRun scans in CI on every pull request and fail the build on new findings via exit codes

   weight 3 · round to Semgrep
   Semgrep✓fullcommunity9/10

   Docs explicitly cover running `semgrep ci` in CI pipelines triggered on push/PR events, with dedicated CI provider guides, SARIF export, and exit-code based build behavior implied by 'ci' command design for gating; community evidence corroborates real-world CI usage. missing for 10: explicit documented mention of exit code semantics for failing builds on new findings (implied but not directly quoted).

   • [claimed-docs] “navigate to the root of your project, and run your first scan: semgrep ci”
   • [claimed-docs] “This guide walks you through creating a Semgrep job in the following CI providers, which are explicitly supported in Semgrep AppSec Platform”
   • [claimed-docs] “your development cycle benefits from the automated scanning of repositories at various events, such as: Push events; Pull requests or merge …”
   • [claimed-docs] “Semgrep is integrated into CI environments by creating a **job** that is run by the CI provider. After a scan, findings are sent to Semgrep …”
   • [claimed-docs] “To export the results to a SARIF file: semgrep ci --sarif --sarif-output=semgrep.sarif”
   • [community] “We've used Semgrep for Zulip's Python codebase for months and really appreciate the semantic checks, especially for security lint rules - it…”

   proof ↗⚑ flag

   Gecko Security✓fullclaimed8/10

   Docs describe PR-triggered scans that post a pass/fail check to GitHub/GitLab, with a configurable fail-on-severity threshold and separation of new vs pre-existing findings, which together enable failing a PR build on new findings. This matches the CI/PR-gating story closely, though it's implemented as a GitHub/GitLab check status rather than an explicit CLI exit code and lacks independent hands-on confirmation of build-failing behavior. Missing for 10: independent verification that the pass/fail check actually blocks merges/builds in practice, and explicit mention of CLI exit-code semantics for non-GitHub CI runners.

   • [claimed-docs] “Gecko posts a pass/fail check back to GitHub or GitLab so reviewers see the security status inline.”
   • [claimed-docs] “Findings introduced by the PR are separated from pre-existing ones, so reviewers focus on what the change adds.”
   • [claimed-docs] “**Fail-on severity**: the severity threshold that fails the check (for example, fail on High and above).”
   • [claimed-docs] “install the Gecko GitHub App and pick repositories”

   proof ↗⚑ flag

2. developerSee scan findings surfaced as pull-request comments or checks in GitHub or GitLab

   weight 2 · round to Gecko Security
   Semgrep✓fullclaimed7/10

   Docs confirm Semgrep integrates into CI at push/PR/MR events, posts PR/MR comments for Secrets findings, and offers a GitHub integration for automating PR review comments, with findings sent to the AppSec Platform after scans. missing for 10: explicit first-party documentation of GitHub/GitLab PR status checks (not just comments) for Semgrep Code/Supply Chain findings, and independent hands-on confirmation of the PR-comment UX beyond docs.

   • [claimed-docs] “informs developers of valid secrets in their PRs and MRs by posting comments directly”
   • [claimed-docs] “your development cycle benefits from the automated scanning of repositories at various events, such as: Push events; Pull requests or merge …”
   • [claimed-docs] “your development cycle benefits from the automated scanning of repositories at various events, such as: Push events, Pull requests or merge …”
   • [claimed-docs] “Semgrep is integrated into CI environments by creating a **job** that is run by the CI provider. After a scan, findings are sent to Semgrep …”
   • [claimed-docs] “You can use Semgrep and its GitHub integration to automate PR comments that you frequently make in code reviews.”
   • [claimed-docs] “This guide walks you through creating a Semgrep job in the following CI providers, which are explicitly supported in Semgrep AppSec Platform”

   proof ↗⚑ flag

   Gecko Security✓fullclaimed8/10

   Docs describe a GitHub/GitLab App integration that posts pass/fail checks and PR security review summaries, separates new vs pre-existing findings, supports fail-on-severity thresholds, and can auto-commit fixes to the PR — a fairly complete PR-check/comment workflow (gecko-security-docs-5, -19, -28, -29, -30). Missing for 10: independent/hands-on confirmation that PR comments/checks actually render as described (community evidence only discusses finding accuracy, not the PR-check UX itself).

   • [claimed-docs] “Gecko posts a pass/fail check back to GitHub or GitLab so reviewers see the security status inline.”
   • [claimed-docs] “PR reviews & fixes: On pull requests, a security review summary and one-click fixes.”
   • [claimed-docs] “Findings introduced by the PR are separated from pre-existing ones, so reviewers focus on what the change adds.”
   • [claimed-docs] “**Fail-on severity**: the severity threshold that fails the check (for example, fail on High and above).”
   • [claimed-docs] “Commit to an existing PR: For findings from a PR scan, Gecko can commit the fix directly to the PR's head branch so the fix rides along with…”

   proof ↗⚑ flag

Pre commit

1. developerRun the scanner as a pre-commit hook so secrets and issues never enter my git history

   weight 3 · round to Semgrep
   Semgrep✓fullclaimed9/10

   Semgrep has explicit first-party documentation for integrating with the pre-commit framework, stating it runs semgrep on commit specifically to prevent secrets and security issues from leaking into git history, matching the story verbatim. Missing for 10: no independent/hands-on community confirmation specifically of the pre-commit hook workflow (community evidence covers CLI/CI use generally, not pre-commit hook experience).

   • [claimed-docs] “The pre-commit framework can run semgrep when you commit changes. This is helpful in preventing secrets and security issues from leaking int…”
   • [claimed-docs] “The pre-commit framework can run `semgrep` when you commit changes. This is helpful in preventing secrets and security issues from leaking i…”
   • [claimed-docs] “The pre-commit framework can run semgr”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko's documented workflow is GitHub App/PR-based scanning with pass/fail PR checks, not a local pre-commit git hook that blocks commits before they enter history — no evidence of a pre-commit hook, CLI hook script, or local git-hook integration exists in the evidence pack.

   ⚑ flag

Containers

1. devops-leadScan container images for OS-package and application-layer vulnerabilities before deploying them

   weight 3 · round drawn
   Semgrep—none0/10

   The evidence pack covers Semgrep Code (SAST), Secrets, and Supply Chain (dependency/OSS SCA) scanning of source repositories, CI pipelines, and manifests, but contains no mention of scanning container images or detecting OS-package vulnerabilities inside image layers — a capability offered by dedicated container scanners. Since Semgrep explicitly compares itself to AppSec competitors (Checkmarx, Snyk) that do offer this, the axis is a fair comparison point but no evidence shows Semgrep delivers it.

   • [claimed-docs] “Semgrep Supply Chain is a software composition analysis (SCA) tool that detects security vulnerabilities in your codebase introduced by open…”
   • [claimed-docs] “Generate reports and software bills of materials (SBOM) that provide a complete inventory of your open source components”
   • [claimed-docs] “Supply Chain's Dynamic Dependency Resolution provides a complete inventory of your project's dependencies using a combination of manifest pa…”
   • [claimed-docs] “Semgrep Secrets scans code to detect exposed API keys, passwords, and other credentials.”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko's documented capability is AI-driven source-code (SAST) analysis across languages like TypeScript, Python, Go, Java, etc., producing call-chain/PoC findings and PR fixes — no evidence describes scanning container images for OS-package (e.g., CVE feeds on apt/rpm packages) or layered image vulnerabilities. The only container-adjacent mention is a changelog entry resolving a version to a digest-pinned image_ref, which is about release artifact resolution, not vulnerability scanning of container layers.

   • [claimed-docs] “Compiler-accurate (precise cross-file analysis) | TypeScript · JavaScript · Python · Go · Java · Scala · C# · Rust”
   • [claimed-docs] “resolves a version to its release... the response includes the digest-pinned image_ref to pull”
   • [claimed-docs] “GET /releases/{channel}/{version} resolves a version to its release.”

   proof ↗⚑ flag

Iac misconfig

1. devops-leadScan Terraform, Kubernetes, and other IaC files for security misconfigurations

   weight 2 · round to Semgrep
   Semgrep~partialclaimed3/10

   Semgrep's docs mention that rules can be written to 'scan configuration files' as one use case (docs-21/35), which implies some IaC coverage, but the evidence pack never explicitly names Terraform, Kubernetes, Dockerfile, or IaC misconfiguration scanning as a supported capability. missing for 10: explicit documentation of Terraform/Kubernetes/Dockerfile rule packs, dedicated IaC scanning mode or product tier, and any community/hands-on confirmation of IaC misconfiguration detection.

   • [claimed-docs] “you can write rules that: Automate code review comments. Identify secure coding violations. Scan configuration files.”
   • [claimed-docs] “Automate code review comments. * Identify secure coding violations. * Scan configuration files.”
   • [claimed-docs] “Semgrep uses rules, which encapsulate pattern matching logic and data flow analysis, to scan your code for security issues, style violations…”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko's documented scanning scope is source-code vulnerability analysis (TypeScript, JavaScript, Python, Go, Java, Scala, C#, Rust) with call-chain/PoC findings; no evidence anywhere mentions Terraform, Kubernetes manifests, or IaC misconfiguration scanning. The axis is plausible for a security-scanning product, but no capability is evidenced, so this is 'none' rather than 'na'.

   • [claimed-docs] “Compiler-accurate (precise cross-file analysis) | TypeScript · JavaScript · Python · Go · Java · Scala · C# · Rust”
   • [claimed-docs] “Repository wiki: An AI-written map of your app: architecture, routing, and security model.”
   • [claimed-docs] “Endpoint map: The HTTP attack surface Gecko discovered in your code.”

   proof ↗⚑ flag

Coverage

1. developerScan projects across all the major languages and package ecosystems my organization uses

   weight 2 · round to Semgrep
   Semgrep~partialcommunity6/10

   Docs and community evidence show Semgrep scanning across many languages (Python, JS/TS, Ruby, YAML, HTML, CSS) and separately covering open-source package ecosystems via Semgrep Supply Chain (docs-11,12,13,23,29,44,55), plus a generic-pattern fallback for unsupported languages (docs-39). However there's no explicit doc listing 'all major languages/ecosystems' coverage, and a hands-on report shows Semgrep's parser failing on valid TypeScript syntax (comm-3), plus performance complaints scanning large multi-language codebases (comm-2, comm-7) that temper the breadth claim. missing for 10: an explicit enumerated language/ecosystem coverage matrix, resolution of the TypeScript parser bug, and independent benchmarking across the full claimed language set.

   • [claimed-docs] “If there's a code pattern in the codebase that matches the vulnerability definition, the finding is flagged as reachable.”
   • [claimed-docs] “Generate reports and software bills of materials (SBOM) that provide a complete inventory of your open source components”
   • [claimed-docs] “Supply Chain's Dynamic Dependency Resolution provides a complete inventory of your project's dependencies using a combination of manifest pa…”
   • [claimed-docs] “Detect malicious dependencies associated with Supply Chain attacks”
   • [claimed-docs] “Semgrep Supply Chain is a software composition analysis (SCA) tool that detects security vulnerabilities in your codebase introduced by open…”
   • [claimed-docs] “Semgrep can match generic patterns in languages that it does **not** yet support.”
   • [community] “Semgrep's parser complains that valid TypeScript using a typed catch clause (`catch (error: unknown)`) is invalid and can't be parsed.”
   • [community] “The underlying tree-sitter parser Semgrep uses is amazing - parsing millions of files across JS, TS, Ruby, YAML, HTML, CSS in about half a m…”
   • [community] “Running the CI ruleset on a tiny Django hobby project spun all cores to 100% after 33% progress and made the OS almost unresponsive; had to …”
   • [community] “We've used Semgrep for Zulip's Python codebase for months and really appreciate the semantic checks, especially for security lint rules - it…”

   proof ↗⚑ flag

   Gecko Security~partialcommunity5/10

   Docs list a defined set of 'compiler-accurate' languages (TypeScript, JavaScript, Python, Go, Java, Scala, C#, Rust) supporting cross-file analysis, but there is no mention of package/dependency-ecosystem scanning (npm, pip, Maven, Cargo, etc.) or of broader language coverage (C/C++, Ruby, PHP, Kotlin, Swift). Community testing on a C project (unsupported language) found nothing, reinforcing that coverage is limited to the stated language list rather than 'all major languages and ecosystems.' Missing for 10: explicit package/dependency-ecosystem scanning support, broader language list, and independent verification of coverage breadth.

   • [claimed-docs] “Compiler-accurate (precise cross-file analysis) | TypeScript · JavaScript · Python · Go · Java · Scala · C# · Rust”
   • [community] “Ran Gecko on a C project after Semgrep AI flagged a minor real issue; Gecko found nothing. After removing the fix and recreating the project…”

   proof ↗⚑ flag

Output formats

1. developerEmit scan results in machine-readable formats like JSON or SARIF for downstream tooling

   weight 2 · round to Semgrep
   Semgrep✓fullcommunity9/10

   Docs explicitly document exporting SARIF output via `semgrep ci --sarif --sarif-output=semgrep.sarif`, and community evidence shows real-world use of Semgrep's SARIF output for downstream tooling (e.g., piping into LLM triage at Microsoft). JSON output is also a standard supported format implied by the CLI's general reporting capabilities, though not separately documented in this pack. missing for 10: explicit docs snippet for --json flag usage.

   • [claimed-docs] “To export the results to a SARIF file:”
   • [claimed-docs] “To export the results to a SARIF file: semgrep ci --sarif --sarif-output=semgrep.sarif”
   • [community] “Built a tool using Semgrep's SARIF output plus prompted LLMs to help prioritize triage; used at Microsoft for the past year to help surface …”

   proof ↗⚑ flag

   Gecko Security~partialprobed4/10

   Gecko's v1 API lets developers read scan, repository, and vulnerability data programmatically (implying JSON responses) and an OpenAPI spec is published, but there is no explicit documentation of a SARIF export format or dedicated 'JSON report' output for CI pipelines. Missing for 10: explicit SARIF format support, documented JSON report export command, and any CI-integration example showing machine-readable output being consumed downstream.

   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [probe] “PROBE openapi: HTTP 200 at https://gecko.security/openapi.json — contains "openapi" key”
   • [claimed-docs] “Gecko can push events to your systems as scans run and findings change: page a channel when a scan fails, open a workflow when a critical la…”

   proof ↗⚑ flag

Dependency fixes

1. developerGet concrete upgrade paths or automated fix pull requests that remediate vulnerable dependencies

   weight 2 · round drawn
   Semgrep—none0/10

   Semgrep Supply Chain (SCA) evidence covers dependency vulnerability detection, reachability analysis, SBOM generation, and malicious dependency detection, but nothing in the evidence pack describes generating concrete upgrade paths or automated fix pull requests for vulnerable dependencies. The --autofix flag ([semgrep-docs-38],[semgrep-docs-53]) applies to Semgrep Code's rule-defined fixes for SAST findings, not dependency remediation.

   • [claimed-docs] “If there's a code pattern in the codebase that matches the vulnerability definition, the finding is flagged as reachable.”
   • [claimed-docs] “Generate reports and software bills of materials (SBOM) that provide a complete inventory of your open source components”
   • [claimed-docs] “Supply Chain's Dynamic Dependency Resolution provides a complete inventory of your project's dependencies using a combination of manifest pa…”
   • [claimed-docs] “Detect malicious dependencies associated with Supply Chain attacks”
   • [claimed-docs] “Semgrep Supply Chain is a software composition analysis (SCA) tool that detects security vulnerabilities in your codebase introduced by open…”
   • [claimed-docs] “You can apply the Rule-defined fix directly to the file using the `--autofix` flag.”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko's documented capability is source-to-sink code vulnerability detection (SAST-style) with a generic 'Request fix' PR mechanism for findings it detects in application code; there is no evidence of dependency/package manifest scanning, CVE-to-package matching, or specific upgrade-path recommendations for vulnerable dependencies (SCA). Missing for 10: any mention of software composition analysis, dependency manifest parsing, package version resolution, or upgrade-path guidance tied to third-party library CVEs.

   • [claimed-docs] “Each finding includes a severity (CVSS 4.0), a confidence score, the full source-to-sink call chain, a proof of concept, and a suggested pat…”
   • [claimed-docs] “Click Request fix to open a gecko/* pull request with the patch applied, or apply the patch yourself.”
   • [claimed-docs] “Click **Request fix** to open a `gecko/*` pull request with the patch applied, or apply the patch yourself.”
   • [claimed-docs] “Compiler-accurate (precise cross-file analysis) | TypeScript · JavaScript · Python · Go · Java · Scala · C# · Rust”

   proof ↗⚑ flag

Sca scanning

1. developerScan my dependency manifests and lockfiles for packages with known vulnerabilities

   weight 3 · round to Semgrep
   Semgrep✓fullcommunity7/10

   Semgrep Supply Chain is explicitly documented as an SCA tool that scans manifests/lockfiles, resolves dependency inventories, flags reachable vulnerabilities, detects malicious packages, and generates SBOMs (semgrep-docs-11,12,13,23,29,55). This directly matches the story of scanning dependency manifests/lockfiles for known vulnerabilities. Missing for 10: independent/hands-on community validation of SCA accuracy or usability (community evidence only covers SAST rule-writing, not the SCA/dependency-scanning feature), and one community comment (semgrep-comm-12) notes SCA isn't open-source and may require uploading data, a caveat not fully addressed in docs.

   • [claimed-docs] “If there's a code pattern in the codebase that matches the vulnerability definition, the finding is flagged as reachable.”
   • [claimed-docs] “Generate reports and software bills of materials (SBOM) that provide a complete inventory of your open source components”
   • [claimed-docs] “Supply Chain's Dynamic Dependency Resolution provides a complete inventory of your project's dependencies using a combination of manifest pa…”
   • [claimed-docs] “Detect malicious dependencies associated with Supply Chain attacks”
   • [claimed-docs] “Semgrep Supply Chain is a software composition analysis (SCA) tool that detects security vulnerabilities in your codebase introduced by open…”
   • [claimed-docs] “Support the enforcement of your business’ open source package licensing requirements”
   • [claimed-docs] “Semgrep Supply Chain is a software composition analysis (SCA) tool that detects security vulnerabilities in”
   • [community] “Regarding Semgrep Supply Chain: "This is not open source, though? It does make a big difference for some whether you're able to run the chec…”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko's documented capability is AI-driven source-code (SAST) analysis — call-chain tracing, endpoint mapping, PR-level code review — with no mention of scanning package manifests, lockfiles, or third-party dependency CVEs. Since Gecko is a security scanner, dependency/SCA scanning is a fair axis to ask about, but no evidence shows it exists.

   ⚑ flag

2. security-engineerPrioritize dependency alerts by whether the vulnerable code is actually reachable from my code

   weight 2 · round to Semgrep
   Semgrep~partialclaimed6/10

   Semgrep Supply Chain explicitly implements reachability analysis, flagging findings as reachable when a matching code pattern exists in the codebase, which directly supports prioritizing dependency alerts by reachability. However, evidence lacks detail on how reachability is surfaced in triage workflows (e.g., filtering/sorting UI, PR gating specifically by reachability) and no independent/hands-on verification of reachability accuracy is present. missing for 10: evidence of UI/workflow for filtering or sorting alerts by reachability status, independent validation of reachability detection accuracy, detail on false negative/positive rates for reachability analysis.

   • [claimed-docs] “If there's a code pattern in the codebase that matches the vulnerability definition, the finding is flagged as reachable.”
   • [claimed-docs] “If there’s a code pattern in the codebase that matches the vulnerability definition, the finding is flagged as reachable.”
   • [claimed-docs] “Semgrep Supply Chain is a software composition analysis (SCA) tool that detects security vulnerabilities in your codebase introduced by open…”
   • [claimed-docs] “Supply Chain's Dynamic Dependency Resolution provides a complete inventory of your project's dependencies using a combination of manifest pa…”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko's docs describe source-to-sink call-chain analysis and reachability for vulnerabilities found in first-party code, but there is no evidence it ingests dependency/SCA alerts (e.g., known-CVE package advisories) or prioritizes them by reachability — it's positioned as a SAST-style code vulnerability scanner, not a dependency/SCA tool.

   • [claimed-docs] “Each finding includes a severity (CVSS 4.0), a confidence score, the full source-to-sink call chain, a proof of concept, and a suggested pat…”
   • [claimed-docs] “Compiler-accurate (precise cross-file analysis) | TypeScript · JavaScript · Python · Go · Java · Scala · C# · Rust”
   • [claimed-docs] “Repository wiki: An AI-written map of your app: architecture, routing, and security model.”
   • [claimed-docs] “Endpoint map: The HTTP attack surface Gecko discovered in your code.”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to Gecko Security
   Semgrep~partialprobed4/10

   Semgrep AppSec Platform ships an API for listing deployments, projects and findings (semgrep-docs-17/59), and CLI/CI features (SARIF export, autofix, rule testing) give some parity with UI actions, but there is no evidence of a comprehensive OpenAPI spec (probe finds 404s at all openapi.json paths) or that triage actions, rule creation/management, policy configuration, or dashboard-only workflows are all exposed via API. missing for 10: documented OpenAPI/API reference covering full UI functionality, API endpoints for rule authoring/publishing and triage actions, independent confirmation that all UI capabilities are API-accessible.

   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep AppSec Platform, and list p…”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”
   • [probe] “PROBE openapi: all candidate paths 404 (https://docs.semgrep.dev/openapi.json, https://docs.semgrep.dev/swagger.json, https://docs.semgrep.d…”
   • [claimed-docs] “Triage findings * Send alerts and notifications or create tickets to track findings identified by Semgrep Code”
   • [claimed-docs] “Users with Semgrep Code's Team or Enterprise tier can publish rules to the Semgrep Registry as private rules that are not visible to those o…”

   proof ↗⚑ flag

   Gecko Security~partialprobed6/10

   Gecko exposes a documented v1 API (and OpenAPI spec) covering scans, repository/vulnerability data, triage, scheduling, token rotation and webhooks, plus an AI-tools/MCP-style integration that can list repos, pull evidence, triage, and file tickets — a substantial subset of UI functionality. However, there's no explicit claim or evidence that every UI feature (e.g., repository wiki generation, endpoint map, PR review summaries, initial GitHub App connection setup) is API-accessible, so full UI/API parity isn't demonstrated. Missing for 10: explicit parity statement or evidence that wiki/endpoint-map/PR-review generation and account/connection setup are API-triggerable, independent confirmation of API completeness.

   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [claimed-docs] “Connect it to an AI tool and the tool can list your repositories and scans, pull full vulnerability evidence (proof of concept, call chain, …”
   • [probe] “PROBE openapi: HTTP 200 at https://gecko.security/openapi.json — contains "openapi" key”
   • [claimed-docs] “Repository wiki: An AI-written map of your app: architecture, routing, and security model.”
   • [claimed-docs] “Endpoint map: The HTTP attack surface Gecko discovered in your code.”
   • [claimed-docs] “PR reviews & fixes: On pull requests, a security review summary and one-click fixes.”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round to Semgrep
   Semgrep~partialclaimed5/10

   Semgrep supports open-format export of scan results (SARIF) and a platform API to gather findings/projects, plus local CLI scanning that never uploads code by default (dry-run, no login required), which supports data portability and avoiding lock-in. However there's no documented comprehensive 'export all your data' feature covering full platform history, configs, and org data in one open bundle. Missing for 10: explicit bulk/account-level data export documentation, confirmation of open formats beyond SARIF (e.g., JSON schema docs), and independent verification of a full data portability workflow.

   • [claimed-docs] “To export the results to a SARIF file:”
   • [claimed-docs] “To export the results to a SARIF file: semgrep ci --sarif --sarif-output=semgrep.sarif”
   • [claimed-docs] “The `--dry-run` flag ensures that your scans are not uploaded to the Semgrep web app.”
   • [claimed-docs] “You do **not** need to be logged in to run a scan.”
   • [claimed-docs] “Semgrep AppSec Platform provides an API that enables you to list deployments, gather findings created by Semgrep”
   • [github] “Semgrep analyzes code locally on your computer or in your build environment: by default, code is never upload”

   proof ↗⚑ flag

   Gecko Security~partialprobed3/10

   The v1 API lets users read scan, repository, and vulnerability data programmatically (JSON via OpenAPI), which provides a technical path to export findings data, but there is no documented bulk 'export all data' feature, no mention of open-format guarantees for full account data, and no offboarding/exit workflow. missing for 10: explicit full-data export tooling, documented open-format (e.g. CSV/JSON) export guarantee, and account-closure/data-portability policy.

   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [probe] “PROBE openapi: HTTP 200 at https://gecko.security/openapi.json — contains "openapi" key”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round to Semgrep
   Semgrep~partialcommunity5/10

   The core Semgrep engine is hosted on a public GitHub repo (semgrep-gh-1/2/3), showing source code is publicly readable, but no evidence pack item states an explicit open-source license (e.g., LGPL) or points to a LICENSE file. Community evidence also notes that a commercial component, Semgrep Supply Chain, is explicitly 'not open source' (semgrep-comm-12), meaning the broader product is not uniformly open-licensed. missing for 10: explicit license text/citation, confirmation covering all product components, independent verification of license terms.

   • [github] “Semgrep would match x = 1; y = x + 1 when searching for 2”
   • [github] “Semgrep analyzes code locally on your computer or in your build environment: by default, code is never upload”
   • [github] “Contextual post-processing of findings with Semgrep Assistant (AI) to further reduce noise by ~20%”
   • [community] “Regarding Semgrep Supply Chain: "This is not open source, though? It does make a big difference for some whether you're able to run the chec…”

   proof ↗⚑ flag

   Gecko Security—none0/10

   No evidence that Gecko Security's source code is published or licensed openly; it is a proprietary SaaS/security scanning product with documented APIs and hybrid deployment, but nothing indicates its codebase is open source.

   ⚑ flag

4. ai-native userSelf-host the core product

   weight 3 · round to Semgrep
   Semgrep✓fullclaimed7/10

   Semgrep's core CLI/engine is open source and runs entirely on local infrastructure by default, never uploading code, and works without an account or login, which is effectively self-hosting the core scanning product (semgrep-gh-2, semgrep-docs-33/34/50/51/27). Missing for 10: explicit documentation of self-hosting the full AppSec Platform/server components, and independent hands-on confirmation of a self-hosted enterprise deployment.

   • [github] “Semgrep analyzes code locally on your computer or in your build environment: by default, code is never upload”
   • [claimed-docs] “The `--dry-run` flag ensures that your scans are not uploaded to the Semgrep web app.”
   • [claimed-docs] “You do **not** need to be logged in to run a scan.”
   • [claimed-docs] “You can run local Semgrep CLI scans with the Semgrep command-line tool.”
   • [claimed-docs] “The --dry-run flag ensures that your scans are not uploaded to the Semgrep web app.”
   • [claimed-docs] “semgrep scan - This is the recommended command for scanning local codebases or scanning a project when you don't have a Semgrep account”

   proof ↗⚑ flag

   Gecko Security~partialclaimed6/10

   Docs confirm a 'hybrid' deployment where the scanner, S3 storage, and AI inference all run inside the customer's AWS account, and the pricing page lists an 'On-prem / self-hosted / private cloud' tier — both indicating genuine self-hosting options exist. However, the same hybrid docs state Gecko still hosts the dashboard, database, and scan orchestration, so the core control plane is not actually self-hostable in the standard offering, and no architecture/setup docs for the full on-prem tier are provided. Missing for 10: detailed on-prem deployment guide/architecture for the fully self-hosted tier, and confirmation that dashboard/orchestration can also run on customer infrastructure.

   • [claimed-docs] “the scanner, S3 storage, and all AI inference run inside your AWS account. Gecko hosts only the dashboard, database, and scan orchestration.”
   • [claimed-docs] “the scanner, S3 storage, and all AI inference run inside your AWS account.”
   • [claimed-docs] “Unlimited scanning * On-prem / self-hosted / private cloud”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round to Gecko Security
   Semgrep—none0/10

   The evidence discusses local scanning (data never uploaded by default) and CLI dry-run options, but there is no mention of data residency, regional hosting options, or ability to choose storage location for the Semgrep AppSec Platform's cloud data. missing for 10: explicit region/residency selection controls, documentation of where cloud-stored findings/data reside, any enterprise data-residency options.

   • [github] “Semgrep analyzes code locally on your computer or in your build environment: by default, code is never upload”
   • [claimed-docs] “The `--dry-run` flag ensures that your scans are not uploaded to the Semgrep web app.”

   proof ↗⚑ flag

   Gecko Security~partialclaimed5/10

   Gecko offers a hybrid deployment where the scanner, S3 storage, and AI inference run inside the customer's own AWS account, and an on-prem/self-hosted/private cloud pricing tier, both of which let customers effectively control where their data resides. However, there's no explicit UI/setting for choosing a region for the Gecko-hosted dashboard/database components, and no documentation of specific supported regions or compliance certifications tied to residency. Missing for 10: explicit region-selection controls for Gecko-hosted components, list of supported regions, and residency/compliance documentation (e.g., GDPR/data-locality guarantees).

   • [claimed-docs] “the scanner, S3 storage, and all AI inference run inside your AWS account. Gecko hosts only the dashboard, database, and scan orchestration.”
   • [claimed-docs] “the scanner, S3 storage, and all AI inference run inside your AWS account.”
   • [claimed-docs] “Unlimited scanning * On-prem / self-hosted / private cloud”

   proof ↗⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round to Semgrep
   Semgrep~partialclaimed4/10

   Semgrep's docs and GitHub note that scanning runs locally by default and code is never uploaded, and the --dry-run flag guarantees no upload to the Semgrep web app, which operationally limits data exposure that could feed model training. However, there is no explicit privacy policy or statement addressing whether data sent to Semgrep's cloud/AI features (e.g., AI-Powered Detection, Semgrep Assistant) is used for AI training, nor any opt-out mechanism specific to AI training. Missing for 10: explicit AI-training data-use policy, opt-out toggle for AI features, and independent confirmation of this guarantee.

   • [github] “Semgrep analyzes code locally on your computer or in your build environment: by default, code is never upload”
   • [claimed-docs] “The `--dry-run` flag ensures that your scans are not uploaded to the Semgrep web app.”
   • [claimed-docs] “The --dry-run flag ensures that your scans are not uploaded to the Semgrep web app.”
   • [claimed-docs] “You do **not** need to be logged in to run a scan.”

   proof ↗⚑ flag

   Gecko Security—none0/10

   No evidence in the pack addresses data-training opt-out, AI training data policies, or any privacy controls governing whether user code/data is used to train AI models; Gecko's docs focus on scanning, deployment, and integrations, not this privacy axis.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round to Semgrep
   Semgrep~partialclaimed4/10

   Docs show privacy-preserving defaults (local-only scanning, --dry-run to avoid upload, no login required), giving users some control over whether data leaves their machine, but there is no evidence of explicit data retention periods, deletion mechanisms, or account/data-erasure controls for data that is uploaded to the Semgrep AppSec Platform. missing for 10: documented retention policy, explicit deletion/erasure controls for platform-stored findings, GDPR/CCPA data handling commitments.

   • [github] “Semgrep analyzes code locally on your computer or in your build environment: by default, code is never upload”
   • [claimed-docs] “The `--dry-run` flag ensures that your scans are not uploaded to the Semgrep web app.”
   • [claimed-docs] “The --dry-run flag ensures that your scans are not uploaded to the Semgrep web app.”
   • [claimed-docs] “You do **not** need to be logged in to run a scan.”

   proof ↗⚑ flag

   Gecko Security—none0/10

   The evidence pack covers scanning, PR fixes, integrations, hybrid deployment, and API/webhooks, but there is no mention of data retention policies, data deletion controls, or user ability to purge stored findings/scan data. Hybrid deployment keeps scanner/AI inference in the customer's AWS account, but this does not address retention/deletion of dashboard-hosted data. missing for 10: explicit data retention policy documentation, a deletion/purge mechanism or API, and any account/data-export or GDPR-style deletion workflow.

   • [claimed-docs] “the scanner, S3 storage, and all AI inference run inside your AWS account. Gecko hosts only the dashboard, database, and scan orchestration.”
   • [claimed-docs] “the scanner, S3 storage, and all AI inference run inside your AWS account.”

   proof ↗⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round to Semgrep
   Semgrep~partialclaimed5/10

   Semgrep's docs and GitHub README indicate privacy-conscious defaults: code is analyzed locally and never uploaded by default, no login is required to run a scan, and the --dry-run flag explicitly prevents scan results from being uploaded to the Semgrep web app. However, there's no explicit documentation of a dedicated 'telemetry/usage tracking' opt-out setting (e.g., an env var or flag specifically for metrics/telemetry) beyond the scan-upload behavior. missing for 10: explicit telemetry/usage-metrics opt-out flag or setting, independent confirmation of what usage data (if any) is collected outside of code content.

   • [github] “Semgrep analyzes code locally on your computer or in your build environment: by default, code is never upload”
   • [claimed-docs] “The `--dry-run` flag ensures that your scans are not uploaded to the Semgrep web app.”
   • [claimed-docs] “You do **not** need to be logged in to run a scan.”
   • [claimed-docs] “The --dry-run flag ensures that your scans are not uploaded to the Semgrep web app.”

   proof ↗⚑ flag

   Gecko Security—none0/10

   No evidence anywhere in the pack of a telemetry/usage-tracking opt-out setting, privacy policy control, or data-collection toggle; the docs focus on scanning, PR fixes, API keys, and deployment but never mention telemetry opt-out.

   ⚑ flag

Ai remediation

1. ai-native userHave AI draft remediation pull requests for security findings that I review and merge

   weight 2 · round to Gecko Security
   Semgrep~partialclaimed4/10

   Semgrep offers rule-defined autofix (--autofix), AI-powered detection/triage/remediation messaging ('Combine AI reasoning with rule-based analysis for detection, triage, and remediation'), and GitHub PR comment automation, but there is no documented evidence of an end-to-end AI-drafted pull request workflow for remediation that a user reviews and merges. missing for 10: evidence of automatic PR/MR creation with AI-generated code fixes, a documented workflow showing AI drafting a remediation diff as a mergeable PR, and independent/hands-on confirmation of this remediation-PR feature working.

   • [claimed-docs] “You can apply the Rule-defined fix directly to the file using the `--autofix` flag.”
   • [claimed-docs] “Combine AI reasoning with rule-based analysis for detection, triage, and remediation”
   • [claimed-docs] “You can use Semgrep and its GitHub integration to automate PR comments that you frequently make in code reviews.”
   • [claimed-docs] “informs developers of valid secrets in their PRs and MRs by posting comments directly”

   proof ↗⚑ flag

   Gecko Security✓fullcommunity8/10

   Docs describe a complete AI-drafted remediation PR workflow: 'Request fix' opens a gecko/* PR with the patch applied, fixes can commit directly to the PR's head branch, and Gecko rechecks and marks 'Fix verified' once merged and the vulnerability is resolved — squarely matching the ai-native review-and-merge story. Community reports raise concerns about finding accuracy/false positives, but do not contradict that the PR-drafting/remediation mechanism itself works as documented. Missing for 10: independent hands-on confirmation that the auto-generated patches are high quality and mergeable without heavy rework.

   • [claimed-docs] “Click Request fix to open a gecko/* pull request with the patch applied, or apply the patch yourself.”
   • [claimed-docs] “Click **Request fix** to open a `gecko/*` pull request with the patch applied, or apply the patch yourself.”
   • [claimed-docs] “When the fix merges, Gecko rechecks the finding and marks it **Fix verified** once the vulnerability is gone.”
   • [claimed-docs] “Commit to an existing PR: For findings from a PR scan, Gecko can commit the fix directly to the PR's head branch so the fix rides along with…”
   • [claimed-docs] “PR reviews & fixes: On pull requests, a security review summary and one-click fixes.”
   • [community] “Tool gave 100% confidence for two vulnerabilities (severity 9.4 and 6.5) that turned out not to be real.”

   proof ↗⚑ flag

Code analysis

1. security-engineerDetect taint-style vulnerabilities that flow across files and function boundaries

   weight 2 · round to Semgrep
   Semgrep✓fullclaimed8/10

   Semgrep explicitly documents taint tracking/taint mode for catching injection-style vulnerabilities (semgrep-docs-41, semgrep-docs-58) and separately documents cross-file (interfile) analysis via Semgrep Code (semgrep-docs-2, semgrep-docs-32) plus interprocedural/interfile constant propagation (semgrep-docs-42), together supporting taint flows across files and function boundaries. Missing for 10: no independent/hands-on evidence specifically validating cross-file taint tracking accuracy (community evidence only covers simpler pattern-matching bypasses, not taint-mode cross-file cases).

   • [claimed-docs] “Taint tracking (known also as taint analysis) enables you to write simple rules that catch complex injection bugs, such as those that can re…”
   • [claimed-docs] “Semgrep supports taint analysis, also known as taint tracking, through taint rules.”
   • [claimed-docs] “To turn on cross-file analysis, which allows you to detect vulnerabilities across files and folders”
   • [claimed-docs] “Learn how to set up Semgrep, scan your project for security issues using Semgrep Code’s interfile analysis, and view your findings in the CL…”
   • [claimed-docs] “Semgrep AppSec Platform supports interprocedural (cross-function), interfile (cross-file) constant propagation.”

   proof ↗⚑ flag

   Gecko Security!disputedcontradicted5/10

   Docs explicitly claim 'compiler-accurate' cross-file analysis and 'full source-to-sink call chain' reporting, which directly describes taint-style detection across file/function boundaries (gecko-security-docs-2, gecko-security-docs-20). However, independent hands-on reports show concrete failures of this claim: 100%-confidence findings that were not real vulnerabilities, missed a real known issue flagged by another tool, and an inaccurate/irrelevant finding on a scanned project (gecko-security-comm-3, gecko-security-comm-4, gecko-security-comm-5). Missing for 10: independent verification of true-positive cross-file taint chains, resolution of the reported false positive/negative cases, and reproducible benchmark data.

   • [claimed-docs] “Each finding includes a severity (CVSS 4.0), a confidence score, the full source-to-sink call chain, a proof of concept, and a suggested pat…”
   • [claimed-docs] “Compiler-accurate (precise cross-file analysis) | TypeScript · JavaScript · Python · Go · Java · Scala · C# · Rust”
   • [community] “Tool gave 100% confidence for two vulnerabilities (severity 9.4 and 6.5) that turned out not to be real.”
   • [community] “Ran Gecko on a C project after Semgrep AI flagged a minor real issue; Gecko found nothing. After removing the fix and recreating the project…”
   • [community] “Uploaded a project (sydbox) to scan; took 10 minutes and found only one vulnerability, an 'RCE: IRC Message Command Execution Bypass' in an …”

   proof ↗⚑ flag

2. developerRun static analysis that finds vulnerability classes like injection and XSS in my source code

   weight 3 · round to Semgrep
   Semgrep✓fullcommunity8/10

   Semgrep explicitly documents SAST scanning with taint tracking designed to catch injection and XSS bugs (semgrep-docs-41, semgrep-docs-58), backed by a CLI scan workflow (semgrep-docs-19, semgrep-docs-27) and corroborated by community use for security lint rules including injection detection (semgrep-comm-1, semgrep-comm-7). Missing for 10: independent benchmark data on vulnerability-class detection rates, and community reports show simple evasions (e.g., function aliasing) can bypass naive rules (semgrep-comm-6, semgrep-comm-10), indicating real-world efficacy caveats.

   • [claimed-docs] “Taint tracking (known also as taint analysis) enables you to write simple rules that catch complex injection bugs, such as those that can re…”
   • [claimed-docs] “Semgrep supports taint analysis, also known as taint tracking, through taint rules.”
   • [claimed-docs] “navigate to the root of your project, and run your first scan: semgrep ci”
   • [claimed-docs] “semgrep scan - This is the recommended command for scanning local codebases or scanning a project when you don't have a Semgrep account”
   • [community] “As a security consultant, I write custom Semgrep rules to match antipatterns unique to a codebase (e.g., controllers missing an authorizatio…”
   • [community] “We've used Semgrep for Zulip's Python codebase for months and really appreciate the semantic checks, especially for security lint rules - it…”
   • [community] “A simple Semgrep 'no-prints' rule can be trivially bypassed with a workaround like `import builtins; builtins.print("whee")`, showing the to…”
   • [community] “Tried the Semgrep editor demo searching for exec(...) in Python; it failed to catch a simple aliasing workaround: `not_exec = exec; not_exec…”

   proof ↗⚑ flag

   Gecko Security!disputedcontradicted5/10

   Gecko's docs describe compiler-accurate, cross-file static analysis with source-to-sink call chains, CVSS severity, and confidence scores that would encompass classes like SQLi/XSS (docs-2, docs-20), and it supports multiple languages. However, independent hands-on testing on Hacker News reports both missed real vulnerabilities and high-confidence false positives (comm-3, comm-4, comm-5), directly contradicting the accuracy implied by the docs. Missing for 10: independent benchmark data on injection/XSS detection rates, and resolution of the false-positive/false-negative reports.

   • [claimed-docs] “Each finding includes a severity (CVSS 4.0), a confidence score, the full source-to-sink call chain, a proof of concept, and a suggested pat…”
   • [claimed-docs] “Compiler-accurate (precise cross-file analysis) | TypeScript · JavaScript · Python · Go · Java · Scala · C# · Rust”
   • [community] “Tool gave 100% confidence for two vulnerabilities (severity 9.4 and 6.5) that turned out not to be real.”
   • [community] “Ran Gecko on a C project after Semgrep AI flagged a minor real issue; Gecko found nothing. After removing the fix and recreating the project…”
   • [community] “Uploaded a project (sydbox) to scan; took 10 minutes and found only one vulnerability, an 'RCE: IRC Message Command Execution Bypass' in an …”

   proof ↗⚑ flag

Custom rules

1. security-engineerWrite custom static-analysis rules in a simple syntax and share them across repositories

   weight 2 · round to Semgrep
   Semgrep✓fullcommunity8/10

   Semgrep's docs extensively cover writing custom rules with a simple YAML-based pattern syntax, testing them, using taint tracking, and sharing rules across repos via the Registry and private rules for teams; community evidence (HN) confirms security engineers actually write custom rules to catch codebase-specific antipatterns and find the syntax readable/flexible across languages. Missing for 10: independent evidence specifically validating cross-repo sharing workflows at scale, and community reports note real limitations (bypassable rules, parser errors, performance issues) that slightly temper confidence in robustness.

   • [claimed-docs] “you can write custom rules to determine what Semgrep detects in your repositories”
   • [claimed-docs] “you can write and test using the [Semgrep Editor]”
   • [claimed-docs] “you can write rules that: Automate code review comments. Identify secure coding violations. Scan configuration files.”
   • [claimed-docs] “Users with Semgrep Code's Team or Enterprise tier can publish rules to the Semgrep Registry as private rules that are not visible to those o…”
   • [claimed-docs] “Semgrep provides a testing mechanism for your rules. You can write code and provide annotations to let Semgrep know where you are or aren't …”
   • [claimed-docs] “Taint tracking (known also as taint analysis) enables you to write simple rules that catch complex injection bugs, such as those that can re…”
   • [claimed-docs] “Semgrep supports taint analysis, also known as taint tracking, through taint rules.”
   • [community] “As a security consultant, I write custom Semgrep rules to match antipatterns unique to a codebase (e.g., controllers missing an authorizatio…”
   • [community] “A simple Semgrep 'no-prints' rule can be trivially bypassed with a workaround like `import builtins; builtins.print("whee")`, showing the to…”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko Security is an AI-driven vulnerability scanner using compiler-accurate cross-file analysis, not a rule-based SAST engine; there is no evidence of a custom rule-writing syntax (e.g., YAML/DSL rules like Semgrep) or any mechanism to define and share such rules across repositories.

   • [claimed-docs] “Compiler-accurate (precise cross-file analysis) | TypeScript · JavaScript · Python · Go · Java · Scala · C# · Rust”
   • [claimed-docs] “Repository wiki: An AI-written map of your app: architecture, routing, and security model.”

   proof ↗⚑ flag

Fix suggestions

1. developerApply suggested or automatic code fixes for findings instead of only reading reports

   weight 2 · round to Gecko Security
   Semgrep~partialclaimed6/10

   Semgrep supports rule-defined autofix via the --autofix flag, letting developers apply suggested fixes directly to files (semgrep-docs-38, semgrep-docs-53), and AppSec Platform mentions 'remediation' and 'fix AI-generated code' (semgrep-docs-45, semgrep-docs-47). However, autofix depends on individual rules defining a fix and is not universal across all findings, and there's no independent/community evidence confirming real-world reliability of autofix. missing for 10: community/hands-on validation of autofix accuracy, evidence of coverage breadth (what % of rules ship with fixes), and any first-party numbers on fix acceptance rates.

   • [claimed-docs] “You can apply the Rule-defined fix directly to the file using the `--autofix` flag.”
   • [claimed-docs] “You can apply the Rule-defined fix directly to the file using the --autofix flag.”
   • [claimed-docs] “Semgrep is integrated into CI environments by creating a **job** that is run by the CI provider. After a scan, findings are sent to Semgrep …”
   • [claimed-docs] “Scan and fix AI-generated code the moment it's written”
   • [claimed-docs] “Combine AI reasoning with rule-based analysis for detection, triage, and remediation”

   proof ↗⚑ flag

   Gecko Security✓fullclaimed8/10

   Docs describe a full remediation loop: suggested patches with 'Request fix' opening a gecko/* PR, ability to commit fixes directly to an existing PR's head branch, and automatic re-scan marking 'Fix verified' once resolved — this goes beyond read-only reporting into applied/automatic fixes. missing for 10: independent/hands-on confirmation that generated fixes actually work correctly (community evidence focuses on detection false positives, not fix application quality).

   • [claimed-docs] “Click Request fix to open a gecko/* pull request with the patch applied, or apply the patch yourself.”
   • [claimed-docs] “Gecko rechecks the finding and marks it Fix verified once the vulnerability is gone.”
   • [claimed-docs] “Commit to an existing PR: For findings from a PR scan, Gecko can commit the fix directly to the PR's head branch so the fix rides along with…”
   • [claimed-docs] “PR reviews & fixes: On pull requests, a security review summary and one-click fixes.”
   • [claimed-docs] “Click **Request fix** to open a `gecko/*` pull request with the patch applied, or apply the patch yourself.”
   • [claimed-docs] “When the fix merges, Gecko rechecks the finding and marks it **Fix verified** once the vulnerability is gone.”

   proof ↗⚑ flag

Custom detection

1. security-engineerDefine custom detection rules for my organization's internal token and secret formats

   weight 2 · round to Semgrep
   Semgrep~partialcommunity6/10

   Semgrep's generic rule-writing engine is well documented (custom rules, pattern matching, Editor for testing) and community evidence confirms security engineers write custom rules for org-specific patterns, which extends naturally to matching internal token/secret formats via pattern or regex-based rules. However, there is no explicit documentation showing how to author custom rules specifically within the Semgrep Secrets product (e.g., custom secret validators or format definitions) — the evidence only shows built-in secret detection (docs-8/9/10) and generic custom rule capability (docs-4/5/21), not a documented workflow for adding custom secret-format rules to Semgrep Secrets. Missing for 10: explicit docs/examples for authoring custom Semgrep Secrets rule types (validators, custom regex-based secret detectors), and independent confirmation of doing so for secrets specifically.

   • [claimed-docs] “you can write custom rules to determine what Semgrep detects in your repositories”
   • [claimed-docs] “you can write and test using the [Semgrep Editor]”
   • [claimed-docs] “you can write rules that: Automate code review comments. Identify secure coding violations. Scan configuration files.”
   • [claimed-docs] “Semgrep Secrets scans code to detect exposed API keys, passwords, and other credentials.”
   • [claimed-docs] “valid secrets are those that have been tested against a web service and confirmed to successfully grant resources or authentication. They ar…”
   • [community] “As a security consultant, I write custom Semgrep rules to match antipatterns unique to a codebase (e.g., controllers missing an authorizatio…”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko Security is a vulnerability-scanning tool focused on code security review (SAST-style findings, PR checks, fixes), and none of the evidence mentions custom secret/token pattern detection rules or any mechanism for defining organization-specific secret formats.

   ⚑ flag

Secret scanning

1. developerScan a repository and its full git history for hardcoded secrets and credentials

   weight 3 · round to Semgrep
   Semgrep~partialclaimed4/10

   Semgrep Secrets is documented to scan code for exposed API keys, passwords, and credentials and validate them (semgrep-docs-8, semgrep-docs-9, semgrep-docs-43), and pre-commit hooks aim to prevent secrets from entering git history (semgrep-docs-16/46), but this only prevents future leaks — none of the evidence shows Semgrep actually scanning the full historical git commit log/diffs for previously committed secrets (a capability tools like gitleaks/trufflehog specialize in). missing for 10: explicit documentation or feature for scanning past git commits/history (not just current working tree), evidence of retroactive history-wide secret detection, independent confirmation this works in practice.

   • [claimed-docs] “Semgrep Secrets scans code to detect exposed API keys, passwords, and other credentials.”
   • [claimed-docs] “valid secrets are those that have been tested against a web service and confirmed to successfully grant resources or authentication. They ar…”
   • [claimed-docs] “Semgrep saves security engineers time and effort by prioritizing valid leaked secrets and informs developer”
   • [claimed-docs] “The pre-commit framework can run semgrep when you commit changes. This is helpful in preventing secrets and security issues from leaking int…”
   • [claimed-docs] “The pre-commit framework can run `semgrep` when you commit changes. This is helpful in preventing secrets and security issues from leaking i…”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko's documented capability is AI-driven vulnerability discovery (source-to-sink call chains, CVSS scoring, PoCs, patches) for application code, not secret/credential scanning or git-history scanning. Missing for 10: any mention of secret/credential detection, hardcoded key scanning, or scanning git commit history for leaked secrets.

   • [claimed-docs] “Each finding includes a severity (CVSS 4.0), a confidence score, the full source-to-sink call chain, a proof of concept, and a suggested pat…”
   • [claimed-docs] “Repository wiki: An AI-written map of your app: architecture, routing, and security model.”
   • [claimed-docs] “Compiler-accurate (precise cross-file analysis) | TypeScript · JavaScript · Python · Go · Java · Scala · C# · Rust”
   • [probe] “PROBE llms.txt: HTTP 200 at https://gecko.security/llms.txt # Gecko Security > Gecko Security is an AI security engineer that finds exploit…”

   proof ↗⚑ flag

Licenses

1. devops-leadDetect the licenses of my dependencies and enforce a license policy in scans

   weight 1 · round drawn
   Semgrep—none0/10

   Semgrep Supply Chain evidence focuses on SCA vulnerability detection, reachability analysis, malicious dependency detection, and SBOM generation, but nothing in the pack mentions license detection or license-policy enforcement for dependencies.

   • [claimed-docs] “Generate reports and software bills of materials (SBOM) that provide a complete inventory of your open source components”
   • [claimed-docs] “Supply Chain's Dynamic Dependency Resolution provides a complete inventory of your project's dependencies using a combination of manifest pa…”
   • [claimed-docs] “Detect malicious dependencies associated with Supply Chain attacks”
   • [claimed-docs] “Semgrep Supply Chain is a software composition analysis (SCA) tool that detects security vulnerabilities in your codebase introduced by open…”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko is positioned as an AI vulnerability/exploit scanner (SAST-style call-chain analysis, patch suggestions) with no mention anywhere in docs, changelog, or API reference of dependency license detection, SBOM generation, or license policy enforcement.

   ⚑ flag

Sbom

1. security-engineerGenerate a software bill of materials (SPDX or CycloneDX) for my project or image

   weight 2 · round to Semgrep
   Semgrep~partialclaimed5/10

   Semgrep Supply Chain docs explicitly state it can 'generate reports and software bills of materials (SBOM)' as part of dependency inventory, confirming the capability exists, but this is gated behind the paid Supply Chain/AppSec Platform tier and no evidence specifies SPDX/CycloneDX format support, CLI flags, or container image SBOM generation. missing for 10: confirmation of specific SPDX/CycloneDX output formats, CLI command/flag documentation for SBOM export, container/image scanning SBOM support, and independent/hands-on corroboration of SBOM generation working as described.

   • [claimed-docs] “Generate reports and software bills of materials (SBOM) that provide a complete inventory of your open source components”
   • [claimed-docs] “Supply Chain's Dynamic Dependency Resolution provides a complete inventory of your project's dependencies using a combination of manifest pa…”
   • [claimed-docs] “Semgrep Supply Chain is a software composition analysis (SCA) tool that detects security vulnerabilities in your codebase introduced by open…”
   • [claimed-docs] “Support the enforcement of your business’ open source package licensing requirements”

   proof ↗⚑ flag

   Gecko Security—none0/10

   Gecko Security is an AI vulnerability scanner focused on code review, exploit findings, and auto-fix PRs; no evidence anywhere mentions generating an SBOM in SPDX or CycloneDX format for a project or container image.

   ⚑ flag

Ai triage

1. ai-native userUse AI to triage findings — explaining them, deduplicating, and flagging likely false positives

   weight 3 · round to Semgrep
   Semgrep~partialcommunity6/10

   Semgrep ships a native AI feature (Semgrep Assistant) explicitly described as 'contextual post-processing of findings...to further reduce noise by ~20%', and marketing docs claim combining 'AI reasoning with rule-based analysis for detection, triage, and remediation,' plus AI-powered detection for complex logic flaws — directly supporting AI-assisted false-positive reduction and triage. However, deduplication specifically is never mentioned, and the strongest independent evidence of AI triage in practice comes from a third-party tool built on Semgrep's SARIF output plus external LLMs (used at Microsoft) rather than the native Assistant, suggesting the first-party AI-triage feature is thin on independent corroboration. Missing for 10: explicit deduplication capability, deeper documentation of Assistant's explanation/false-positive-flagging behavior, independent hands-on evidence validating the native AI triage feature itself.

   • [github] “Contextual post-processing of findings with Semgrep Assistant (AI) to further reduce noise by ~20%”
   • [claimed-docs] “With AI-Powered Detection, you can automatically identify complex business logic flaws, such as IDORs and broken authorization.”
   • [claimed-docs] “Combine AI reasoning with rule-based analysis for detection, triage, and remediation”
   • [community] “Built a tool using Semgrep's SARIF output plus prompted LLMs to help prioritize triage; used at Microsoft for the past year to help surface …”

   proof ↗⚑ flag

   Gecko Security!disputedcontradicted4/10

   Docs describe confidence scores, severity, call chains, and PR-introduced vs pre-existing separation, plus an AI-tools API explicitly for triaging findings (gecko-security-docs-2, -9, -28), which maps to explaining/deduplicating/flagging false positives. However, hands-on community reports directly contradict the false-positive flagging claim: the tool assigned 100% confidence to two non-existent vulnerabilities and flagged an implausible 'RCE' in a CTF IRC bot script as high severity, while missing a real issue found by another scanner (gecko-security-comm-3, gecko-security-comm-5, gecko-security-comm-4). Missing for 10: independent validation that confidence scoring reliably suppresses false positives, and resolution of the reported inaccurate high-confidence findings.

   • [claimed-docs] “Each finding includes a severity (CVSS 4.0), a confidence score, the full source-to-sink call chain, a proof of concept, and a suggested pat…”
   • [claimed-docs] “Connect it to an AI tool and the tool can list your repositories and scans, pull full vulnerability evidence (proof of concept, call chain, …”
   • [claimed-docs] “Findings introduced by the PR are separated from pre-existing ones, so reviewers focus on what the change adds.”
   • [community] “Tool gave 100% confidence for two vulnerabilities (severity 9.4 and 6.5) that turned out not to be real.”
   • [community] “Uploaded a project (sydbox) to scan; took 10 minutes and found only one vulnerability, an 'RCE: IRC Message Command Execution Bypass' in an …”
   • [community] “Ran Gecko on a C project after Semgrep AI flagged a minor real issue; Gecko found nothing. After removing the fix and recreating the project…”

   proof ↗⚑ flag

Fp management

1. security-engineerBaseline or ignore existing and false-positive findings so only new issues alert

   weight 3 · round to Gecko Security
   Semgrep~partialclaimed4/10

   Docs mention a 'Triage findings' capability (alerts/tickets for tracking findings) and Semgrep Assistant's contextual post-processing to reduce noise by ~20%, implying some findings-management workflow, but there is no explicit evidence of baseline scanning (e.g., baseline-commit diffing), ignore-comments (nosemgrep), or a documented false-positive suppression/triage-state workflow. missing for 10: explicit baseline-commit/diff scanning documentation, .semgrepignore or inline nosemgrep suppression evidence, and a documented 'mark as false positive / ignored' triage state in the platform.

   • [claimed-docs] “Triage findings * Send alerts and notifications or create tickets to track findings identified by Semgrep Code”
   • [claimed-docs] “Triage findings Send alerts and notifications or create tickets to track findings identified by Semgrep Code”
   • [github] “Contextual post-processing of findings with Semgrep Assistant (AI) to further reduce noise by ~20%”

   proof ↗⚑ flag

   Gecko Security~partialcommunity6/10

   Docs show PR scans separate pre-existing findings from new ones so reviewers focus on what changed (docs-28), and the API supports 'triage findings' (docs-10), implying some noise-reduction workflow. However, there's no explicit documentation of a false-positive marking/ignore mechanism or a persistent baseline snapshot feature, and community reports flag inaccurate high-confidence findings (comm-3) without confirming a suppression workflow exists. missing for 10: explicit 'mark as false positive'/dismiss UI, documented baseline snapshotting across scans, independent confirmation that dismissed findings stay suppressed on rescans.

   • [claimed-docs] “Findings introduced by the PR are separated from pre-existing ones, so reviewers focus on what the change adds.”
   • [claimed-docs] “Gecko's v1 API lets you trigger scans, read scan, repository, and vulnerability data, triage findings, manage scan schedules, rotate your Gi…”
   • [claimed-docs] “create a ticket in Jira, Linear, ClickUp, or Shortcut”
   • [community] “Tool gave 100% confidence for two vulnerabilities (severity 9.4 and 6.5) that turned out not to be real.”

   proof ↗⚑ flag

2. developerFilter and gate on findings by severity or confidence so low-value noise does not block builds

   weight 2 · round to Gecko Security
   Semgrep—none0/10

   The evidence pack describes general triage/noise features (AI post-processing to reduce noise, ticketing/alerts, reachability analysis) but never mentions severity levels, confidence scores, or a mechanism to filter/gate CI builds based on them. Missing for 10: explicit severity/confidence filtering flags (e.g., --severity), CI gating configuration based on severity/confidence thresholds, and documentation tying these to build pass/fail behavior.

   ⚑ flag

   Gecko Security~partialclaimed6/10

   Docs confirm a 'fail-on severity' threshold for PR checks (e.g., fail on High and above) and that findings carry both severity and confidence scores, supporting build-gating by severity. However, there's no documented mechanism to gate/filter specifically by confidence score, only severity threshold is described as configurable. Missing for 10: explicit confidence-based filtering/gating controls, and independent verification that severity gating reliably suppresses low-value noise.

   • [claimed-docs] “Each finding includes a severity (CVSS 4.0), a confidence score, the full source-to-sink call chain, a proof of concept, and a suggested pat…”
   • [claimed-docs] “**Fail-on severity**: the severity threshold that fails the check (for example, fail on High and above).”
   • [claimed-docs] “Findings introduced by the PR are separated from pre-existing ones, so reviewers focus on what the change adds.”

   proof ↗⚑ flag