Pilot vs Bench

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to Pilot
   Pilot~partialprobed4/10

   A probe confirms Pilot serves an llms.txt file at HTTP 200, so an agent can indeed be pointed at it — but the content shown is minimal (just entity/category metadata, no structured docs or API references), and other agent-doc formats (docs-md, openapi) return 404. Missing for 10: substantive llms.txt content beyond basic metadata, markdown docs mirror, and any first-party statement about agent-oriented documentation strategy.

   • [probe] “PROBE llms.txt: HTTP 200 at https://pilot.com/llms.txt # Pilot [Entity: Pilot] [Category: Professional Services] [Subcategory: Acco…”
   • [probe] “PROBE docs-md: HTTP 404 at https://pilot.com/faq.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a bookkeeping service, not a dev tool/platform, but the axis still applies since any web product could publish agent-oriented docs; probes explicitly show llms.txt returns 404, docs-as-markdown 404, and no OpenAPI spec found. There is no evidence of any agent-oriented documentation surface.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round drawn
   Pilot—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Bench—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

3. ai-native userConnect an agent via an official MCP server

   weight 3 · round drawn
   Pilot—none0/10

   Pilot is a bookkeeping/accounting service with no evidence of any MCP server offering; docs mention integrations (Bill.com, Stripe, Gusto) but nothing about AI agent connectivity via MCP, and API/openapi probes returned 404s.

   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”
   • [probe] “PROBE docs-md: HTTP 404 at https://pilot.com/faq.md”
   • [claimed-docs] “We integrate with hundreds of tools like Bill.com, Stripe, and Gusto.”

   proof ↗⚑ flag

   Bench—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

4. ai-native userDrive the product through a documented public API

   weight 3 · round drawn
   Pilot—none0/10

   No evidence of a public API for Pilot; OpenAPI/swagger probes all returned 404 and no docs mention an API for driving the product programmatically. Pilot is a human-service accounting/bookkeeping product with integrations, not an API-driven platform.

   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”
   • [probe] “PROBE docs-md: HTTP 404 at https://pilot.com/faq.md”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a bookkeeping/accounting service with no evidence of a documented public API; direct probes for llms.txt, docs.md, and OpenAPI/swagger endpoints all returned 404, and no docs mention an API for AI-native integration.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

5. ai-native userIssue scoped/least-privilege API credentials for an agent

   weight 2 · round drawn
   Pilot—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Bench—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

6. ai-native userBuild against official SDKs

   weight 2 · round drawn
   Pilot—none0/10

   Pilot is a bookkeeping/accounting service with no evidence of any official SDKs; probes for API/OpenAPI specs returned 404s, and no docs mention developer SDKs.

   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”
   • [probe] “PROBE docs-md: HTTP 404 at https://pilot.com/faq.md”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a bookkeeping/tax service with no evidence of any public SDK, API, or developer library; probes for llms.txt, docs-md, and OpenAPI specs all returned 404. Missing for 10: any official SDK, API documentation, or developer integration surface.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

7. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   Pilot—none0/10

   No evidence of any webhook subscription capability or event API; Pilot is an accounting/bookkeeping service and probes show no OpenAPI or developer API surface exists.

   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”
   • [probe] “PROBE docs-md: HTTP 404 at https://pilot.com/faq.md”

   proof ↗⚑ flag

   Bench—none0/10

   No evidence of any webhook subscription mechanism, API, or event system; probes for llms.txt, docs-md, and OpenAPI specs all return 404, and Bench is a human-plus-software bookkeeping service with no developer/webhook offering documented.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Pilot
   Pilot~partialclaimed4/10

   Pilot's marketing mentions 'Instant AI answers about past transactions and current cash flow, plus run scenarios to plan ahead' (pilot-docs-7), which points to some AI-generated insight capability inside the product, and scenario/cash-flow modeling features (pilot-docs-8, pilot-docs-21) suggest similar analytical output, though these are framed as human CFO-delivered rather than explicitly AI-driven. There is no independent/hands-on corroboration of the AI feature actually working, and community reviews focus on bookkeeping accuracy/time savings rather than AI insights. Missing for 10: detailed documentation of the AI feature's scope/accuracy, independent user testimony specifically about the AI insights, and clarity on whether insights are AI-generated vs. human-analyst-generated.

   • [claimed-docs] “Instant AI answers about past transactions and current cash flow, plus run scenarios to plan ahead”
   • [claimed-docs] “Model cash flow scenarios Know if you can afford new hires well before you make the offers.”
   • [claimed-docs] “Know if you can afford new hires well before you make the offers.”
   • [claimed-docs] “Track company expenses in real time Find the $2,000/month you didn't know you were wasting.”
   • [claimed-docs] “Find the $2,000/month you didn't know you were wasting.”

   proof ↗⚑ flag

   Bench—none0/10

   Bench's evidence describes automation (data entry, categorization) and human bookkeeper support, but nothing describes AI-generated insights, recommendations, or suggestions surfaced to the user from their data — 'smart automation' refers to categorization mechanics, not generative analysis or advisory insights. Missing for 10: any mention of AI-generated commentary, anomaly detection, forecasting, or proactive suggestions derived from financial data.

   • [claimed-docs] “Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time”
   • [claimed-docs] “Accurate transaction categorization, powered by smart automation with instant guidance”
   • [claimed-docs] “"Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time", with "Accurate transaction c…”
   • [claimed-docs] “Real-time reporting. Access or download your updated income statement or balance sheet at all times”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round drawn
   Pilot—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Bench—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round to Pilot
   Pilot~partialclaimed3/10

   Pilot's pricing page mentions an AI feature offering 'Instant AI answers about past transactions and current cash flow, plus run scenarios to plan ahead,' which is a thin, single-mention hint of an in-product AI assistant capable of some task-like actions (scenario modeling). There's no further documentation on scope, autonomy, or how tasks are delegated to it, and community evidence focuses on human bookkeepers/CFOs rather than the AI assistant. Missing for 10: detailed documentation of the AI assistant's task-delegation capabilities, independent/hands-on corroboration of it actually performing delegated tasks, and clarity on its scope beyond Q&A.

   • [claimed-docs] “Instant AI answers about past transactions and current cash flow, plus run scenarios to plan ahead”

   proof ↗⚑ flag

   Bench—none0/10

   Bench offers human bookkeepers plus automation for data entry and categorization, but there is no evidence of a built-in AI assistant that a user can delegate tasks to conversationally — communication is described as messaging with a human team, not an AI agent. Missing for 10: any mention of an AI assistant/chat feature, agentic task delegation, or AI-driven interaction surface within the product.

   • [claimed-docs] “Effortless messaging with your Bench team for unlimited support or advice, anytime, anywhere”
   • [claimed-docs] “Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time”
   • [claimed-docs] “Accurate transaction categorization, powered by smart automation with instant guidance”
   • [claimed-docs] “Bench is a human-plus-software service, now under new ownership: "Bench simplifies your small business accounting by combining intuitive sof…”

   proof ↗⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round to Pilot
   Pilot~partialclaimed3/10

   Pilot's marketing mentions 'Instant AI answers about past transactions and current cash flow, plus run scenarios to plan ahead,' implying some natural-language Q&A interface, but there's no evidence of broader natural-language command execution (e.g., initiating actions, workflows) or any API/CLI to support it. missing for 10: evidence of NL commands beyond Q&A, independent/hands-on corroboration, technical documentation of the AI interface, API or programmatic access.

   • [claimed-docs] “Instant AI answers about past transactions and current cash flow, plus run scenarios to plan ahead”

   proof ↗⚑ flag

   Bench—none0/10

   Bench offers human-support messaging and a client dashboard, but there is no evidence of a natural-language command interface (chat-driven actions, NL query of financial data, etc.); probes for llms.txt, docs-md, and OpenAPI all 404, indicating no AI-native interface layer. missing for 10: any natural-language command/chat-to-action capability, NL query support, or API/agent interface evidence.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”
   • [claimed-docs] “Effortless messaging with your Bench team for unlimited support or advice, anytime, anywhere”

   proof ↗⚑ flag

Api quality

1. ai-native userExplore an interactive API reference with runnable examples

   weight 2 · round drawn
   Pilot—none0/10

   Pilot is an accounting/bookkeeping service with no evidence of any developer API reference or runnable examples; probes for OpenAPI/docs-md all returned 404, and no evidence pack items mention an API reference.

   • [probe] “PROBE docs-md: HTTP 404 at https://pilot.com/faq.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a bookkeeping/accounting service with no API, no interactive API reference, and no runnable examples; probes for llms.txt, docs.md, and OpenAPI specs all returned 404. missing for 10: any API reference, interactive docs, runnable code examples.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

2. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   Pilot—none0/10

   Probe explicitly checked common OpenAPI/swagger endpoints and all returned 404, and no documentation mentions a machine-readable API spec. Missing for 10: any published OpenAPI/Swagger spec, API documentation, or developer portal.

   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”

   proof ↗⚑ flag

   Bench—none0/10

   Probes explicitly show llms.txt, docs.md, and all openapi/swagger candidate paths return 404, and no docs mention any API spec; Bench is a bookkeeping service with no evidence of a machine-readable API for AI agents.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

3. ai-native userTest against a sandbox environment without touching production data

   weight 1 · round drawn
   Pilot—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Bench—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

4. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   Pilot—none0/10

   Pilot is a bookkeeping/accounting service with no evidence of any public API, versioning scheme, or deprecation policy; probes for OpenAPI specs all returned 404s. missing for 10: any public API documentation, versioning scheme, deprecation policy, or developer-facing API references.

   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”
   • [probe] “PROBE docs-md: HTTP 404 at https://pilot.com/faq.md”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a bookkeeping/tax service with no evidence of a public API at all — probes for OpenAPI specs, docs, and llms.txt all returned 404. No mention of versioning or a deprecation policy anywhere in the evidence pack. Missing for 10: any public API, version numbers, changelog, or deprecation policy documentation.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round drawn
   Pilot—none0/10

   Pilot is a bookkeeping/accounting service; no evidence describes any bulk-operation or batch-processing capability across items (transactions, invoices, etc.) for AI-native automation. Missing for 10: any mention of bulk edit/import/export tools, batch API operations, or automation-scale item processing.

   ⚑ flag

   Bench—none0/10

   Bench is a human-plus-software bookkeeping service focused on categorization, reporting, and tax filing for individual accounts; no evidence describes any bulk/batch operation capability across many items (e.g., bulk transaction edits, bulk uploads, batch API operations) for AI-native or power users. Probes also show no API/docs surface for programmatic bulk actions.

   • [claimed-docs] “Accurate transaction categorization, powered by smart automation with instant guidance”
   • [claimed-docs] “"Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time", with "Accurate transaction c…”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

2. ai-native userDefine rules that trigger actions automatically on events

   weight 3 · round drawn
   Pilot—none0/10

   Pilot is a bookkeeping/accounting service with dedicated human experts and integrations, but there is no evidence of a rules engine or automation-trigger system letting users define event-based automated actions; features described are AI Q&A and integrations, not user-defined trigger/action rules.

   ⚑ flag

   Bench—none0/10

   Bench is a bookkeeping/accounting service with backend automation for categorization and reminders, but there is no evidence of a user-facing rule engine where AI-native users can define custom triggers/actions on events; the automation described is fixed internal workflow, not configurable rules.

   ⚑ flag

3. ai-native userSchedule recurring jobs or workflows

   weight 2 · round drawn
   Pilot—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

   Bench—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

Ai reconciliation

1. ai-native userHave an agent reconcile a month end-to-end, matching transactions, flagging exceptions, and preparing the close for human review

   weight 3 · round to Pilot
   Pilot~partialcommunity3/10

   Pilot's evidence shows human bookkeepers finalize monthly books with integration-driven transaction classification (pilot-comm-1, pilot-docs-18, pilot-docs-26), which is directionally toward automated reconciliation, but there is no evidence of an autonomous agent matching transactions, flagging exceptions, and preparing a close package end-to-end without human bookkeeper involvement. Missing for 10: explicit agentic exception-flagging workflow, evidence of AI (not human staff) doing the matching/reconciliation, and documentation of an automated close-prep output for review.

   • [community] “I'm a Pilot customer... Pilot basically solved that problem. I just do my normal business and their integrations with my banks, expensify, c…”
   • [claimed-docs] “Communicate directly with your team through our user-friendly portal, where you can ask questions and provide feedback on any outstanding it…”
   • [claimed-docs] “Plus stay up to date with real time account data directly in the portal.”
   • [claimed-docs] “stay up to date with real time account data directly in the portal”

   proof ↗⚑ flag

   Bench—none0/10

   Bench's evidence describes automated data entry/categorization plus human bookkeepers preparing reports, not an autonomous agent that reconciles a month end-to-end, flags exceptions, and stages a close for human review. No documentation of exception-flagging logic, agent-driven reconciliation workflow, or AI-native interface exists in the pack. Missing for 10: evidence of automated exception detection, an agentic reconciliation workflow, and any API/agent interface for month-end close.

   • [claimed-docs] “"Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time", with "Accurate transaction c…”
   • [claimed-docs] “Accurate transaction categorization, powered by smart automation with instant guidance”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”
   • [claimed-docs] “Bench is a human-plus-software service, now under new ownership: "Bench simplifies your small business accounting by combining intuitive sof…”

   proof ↗⚑ flag

Bank feeds

1. founderConnect bank and credit-card accounts so transactions import automatically every day

   weight 3 · round drawn
   Pilot✓fullcommunity7/10

   Pilot's docs and a first-hand customer testimonial confirm bank/credit-card/expense integrations automatically feed transaction data for classification, with real-time account data in the portal, eliminating manual founder bookkeeping work. missing for 10: no explicit list of supported banks, no detail on daily-sync frequency or setup flow, and no independent technical documentation beyond marketing/community claims.

   • [community] “I'm a Pilot customer... Pilot basically solved that problem. I just do my normal business and their integrations with my banks, expensify, c…”
   • [claimed-docs] “We integrate with hundreds of tools like Bill.com, Stripe, and Gusto.”
   • [claimed-docs] “stay up to date with real time account data directly in the portal”
   • [claimed-docs] “Plus stay up to date with real time account data directly in the portal.”

   proof ↗⚑ flag

   Bench✓fullclaimed7/10

   Bench's docs explicitly describe connecting all financial accounts to automate daily data entry and transaction categorization, which directly matches the story of automatic daily import via connected bank/credit card accounts. Missing for 10: no independent/hands-on corroboration of connection reliability, no detail on supported bank list or sync frequency specifics beyond marketing copy.

   • [claimed-docs] “Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time”
   • [claimed-docs] “Accurate transaction categorization, powered by smart automation with instant guidance”
   • [claimed-docs] “"Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time", with "Accurate transaction c…”

   proof ↗⚑ flag

Categorization

1. founderGet automatic categorization suggestions for imported transactions that learn from my corrections

   weight 2 · round to Bench
   Pilot—none0/10

   Evidence shows Pilot's transaction classification is handled by integrations plus human bookkeepers/accountants (pilot-comm-1, pilot-docs-17), not by an AI categorization engine that learns from founder corrections. No documentation describes a self-serve ML suggestion-and-correction loop. Missing for 10: any mention of an AI/ML categorization suggestion feature, evidence of a correction/feedback loop, or founder-facing UI for adjusting categorizations.

   • [community] “I'm a Pilot customer... Pilot basically solved that problem. I just do my normal business and their integrations with my banks, expensify, c…”
   • [claimed-docs] “We integrate with hundreds of tools like Bill.com, Stripe, and Gusto.”
   • [claimed-docs] “stay up to date with real time account data directly in the portal”

   proof ↗⚑ flag

   Bench~partialclaimed4/10

   Bench documents automated, 'smart automation' transaction categorization with 'instant guidance' (bench-docs-4, bench-supp-automation), which covers the core categorization-suggestion capability, but since Bench is a human-plus-software bookkeeping service, human bookkeepers—not the software—typically apply corrections, and there is no evidence describing a feedback loop where the system learns from founder corrections. Missing for 10: any documentation of a correction-learning mechanism, founder-facing feedback UI, or improvement in suggestion accuracy over time.

   • [claimed-docs] “Accurate transaction categorization, powered by smart automation with instant guidance”
   • [claimed-docs] “"Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time", with "Accurate transaction c…”
   • [claimed-docs] “Bench is a human-plus-software service, now under new ownership: "Bench simplifies your small business accounting by combining intuitive sof…”

   proof ↗⚑ flag

Reconciliation

1. accountantReconcile each bank account against the ledger with matched, unmatched, and duplicate transactions surfaced for review

   weight 3 · round drawn
   Pilot—none0/10

   Pilot's evidence describes bank/tool integrations and automated transaction classification (pilot-comm-1, pilot-docs-17) but never describes a reconciliation workflow that surfaces matched, unmatched, and duplicate transactions for accountant review. Missing for 10: any documentation or user report of a reconciliation screen or matched/unmatched/duplicate transaction handling.

   • [community] “I'm a Pilot customer... Pilot basically solved that problem. I just do my normal business and their integrations with my banks, expensify, c…”
   • [claimed-docs] “We integrate with hundreds of tools like Bill.com, Stripe, and Gusto.”
   • [claimed-docs] “stay up to date with real time account data directly in the portal”

   proof ↗⚑ flag

   Bench—none0/10

   Bench's evidence covers account connection, automated categorization, and reporting, but no citation describes a reconciliation workflow that surfaces matched, unmatched, and duplicate transactions for accountant review. missing for 10: explicit reconciliation UI/workflow, matched/unmatched transaction views, duplicate detection and review surface.

   • [claimed-docs] “Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time”
   • [claimed-docs] “Accurate transaction categorization, powered by smart automation with instant guidance”
   • [claimed-docs] “"Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time", with "Accurate transaction c…”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”

   proof ↗⚑ flag

Bill management

1. accountantCapture vendor bills, route them for approval, and schedule payments from one workflow

   weight 2 · round to Pilot
   Pilot~partialclaimed3/10

   Pilot's pricing page lists 'Full accounts payable' as a service line item and mentions integration with Bill.com, implying vendor bill handling, but there is no documented detail on bill capture, approval routing, or payment scheduling mechanics within a single workflow — AP appears to be a done-for-you service rather than a described self-serve workflow. Missing for 10: bill capture/OCR details, approval routing configuration, payment scheduling functionality, and any hands-on/independent confirmation of the AP workflow.

   • [claimed-docs] “Full accounts receivable Full accounts payable Payroll administration”
   • [claimed-docs] “Full accounts receivable Full accounts payable”
   • [claimed-docs] “We integrate with hundreds of tools like Bill.com, Stripe, and Gusto.”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a bookkeeping/tax service focused on categorization, reporting, and human bookkeeper support; no evidence of bill capture (AP inbox/OCR), approval routing, or scheduled bill payment workflows anywhere in the docs, pricing, or banking pages.

   • [claimed-docs] “Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time”
   • [claimed-docs] “Accurate transaction categorization, powered by smart automation with instant guidance”
   • [claimed-docs] “"Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time", with "Accurate transaction c…”
   • [claimed-docs] “"Banking by Mainstreet — Get dedicated business accounts, debit cards, and automated financial management tools that integrate seamlessly wi…”
   • [claimed-docs] “Published service pricing: Bookkeeping Grow from $199/month ("An accessible, lighter-touch plan—built for lean businesses seeking financial …”

   proof ↗⚑ flag

Payables

1. accountantSee accounts payable aging and vendor balances at any point in time

   weight 2 · round to Pilot
   Pilot~partialclaimed4/10

   Pilot advertises 'Full accounts payable' as a service feature and real-time account data access via its portal, implying some visibility into AP status, but there is no specific mention of AP aging reports or vendor balance tracking at a point in time. missing for 10: explicit AP aging report feature, vendor balance detail/history, point-in-time reporting capability, screenshots or hands-on evidence of AP aging views.

   • [claimed-docs] “Full accounts receivable Full accounts payable Payroll administration”
   • [claimed-docs] “Full accounts receivable Full accounts payable”
   • [claimed-docs] “stay up to date with real time account data directly in the portal”
   • [claimed-docs] “Plus stay up to date with real time account data directly in the portal.”

   proof ↗⚑ flag

   Bench—none0/10

   Bench's evidence covers bookkeeping automation, categorization, income statement/balance sheet reporting, banking and tax prep, but nowhere mentions accounts payable aging, vendor balances, or bill-tracking reports at a point in time. Bench appears to focus on income statement/balance sheet reporting rather than AP-specific aging reports.

   • [claimed-docs] “Real-time reporting. Access or download your updated income statement or balance sheet at all times”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”
   • [claimed-docs] “Accurate transaction categorization, powered by smart automation with instant guidance”

   proof ↗⚑ flag

Close process

1. accountantLock closed periods so prior-period numbers cannot change without an explicit reopening

   weight 2 · round drawn
   Pilot—none0/10

   No evidence in the pack mentions period locking, closing periods, or preventing retroactive changes to prior-period financials; the evidence only covers bookkeeping, tax, CFO, and integration features. Missing for 10: any mention of period-lock/close controls, reopening workflow, or audit trail preventing prior-period edits.

   ⚑ flag

   Bench—none0/10

   No evidence describes period locking, close controls, or reopening workflows; Bench is presented as a human-plus-software bookkeeping service with reporting/downloads, not a ledger system with period-lock controls exposed to the accountant/customer. missing for 10: any mention of period closing, locking prior periods, or explicit reopening controls.

   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”
   • [claimed-docs] “Real-time reporting. Access or download your updated income statement or balance sheet at all times”

   proof ↗⚑ flag

2. accountantRun a month-end close against a checklist that tracks reconciliations, accruals, and review sign-offs

   weight 3 · round to Pilot
   Pilot~partialclaimed3/10

   Pilot's monthly bookkeeping service closes books and lets customers communicate on 'outstanding items needed to finalize your books every month' via a portal, implying some close-tracking process, but there is no evidence of an actual checklist artifact tracking reconciliations, accruals, and review sign-offs as distinct trackable steps. missing for 10: explicit reconciliation checklist UI, accrual tracking, and sign-off/approval workflow evidence.

   • [claimed-docs] “Communicate directly with your team through our user-friendly portal, where you can ask questions and provide feedback on any outstanding it…”
   • [claimed-docs] “stay up to date with real time account data directly in the portal”
   • [claimed-docs] “Plus stay up to date with real time account data directly in the portal.”

   proof ↗⚑ flag

   Bench—none0/10

   Bench markets itself as a done-for-you bookkeeping/tax service with messaging, document upload, categorization, and real-time reports, but nothing in the evidence describes a month-end close checklist tracking reconciliations, accruals, or review sign-offs — only generic 'timely reminders' are mentioned, which is not the same as a structured close workflow.

   • [claimed-docs] “Get timely reminders to stay on top of your financial tasks and deadlines”
   • [claimed-docs] “"Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time", with "Accurate transaction c…”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”

   proof ↗⚑ flag

Collaboration

1. founderInvite my external accountant or bookkeeper with role-scoped access to review and adjust the books

   weight 3 · round drawn
   Pilot—none0/10

   Pilot's evidence describes its own bookkeeping team communicating with the founder via a portal (docs-18, docs-26) and lets customers keep QuickBooks data so any tax preparer can access it (docs-24), but there is no evidence of a feature to invite an external accountant/bookkeeper into Pilot's own system with role-scoped permissions to review or adjust books. missing for 10: any documentation of user/role invitation, permission scoping, or external collaborator access controls within Pilot's platform.

   • [claimed-docs] “Communicate directly with your team through our user-friendly portal, where you can ask questions and provide feedback on any outstanding it…”
   • [claimed-docs] “Virtually every tax preparer in the US knows how to work with QuickBooks. You’ll have the freedom to choose any tax preparer you want.”
   • [claimed-docs] “Plus stay up to date with real time account data directly in the portal.”

   proof ↗⚑ flag

   Bench—none0/10

   Bench's evidence describes a done-for-you bookkeeping service where Bench's own staff manage the books and communicate with the founder, with no mention of inviting an external accountant/bookkeeper as a role-scoped collaborator with adjustable permissions inside a shared ledger. The 'Add a bookkeeper to your team' claim refers to Bench's own hire-a-bookkeeper offering working inside the founder's existing systems, not the founder granting scoped access to their own outside accountant.

   • [claimed-docs] “Add a bookkeeper to your team who works with your existing systems”
   • [claimed-docs] “Published service pricing: Bookkeeping Grow from $199/month ("An accessible, lighter-touch plan—built for lean businesses seeking financial …”
   • [claimed-docs] “Bench is a human-plus-software service, now under new ownership: "Bench simplifies your small business accounting by combining intuitive sof…”

   proof ↗⚑ flag

Data access

1. developerPull the full general ledger and trial balance through a documented API in open formats

   weight 3 · round drawn
   Pilot—none0/10

   No evidence of a documented API for pulling general ledger or trial balance data; probes for OpenAPI/docs endpoints returned 404s and no API documentation is mentioned anywhere in the evidence pack.

   • [probe] “PROBE docs-md: HTTP 404 at https://pilot.com/faq.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”

   proof ↗⚑ flag

   Bench—none0/10

   Bench only offers report viewing/downloads (income statement, balance sheet) via its web app; there is no documented API, no OpenAPI/Swagger spec, and probes for llms.txt, markdown docs, and OpenAPI endpoints all returned 404s, so developers have no documented way to pull the GL or trial balance in open formats.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”

   proof ↗⚑ flag

Integrations

1. founderConnect my banking, payments, and payroll systems so their activity flows into the ledger automatically

   weight 2 · round to Pilot
   Pilot✓fullcommunity8/10

   Pilot explicitly integrates with banking, payments, and payroll systems (e.g. Bill.com, Stripe, Gusto, 10k+ integrations) to feed real-time transaction data into the ledger, and community testimonials confirm bank/card/payment integrations auto-handle transaction classification with minimal founder effort. missing for 10: no explicit first-party detail on the breadth/setup process of bank feed connections beyond named partners, no independent audit of integration reliability at scale.

   • [claimed-docs] “We integrate with hundreds of tools like Bill.com, Stripe, and Gusto.”
   • [claimed-docs] “10k+ Integrations”
   • [claimed-docs] “If you use Gusto, we’ve partnered with Gusto to automatically apply the credit, saving you time.”
   • [community] “I'm a Pilot customer... Pilot basically solved that problem. I just do my normal business and their integrations with my banks, expensify, c…”
   • [claimed-docs] “stay up to date with real time account data directly in the portal”

   proof ↗⚑ flag

   Bench~partialclaimed6/10

   Bench's docs claim customers can 'connect all your financial accounts to automate data entry' and offers integrated banking (Banking by Bench/Mainstreet), which supports bank/payment account linking flowing into bookkeeping. However, payroll integration is never explicitly mentioned, and the ledger itself is not customer-facing/editable — it's a human-plus-software service producing reports rather than a live self-serve ledger. missing for 10: explicit payroll system integration, technical detail on connection breadth (which banks/payment processors), independent/hands-on confirmation of automated data flow reliability.

   • [claimed-docs] “Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time”
   • [claimed-docs] “"Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time", with "Accurate transaction c…”
   • [claimed-docs] “Get dedicated business accounts, debit cards, and automated financial management tools that integrate seamlessly with your bookkeeping opera…”
   • [claimed-docs] “"Banking by Mainstreet — Get dedicated business accounts, debit cards, and automated financial management tools that integrate seamlessly wi…”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”

   proof ↗⚑ flag

Migration

1. founderMigrate historical books from another accounting system without losing transaction detail

   weight 1 · round drawn
   Pilot~partialclaimed5/10

   Pilot markets that switching providers is supported and mentions keeping existing QuickBooks account and historical data, implying migration support, but there's no detail on how historical transaction-level detail is preserved or migrated from other systems (e.g., Xero, Wave, spreadsheets) beyond QuickBooks. missing for 10: documented migration process, evidence of no data/detail loss, support for non-QuickBooks source systems, independent confirmation of successful migrations.

   • [claimed-docs] “Whether you’re starting fresh or switching providers, your dedicated accounting expert is with you from day one handling onboarding, underst…”
   • [claimed-docs] “Keep your existing QuickBooks account and historical data”
   • [claimed-docs] “Whether you're starting fresh or switching providers, your dedicated accounting expert is with you from day one handling onboarding, underst…”
   • [claimed-docs] “Whether you’re starting fresh or switching providers, your dedicated accounting expert is with you from day one handling onboarding”

   proof ↗⚑ flag

   Bench~partialclaimed5/10

   Bench explicitly offers 'Catch Up Bookkeeping' for businesses behind on their books, implying it can reconstruct/backfill historical transaction data, and its human-bookkeeper model suggests manual handling of past records. However, there is no explicit documentation of migrating data from another accounting system (e.g., QuickBooks, Xero) with transaction-level fidelity preserved, no import tooling, and no evidence addressing loss/retention of transaction detail during migration. Missing for 10: explicit migration/import process from other platforms, evidence of transaction-level data preservation, any tooling or API for historical data transfer.

   • [claimed-docs] “Catch up bookkeeping services for small businesses, no matter how far behind they are”
   • [claimed-docs] “Add a bookkeeper to your team who works with your existing systems”
   • [claimed-docs] “The buying motion is demo-led for service fit: "Book a demo with our friendly team of experts. Not sure where to start or which accounting s…”

   proof ↗⚑ flag

Invoicing

1. founderCreate and send invoices with online payment options and see when they are viewed and paid

   weight 3 · round drawn
   Pilot—none0/10

   Evidence only shows Pilot offers 'full accounts receivable' as a bookkeeping service item, but there is no mention of self-serve invoice creation, online payment collection, or view/paid tracking features for founders—Pilot is a done-for-you bookkeeping service, not an invoicing tool. Missing for 10: invoice creation UI, online payment processing, read-receipt/paid-status tracking evidence.

   • [claimed-docs] “Full accounts receivable Full accounts payable Payroll administration”
   • [claimed-docs] “Full accounts receivable Full accounts payable”

   proof ↗⚑ flag

   Bench—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

2. founderSet up recurring invoices and automatic payment reminders for overdue customers

   weight 2 · round drawn
   Pilot—none0/10

   Evidence mentions 'Full accounts receivable' as a service but nothing about recurring invoicing or automated overdue payment reminders specifically. Pilot is positioned as a bookkeeping/accounting service, not an invoicing tool with these automation features documented anywhere in the pack.

   • [claimed-docs] “Full accounts receivable Full accounts payable Payroll administration”
   • [claimed-docs] “Full accounts receivable Full accounts payable”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a bookkeeping/tax service focused on categorization, reporting, and tax filing; there is no mention of invoicing, accounts receivable, recurring invoices, or customer payment reminders anywhere in the evidence pack.

   ⚑ flag

Receivables

1. accountantTrack accounts receivable with aging reports and per-customer balances

   weight 2 · round to Pilot
   Pilot~partialclaimed4/10

   Pilot's pricing page lists 'Full accounts receivable' as a service offering, confirming AR management is part of the product, but there is no evidence of specific features like aging reports or per-customer balance tracking/reporting in the evidence pack. missing for 10: documentation of aging reports, per-customer balance views, AR-specific dashboards or reports, and independent/hands-on confirmation of these features.

   • [claimed-docs] “Full accounts receivable Full accounts payable Payroll administration”
   • [claimed-docs] “Full accounts receivable Full accounts payable”

   proof ↗⚑ flag

   Bench—none0/10

   Bench's evidence covers reporting (income statement, balance sheet), categorization, and banking, but there is no mention of accounts receivable aging reports or per-customer balance tracking anywhere in the docs or probes. Missing for 10: any AR aging report feature, per-customer balance tracking, invoicing/AR module documentation.

   • [claimed-docs] “Real-time reporting. Access or download your updated income statement or balance sheet at all times”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”

   proof ↗⚑ flag

Ai ledger

1. ai-native userHave an agent draft and post journal entries via the API and leave them in a review queue for my accountant

   weight 3 · round drawn
   Pilot—none0/10

   No evidence of any API, agent integration, or journal entry review queue functionality; Pilot is a human-service bookkeeping/accounting company with no documented API (openapi probes 404). Missing for 10: any API existence, journal entry draft/post capability, review queue workflow, agent-facing endpoints.

   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”
   • [probe] “PROBE docs-md: HTTP 404 at https://pilot.com/faq.md”
   • [probe] “PROBE llms.txt: HTTP 200 at https://pilot.com/llms.txt # Pilot [Entity: Pilot] [Category: Professional Services] [Subcategory: Acco…”

   proof ↗⚑ flag

   Bench—none0/10

   The axis applies to this product kind (peer products hold positive or none verdicts on this story), so lack of evidence for an applicable capability is "none", never "na". (na/none harmonized at arena bring-up — see pipeline/scripts/na-harmonize.ts.)

   ⚑ flag

Chart of accounts

1. accountantCustomize the chart of accounts with account types, sub-accounts, and class or tag dimensions

   weight 3 · round drawn
   Pilot—none0/10

   Pilot is a done-for-you bookkeeping/accounting service, so ledger customization is a fair question, but the evidence pack only mentions generic 'custom workflows and reporting' with no mention of chart of accounts structure, account types, sub-accounts, or class/tag dimensions. Missing for 10: any documentation of COA setup, account type hierarchy, sub-account creation, or class/tag dimension tagging controlled by the accountant.

   • [claimed-docs] “Custom workflows and reporting”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a done-for-you bookkeeping service where a human bookkeeper manages the books; the evidence pack shows no self-serve chart-of-accounts customization, account types, sub-accounts, or class/tag dimension configuration exposed to the accountant/customer. The client-facing surface is reports and downloads, not ledger/CoA editing. Missing for 10: any mention of CoA management, account type/sub-account creation, or class/tag dimension tagging capability.

   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”
   • [claimed-docs] “Real-time reporting. Access or download your updated income statement or balance sheet at all times”
   • [claimed-docs] “Bench is a human-plus-software service, now under new ownership: "Bench simplifies your small business accounting by combining intuitive sof…”

   proof ↗⚑ flag

Journal entries

1. accountantPost manual journal entries with balanced debits, credits, and supporting memos

   weight 3 · round drawn
   Pilot—none0/10

   Pilot's evidence describes a done-for-you bookkeeping/CFO service built on QuickBooks with dedicated experts handling books, but there is no mention of any interface for accountants to post manual journal entries with debits/credits and memos. Missing for 10: any UI or workflow for manual JE creation, debit/credit balancing, or memo attachment.

   • [claimed-docs] “Keep your existing QuickBooks account and historical data”
   • [claimed-docs] “Virtually every tax preparer in the US knows how to work with QuickBooks. You’ll have the freedom to choose any tax preparer you want.”
   • [claimed-docs] “Communicate directly with your team through our user-friendly portal, where you can ask questions and provide feedback on any outstanding it…”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a done-for-you bookkeeping service where Bench's own staff (not the customer) manage the books; evidence explicitly states the client-facing surface is reports/downloads, 'not a ledger the customer posts to' (bench-supp-reporting), with no mention of a journal-entry interface, debit/credit balancing, or memo fields for accountant users. Missing for 10: any UI or workflow for entering/posting journal entries, debit/credit validation, and memo/supporting-documentation attachment on entries.

   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”
   • [claimed-docs] “Real-time reporting. Access or download your updated income statement or balance sheet at all times”
   • [claimed-docs] “Bench is a human-plus-software service, now under new ownership: "Bench simplifies your small business accounting by combining intuitive sof…”

   proof ↗⚑ flag

Ai audit

1. ai-native userHave an agent monitor the ledger and flag anomalies in spend or duplicate transactions before the close

   weight 3 · round drawn
   Pilot—none0/10

   Pilot's evidence covers bookkeeping, tax, CFO services, and AI Q&A on transactions, but there is no mention of an autonomous agent that monitors the ledger, flags spend anomalies, or detects duplicate transactions before close. missing for 10: anomaly-detection feature, duplicate-transaction flagging, any agentic monitoring capability, pre-close audit workflow evidence.

   • [claimed-docs] “Instant AI answers about past transactions and current cash flow, plus run scenarios to plan ahead”
   • [claimed-docs] “Track company expenses in real time Find the $2,000/month you didn't know you were wasting.”
   • [claimed-docs] “Communicate directly with your team through our user-friendly portal, where you can ask questions and provide feedback on any outstanding it…”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a human-plus-software bookkeeping service with automated categorization and real-time reporting, but there is no evidence of an AI agent that actively monitors the ledger for anomalies or duplicate transactions ahead of close; the closest capability is human bookkeeper review and periodic reminders. missing for 10: agent-based anomaly detection, duplicate-transaction flagging, any pre-close audit automation.

   • [claimed-docs] “Accurate transaction categorization, powered by smart automation with instant guidance”
   • [claimed-docs] “"Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time", with "Accurate transaction c…”
   • [claimed-docs] “Real-time reporting. Access or download your updated income statement or balance sheet at all times”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”

   proof ↗⚑ flag

Audit trail

1. accountantSee a complete audit trail of who changed what and when across the entire ledger

   weight 3 · round drawn
   Pilot—none0/10

   No evidence in the pack describes any audit trail, change history, or user-activity logging feature within Pilot's platform; the material focuses on bookkeeping services, integrations, CFO advisory, and tax credits. Missing for 10: any mention of audit logs, change tracking, user attribution, or timestamped edit history across the ledger.

   ⚑ flag

   Bench—none0/10

   Bench is a service where a human bookkeeper does the books, and the client-facing surface is reports/downloads rather than a ledger with visible change history; no evidence of any audit-trail, version history, or user-attribution feature is present anywhere in the pack, and probes confirm no docs/API surfacing such capability. missing for 10: any mention of audit logs, change tracking, user attribution, or timestamped edit history across the ledger.

   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

Multi entity

1. accountantManage multiple legal entities and produce consolidated financials across them

   weight 2 · round to Pilot
   Pilot~partialclaimed4/10

   Pilot's FAQ mentions it can provide consolidated reporting for US entities with foreign subsidiaries, but there is no evidence of multi-entity management for domestic entities, no detail on consolidation methodology, and no independent/hands-on corroboration of this feature. missing for 10: multi-entity dashboard/management capability, domestic multi-entity consolidation support, detailed consolidation workflow, independent verification.

   • [claimed-docs] “We can also provide US entities with foreign subsidiaries with consolidated reporting.”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a single-entity small-business bookkeeping/tax service; nothing in the evidence mentions multi-entity management, consolidation, or intercompany eliminations. Missing for 10: multi-entity account structure, consolidated financial statements across entities, intercompany reconciliation/eliminations.

   • [claimed-docs] “Real-time reporting. Access or download your updated income statement or balance sheet at all times”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”
   • [claimed-docs] “Bench is a human-plus-software service, now under new ownership: "Bench simplifies your small business accounting by combining intuitive sof…”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round drawn
   Pilot—none0/10

   No evidence of any public API for Pilot; probes for OpenAPI/swagger endpoints all returned 404, and all documentation describes only the human portal/UI and service offerings. missing for 10: any documented API, API parity claims, or developer endpoints.

   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”
   • [probe] “PROBE docs-md: HTTP 404 at https://pilot.com/faq.md”

   proof ↗⚑ flag

   Bench—none0/10

   No API, OpenAPI spec, or developer documentation exists; probes for llms.txt, docs.md, and openapi.json all return 404s, and all evidence describes a UI/human-service product with no programmatic access surface.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   Pilot—none0/10

   While Pilot mentions QuickBooks compatibility and tax-preparer portability, there is no evidence of a dedicated data export feature, open-format export (CSV/JSON/API), or a documented process to fully extract and leave with all financial data; probes even show no public API/OpenAPI spec.

   • [claimed-docs] “Keep your existing QuickBooks account and historical data”
   • [claimed-docs] “Virtually every tax preparer in the US knows how to work with QuickBooks. You’ll have the freedom to choose any tax preparer you want.”
   • [probe] “PROBE docs-md: HTTP 404 at https://pilot.com/faq.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”

   proof ↗⚑ flag

   Bench—none0/10

   No evidence of data export in open/portable formats or account portability; reporting is limited to viewing/downloading income statements or balance sheets, not a full data export mechanism, and probes for llms.txt/openapi/docs all 404. missing for 10: explicit data export feature (CSV/QBO/API), documented data portability policy, evidence of ability to leave with full underlying data.

   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   Pilot—none0/10

   No evidence in the pack addresses data residency, region selection, or storage location controls; Pilot is an accounting/bookkeeping service with no mention of infrastructure or data-residency options. Missing for 10: any documentation of data storage regions, residency guarantees, or configurable data location.

   ⚑ flag

   Bench—none0/10

   No evidence in the pack mentions data residency, region selection, or storage location controls; Bench's marketing focuses on bookkeeping automation, tax, and banking, with no data-residency documentation and 404s on technical/API probes that might have revealed such settings.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   Pilot—none0/10

   Pilot is an accounting/bookkeeping service; the evidence pack contains no privacy policy statements, opt-out controls, or AI-training data usage terms addressing whether customer financial data is used to train AI models. This is a fair question given Pilot advertises 'AI answers' features, but no evidence confirms any data-training opt-out or exclusion mechanism. Missing for 10: privacy policy/ToS language on AI training data usage, an opt-out mechanism, or any statement on data handling for AI features.

   • [claimed-docs] “Instant AI answers about past transactions and current cash flow, plus run scenarios to plan ahead”

   proof ↗⚑ flag

   Bench—none0/10

   Bench is a bookkeeping/accounting service and no evidence pack item mentions AI training data usage, opt-out controls, or any privacy policy addressing model training; this is an applicable privacy-posture axis for any modern SaaS handling customer data, but no supporting or contradicting evidence exists. Missing for 10: any privacy policy statement, opt-out mechanism, or terms-of-service language regarding AI/ML training on customer data.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round drawn
   Pilot—none0/10

   No evidence in the pack addresses data retention policies, deletion controls, or privacy settings for Pilot's AI features or data; the pack only covers accounting, bookkeeping, and tax service offerings. Missing for 10: any documentation of data retention/deletion controls, privacy settings, or data handling policies for AI or customer data.

   ⚑ flag

   Bench—none0/10

   Bench is a human-plus-software bookkeeping service; no evidence in the pack addresses data retention policies, deletion controls, or export/delete-your-data mechanisms for AI-native users. Probes for docs/llms.txt/openapi all 404, and no privacy/retention page is cited. missing for 10: retention policy documentation, data deletion/export controls, privacy posture disclosures, any AI-specific data handling terms.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”

   proof ↗⚑ flag

Ai insights

1. ai-native userAsk an agent "can we afford X?" and get an answer grounded in live ledger data pulled through the API

   weight 3 · round to Pilot
   Pilot~partialprobed4/10

   Pilot advertises an in-app AI feature offering 'Instant AI answers about past transactions and current cash flow, plus run scenarios to plan ahead,' which aligns with the 'can we afford X' query use case, but there is no evidence this is exposed via an API for external agents to query — the openapi probe returned 404s and no API/agent-integration docs exist. missing for 10: evidence of a public API/endpoint an external AI agent could call, documentation of grounding in live ledger data via API, and any independent confirmation the AI answer feature works as described.

   • [claimed-docs] “Instant AI answers about past transactions and current cash flow, plus run scenarios to plan ahead”
   • [claimed-docs] “Model cash flow scenarios Know if you can afford new hires well before you make the offers.”
   • [claimed-docs] “Know if you can afford new hires well before you make the offers.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://pilot.com/openapi.json, https://pilot.com/swagger.json, https://pilot.com/api/openapi.json, …”

   proof ↗⚑ flag

   Bench—none0/10

   No evidence of any public API, MCP server, or AI-agent integration exposing ledger data; probes for llms.txt, docs-md, and OpenAPI all returned 404s, and reporting is described as human-facing downloads/dashboards, not programmatic access. missing for 10: any API/endpoint for ledger data, agent integration, documented query interface for financial questions.

   • [probe] “PROBE llms.txt: HTTP 404 at https://www.bench.co/llms.txt”
   • [probe] “PROBE docs-md: HTTP 404 at https://www.bench.co/how-it-works.md”
   • [probe] “PROBE openapi: all candidate paths 404 (https://www.bench.co/openapi.json, https://www.bench.co/swagger.json, https://www.bench.co/api/opena…”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”

   proof ↗⚑ flag

Custom reports

1. accountantBuild custom reports filtered and grouped by class, location, or tag dimensions

   weight 2 · round drawn
   Pilot—none0/10

   Evidence only mentions generic 'custom workflows and reporting' (pilot-docs-10) with no mention of class, location, or tag dimension filtering/grouping capabilities typical of QuickBooks-style reports; missing for 10: any documentation of dimension-based (class/location/tag) filtering or grouping in reports.

   • [claimed-docs] “Custom workflows and reporting”

   proof ↗⚑ flag

   Bench—none0/10

   Evidence only shows basic reporting (income statement, balance sheet) delivered as static downloads/reports, with no mention of custom report building or filtering/grouping by class, location, or tag dimensions. Bench is positioned as a done-for-you bookkeeping service, not a self-serve reporting tool with dimensional tagging capabilities.

   • [claimed-docs] “Real-time reporting. Access or download your updated income statement or balance sheet at all times”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”

   proof ↗⚑ flag

Statements

1. founderGenerate an accurate P&L, balance sheet, and cash-flow statement for any period on demand

   weight 3 · round to Bench
   Pilot~partialcommunity5/10

   Pilot's bookkeeping and CFO service pages reference real-time account data, custom workflows/reporting, and budget-vs-actuals/scenario planning, implying standard financial statements are produced as part of monthly bookkeeping, and community reviews confirm books are kept accurate and up to date. However, no evidence explicitly confirms generation of a P&L, balance sheet, and cash-flow statement for any arbitrary period on demand. Missing for 10: explicit documentation of the three core statements, confirmation of on-demand/any-period generation (vs. standard monthly close), and independent verification of statement accuracy/timeliness.

   • [claimed-docs] “Custom workflows and reporting”
   • [claimed-docs] “stay up to date with real time account data directly in the portal”
   • [claimed-docs] “Custom financial model with strategic insights Budget vs actuals variance analysis Scenario planning model”
   • [claimed-docs] “Plus stay up to date with real time account data directly in the portal.”
   • [community] “I'm a Pilot customer... Pilot basically solved that problem. I just do my normal business and their integrations with my banks, expensify, c…”
   • [community] “Pilot customer here; they are amazing. It's pretty different from a regular bookkeeper largely in that Pilot is super accurate and timely. M…”

   proof ↗⚑ flag

   Bench✓fullclaimed8/10

   Bench explicitly markets real-time reporting with income statement (P&L) and balance sheet access/download at all times, backed by automated categorization and human bookkeeper review for accuracy. Cash-flow statement generation is not explicitly named, and there's no independent/hands-on corroboration of accuracy beyond vendor claims. Missing for 10: explicit mention of cash-flow statement as a deliverable report, and third-party or user corroboration of report accuracy.

   • [claimed-docs] “Real-time reporting. Access or download your updated income statement or balance sheet at all times”
   • [claimed-docs] “"Real-time reporting. Access or download your updated income statement or balance sheet at all times" — the client-facing surface is reports…”
   • [claimed-docs] “Accurate transaction categorization, powered by smart automation with instant guidance”
   • [claimed-docs] “"Connect all your financial accounts to automate data entry, speed up your books, reduce errors and save time", with "Accurate transaction c…”
   • [claimed-docs] “Bench is a human-plus-software service, now under new ownership: "Bench simplifies your small business accounting by combining intuitive sof…”

   proof ↗⚑ flag