FutureHouse Platform vs Sakana Marlin

Agent access

1. ai-native userPoint an agent at llms.txt or agent-oriented docs

   weight 2 · round to FutureHouse Platform
   FutureHouse Platform✓fullprobed7/10

   A working llms.txt file is directly confirmed via probe at futurehouse.gitbook.io/llms.txt returning HTTP 200 with structured agent-oriented documentation links, satisfying the core ask of pointing an agent at llms.txt. missing for 10: markdown-per-page fallback (docs-md probe returned 404), no OpenAPI/machine-readable schema confirmation, and no independent third-party corroboration of agents actually consuming this file successfully.

   • [probe] “PROBE llms.txt: HTTP 200 at https://futurehouse.gitbook.io/llms.txt # Edison Scientific Documentation ## Home - [Overview](https://docs.ed…”
   • [probe] “PROBE docs-md: HTTP 200 at https://futurehouse.gitbook.io/futurehouse-cookbook.md # Page Not Found The URL `futurehouse-cookbook` does not …”
   • [probe] “PROBE openapi: all candidate paths 404 (https://futurehouse.gitbook.io/openapi.json, https://futurehouse.gitbook.io/swagger.json, https://fu…”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   The llms.txt probe returned a 404, and no other evidence shows agent-oriented docs (like an API spec or agent-readable documentation) for Marlin; the evidence pack is entirely marketing copy about the product's research capabilities, not machine-readable docs.

   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”

   proof ↗⚑ flag

2. ai-native userRun the product headlessly / in CI for automation

   weight 2 · round to FutureHouse Platform
   FutureHouse Platform✓fullclaimed7/10

   The Edison client is a pip-installable Python SDK (`edison-client`) authenticated via API key, with a scriptable interface (`client.run_tasks_until_done(task_data)`) that requires no UI, making it straightforward to invoke headlessly or from CI pipelines. Missing for 10: explicit CI/automation examples (e.g., GitHub Actions workflow), documentation of exit codes/async polling patterns for pipelines, and independent confirmation of real-world CI usage.

   • [claimed-docs] “uv pip install edison-client”
   • [claimed-docs] “Authentication is done by providing an API key, which can be obtained directly from your profile page in the Edison platform”
   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   Marlin is presented as a research-report generation web product with a UI and pay-per-use credits, with no CLI, API, SDK, or webhook documentation for headless/CI usage; probes for llms.txt and OpenAPI specs both returned 404, indicating no programmatic interface is exposed.

   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”
   • [claimed-docs] “Add your card. Start right away.”

   proof ↗⚑ flag

3. ai-native userUse an official CLI

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   Evidence shows only a Python client library (edison-client, installed via pip, used programmatically with client.run_tasks_until_done) rather than a command-line interface; no CLI tool, command syntax, or terminal usage is documented anywhere in the pack.

   • [claimed-docs] “uv pip install edison-client”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   No evidence of an official CLI; Marlin appears to be a web-based research tool with a pay-per-credit UI, and probes for API/llms.txt endpoints returned 404s, suggesting no developer-facing interface is exposed.

   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”

   proof ↗⚑ flag

4. ai-native userDrive the product through a documented public API

   weight 3 · round to FutureHouse Platform
   FutureHouse Platform✓fullprobed7/10

   Docs describe an installable Python client (edison-client) with API-key/token auth and a documented workflow (client.run_tasks_until_done) to submit tasks like literature review, matching a genuine public API for driving the platform programmatically. Missing for 10: no OpenAPI/swagger spec found (404s), and no independent/hands-on corroboration beyond first-party docs.

   • [claimed-docs] “uv pip install edison-client”
   • [claimed-docs] “Authentication is done by providing an API key, which can be obtained directly from your profile page in the Edison platform”
   • [claimed-docs] “Under "API Tokens", click "Create New Token."”
   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”
   • [probe] “PROBE openapi: all candidate paths 404 (https://futurehouse.gitbook.io/openapi.json, https://futurehouse.gitbook.io/swagger.json, https://fu…”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   No documented public API is evidenced; probes for llms.txt and OpenAPI/swagger endpoints all returned 404s, and all other evidence describes the product's research capabilities, not a programmatic interface.

   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”

   proof ↗⚑ flag

5. ai-native userBuild against official SDKs

   weight 2 · round to FutureHouse Platform
   FutureHouse Platform~partialprobed6/10

   There is a documented official Python client (edison-client) with install, auth via API key, and task submission/run examples, showing a real SDK path for programmatic/agentic use. However, no OpenAPI spec was found (all probed paths 404), and the docs page itself returned 404 on a direct fetch, suggesting incomplete or unstable documentation coverage. Missing for 10: verified OpenAPI/REST spec, broader SDK language coverage, independent developer corroboration of SDK reliability.

   • [claimed-docs] “uv pip install edison-client”
   • [claimed-docs] “Authentication is done by providing an API key, which can be obtained directly from your profile page in the Edison platform”
   • [claimed-docs] “Under "API Tokens", click "Create New Token."”
   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”
   • [probe] “PROBE openapi: all candidate paths 404 (https://futurehouse.gitbook.io/openapi.json, https://futurehouse.gitbook.io/swagger.json, https://fu…”
   • [probe] “PROBE docs-md: HTTP 200 at https://futurehouse.gitbook.io/futurehouse-cookbook.md # Page Not Found The URL `futurehouse-cookbook` does not …”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   No evidence of any official SDK, API, or developer library for Marlin; probes for llms.txt and OpenAPI specs both returned 404, and all docs describe an end-user research product with no mention of programmatic/SDK access.

   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”

   proof ↗⚑ flag

6. ai-native userSubscribe to events via webhooks

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   No mention of webhooks, event subscriptions, or callback mechanisms anywhere in the docs; the client is a polling/run-tasks style API and OpenAPI probe returned 404s, giving no evidence of webhook support.

   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”
   • [probe] “PROBE openapi: all candidate paths 404 (https://futurehouse.gitbook.io/openapi.json, https://futurehouse.gitbook.io/swagger.json, https://fu…”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   No evidence of a webhook or event subscription mechanism; probes for API/OpenAPI specs returned 404s and docs focus only on research report generation. Missing for 10: any webhook documentation, event subscription API, or callback mechanism.

   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”

   proof ↗⚑ flag

Agentic features

1. ai-native userGet AI-generated insights and suggestions from my data inside the product

   weight 2 · round to Sakana Marlin
   FutureHouse Platform✓fullprobed7/10

   The platform's core agents (Kosmos, Precedent, Molecules) are documented as autonomously generating insights, hypotheses, and cited reports directly from user-supplied data and literature, matching the story closely (e.g., Kosmos 'autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report'). Missing for 10: independent/hands-on corroboration beyond vendor docs, and some doc links returned 404 in probes, weakening confidence in documentation completeness.

   • [claimed-docs] “Kosmos autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report.”
   • [claimed-docs] “Given a research objective and one or more datasets, Kosmos autonomously reads literature, writes and executes analysis code, generates hypo…”
   • [claimed-docs] “You can ask it a complex scientific question and get a high-accuracy, cited response, or task it with a deep literature review synthesizing …”
   • [claimed-docs] “It turns raw data into detailed analyses, statistical results, and publication-ready figures.”
   • [claimed-docs] “Precedent determines whether a research idea has been tried before. It searches across fields to assess novelty and identify gaps”
   • [claimed-docs] “Every conclusion is fully auditable. You can trace any finding back to the specific code or literature passage that produced it.”
   • [probe] “PROBE docs-md: HTTP 200 at https://futurehouse.gitbook.io/futurehouse-cookbook.md # Page Not Found The URL `futurehouse-cookbook` does not …”

   proof ↗⚑ flag

   Sakana Marlin✓fullclaimed8/10

   Marlin autonomously researches user-provided topics, mapping causal relationships, comparing hypotheses, and generating structured strategic insights and reports with citations, going beyond simple summarization. Missing for 10: independent/hands-on third-party validation beyond vendor-curated testimonials, and no visibility into underlying data/insight quality benchmarks.

   • [claimed-docs] “Sakana Marlin maps the causal relationships at work in complex business environments and organizes them into structured strategic options.”
   • [claimed-docs] “It does more than summarize. Sakana Marlin maps the causal relationships at work in complex business environments and organizes them into st…”
   • [claimed-docs] “Rather than merely aggregating information, it compares and evaluates multiple hypotheses to provide deep insights.”
   • [claimed-docs] “From ideation and data gathering to resolving contradictions and structuring the report, the AI autonomously executes the entire research pr…”
   • [claimed-docs] “The research was of extremely high quality, grounded strictly in primary sources, resulting in a highly convincing and reliable final report…”
   • [claimed-docs] “Give it a research topic, and Marlin works autonomously for up to roughly eight hours, crafting a detailed strategy report up to a hundred p…”

   proof ↗⚑ flag

2. ai-native userSet up automations that run autonomously in the background

   weight 2 · round to Sakana Marlin
   FutureHouse Platform—none0/10

   The evidence describes an API client for submitting tasks/queries and autonomous agents that complete a single task end-to-end, but there is no mention of scheduling, triggers, or background automations that run without explicit invocation. missing for 10: scheduling/cron mechanism, event-driven triggers, persistent background job management, and any docs describing recurring or unattended automation setup.

   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”
   • [claimed-docs] “Kosmos autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report.”

   proof ↗⚑ flag

   Sakana Marlin~partialclaimed6/10

   Marlin's docs clearly describe a single research task running autonomously for up to ~8 hours without further human input once a topic is set (docs-9, docs-10, docs-16), which matches the 'runs in background autonomously' idea. However, this is a one-shot session, not a recurring/scheduled automation you configure and forget — there's no evidence of triggers, schedules, or multi-run automation management typical of 'set up automations.' Missing for 10: scheduled/recurring automation setup, background job management UI, independent hands-on corroboration of unattended runtime.

   • [claimed-docs] “Give it a research topic, and Marlin works autonomously for up to roughly eight hours, crafting a detailed strategy report up to a hundred p…”
   • [claimed-docs] “Sakana Marlin sharpens the direction of the investigation through a brief exchange with the user. Once the course is set, it works without f…”
   • [claimed-docs] “From ideation and data gathering to resolving contradictions and structuring the report, the AI autonomously executes the entire research pr…”
   • [claimed-docs] “It is designed for use cases where you want to delegate everything up to "issue framing (agenda setting)" prior to decision-making to the AI…”

   proof ↗⚑ flag

3. ai-native userDelegate tasks to a built-in AI assistant inside the product

   weight 3 · round drawn
   FutureHouse Platform✓fullclaimed8/10

   The platform ships built-in autonomous agents (Kosmos, Precedent, Molecules, literature review agent) that users can delegate research tasks to via API/client, with docs describing task submission and autonomous execution producing cited reports. Missing for 10: independent/hands-on corroboration beyond vendor docs, and clearer in-product (non-API) delegation UI evidence.

   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “Kosmos autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report.”
   • [claimed-docs] “You can ask it a complex scientific question and get a high-accuracy, cited response, or task it with a deep literature review synthesizing …”
   • [claimed-docs] “Precedent determines whether a research idea has been tried before. It searches across fields to assess novelty and identify gaps”
   • [claimed-docs] “Molecules is a chemistry-focused agent for molecular design and analysis.”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”
   • [claimed-docs] “Given a research objective and one or more datasets, Kosmos autonomously reads literature, writes and executes analysis code, generates hypo…”

   proof ↗⚑ flag

   Sakana Marlin✓fullclaimed8/10

   Marlin is explicitly designed as a built-in AI agent that users delegate entire research/strategy tasks to, working autonomously for hours with minimal human input beyond initial framing (docs-1, docs-9, docs-10, docs-16, docs-21). This directly matches the story of delegating tasks to a built-in assistant within the product. Missing for 10: independent/hands-on verification beyond vendor testimonials, and detail on interactive control while a task is delegated.

   • [claimed-docs] “Once you set a research topic, it works without further human input: repeatedly forming hypotheses, gathering information, navigating the we…”
   • [claimed-docs] “Give it a research topic, and Marlin works autonomously for up to roughly eight hours, crafting a detailed strategy report up to a hundred p…”
   • [claimed-docs] “Sakana Marlin sharpens the direction of the investigation through a brief exchange with the user. Once the course is set, it works without f…”
   • [claimed-docs] “From ideation and data gathering to resolving contradictions and structuring the report, the AI autonomously executes the entire research pr…”
   • [claimed-docs] “It is designed for use cases where you want to delegate everything up to "issue framing (agenda setting)" prior to decision-making to the AI…”

   proof ↗⚑ flag

4. ai-native userOperate the product with natural-language commands

   weight 2 · round to FutureHouse Platform
   FutureHouse Platform~partialclaimed7/10

   The platform's core interaction model is natural-language queries (e.g., the query string "Which neglected diseases had a treatment developed by artificial intelligence?") passed to autonomous agents like Kosmos, Precedent, and Molecules, and docs explicitly say users can 'ask it a complex scientific question' or 'task it with a deep literature review.' However, the demonstrated flow is via a Python client (task_data dict + run_tasks_until_done), not a conversational chat UI, so it's NL-as-input-parameter rather than full conversational natural-language operation of the whole platform (e.g., no evidence of multi-turn dialogue or a chat interface). Missing for 10: evidence of a conversational/chat UI, multi-turn NL interaction, or NL control over platform features beyond the initial query.

   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “Kosmos autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report.”
   • [claimed-docs] “You can ask it a complex scientific question and get a high-accuracy, cited response, or task it with a deep literature review synthesizing …”
   • [claimed-docs] “Given a research objective and one or more datasets, Kosmos autonomously reads literature, writes and executes analysis code, generates hypo…”
   • [claimed-docs] “task it with a deep literature review synthesizing conflicting evidence across hundreds of papers”

   proof ↗⚑ flag

   Sakana Marlin~partialclaimed5/10

   Marlin is initiated by giving it a research topic and a brief natural-language exchange to set direction (docs-10), suggesting natural-language input drives its operation, but there is no evidence of a broader natural-language command interface (e.g., chat-style control, follow-up instructions, or command syntax) beyond initial topic-setting. Missing for 10: documentation of ongoing NL command/control during execution, examples of varied NL commands, independent/hands-on confirmation of NL interaction quality.

   • [claimed-docs] “Sakana Marlin sharpens the direction of the investigation through a brief exchange with the user. Once the course is set, it works without f…”
   • [claimed-docs] “Give it a research topic, and Marlin works autonomously for up to roughly eight hours, crafting a detailed strategy report up to a hundred p…”
   • [claimed-docs] “It is designed for use cases where you want to delegate everything up to "issue framing (agenda setting)" prior to decision-making to the AI…”

   proof ↗⚑ flag

Api quality

1. ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   Probes for OpenAPI/swagger spec paths all returned 404, and no docs reference a machine-readable API spec; only a Python client library and manual REST auth flow are documented. Missing for 10: any OpenAPI/Swagger endpoint or downloadable spec file, any documentation mentioning API schema export.

   • [probe] “PROBE openapi: all candidate paths 404 (https://futurehouse.gitbook.io/openapi.json, https://futurehouse.gitbook.io/swagger.json, https://fu…”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   Probes for OpenAPI/swagger spec files and llms.txt all returned 404, and no documentation mentions a machine-readable API spec. Missing for 10: any published OpenAPI/Swagger file, API reference docs, or equivalent machine-readable spec.

   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”

   proof ↗⚑ flag

2. ai-native userRely on versioned APIs with a documented deprecation policy

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   No evidence of API versioning scheme, changelog, or deprecation policy anywhere in the docs; OpenAPI spec probes all 404 and no versioning documentation is present. missing for 10: versioned API endpoints, changelog/release notes, documented deprecation policy or timelines.

   • [probe] “PROBE openapi: all candidate paths 404 (https://futurehouse.gitbook.io/openapi.json, https://futurehouse.gitbook.io/swagger.json, https://fu…”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   No evidence of any public API, versioning scheme, or deprecation policy; probes for OpenAPI/llms.txt endpoints returned 404s, and all docs describe the research product itself, not a developer API. Missing for 10: any API documentation, versioning scheme, deprecation policy.

   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”

   proof ↗⚑ flag

1. ai-native userPerform bulk operations across many items at once

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   The docs show only single-task submission via task_data/run_tasks_until_done, with no mention of batch endpoints, bulk task lists, or parallel multi-item processing APIs; missing for 10: batch/bulk submission API, documented loop or list-based task queuing, evidence of concurrent multi-item execution.

   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   All evidence describes Marlin running a single autonomous research task per topic (one report, one investigation) rather than processing many items/topics in bulk; no mention of batch submission, multi-item queues, or parallel bulk execution.

   ⚑ flag

2. ai-native userSchedule recurring jobs or workflows

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   Evidence covers running one-off tasks via the client (run_tasks_until_done) but there is no mention of scheduling, cron-like recurrence, or persistent workflow automation anywhere in the docs or probes.

   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   Sakana Marlin is a single-run autonomous research/report tool triggered by a user topic; no evidence of scheduling, recurrence, cron-like triggers, or workflow automation for repeated jobs. Probes also show no API/OpenAPI surface that could support scheduled invocation.

   • [claimed-docs] “Give it a research topic, and Marlin works autonomously for up to roughly eight hours, crafting a detailed strategy report up to a hundred p…”
   • [claimed-docs] “Sakana Marlin sharpens the direction of the investigation through a brief exchange with the user. Once the course is set, it works without f…”
   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”

   proof ↗⚑ flag

Sharing

1. analystShare a research session or report with collaborators who can view or build on it

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   No evidence in the pack mentions sharing sessions, reports, or collaborative access controls; documentation focuses on API usage, agents, and task execution, not collaboration features. Missing for 10: any mention of sharing/exporting reports, collaborator invites, or team/workspace features.

   ⚑ flag

   Sakana Marlin—none0/10

   No evidence describes any collaboration or sharing features—no mention of shared workspaces, links, comments, or multi-user access to reports/sessions; the pack only covers autonomous research generation, pricing tiers, and API probes returning 404s.

   • [claimed-docs] “We have made Marlin available as a pay-per-use tier to monthly Pro, Team, and Enterprise-tier plans.”
   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”

   proof ↗⚑ flag

Alerts

1. researcherSet up standing searches or alerts that surface new relevant sources as they appear

   weight 1 · round drawn
   FutureHouse Platform—none0/10

   The evidence describes on-demand literature search, review, and analysis agents (Kosmos, Precedent, etc.) invoked via API calls, but nothing describes standing searches, saved alerts, or recurring notifications when new relevant sources appear.

   ⚑ flag

   Sakana Marlin—none0/10

   Marlin is a one-shot deep-research report generator that runs autonomously for a single session (up to ~8 hours) and produces a report; there is no evidence of standing searches, saved alerts, or recurring monitoring for new sources over time.

   ⚑ flag

Corpus

1. researcherUpload my own PDFs or corpus and have the agent research over them

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   The docs describe FutureHouse's agents (Kosmos, literature review, Precedent) searching a fixed 175M+ paper corpus and analyzing 'datasets' for experiments, but nowhere describe a mechanism for a researcher to upload their own PDFs or personal corpus for the agent to search/reason over. Without evidence of a file-upload or custom-corpus ingestion feature, this capability is not shown.

   • [claimed-docs] “Given a research objective and one or more datasets, Kosmos autonomously reads literature, writes and executes analysis code, generates hypo…”
   • [claimed-docs] “It accesses 175M+ papers, trials, and patents with native understanding of citation graphs, journal quality, and clinical trial data.”
   • [claimed-docs] “specializes in processing complex experimental data, including flow cytometry, RNA-seq, and other biological datasets”
   • [claimed-docs] “task it with a deep literature review synthesizing conflicting evidence across hundreds of papers”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   Evidence describes Marlin as an autonomous web-research agent that gathers information via web navigation and generates reports, but there is no mention of uploading a user's own PDFs or corpus for the agent to research over. Missing for 10: any document/file upload feature, corpus ingestion, or evidence of researching over user-supplied materials rather than open web sources.

   ⚑ flag

Reviews

1. researcherRun a systematic screening and extraction workflow across many papers with consistent criteria

   weight 2 · round to FutureHouse Platform
   FutureHouse Platform~partialprobed5/10

   The API client supports programmatic task submission (e.g., JobNames.LITERATURE) and batch execution via run_tasks_until_done, and the docs describe a literature agent that can synthesize evidence across hundreds of papers with citations, which could be scripted for repeated, consistent queries. However, there is no explicit documented workflow for structured screening/extraction with defined inclusion/exclusion criteria across a large corpus (e.g., systematic review style tables), and the cookbook page itself returns 404 on the probe, reducing confidence in the full pipeline. missing for 10: explicit systematic-review/extraction workflow docs, example of consistent-criteria batch screening, working cookbook page confirming the API details.

   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”
   • [claimed-docs] “task it with a deep literature review synthesizing conflicting evidence across hundreds of papers”
   • [claimed-docs] “You can ask it a complex scientific question and get a high-accuracy, cited response, or task it with a deep literature review synthesizing …”
   • [probe] “PROBE docs-md: HTTP 200 at https://futurehouse.gitbook.io/futurehouse-cookbook.md # Page Not Found The URL `futurehouse-cookbook` does not …”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   Marlin is positioned as an autonomous business/market strategy research agent producing single deep-dive reports, not as a tool for systematic multi-paper screening/extraction with consistent criteria (a literature-review workflow). No evidence describes handling many papers, applying consistent inclusion/extraction criteria, or batch processing across a corpus — the described unit of work is one topic producing one report.

   • [claimed-docs] “Give it a research topic, and Marlin works autonomously for up to roughly eight hours, crafting a detailed strategy report up to a hundred p…”
   • [claimed-docs] “It is designed for use cases where you want to delegate everything up to "issue framing (agenda setting)" prior to decision-making to the AI…”
   • [claimed-docs] “putting Marlin to work on real tasks such as strategy formulation, market research, risk analysis, and competitive analysis”

   proof ↗⚑ flag

1. ai-native userDo everything through the API that I can do in the UI

   weight 2 · round to FutureHouse Platform
   FutureHouse Platform~partialprobed5/10

   The edison-client library and quickstart docs show that core agent tasks (e.g., JobNames.LITERATURE) can be invoked via API with authentication and task submission (docs-1–4,12), and the cookbook describes multiple agents (Kosmos, Precedent, Molecules) that presumably map to API job types, suggesting broad functional overlap with the UI. However, there is no explicit documentation stating full feature parity between UI and API, no OpenAPI/swagger spec discoverable (probe-3 shows 404s across all candidate paths), and no confirmation that all UI-exposed capabilities (e.g., account management, dataset uploads, report viewing) are also API-accessible. missing for 10: explicit parity statement, discoverable OpenAPI schema, evidence covering non-task UI features (billing, dataset management, report exports) via API.

   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”
   • [claimed-docs] “Kosmos autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report.”
   • [claimed-docs] “Precedent determines whether a research idea has been tried before. It searches across fields to assess novelty and identify gaps”
   • [claimed-docs] “Molecules is a chemistry-focused agent for molecular design and analysis.”
   • [probe] “PROBE openapi: all candidate paths 404 (https://futurehouse.gitbook.io/openapi.json, https://futurehouse.gitbook.io/swagger.json, https://fu…”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   No evidence of any public API for Sakana Marlin; probes for llms.txt and OpenAPI spec both returned 404, and all documentation describes only UI/credit-based access.

   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”
   • [claimed-docs] “Add your card. Start right away.”

   proof ↗⚑ flag

2. ai-native userExport all of my data in open formats and leave

   weight 3 · round drawn
   FutureHouse Platform—none0/10

   No evidence of any data export capability, open-format export, or account portability/deletion feature; documentation covers agent/task usage and API access but nothing about exporting user data or leaving with it.

   ⚑ flag

   Sakana Marlin—none0/10

   No evidence of data export, open-format download, or account portability features; probes for llms.txt and OpenAPI both returned 404, and docs only describe generated reports/slides, not export of underlying user data.

   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”

   proof ↗⚑ flag

3. ai-native userRead the product's source under an open license

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   No evidence pack items reference an open-source license or public repository for the FutureHouse Platform itself; only client library install instructions and product docs are provided, with no license information. missing for 10: evidence of open-source license, public source repo link, license file or statement.

   ⚑ flag

   Sakana Marlin—none0/10

   No evidence of any open-source license or public source code repository for Marlin; it is presented as a paid SaaS research product, and probes for open API/docs artifacts returned 404s.

   • [claimed-docs] “We have made Marlin available as a pay-per-use tier to monthly Pro, Team, and Enterprise-tier plans.”
   • [probe] “PROBE llms.txt: HTTP 404 at https://sakana.ai/llms.txt”
   • [probe] “PROBE openapi: all candidate paths 404 (https://sakana.ai/openapi.json, https://sakana.ai/swagger.json, https://sakana.ai/api/openapi.json, …”

   proof ↗⚑ flag

Pricing

1. researcherTry the product meaningfully on a free tier or trial

   weight 1 · round to FutureHouse Platform
   FutureHouse Platform✓fullclaimed7/10

   Docs explicitly state Edison maintains a generous free tier for academics, with paid plans only needed for higher rate limits/additional features, and the quickstart shows a full API key + client workflow to run real tasks (literature review job). Missing for 10: independent/hands-on confirmation of free tier limits and no clarity on exact quota/duration restrictions.

   • [claimed-docs] “Edison maintains a generous free tier for academics. Researchers who need higher rate limits or additional features can subscribe to paid pl…”
   • [claimed-docs] “Researchers who need higher rate limits or additional features can subscribe to paid plans.”
   • [claimed-docs] “Authentication is done by providing an API key, which can be obtained directly from your profile page in the Edison platform”
   • [claimed-docs] “Under "API Tokens", click "Create New Token."”
   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”

   proof ↗⚑ flag

   Sakana Marlin—none0/10

   Marlin is explicitly pay-per-use available only to paid Pro/Team/Enterprise plans, requires adding a card to start, and cancelling mid-run still consumes credits — there is no free tier or trial for researchers to test it meaningfully.

   • [claimed-docs] “We have made Marlin available as a pay-per-use tier to monthly Pro, Team, and Enterprise-tier plans.”
   • [claimed-docs] “Add your card. Start right away.”
   • [claimed-docs] “You can cancel at any time during execution, but please note that credits will still be consumed.”
   • [claimed-docs] “Add-on credits available (¥98 / credit)”

   proof ↗⚑ flag

2. researcherUnderstand plan pricing and usage limits before committing

   weight 2 · round drawn
   FutureHouse Platform~partialclaimed4/10

   Docs confirm a free tier for academics and paid plans for higher rate limits/features, but no concrete pricing figures, tier breakdowns, or specific usage limit numbers are provided anywhere in the pack. missing for 10: actual price points, detailed rate-limit thresholds, feature comparison table across tiers, billing/usage dashboard evidence.

   • [claimed-docs] “Researchers who need higher rate limits or additional features can subscribe to paid plans.”
   • [claimed-docs] “Edison maintains a generous free tier for academics. Researchers who need higher rate limits or additional features can subscribe to paid pl…”

   proof ↗⚑ flag

   Sakana Marlin~partialclaimed4/10

   Marlin's docs mention it's offered as a pay-per-use add-on to Pro/Team/Enterprise plans, credits are consumed even if cancelled mid-run, and additional credits cost ¥98 each, giving a researcher some pricing signal. However, there's no concrete breakdown of how many credits a typical run consumes, no explicit usage caps, and no comparison table of plan tiers — missing for 10: full plan pricing table, credit-consumption-per-task estimates, explicit usage limits, independent/hands-on cost verification.

   • [claimed-docs] “You can cancel at any time during execution, but please note that credits will still be consumed.”
   • [claimed-docs] “We have made Marlin available as a pay-per-use tier to monthly Pro, Team, and Enterprise-tier plans.”
   • [claimed-docs] “Add-on credits available (¥98 / credit)”
   • [claimed-docs] “Add your card. Start right away.”

   proof ↗⚑ flag

1. ai-native userChoose where my data is stored (region/residency)

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   No evidence in the pack addresses data residency, regional storage options, or geographic control over where data is processed/stored; documentation only covers API usage, agents, and pricing tiers.

   ⚑ flag

   Sakana Marlin—none0/10

   No evidence in the pack mentions data residency, region selection, or storage location controls for Sakana Marlin; the evidence covers only research capabilities and pricing.

   ⚑ flag

2. ai-native userPrevent my data from being used to train AI models

   weight 3 · round drawn
   FutureHouse Platform—none0/10

   No evidence in the pack addresses data usage for AI training opt-out, data privacy controls, or any training-data policy; the docs focus entirely on product features and API usage.

   ⚑ flag

   Sakana Marlin—none0/10

   No evidence in the pack addresses data-training opt-out, privacy controls, or data usage policy for Sakana Marlin; missing for 10: any mention of training opt-out settings, privacy policy, or data retention controls.

   ⚑ flag

3. ai-native userControl data retention and deletion

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   No evidence in the pack addresses data retention policies, deletion controls, or privacy/data lifecycle management; documentation focuses on client setup, agents, and use cases only.

   ⚑ flag

   Sakana Marlin—none0/10

   No evidence in the pack addresses data retention, deletion controls, or privacy settings for Marlin; the evidence pack only covers research capabilities, pricing, and product marketing.

   ⚑ flag

4. ai-native userOpt out of telemetry and usage tracking

   weight 2 · round drawn
   FutureHouse Platform—none0/10

   No evidence in the pack mentions telemetry, usage tracking, analytics collection, or an opt-out mechanism anywhere in the FutureHouse docs or probes; the documentation focuses on client setup and agent features only.

   ⚑ flag

   Sakana Marlin—none0/10

   No evidence in the pack addresses telemetry, usage tracking, or any privacy/opt-out controls for Sakana Marlin. Missing for 10: any documentation of telemetry practices, an opt-out mechanism, or privacy policy details.

   ⚑ flag

Reports

1. researcherExport results to common formats, including documents, spreadsheets, and reference-manager files

   weight 1 · round to Sakana Marlin
   FutureHouse Platform—none0/10

   The evidence describes cited reports and analyses produced by Kosmos/agents but nowhere mentions exporting to document, spreadsheet, or reference-manager formats (e.g., PDF/DOCX, CSV/XLSX, BibTeX/RIS); the docs-cookbook page itself is even reported as not found in the probe.

   • [claimed-docs] “Kosmos autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report.”
   • [claimed-docs] “Every conclusion is fully auditable. You can trace any finding back to the specific code or literature passage that produced it.”
   • [claimed-docs] “It turns raw data into detailed analyses, statistical results, and publication-ready figures.”
   • [probe] “PROBE docs-md: HTTP 200 at https://futurehouse.gitbook.io/futurehouse-cookbook.md # Page Not Found The URL `futurehouse-cookbook` does not …”

   proof ↗⚑ flag

   Sakana Marlin~partialclaimed4/10

   Marlin auto-generates full reports with references and PowerPoint slides (docs-3, docs-9, docs-20), covering the 'documents' part of the story, but there is no evidence of spreadsheet export or reference-manager file formats (e.g., BibTeX/RIS/EndNote) for citations. missing for 10: spreadsheet export, reference-manager file export (BibTeX/RIS/EndNote), independent corroboration of export formats.

   • [claimed-docs] “Everything from the main body to appendices, references, and presentation slides is automatically generated — at a quality that stands along…”
   • [claimed-docs] “Give it a research topic, and Marlin works autonomously for up to roughly eight hours, crafting a detailed strategy report up to a hundred p…”
   • [claimed-docs] “From a fully referenced report to PowerPoint slides, everything is automatically generated — at a quality that stands alongside professional…”

   proof ↗⚑ flag

2. analystGet a structured report with sections, tables, and a summary that I can share with stakeholders

   weight 3 · round to Sakana Marlin
   FutureHouse Platform~partialclaimed6/10

   Docs describe Kosmos/agents producing a 'comprehensive cited report' with statistical results, publication-ready figures, and traceable citations, which implies structured, shareable output, but there's no explicit mention of report sections/tables format or export/sharing mechanisms for stakeholders. missing for 10: explicit description of report structure (sections/tables), export or sharing functionality, and independent confirmation of report format quality.

   • [claimed-docs] “Kosmos autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report.”
   • [claimed-docs] “Every conclusion is fully auditable. You can trace any finding back to the specific code or literature passage that produced it.”
   • [claimed-docs] “It turns raw data into detailed analyses, statistical results, and publication-ready figures.”
   • [claimed-docs] “Given a research objective and one or more datasets, Kosmos autonomously reads literature, writes and executes analysis code, generates hypo…”
   • [claimed-docs] “You can trace any finding back to the specific code or literature passage that produced it.”

   proof ↗⚑ flag

   Sakana Marlin✓fullclaimed7/10

   Docs describe autonomous generation of a full structured report with main body, appendices, references, and presentation slides/executive summary — directly matching sections, tables (implied by structured strategic options), and summary needs for stakeholder sharing. Missing for 10: no independent/hands-on verification of table formatting or actual sample report shown, and no evidence beyond vendor marketing copy.

   • [claimed-docs] “Everything from the main body to appendices, references, and presentation slides is automatically generated — at a quality that stands along…”
   • [claimed-docs] “Give it a research topic, and Marlin works autonomously for up to roughly eight hours, crafting a detailed strategy report up to a hundred p…”
   • [claimed-docs] “From a fully referenced report to PowerPoint slides, everything is automatically generated — at a quality that stands alongside professional…”
   • [claimed-docs] “Sakana Marlin maps the causal relationships at work in complex business environments and organizes them into structured strategic options.”

   proof ↗⚑ flag

Agent runs

1. researcherPose a research question and get an autonomous multi-step investigation, not just a single-pass summary

   weight 3 · round drawn
   FutureHouse Platform✓fullclaimed8/10

   Docs describe Kosmos and other agents autonomously reading literature, writing/executing analysis code, generating hypotheses, and producing cited multi-step reports (not single-pass summaries), with auditable traceability of findings and multi-agent capabilities (Precedent, Molecules, literature review across hundreds of papers). This directly matches the story of posing a question and getting an autonomous multi-step investigation. Missing for 10: independent/hands-on verification of the multi-step autonomy claims (probe shows docs page 404'd, reducing corroboration strength) and no third-party case study confirming real-world multi-step behavior.

   • [claimed-docs] “Kosmos autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report.”
   • [claimed-docs] “Given a research objective and one or more datasets, Kosmos autonomously reads literature, writes and executes analysis code, generates hypo…”
   • [claimed-docs] “Every conclusion is fully auditable. You can trace any finding back to the specific code or literature passage that produced it.”
   • [claimed-docs] “You can ask it a complex scientific question and get a high-accuracy, cited response, or task it with a deep literature review synthesizing …”
   • [claimed-docs] “task it with a deep literature review synthesizing conflicting evidence across hundreds of papers”
   • [claimed-docs] “Precedent determines whether a research idea has been tried before. It searches across fields to assess novelty and identify gaps”

   proof ↗⚑ flag

   Sakana Marlin✓fullclaimed8/10

   Vendor docs describe autonomous multi-step research: forming hypotheses, gathering info, resolving contradictions, running for hours across thousands of cycles without further human input, producing a full structured report - directly matching the story. This is corroborated by beta-tester quotes praising depth beyond chat-based research tools, though all evidence is vendor-published/testimonial rather than independent hands-on verification. missing for 10: independent third-party evaluation or benchmark of the autonomous multi-step process, technical detail on how contradictions/hypotheses are actually verified

   • [claimed-docs] “Once you set a research topic, it works without further human input: repeatedly forming hypotheses, gathering information, navigating the we…”
   • [claimed-docs] “Give it a research topic, and Marlin works autonomously for up to roughly eight hours, crafting a detailed strategy report up to a hundred p…”
   • [claimed-docs] “Sakana Marlin sharpens the direction of the investigation through a brief exchange with the user. Once the course is set, it works without f…”
   • [claimed-docs] “From ideation and data gathering to resolving contradictions and structuring the report, the AI autonomously executes the entire research pr…”
   • [claimed-docs] “Many told us that Marlin was more practical at digging deeply into information than the chat-based research tools they had used before”
   • [claimed-docs] “Rather than merely aggregating information, it compares and evaluates multiple hypotheses to provide deep insights.”

   proof ↗⚑ flag

2. analystStart a long research job that keeps working unattended and notifies me when the result is ready

   weight 2 · round to Sakana Marlin
   FutureHouse Platform~partialclaimed5/10

   The client SDK exposes a task submission and `run_tasks_until_done` pattern (docs-4, docs-12) plus descriptions of Kosmos autonomously running long, multi-step research (literature review, code execution, hypothesis generation) unattended (docs-5, docs-13, docs-19), supporting the 'long unattended job' half of the story. However there is no evidence of any notification mechanism (email, webhook, push) when a job completes — the only documented pattern is a client-side polling call that blocks until done, not an async notify-on-completion flow. Missing for 10: explicit notification/webhook/email-on-completion feature, evidence of true async fire-and-forget job submission separate from polling.

   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”
   • [claimed-docs] “Kosmos autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report.”
   • [claimed-docs] “Given a research objective and one or more datasets, Kosmos autonomously reads literature, writes and executes analysis code, generates hypo…”
   • [claimed-docs] “task it with a deep literature review synthesizing conflicting evidence across hundreds of papers”

   proof ↗⚑ flag

   Sakana Marlin~partialclaimed6/10

   Vendor docs clearly describe long unattended autonomous research runs (~8 hours) producing full reports, which supports the core of the story, but there is no mention of a notification mechanism when results are ready and no independent/hands-on corroboration beyond marketing copy. missing for 10: evidence of a completion notification/alert feature, independent verification of unattended runtime and reliability, and API/technical docs confirming job-control (start/monitor/cancel) beyond the marketing page.

   • [claimed-docs] “Give it a research topic, and Marlin works autonomously for up to roughly eight hours, crafting a detailed strategy report up to a hundred p…”
   • [claimed-docs] “Sakana Marlin sharpens the direction of the investigation through a brief exchange with the user. Once the course is set, it works without f…”
   • [claimed-docs] “Once you set a research topic, it works without further human input: repeatedly forming hypotheses, gathering information, navigating the we…”
   • [claimed-docs] “You can cancel at any time during execution, but please note that credits will still be consumed.”

   proof ↗⚑ flag

3. researcherSteer the depth, effort, and scope of a research run before or while it executes

   weight 1 · round to Sakana Marlin
   FutureHouse Platform—none0/10

   Evidence shows job submission with a query and job name, plus a mid-run polling helper (run_tasks_until_done), but no documented parameters or controls for adjusting depth, effort level, or scope of a run before or during execution.

   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”
   • [claimed-docs] “task_response = client.run_tasks_until_done(task_data)”

   proof ↗⚑ flag

   Sakana Marlin~partialclaimed4/10

   Marlin only allows a brief initial exchange to set direction before running fully autonomously for up to ~8 hours with no mid-run steering, and there's no documented control over depth/effort/scope parameters (e.g., report length, time budget, source breadth) beyond the initial topic framing; cancellation is possible but not adjustment. missing for 10: mid-execution steering controls, explicit depth/effort/scope parameters or settings, independent evidence of pre-run configurability beyond a 'brief exchange'.

   • [claimed-docs] “Sakana Marlin sharpens the direction of the investigation through a brief exchange with the user. Once the course is set, it works without f…”
   • [claimed-docs] “Give it a research topic, and Marlin works autonomously for up to roughly eight hours, crafting a detailed strategy report up to a hundred p…”
   • [claimed-docs] “You can cancel at any time during execution, but please note that credits will still be consumed.”
   • [claimed-docs] “It is designed for use cases where you want to delegate everything up to "issue framing (agenda setting)" prior to decision-making to the AI…”

   proof ↗⚑ flag

Citations

1. researcherSee citations for every substantive claim so I can verify it against the underlying source

   weight 3 · round to FutureHouse Platform
   FutureHouse Platform✓fullclaimed7/10

   Docs state that outputs are cited and auditable, with claims traceable back to specific literature passages or code, and that literature review/answer agents produce cited responses. This is first-party documentation only, with no independent/hands-on corroboration of citation accuracy or completeness. missing for 10: independent verification of citation accuracy, example of an actual cited report output, detail on citation format/traceability mechanism.

   • [claimed-docs] “Every conclusion is fully auditable. You can trace any finding back to the specific code or literature passage that produced it.”
   • [claimed-docs] “You can trace any finding back to the specific code or literature passage that produced it.”
   • [claimed-docs] “You can ask it a complex scientific question and get a high-accuracy, cited response, or task it with a deep literature review synthesizing …”
   • [claimed-docs] “Kosmos autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report.”
   • [claimed-docs] “Given a research objective and one or more datasets, Kosmos autonomously reads literature, writes and executes analysis code, generates hypo…”

   proof ↗⚑ flag

   Sakana Marlin~partialclaimed5/10

   Vendor docs claim Marlin generates 'fully referenced reports' grounded in primary sources with appendices and references, and a testimonial praises its higher-quality citations to primary vs secondary sources, suggesting citation support exists. However, there is no independent verification, no example of inline citation format, and no detail on how claims map to sources for auditability. Missing for 10: independent hands-on verification of citation accuracy, example output showing citation linking, and confirmation citations are traceable/clickable to primary sources.

   • [claimed-docs] “The research was of extremely high quality, grounded strictly in primary sources, resulting in a highly convincing and reliable final report…”
   • [claimed-docs] “他の生成AIと比較して、引用される情報の量と質が高く、二次情報ではなく一次情報を参照できている点に優位性を感じました。”
   • [claimed-docs] “From a fully referenced report to PowerPoint slides, everything is automatically generated — at a quality that stands alongside professional…”
   • [claimed-docs] “Everything from the main body to appendices, references, and presentation slides is automatically generated — at a quality that stands along…”

   proof ↗⚑ flag

Corpus

1. researcherSearch scholarly literature and primary sources, not just the open web

   weight 2 · round to FutureHouse Platform
   FutureHouse Platform✓fullclaimed8/10

   Docs describe agents (LITERATURE task, Kosmos, Precedent) that explicitly search scholarly literature, trials, and patents, accessing '175M+ papers, trials, and patents' with citation graph and journal quality awareness, and produce cited reports auditable back to specific literature passages. This directly matches the researcher story of searching scholarly/primary sources beyond the open web. Missing for 10: independent/hands-on corroboration of search coverage and source quality beyond vendor docs, and clarity on which specific databases/indices are used.

   • [claimed-docs] “It accesses 175M+ papers, trials, and patents with native understanding of citation graphs, journal quality, and clinical trial data.”
   • [claimed-docs] “Precedent determines whether a research idea has been tried before. It searches across fields to assess novelty and identify gaps”
   • [claimed-docs] “Precedent determines whether a research idea has been tried before. It searches across fields to assess novelty and identify gaps, helping y…”
   • [claimed-docs] “Kosmos autonomously reads literature, writes and executes analysis code, generates hypotheses, and produces a comprehensive cited report.”
   • [claimed-docs] “Every conclusion is fully auditable. You can trace any finding back to the specific code or literature passage that produced it.”
   • [claimed-docs] “task_data = { "name": JobNames.LITERATURE, "query": "Which neglected diseases had a treatment developed by artificial intelligence?"…”

   proof ↗⚑ flag

   Sakana Marlin~partialclaimed6/10

   Vendor docs claim research is 'grounded strictly in primary sources' and a testimonial notes it references primary rather than secondary information compared to other generative AI tools, but there is no evidence of dedicated scholarly database access (e.g., PubMed, arXiv, JSTOR integration) or citation verification—only general web navigation is described. missing for 10: evidence of scholarly/academic database integration, independent verification of primary-source claim, and details on how it distinguishes scholarly vs open-web sources.

   • [claimed-docs] “The research was of extremely high quality, grounded strictly in primary sources, resulting in a highly convincing and reliable final report…”
   • [claimed-docs] “他の生成AIと比較して、引用される情報の量と質が高く、二次情報ではなく一次情報を参照できている点に優位性を感じました。”
   • [claimed-docs] “Once you set a research topic, it works without further human input: repeatedly forming hypotheses, gathering information, navigating the we…”
   • [claimed-docs] “From ideation and data gathering to resolving contradictions and structuring the report, the AI autonomously executes the entire research pr…”

   proof ↗⚑ flag

Synthesis

1. analystSee where sources agree and disagree instead of a single unqualified answer

   weight 2 · round to FutureHouse Platform
   FutureHouse Platform~partialclaimed5/10

   Docs mention tasking the platform with 'a deep literature review synthesizing conflicting evidence across hundreds of papers' and full auditability back to source passages, suggesting some capability to surface disagreement across sources, but there is no dedicated UI/feature shown for explicitly contrasting agreeing vs disagreeing sources, no example output demonstrating this, and no independent/hands-on verification. missing for 10: concrete example output showing conflicting evidence highlighted, dedicated UI feature for source agreement/disagreement, independent corroboration beyond vendor docs.

   • [claimed-docs] “You can ask it a complex scientific question and get a high-accuracy, cited response, or task it with a deep literature review synthesizing …”
   • [claimed-docs] “task it with a deep literature review synthesizing conflicting evidence across hundreds of papers”
   • [claimed-docs] “Every conclusion is fully auditable. You can trace any finding back to the specific code or literature passage that produced it.”
   • [claimed-docs] “You can trace any finding back to the specific code or literature passage that produced it.”

   proof ↗⚑ flag

   Sakana Marlin~partialclaimed4/10

   Marketing copy claims Marlin 'resolves contradictions' and 'compares and evaluates multiple hypotheses' rather than merely aggregating, implying some handling of conflicting sources, but there is no evidence of a UI feature or report section that explicitly surfaces where sources agree/disagree to the analyst. Missing for 10: concrete example of a report showing conflicting source viewpoints, screenshot/description of how disagreement is presented, independent corroboration beyond vendor marketing.

   • [claimed-docs] “Once you set a research topic, it works without further human input: repeatedly forming hypotheses, gathering information, navigating the we…”
   • [claimed-docs] “From ideation and data gathering to resolving contradictions and structuring the report, the AI autonomously executes the entire research pr…”
   • [claimed-docs] “Rather than merely aggregating information, it compares and evaluates multiple hypotheses to provide deep insights.”

   proof ↗⚑ flag