Search Infrastructure arenaBuyer checklist
Every requirement we judge search infrastructure products against, as a ready-to-send RFP checklist — with each item's priority, why it matters, and how the top-ranked products score on it today.
53 requirements · 13 themes · verdicts for 5 products · updated 2026-09-16 · priorities mirror the story weights our scoring uses (methodology)
Show the markdown export
# Search Infrastructure — buyer checklist (RFP) Derived from ProductArena's evidence-graded user-story taxonomy for Search Infrastructure: 53 judged requirements. Priorities mirror story weights (3 = must-have, 2 = should-have, 1 = nice-to-have). ## Agent search - [ ] **[must-have]** My coding agent can create an index, add documents, and run queries end to end — through the API, CLI, or MCP without touching a dashboard - [ ] **[should-have]** Agents can use my search indexes as a tool — an MCP server or tool-calling surface exposes query, analytics, and index operations ## Agenticness - [ ] **[must-have]** Plug MCP servers into this product so it can use their tools - [ ] **[must-have]** Connect an agent via an official MCP server - [ ] **[must-have]** Drive the product through a documented public API - [ ] **[must-have]** Delegate tasks to a built-in AI assistant inside the product - [ ] **[should-have]** Point an agent at llms.txt or agent-oriented docs - [ ] **[should-have]** Run the product headlessly / in CI for automation - [ ] **[should-have]** Use an official CLI - [ ] **[should-have]** Issue scoped/least-privilege API credentials for an agent - [ ] **[should-have]** Build against official SDKs - [ ] **[should-have]** Subscribe to events via webhooks - [ ] **[should-have]** Get AI-generated insights and suggestions from my data inside the product - [ ] **[should-have]** Set up automations that run autonomously in the background - [ ] **[should-have]** Operate the product with natural-language commands - [ ] **[should-have]** Explore an interactive API reference with runnable examples - [ ] **[should-have]** Download a machine-readable API spec (OpenAPI or equivalent) - [ ] **[should-have]** Rely on versioned APIs with a documented deprecation policy - [ ] **[nice-to-have]** Test against a sandbox environment without touching production data ## Ai search - [ ] **[must-have]** Run hybrid search — semantic vector similarity fused with keyword matching — in a single query - [ ] **[should-have]** Use built-in or managed embedders so documents and queries are vectorized without running my own embedding pipeline - [ ] **[nice-to-have]** Power RAG and conversational answers on top of my indexes with documented retrieval or answer APIs ## Automation depth - [ ] **[must-have]** Define rules that trigger actions automatically on events - [ ] **[should-have]** Perform bulk operations across many items at once - [ ] **[should-have]** Schedule recurring jobs or workflows - [ ] **[nice-to-have]** Version, review, and roll back my automations ## Developer experience - [ ] **[must-have]** Create an index, add documents, and run my first search within minutes of starting the quickstart - [ ] **[should-have]** Official SDKs cover my language and framework, kept current across JavaScript, Python, PHP, Ruby, Go, and more - [ ] **[should-have]** Official UI component libraries let me assemble a full search interface — box, results, facets, pagination — without building it from scratch ## Indexing pipelines - [ ] **[should-have]** Bulk-import millions of documents quickly, with async task tracking to know when indexing completes - [ ] **[should-have]** Document adds, updates, and deletes become searchable in near real time without a full reindex - [ ] **[nice-to-have]** Ingest content with an official crawler or connectors instead of writing my own indexing pipeline ## Openness - [ ] **[must-have]** Export all of my data in open formats and leave - [ ] **[must-have]** Self-host the core product - [ ] **[should-have]** Do everything through the API that I can do in the UI - [ ] **[should-have]** Read the product's source under an open license ## Operations scale - [ ] **[must-have]** Self-host the full engine — same features as the hosted product — on my own infrastructure - [ ] **[should-have]** Built-in analytics show top queries, no-result queries, and click-through so I know what users search for and miss - [ ] **[should-have]** Documented scaling paths — clustering, replication, high availability — carry me from prototype to hundreds of millions of documents ## Pricing plans - [ ] **[must-have]** Published per-unit pricing (searches, records, or nodes) lets me predict what search will cost before committing - [ ] **[should-have]** Costs stay predictable as records and query volume grow — no surprise per-request cliffs ## Privacy posture - [ ] **[must-have]** Prevent my data from being used to train AI models - [ ] **[should-have]** Choose where my data is stored (region/residency) - [ ] **[should-have]** Control data retention and deletion - [ ] **[should-have]** Opt out of telemetry and usage tracking ## Relevance tuning - [ ] **[should-have]** Define synonyms and curate results — pin, boost, or hide specific hits for specific queries - [ ] **[should-have]** Shape relevance with custom ranking rules and business signals (popularity, recency, margin) beyond textual matching - [ ] **[nice-to-have]** Inspect ranking scores or explanations to understand exactly why a result ranked where it did ## Search experience - [ ] **[must-have]** Deliver as-you-type instant search with millisecond responses so results update on every keystroke - [ ] **[must-have]** Searches tolerate typos and misspellings out of the box, with tunable rules for when and how fuzzy matching applies - [ ] **[should-have]** Serve query suggestions and autocomplete backed by real search traffic or a suggestions index - [ ] **[should-have]** Build faceted navigation — filters with live counts across categories, ranges, and attributes — from a single query ## Security multitenancy - [ ] **[should-have]** Scoped or tenant tokens restrict each end user's searches to their own documents without separate indexes per user --- Source: https://ultrametric.ai/productarena/arena/search-infra (evidence-graded verdicts for 5 products) · methodology: https://ultrametric.ai/productarena/methodology
Chips show the top 5 ranked products' current verdict on each requirement — ✓ full · ~ partial · ! disputed · — none · n/a not applicable.
Agent search — stories about agent search in this arenaAgent search· 2 items
Stories about agent search in this arena
- must-have
ai-native userMy coding agent can create an index, add documents, and run queries end to end — through the API, CLI, or MCP without touching a dashboard
Core requirement — weighs 3× in arena scoring · 3 of 5 products fully deliver this today
- should-have
ai-native userAgents can use my search indexes as a tool — an MCP server or tool-calling surface exposes query, analytics, and index operations
Important, not disqualifying — weighs 2× in arena scoring · all 5 products fully deliver this today
Agenticness — how well agents can access and operate the productAgenticness· 17 items
How well agents can access and operate the product
- must-have
ai-native userPlug MCP servers into this product so it can use their tools
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- must-have
ai-native userConnect an agent via an official MCP server
Core requirement — weighs 3× in arena scoring · 4 of 5 products fully deliver this today
- must-have
ai-native userDrive the product through a documented public API
Core requirement — weighs 3× in arena scoring · 3 of 5 products fully deliver this today
- must-have
ai-native userDelegate tasks to a built-in AI assistant inside the product
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- should-have
ai-native userPoint an agent at llms.txt or agent-oriented docs
Important, not disqualifying — weighs 2× in arena scoring · 4 of 5 products fully deliver this today
- should-have
ai-native userRun the product headlessly / in CI for automation
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
- should-have
ai-native userUse an official CLI
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
- should-have
ai-native userIssue scoped/least-privilege API credentials for an agent
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
- should-have
ai-native userBuild against official SDKs
Important, not disqualifying — weighs 2× in arena scoring · 4 of 5 products fully deliver this today
- should-have
ai-native userSubscribe to events via webhooks
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userGet AI-generated insights and suggestions from my data inside the product
Important, not disqualifying — weighs 2× in arena scoring · 2 of 5 products fully deliver this today
- should-have
ai-native userSet up automations that run autonomously in the background
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userOperate the product with natural-language commands
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
- should-have
ai-native userExplore an interactive API reference with runnable examples
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userRely on versioned APIs with a documented deprecation policy
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- nice-to-have
ai-native userTest against a sandbox environment without touching production data
Differentiator, not a dealbreaker — weighs 1× in arena scoring · no product fully delivers this yet
Ai search — stories about ai search in this arenaAi search· 3 items
Stories about ai search in this arena
- must-have
developerRun hybrid search — semantic vector similarity fused with keyword matching — in a single query
Core requirement — weighs 3× in arena scoring · all 5 products fully deliver this today
- should-have
developerUse built-in or managed embedders so documents and queries are vectorized without running my own embedding pipeline
Important, not disqualifying — weighs 2× in arena scoring · 2 of 5 products fully deliver this today
- nice-to-have
developerPower RAG and conversational answers on top of my indexes with documented retrieval or answer APIs
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 2 of 5 products fully deliver this today
Automation depth — how much of the product can run unattendedAutomation depth· 4 items
How much of the product can run unattended
- must-have
ai-native userDefine rules that trigger actions automatically on events
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- should-have
ai-native userPerform bulk operations across many items at once
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userSchedule recurring jobs or workflows
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- nice-to-have
ai-native userVersion, review, and roll back my automations
Differentiator, not a dealbreaker — weighs 1× in arena scoring · no product fully delivers this yet
Developer experience — stories about developer experience in this arenaDeveloper experience· 3 items
Stories about developer experience in this arena
- must-have
developerCreate an index, add documents, and run my first search within minutes of starting the quickstart
Core requirement — weighs 3× in arena scoring · 2 of 5 products fully deliver this today
- should-have
developerOfficial SDKs cover my language and framework, kept current across JavaScript, Python, PHP, Ruby, Go, and more
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
- should-have
developerOfficial UI component libraries let me assemble a full search interface — box, results, facets, pagination — without building it from scratch
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
Indexing pipelines — stories about indexing pipelines in this arenaIndexing pipelines· 3 items
Stories about indexing pipelines in this arena
- should-have
platform-engineerBulk-import millions of documents quickly, with async task tracking to know when indexing completes
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
developerDocument adds, updates, and deletes become searchable in near real time without a full reindex
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- nice-to-have
founderIngest content with an official crawler or connectors instead of writing my own indexing pipeline
Differentiator, not a dealbreaker — weighs 1× in arena scoring · 1 of 5 products fully deliver this today
Openness — open source, data portability, and self-hosting storiesOpenness· 4 items
Open source, data portability, and self-hosting stories
- must-have
ai-native userExport all of my data in open formats and leave
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- must-have
ai-native userSelf-host the core product
Core requirement — weighs 3× in arena scoring · 4 of 5 products fully deliver this today
- should-have
ai-native userDo everything through the API that I can do in the UI
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userRead the product's source under an open license
Important, not disqualifying — weighs 2× in arena scoring · 2 of 5 products fully deliver this today
Operations scale — stories about operations scale in this arenaOperations scale· 3 items
Stories about operations scale in this arena
- must-have
platform-engineerSelf-host the full engine — same features as the hosted product — on my own infrastructure
Core requirement — weighs 3× in arena scoring · 1 of 5 products fully deliver this today
- should-have
founderBuilt-in analytics show top queries, no-result queries, and click-through so I know what users search for and miss
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
- should-have
platform-engineerDocumented scaling paths — clustering, replication, high availability — carry me from prototype to hundreds of millions of documents
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
Pricing plans — plan structure and value — what each tier costs and what it unlocksPricing plans· 2 items
Plan structure and value — what each tier costs and what it unlocks
- must-have
founderPublished per-unit pricing (searches, records, or nodes) lets me predict what search will cost before committing
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- should-have
founderCosts stay predictable as records and query volume grow — no surprise per-request cliffs
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
Privacy posture — data-handling and privacy storiesPrivacy posture· 4 items
Data-handling and privacy stories
- must-have
ai-native userPrevent my data from being used to train AI models
Core requirement — weighs 3× in arena scoring · no product fully delivers this yet
- should-have
ai-native userChoose where my data is stored (region/residency)
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userControl data retention and deletion
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
ai-native userOpt out of telemetry and usage tracking
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
Relevance tuning — stories about relevance tuning in this arenaRelevance tuning· 3 items
Stories about relevance tuning in this arena
- should-have
developerDefine synonyms and curate results — pin, boost, or hide specific hits for specific queries
Important, not disqualifying — weighs 2× in arena scoring · 3 of 5 products fully deliver this today
- should-have
developerShape relevance with custom ranking rules and business signals (popularity, recency, margin) beyond textual matching
Important, not disqualifying — weighs 2× in arena scoring · 3 of 5 products fully deliver this today
- nice-to-have
platform-engineerInspect ranking scores or explanations to understand exactly why a result ranked where it did
Differentiator, not a dealbreaker — weighs 1× in arena scoring · no product fully delivers this yet
Search experience — stories about search experience in this arenaSearch experience· 4 items
Stories about search experience in this arena
- must-have
developerDeliver as-you-type instant search with millisecond responses so results update on every keystroke
Core requirement — weighs 3× in arena scoring · 3 of 5 products fully deliver this today
- must-have
developerSearches tolerate typos and misspellings out of the box, with tunable rules for when and how fuzzy matching applies
Core requirement — weighs 3× in arena scoring · 3 of 5 products fully deliver this today
- should-have
developerServe query suggestions and autocomplete backed by real search traffic or a suggestions index
Important, not disqualifying — weighs 2× in arena scoring · no product fully delivers this yet
- should-have
developerBuild faceted navigation — filters with live counts across categories, ranges, and attributes — from a single query
Important, not disqualifying — weighs 2× in arena scoring · 4 of 5 products fully deliver this today
Security multitenancy — stories about security multitenancy in this arenaSecurity multitenancy· 1 item
Stories about security multitenancy in this arena
- should-have
developerScoped or tenant tokens restrict each end user's searches to their own documents without separate indexes per user
Important, not disqualifying — weighs 2× in arena scoring · 1 of 5 products fully deliver this today
Full evidence behind every verdict lives on the arena page and each product page — chips above deep-link straight to the judged story.