Fly.io vs Railway
Fly.io
Fly.io, Inc.
Railway wins · 23–24 (33 drawn)
Agenticness — how well agents can access and operate the productAgenticness
How well agents can access and operate the product
Agent access
ai-native userPoint an agent at llms.txt or agent-oriented docs
weight 2 · round to RailwayA direct probe confirms Fly.io serves a working llms.txt file at https://fly.io/llms.txt (HTTP 200) with structured documentation content, and Fly.io's docs also explicitly cater to agent-oriented workflows (e.g. Sprites, MCP server hosting). Missing for 10: no evidence of docs.md fallback or broader agent-oriented docs discoverability beyond the single llms.txt file, and no independent confirmation of agents actually consuming it successfully.
- [probe] “PROBE llms.txt: HTTP 200 at https://fly.io/llms.txt # Fly.io Documentation > Fly.io is a global public cloud for running full-stack and AI …”
- [claimed-docs] “Every agent gets its own Linux box. A Sprite checkpoints itself while the agent works, so harnesses like Claude and Codex find everything wh…”
- [claimed-docs] “Run every MCP server in its own Sprite, with persistent disk, and egress policy. Pay only for the tool calls you use.”
Railway hosts a live llms.txt confirmed by direct probe (HTTP 200) plus dedicated agent-oriented docs like agents.md and cloud-agents.md pages designed for AI assistants to consume. missing for 10: independent third-party confirmation that agents actually use these files successfully in practice.
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.railway.com/llms.txt # Railway Documentation > Railway is an all-in-one intelligent cloud provider…”
- [claimed-docs] “Install the Railway CLI and configure agent support — skills, MCP, and authentication — in one command.”
- [claimed-docs] “It lets AI assistants create projects, deploy templates, manage environments, pull variables, and redeploy services.”
- [claimed-docs] “The use-railway skill teaches AI coding agents how to operate Railway.”
ai-native userRun the product headlessly / in CI for automation
weight 2 · round to Fly.ioFly.io ships a scriptable CLI (flyctl) with single-line commands like `fly launch`, `fly machine run`, and `fly scale`, plus a full REST API for Machines that can provision and manage apps programmatically — both of which are usable headlessly in automation/CI pipelines. However, the evidence pack never explicitly documents CI integration (e.g., GitHub Actions, deploy tokens for automated pipelines) or shows a CI-specific workflow example. Missing for 10: explicit CI/CD integration docs (GitHub Actions, deploy tokens), independent hands-on confirmation of running flyctl in CI, and any automation-specific guide beyond general CLI/API docs.
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
- [claimed-docs] “The `fly machine run` command is a tool to configure, build, and start a new Machine in one line.”
- [claimed-docs] “The Fly Machines REST API provides resources to provision and manage Fly Apps, Fly Machines, and Fly Volumes.”
- [claimed-docs] “Fly Machines are fast-launching VMs with a simple REST API.”
- [probe] “official CLI documented at https://fly.io/docs/flyctl/”
Railway ships an official CLI for local development and deploying from the terminal, which implies scriptable/headless usage in automation contexts, but the evidence pack has no explicit documentation of CI/CD pipeline integration, GitHub Actions examples, or non-interactive auth tokens for CI. missing for 10: explicit CI/CD integration docs, non-interactive auth/token flow for automation, examples of headless CI usage.
- [claimed-docs] “Develop locally and deploy from your terminal with the Railway CLI.”
- [probe] “official CLI documented at https://docs.railway.com/guides/cli”
ai-native userConnect an agent via an official MCP server
weight 3 · round to RailwayFly.io ships an official, first-party MCP server via `flyctl mcp` that lets agents connect to Fly's platform, but the docs explicitly flag it as experimental with no independent corroboration of stability or broader agent-integration support. Missing for 10: evidence of production-readiness, hands-on validation from third parties, and details on capabilities/tools exposed beyond the experimental flag.
- [claimed-docs] “[experimental] Start a flyctl MCP server”
- [claimed-docs] “Fly Machines are tightly isolated VMs that are perfect for running MCP servers.”
Railway ships an official documented MCP server (docs.railway.com/ai/mcp-server.md) that lets AI assistants create projects, deploy templates, manage environments, pull variables, and redeploy services, plus a one-command CLI setup for agent/MCP/auth configuration and an 'use-railway' skill teaching agents to operate the platform. This is first-party, well-documented, and clearly targets AI-native agent connectivity. missing for 10: independent/hands-on third-party corroboration of the MCP server working in practice, and more detail on auth/security model for remote agent connections.
- [claimed-docs] “Build with Railway using Agent Skills and the MCP server for AI-powered workflows.”
- [claimed-docs] “Install the Railway CLI and configure agent support — skills, MCP, and authentication — in one command.”
- [claimed-docs] “It lets AI assistants create projects, deploy templates, manage environments, pull variables, and redeploy services.”
- [claimed-docs] “The use-railway skill teaches AI coding agents how to operate Railway.”
ai-native userUse an official CLI
weight 2 · round to RailwayFly.io ships flyctl, a fully documented official CLI (fly launch, fly machine run, fly scale, fly doctor, etc.) that is AI-native aware — it even has an experimental built-in MCP server mode (flyctl mcp) and the whole platform markets itself around agent workflows (Sprites for Claude/Codex harnesses). missing for 10: no independent/hands-on corroboration of the CLI's AI-native workflows specifically, and the MCP server mode is explicitly labeled experimental rather than GA.
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
- [claimed-docs] “Run diagnostics: `fly doctor` — checks WireGuard, IPs, and Docker.”
- [claimed-docs] “The `fly machine run` command is a tool to configure, build, and start a new Machine in one line.”
- [claimed-docs] “[experimental] Start a flyctl MCP server”
- [probe] “official CLI documented at https://fly.io/docs/flyctl/”
- [claimed-docs] “Every agent gets its own Linux box. A Sprite checkpoints itself while the agent works, so harnesses like Claude and Codex find everything wh…”
Railway ships a well-documented official CLI (docs-15, docs-18, probe-3) and explicitly extends it for AI-native workflows via a one-command agent setup that installs skills, MCP, and authentication (docs-19), making it directly usable by AI coding agents (docs-21, docs-16). missing for 10: independent/hands-on community confirmation of the CLI's AI-native agent setup specifically.
- [claimed-docs] “Develop locally and deploy from your terminal with the Railway CLI.”
- [claimed-docs] “Railway is an all-in-one intelligent cloud provider that makes it easy to provision infrastructure, develop locally, and deploy to the cloud…”
- [claimed-docs] “Install the Railway CLI and configure agent support — skills, MCP, and authentication — in one command.”
- [claimed-docs] “Build with Railway using Agent Skills and the MCP server for AI-powered workflows.”
- [claimed-docs] “The use-railway skill teaches AI coding agents how to operate Railway.”
- [probe] “official CLI documented at https://docs.railway.com/guides/cli”
ai-native userDrive the product through a documented public API
weight 3 · round drawnFly.io documents a public REST API ("Fly Machines REST API provides resources to provision and manage Fly Apps, Fly Machines, and Fly Volumes" and "fast-launching VMs with a simple REST API"), giving AI-native users a genuine programmatic interface beyond the CLI. However, probes show no discoverable OpenAPI/swagger spec at standard locations, and the primary documented interface for automation is flyctl (CLI) with only an experimental MCP wrapper, not a fully machine-consumable API spec. missing for 10: a published OpenAPI/swagger schema, independent developer corroboration of API robustness, and clarity on API versioning/stability guarantees.
- [claimed-docs] “The Fly Machines REST API provides resources to provision and manage Fly Apps, Fly Machines, and Fly Volumes.”
- [claimed-docs] “Fly Machines are fast-launching VMs with a simple REST API.”
- [claimed-docs] “Fly Machines are tightly isolated VMs that are perfect for running MCP servers.”
- [probe] “PROBE openapi: all candidate paths 404 (https://fly.io/openapi.json, https://fly.io/swagger.json, https://fly.io/api/openapi.json, https://f…”
- [claimed-docs] “[experimental] Start a flyctl MCP server”
Railway documents an MCP server and CLI that let AI agents create projects, deploy templates, manage environments, and redeploy services (railway-docs-20, railway-docs-15), which functions as a programmatic interface for AI-native drivers. However, probing for a standard public API spec (OpenAPI/Swagger) returned 404 on all candidate paths, indicating no conventional documented REST/GraphQL API surface (railway-probe-2). Missing for 10: a discoverable OpenAPI/GraphQL API reference, independent confirmation of API completeness beyond MCP tool coverage.
- [claimed-docs] “It lets AI assistants create projects, deploy templates, manage environments, pull variables, and redeploy services.”
- [claimed-docs] “Develop locally and deploy from your terminal with the Railway CLI.”
- [claimed-docs] “Build with Railway using Agent Skills and the MCP server for AI-powered workflows.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.railway.com/openapi.json, https://docs.railway.com/swagger.json, https://docs.railway.c…”
ai-native userIssue scoped/least-privilege API credentials for an agent
weight 2 · round to Fly.ioFly.io docs describe least-privilege credential scoping for agent sandboxes ('Sprites'): grant access by Sprite name, label, or single endpoint so an agent 'can post messages and never touch admin,' plus per-Sprite rotation/revocation without affecting others, and secrets encrypted in a vault. This directly matches issuing scoped, revocable credentials to an agent. Missing for 10: independent/hands-on corroboration of this scoping in practice, and no detail on underlying token/macaroon API mechanics.
- [claimed-docs] “Connect a service once for your whole org, and every Sprite can reach it. Nobody pastes a token.”
- [claimed-docs] “Grant access on purpose: by Sprite name, by label, or down to a single endpoint, so a Slack bot can post messages and never touch admin.”
- [claimed-docs] “Rotate once and every Sprite has the new credential. Cut off a single Sprite and the rest never notice.”
- [claimed-docs] “Secrets allow sensitive values, such as credentials, to be passed securely to your Fly App. The secret is encrypted and stored in a vault.”
Railwaynone0/10Evidence shows Railway offers an MCP server and CLI-based agent authentication (railway-docs-19, railway-docs-20) but nothing describes scoped or least-privilege credential issuance — the agent integration docs only mention broad capabilities (create projects, deploy templates, manage environments, pull variables) with no token-scoping or permission-tiering language. missing for 10: any mention of scoped/limited API tokens, role-based or read-only credentials, or documentation of least-privilege agent authentication.
- [claimed-docs] “Install the Railway CLI and configure agent support — skills, MCP, and authentication — in one command.”
- [claimed-docs] “It lets AI assistants create projects, deploy templates, manage environments, pull variables, and redeploy services.”
- [claimed-docs] “Railway prepares the environment and carries available coding-agent credentials from your machine. Your files stay on the agent's disk, and …”
ai-native userBuild against official SDKs
weight 2 · round drawnFly.ionone0/10Evidence shows Fly.io provides a CLI (flyctl), a REST API (Fly Machines API), and an experimental MCP server, but no official language SDKs (e.g., Python/JS/Go client libraries) are mentioned anywhere in the evidence pack; probes for openapi/SDK-related pages also failed (404s).
- [claimed-docs] “The Fly Machines REST API provides resources to provision and manage Fly Apps, Fly Machines, and Fly Volumes.”
- [probe] “PROBE openapi: all candidate paths 404 (https://fly.io/openapi.json, https://fly.io/swagger.json, https://fly.io/api/openapi.json, https://f…”
- [probe] “official CLI documented at https://fly.io/docs/flyctl/”
Railwaynone0/10Railway documents a CLI, MCP server, and agent skills for AI-driven workflows, but no evidence of official SDKs (client libraries) for building against Railway programmatically; a probe for OpenAPI/SDK specs returned 404s. Missing for 10: any first-party SDK (Python, Node, Go, etc.), API client library documentation, or SDK release notes.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.railway.com/openapi.json, https://docs.railway.com/swagger.json, https://docs.railway.c…”
- [claimed-docs] “Develop locally and deploy from your terminal with the Railway CLI.”
- [claimed-docs] “Build with Railway using Agent Skills and the MCP server for AI-powered workflows.”
ai-native userSubscribe to events via webhooks
weight 2 · round drawnFly.ionone0/10No evidence Fly.io offers a webhook subscription/event system for its own platform events (deploys, machine state changes, etc.); the only webhook-related item describes Fly.io as a place to receive/ingest incoming webhooks, not a source of subscribable events.
- [claimed-docs] “Fly.io converts Docker images to fast-booting VMs and runs them globally. This is great or ingesting incoming data!”
Agentic features
ai-native userSet up automations that run autonomously in the background
weight 2 · round to RailwayFly.io markets 'Sprites' that sleep until needed, checkpoint themselves, and wake up remembering state, plus Machine restart policies that keep processes running autonomously — a reasonable base for background agent automation (fly-io-docs-4, fly-io-docs-5, fly-io-docs-26). However, community feedback explicitly flags that Fly lacks a built-in scheduler: 'the missing bit in Fly is scheduled tasks... basic cron-like scheduling should be part of the platform' (fly-io-comm-20), meaning real automation triggers require external glue rather than a native mechanism. Missing for 10: native cron/scheduled-trigger support, and independent hands-on evidence that long-running agent automations are reliable in production (reliability complaints exist elsewhere in the evidence pack, though not specific to automation).
- [claimed-docs] “Every agent gets its own Linux box. A Sprite checkpoints itself while the agent works, so harnesses like Claude and Codex find everything wh…”
- [claimed-docs] “Give it a Sprite that sleeps until it's needed and wakes up remembering everything.”
- [claimed-docs] “The Machine restart policy defines whether and how flyd restarts a Machine after its main process exits.”
- [community] “For me the missing bit in Fly is scheduled tasks. I know how to solve this by spinning up an app that runs permanently as a scheduler, but b…”
Railway's Cloud Agents keep coding-agent machines running persistently in the background even after disconnecting, and the built-in Railway Agent can autonomously diagnose failures, create/configure services, and open PRs, plus condition-based Slack/Discord/email alerts run without manual triggering. However, evidence shows these are mostly agent-assisted or chat-triggered rather than fully scheduled/cron-style autonomous workflows, and there's no independent corroboration of unattended background automation reliability. Missing for 10: explicit scheduled/triggered automation framework (e.g. cron jobs), independent hands-on verification of autonomous background runs.
- [claimed-docs] “Railway prepares the environment and carries available coding-agent credentials from your machine. Your files stay on the agent's disk, and …”
- [claimed-docs] “The Railway Agent is a chat-based AI assistant built into the Railway dashboard. It can operate the platform on your behalf: creating and co…”
- [claimed-docs] “If the fix is in your code, the agent can open a pull request with the change so you can review and merge it.”
- [claimed-docs] “Slack, Discord, or emails the moment conditions you specify are met.”
ai-native userOperate the product with natural-language commands
weight 2 · round to RailwayFly.io exposes an experimental flyctl MCP server (docs-29) that lets AI agents/assistants issue natural-language-driven commands against the platform, and the broader Sprites/agent-hosting narrative (docs-4–13, docs-28) shows agent-first design. However, the primary interface remains the traditional flyctl CLI and fly.toml config (docs-1, docs-16, docs-27), with no dedicated natural-language command layer for end users themselves. Missing for 10: a first-party natural-language chat/command interface for operating Fly.io (not just an experimental MCP passthrough), and independent corroboration that the MCP server reliably interprets NL commands.
- [claimed-docs] “[experimental] Start a flyctl MCP server”
- [claimed-docs] “Give your agent a real computer and get back to building.”
- [claimed-docs] “Every agent gets its own Linux box. A Sprite checkpoints itself while the agent works, so harnesses like Claude and Codex find everything wh…”
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
Railway ships a chat-based 'Railway Agent' in-dashboard and via Slack (@Railway) that lets users create/configure services, inspect deployments, diagnose failures, and open PRs using natural language, plus an MCP server and Agent Skills for AI assistants to manage projects, deploy templates, and redeploy services conversationally. This directly matches the ai-native/agenticness story of operating the product via natural-language commands. Missing for 10: independent/hands-on validation of the Railway Agent's reliability and accuracy beyond first-party docs.
- [claimed-docs] “The Railway Agent is a chat-based AI assistant built into the Railway dashboard. It can operate the platform on your behalf: creating and co…”
- [claimed-docs] “If the fix is in your code, the agent can open a pull request with the change so you can review and merge it.”
- [claimed-docs] “Connect Notion, Linear, or Sentry from the catalog, or point Railway at any remote MCP server you run, and the agent can read from it while …”
- [claimed-docs] “Mention @Railway in Slack to ask the Railway Agent about your projects, inspect deployments, read logs, and make changes without leaving the…”
- [claimed-docs] “It lets AI assistants create projects, deploy templates, manage environments, pull variables, and redeploy services.”
- [claimed-docs] “Install the Railway CLI and configure agent support — skills, MCP, and authentication — in one command.”
- [claimed-docs] “The use-railway skill teaches AI coding agents how to operate Railway.”
Api quality
ai-native userExplore an interactive API reference with runnable examples
weight 2 · round drawnFly.ionone0/10No evidence of an interactive API reference or runnable examples; the openapi probe returned 404s across all candidate paths and docs only describe REST API resources in prose form.
- [claimed-docs] “The Fly Machines REST API provides resources to provision and manage Fly Apps, Fly Machines, and Fly Volumes.”
- [probe] “PROBE openapi: all candidate paths 404 (https://fly.io/openapi.json, https://fly.io/swagger.json, https://fly.io/api/openapi.json, https://f…”
Railwaynone0/10No evidence of an interactive API reference or runnable examples; the openapi probe explicitly returned 404s for all candidate spec paths, and no docs item mentions an API explorer or try-it-out console.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.railway.com/openapi.json, https://docs.railway.com/swagger.json, https://docs.railway.c…”
ai-native userDownload a machine-readable API spec (OpenAPI or equivalent)
weight 2 · round drawnFly.ionone0/10Fly.io documents a REST API for Machines (fly-io-docs-31) but explicit probes for OpenAPI/swagger spec files at standard locations all return 404 (fly-io-probe-3), and no downloadable machine-readable spec is referenced anywhere in the docs.
- [probe] “PROBE openapi: all candidate paths 404 (https://fly.io/openapi.json, https://fly.io/swagger.json, https://fly.io/api/openapi.json, https://f…”
- [claimed-docs] “The Fly Machines REST API provides resources to provision and manage Fly Apps, Fly Machines, and Fly Volumes.”
Railwaynone0/10The probe evidence explicitly shows all standard OpenAPI spec paths (openapi.json, swagger.json, etc.) return 404, and no docs mention a downloadable machine-readable API spec; only an llms.txt file and CLI/MCP docs are found. missing for 10: a hosted OpenAPI/Swagger spec, any documented machine-readable API schema.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.railway.com/openapi.json, https://docs.railway.com/swagger.json, https://docs.railway.c…”
- [probe] “PROBE llms.txt: HTTP 200 at https://docs.railway.com/llms.txt # Railway Documentation > Railway is an all-in-one intelligent cloud provider…”
ai-native userTest against a sandbox environment without touching production data
weight 1 · round to Fly.ioFly.io explicitly offers hardware-isolated, disposable VMs/Sprites with 'a clean baseline per request, so no two users ever share state,' plus checkpoint/rollback features that support spinning up isolated test environments without affecting production (fly-io-docs-7, fly-io-docs-9, fly-io-docs-4/5). However, this is all first-party marketing language with no independent or hands-on evidence confirming a documented staging/production separation workflow, and community threads focus on unrelated reliability complaints rather than sandbox isolation. Missing for 10: independent corroboration of sandbox isolation in practice, explicit staging-vs-production workflow documentation.
- [claimed-docs] “Run it in a hardware-isolated VM. A clean baseline per request, so no two users ever share state. Egress-locked and disposable.”
- [claimed-docs] “Roll back to any checkpoint in seconds”
- [claimed-docs] “Every agent gets its own Linux box. A Sprite checkpoints itself while the agent works, so harnesses like Claude and Codex find everything wh…”
- [claimed-docs] “Give it a Sprite that sleeps until it's needed and wakes up remembering everything.”
Railway supports isolated per-branch/PR preview environments and 'unlimited environments' for separate dev/staging vs production, which lets a user test changes without touching production (railway-docs-11, railway-docs-12). Cloud agents also run in isolated, ephemeral environments with their own disk (railway-docs-22). However, there is no explicit documentation of sandboxed test data, database seeding/copying, or safeguards preventing accidental production data access. Missing for 10: explicit sandbox/test-data isolation guidance, database cloning/seeding for previews, and independent confirmation that preview environments never touch production data.
- [claimed-docs] “Unlimited environments mean everyone can ship simultaneously.”
- [claimed-docs] “Every pull request gets its own preview. No surprises after merge.”
- [claimed-docs] “Railway prepares the environment and carries available coding-agent credentials from your machine. Your files stay on the agent's disk, and …”
ai-native userRely on versioned APIs with a documented deprecation policy
weight 2 · round drawnFly.ionone0/10No evidence of API versioning scheme or a documented deprecation policy; OpenAPI spec probes returned 404s and no docs mention versioning/deprecation commitments.
Railwaynone0/10No evidence of API versioning scheme or a documented deprecation policy; the openapi probe returned 404s on all candidate paths and no docs mention API versioning/deprecation commitments.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.railway.com/openapi.json, https://docs.railway.com/swagger.json, https://docs.railway.c…”
developerThe documented rate limits and concurrency caps for APIs and functions so I can plan for scale
weight 3 · round drawnFly.ionone0/10No evidence pack items document specific rate limits, concurrency caps, or throughput quotas for Fly.io's APIs, Machines, or MCP/Sprite offerings; docs focus on scaling primitives and pricing model, not documented numeric limits/caps for planning capacity.
Railwaynone0/10No evidence pack item documents rate limits or concurrency caps for Railway's APIs, MCP server, or functions; OpenAPI probe returned 404s and no rate-limit documentation is cited.
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.railway.com/openapi.json, https://docs.railway.com/swagger.json, https://docs.railway.c…”
Automation depth — how much of the product can run unattendedAutomation depth
How much of the product can run unattended
ai-native userPerform bulk operations across many items at once
weight 2 · round to Fly.ioFly.io exposes a REST API for provisioning/managing many Fly Apps, Machines, and Volumes (fly-io-docs-31) and CLI primitives like `fly machine run`/`fly scale` (fly-io-docs-27, fly-io-docs-24), which could be scripted for bulk actions, but there is no documented bulk/batch operation feature (e.g., a single command or endpoint acting on many resources at once) demonstrated in the evidence. missing for 10: explicit bulk/batch API endpoints or CLI flags for multi-resource operations, examples of scripted bulk workflows, independent confirmation of bulk automation use.
- [claimed-docs] “The Fly Machines REST API provides resources to provision and manage Fly Apps, Fly Machines, and Fly Volumes.”
- [claimed-docs] “The `fly machine run` command is a tool to configure, build, and start a new Machine in one line.”
- [claimed-docs] “the `fly scale` command for quick horizontal and vertical scaling”
Railwaynone0/10Railway's docs describe MCP/CLI actions on individual projects, deployments, and variables (e.g., create projects, deploy templates, manage environments, pull variables, redeploy services), but no evidence shows any bulk or batch operation across many items at once. Missing for 10: documentation of batch/bulk API calls, multi-service scripting examples, or CLI flags for operating on multiple resources simultaneously.
- [claimed-docs] “It lets AI assistants create projects, deploy templates, manage environments, pull variables, and redeploy services.”
- [probe] “official CLI documented at https://docs.railway.com/guides/cli”
ai-native userDefine rules that trigger actions automatically on events
weight 3 · round to RailwayFly.ionone0/10Fly.io's evidence only shows infrastructure-level triggers like Machine restart policies (restart on process exit) — not a general-purpose rules/automation engine for reacting to arbitrary events. Community feedback explicitly notes the platform lacks built-in scheduled/cron-like automation, requiring users to build their own scheduler apps, indicating no native event-rule automation capability exists.
- [claimed-docs] “The Machine restart policy defines whether and how flyd restarts a Machine after its main process exits.”
- [community] “For me the missing bit in Fly is scheduled tasks. I know how to solve this by spinning up an app that runs permanently as a scheduler, but b…”
Railway supports notification-style alerts ('Slack, Discord, or emails the moment conditions you specify are met') which is a basic trigger-action mechanism, and its Railway Agent/MCP ecosystem allows AI-driven actions like opening PRs on failed deployments, but there's no evidence of a general user-defined rules engine (custom conditions/triggers mapped to arbitrary actions) beyond alerting. missing for 10: a documented rules/automation engine letting users define arbitrary event-condition-action workflows, evidence of custom trigger configuration UI/API, and independent confirmation of reliability of these triggers.
- [claimed-docs] “Slack, Discord, or emails the moment conditions you specify are met.”
- [claimed-docs] “The Railway Agent is a chat-based AI assistant built into the Railway dashboard. It can operate the platform on your behalf: creating and co…”
- [claimed-docs] “If the fix is in your code, the agent can open a pull request with the change so you can review and merge it.”
ai-native userSchedule recurring jobs or workflows
weight 2 · round drawnFly.ionone0/10No documentation evidence shows native recurring job/cron scheduling on Fly.io; a community comment explicitly states scheduled tasks are missing from the platform and must be built manually.
- [community] “For me the missing bit in Fly is scheduled tasks. I know how to solve this by spinning up an app that runs permanently as a scheduler, but b…”
Railwaynone0/10The evidence pack covers deployments, scaling, environments, alerts, and AI agent integrations, but contains no mention of cron jobs, scheduled tasks, or recurring workflow execution capabilities. Missing for 10: any documentation of a cron/scheduler feature, recurring job configuration, or workflow automation triggers.
ai-native userVersion, review, and roll back my automations
weight 1 · round to RailwayFly.io documents checkpoint-based rollback for its Sprite/agent runtime ('Roll back to any checkpoint in seconds', 'A Sprite checkpoints itself while the agent works'), which covers rollback of agent state, but there is no evidence of version history, diffing, or a review workflow for automations themselves. missing for 10: version history/diff view, review workflow before rollback, independent hands-on confirmation of checkpoint rollback working reliably
- [claimed-docs] “Roll back to any checkpoint in seconds”
- [claimed-docs] “Every agent gets its own Linux box. A Sprite checkpoints itself while the agent works, so harnesses like Claude and Codex find everything wh…”
- [claimed-docs] “Give it a Sprite that sleeps until it's needed and wakes up remembering everything.”
Railway explicitly supports git-based versioning of deployments, per-PR preview reviews before merge, and instant rollback to any previous version, and its Railway Agent can open pull requests for review before changes are merged — covering version/review/rollback of automations end-to-end. missing for 10: independent/hands-on corroboration that rollback and PR-review flows work reliably in practice (community evidence focuses on platform outages, not rollback failures).
- [claimed-docs] “Every pull request gets its own preview. No surprises after merge.”
- [claimed-docs] “Rollback to any previous version instantly when something breaks.”
- [claimed-docs] “Unlimited environments mean everyone can ship simultaneously.”
- [claimed-docs] “The Railway Agent is a chat-based AI assistant built into the Railway dashboard. It can operate the platform on your behalf: creating and co…”
- [claimed-docs] “If the fix is in your code, the agent can open a pull request with the change so you can review and merge it.”
Deploy workflow — the commit-to-production path — builds, previews, rollbacksDeploy workflow
The commit-to-production path — builds, previews, rollbacks
Cli tooling
developerDeploy my project directly from the command line without using a web dashboard
weight 2 · round to Fly.ioFly.io's core deploy workflow is CLI-first via flyctl: `fly launch` creates, configures, and deploys apps directly from the project directory, with additional CLI commands (`fly machine run`, `fly scale`, `fly doctor`) covering the full deploy lifecycle without needing a dashboard. Missing for 10: independent hands-on confirmation that dashboard is never required for any step (community evidence focuses on reliability, not on CLI-vs-dashboard workflow).
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
- [claimed-docs] “The `fly machine run` command is a tool to configure, build, and start a new Machine in one line.”
- [claimed-docs] “the `fly scale` command for quick horizontal and vertical scaling”
- [probe] “official CLI documented at https://fly.io/docs/flyctl/”
Railway documents a CLI that lets developers develop locally and deploy from the terminal, with a dedicated CLI guide page confirming this is an official, supported workflow. missing for 10: no independent/hands-on corroboration of CLI deploy experience, and no detail on full command coverage (e.g., env management, rollbacks) via CLI alone versus dashboard.
- [claimed-docs] “Develop locally and deploy from your terminal with the Railway CLI.”
- [claimed-docs] “Railway is an all-in-one intelligent cloud provider that makes it easy to provision infrastructure, develop locally, and deploy to the cloud…”
- [probe] “official CLI documented at https://docs.railway.com/guides/cli”
developerScaffold and deploy a new project with a single CLI command
weight 2 · round to Fly.ioFirst-party docs explicitly describe `fly launch` as a single command run from the project directory that creates, configures, and deploys a new app, matching the story precisely (fly-io-docs-16, fly-io-docs-1). However, there is no independent/hands-on confirmation of this specific scaffold-and-deploy flow, and several community reports describe general deploy flakiness on the platform (fly-io-comm-5, fly-io-comm-19) that add some uncertainty to the smoothness of the workflow. missing for 10: independent hands-on verification of `fly launch` succeeding cleanly, and resolution of community reports of deploy failures.
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [community] “We tried to migrate all our staging environments to Fly last year but it was the flakiest experience I've experienced on any PaaS. Pushing s…”
- [community] “Fly has a nice DX but unfortunately the platform still has major issues with reliability and networking. There are often problems with deplo…”
Docs confirm a dedicated CLI for local development and terminal deploys, plus one-click templates/starters, but no evidence details a single command that both scaffolds a new project and deploys it end-to-end. missing for 10: explicit CLI command sequence (e.g. init+up) demonstrating one-command scaffold-and-deploy, independent hands-on confirmation of speed/simplicity.
- [claimed-docs] “Develop locally and deploy from your terminal with the Railway CLI.”
- [claimed-docs] “One-click deployable applications and starters for common use cases.”
- [probe] “official CLI documented at https://docs.railway.com/guides/cli”
developerDevelop and test my app locally using a simulator that matches the production runtime
weight 2 · round to RailwayFly.ionone0/10The evidence pack covers flyctl launch/deploy workflow, Fly Machines, Sprites, and MCP server support, but nowhere mentions a local simulator or emulator that replicates the production runtime for local development/testing. This is a reasonable capability to expect from a deploy platform, but no such tool (e.g., local Machines emulator) is documented.
Railway's CLI lets developers 'develop locally and deploy from your terminal' and inject production-like environment variables, but there's no evidence of an actual local simulator/emulator that mirrors the production runtime (e.g., container parity, network topology, or service mesh simulation). Missing for 10: documentation of a local runtime emulator, evidence of environment parity beyond env-var injection, and any independent confirmation that local dev closely mirrors production behavior.
- [claimed-docs] “Develop locally and deploy from your terminal with the Railway CLI.”
- [probe] “official CLI documented at https://docs.railway.com/guides/cli”
Deployment strategies
developerDeploy applications packaged as Docker containers
weight 2 · round to Fly.ioFly.iodisputedcontradicted5/10Fly.io's core docs clearly describe Docker-based deployment (fly launch/fly deploy converting Docker images into VMs, fly machine run, fly doctor checking Docker) [fly-io-docs-1, fly-io-docs-16, fly-io-docs-27, fly-io-docs-32, fly-io-docs-22], but hands-on community reports describe container deploys failing at high rates with unclear errors and stuck/unreachable instances, directly contradicting a smooth deploy workflow [fly-io-comm-5, fly-io-comm-19]. missing for 10: independent verification that Docker deploys work reliably at scale, and resolution of documented deploy failure reports.
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
- [claimed-docs] “The `fly machine run` command is a tool to configure, build, and start a new Machine in one line.”
- [claimed-docs] “Fly.io converts Docker images to fast-booting VMs and runs them globally. This is great or ingesting incoming data!”
- [claimed-docs] “Run diagnostics: `fly doctor` — checks WireGuard, IPs, and Docker.”
- [community] “We tried to migrate all our staging environments to Fly last year but it was the flakiest experience I've experienced on any PaaS. Pushing s…”
- [community] “Fly has a nice DX but unfortunately the platform still has major issues with reliability and networking. There are often problems with deplo…”
Railwaynone0/10The evidence pack describes Railway's git-based auto-config, canvas UI, CLI, and AI agent features, but no citation explicitly confirms deploying pre-built Docker container images or Dockerfiles as a supported workflow. missing for 10: explicit docs on Dockerfile/Docker image deployment support, evidence of container registry integration.
devops-leadRoll out new deployments incrementally to reduce blast radius
weight 2 · round drawnFly.ionone0/10The evidence pack shows generic deploy commands (`fly launch`, `fly machine run`, `fly scale`) and rollback/checkpoint features for Sprites, but nothing documents incremental rollout mechanisms like canary, rolling, or blue-green deployment strategies for reducing blast radius. Deployment platforms commonly support this, so absence of evidence means the axis is unmet rather than inapplicable.
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
- [claimed-docs] “The `fly machine run` command is a tool to configure, build, and start a new Machine in one line.”
- [claimed-docs] “the `fly scale` command for quick horizontal and vertical scaling”
- [claimed-docs] “The Machine restart policy defines whether and how flyd restarts a Machine after its main process exits.”
Git integration
developerAutomatically deploy my app on every Git push
weight 3 · round to RailwayFly.ionone0/10Fly.io's docs describe CLI-based deploys (`fly launch`, `fly deploy`, `fly machine run`) but nothing in the evidence pack mentions Git integration, GitHub Actions, or any automatic deploy-on-push workflow. This is a fair axis for a PaaS like Fly.io, but the evidence shows only manual/CLI-driven deploys, missing for 10: any mention of Git-triggered CI/CD, GitHub App integration, or auto-deploy hooks.
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
- [claimed-docs] “The `fly machine run` command is a tool to configure, build, and start a new Machine in one line.”
Railway's docs explicitly describe connecting a repo for auto-config deploys, per-PR preview environments, and unlimited environments so pushes deploy simultaneously, which directly covers git-push-triggered deployment workflows. Missing for 10: independent hands-on confirmation of push-to-deploy behavior and explicit mention of branch-based auto-deploy configuration options.
- [claimed-docs] “Connect your repo, Railway handles the rest. Auto-config, instant previews, no new tools to learn.”
- [claimed-docs] “Every pull request gets its own preview. No surprises after merge.”
- [claimed-docs] “Unlimited environments mean everyone can ship simultaneously.”
- [claimed-docs] “Railway is an all-in-one intelligent cloud provider that makes it easy to provision infrastructure, develop locally, and deploy to the cloud…”
developerDeploy my app by simply connecting my repo without manual build configuration
weight 3 · round to RailwayFly.io's `fly launch` auto-detects app type and generates config/deploys with minimal manual setup (fly-io-docs-1, fly-io-docs-16), but this is a CLI-driven flow from a local project directory, not a GitHub-repo-connect auto-deploy UX like Vercel/Heroku. No evidence of a 'connect your repo and we build/deploy automatically' web integration. Community reports also cite frequent build/deploy failures undermining the 'no manual configuration' promise (fly-io-comm-5, fly-io-comm-19). Missing for 10: native repo-connect (GitHub App/webhook) auto-deploy flow, evidence of zero manual config across frameworks, and independent confirmation deploys succeed reliably without intervention.
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
- [community] “We tried to migrate all our staging environments to Fly last year but it was the flakiest experience I've experienced on any PaaS. Pushing s…”
- [community] “Fly has a nice DX but unfortunately the platform still has major issues with reliability and networking. There are often problems with deplo…”
Railway's docs explicitly state that connecting a repo triggers auto-configuration with no manual setup needed, YAML being optional, and instant PR previews without extra tooling — directly matching the story. Community evidence discusses reliability/outages but does not contradict the repo-connect/auto-config workflow itself. Missing for 10: independent hands-on developer confirmation of zero-config deploys and more detail on edge cases where manual config is still required.
- [claimed-docs] “Connect your repo, Railway handles the rest. Auto-config, instant previews, no new tools to learn.”
- [claimed-docs] “Modify settings directly from the canvas. YAML optional.”
- [claimed-docs] “Every pull request gets its own preview. No surprises after merge.”
- [claimed-docs] “Railway is an all-in-one intelligent cloud provider that makes it easy to provision infrastructure, develop locally, and deploy to the cloud…”
Infra as code
devops-leadDefine, deploy, and version my entire architecture with a single configuration file
weight 2 · round to Fly.ioFly.io's `fly launch` workflow creates, configures, and deploys an app from a single project directory (fly-io-docs-16, fly-io-docs-1), and `fly machine run` further shows one-line create+deploy semantics (fly-io-docs-27), implying a single fly.toml-style config drives the whole architecture. However the evidence never explicitly names or documents the fly.toml config schema, nor shows how that file is versioned/rolled back (the 'roll back to any checkpoint' claim, fly-io-docs-9, refers to Sprite/agent checkpoints, not app config versioning), and 'fly scale' (fly-io-docs-24) is a separate CLI action rather than config-driven scaling. Missing for 10: explicit fly.toml schema/documentation, config-file-based versioning or rollback, multi-service/region architecture defined in one file, independent hands-on confirmation.
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “The `fly machine run` command is a tool to configure, build, and start a new Machine in one line.”
- [claimed-docs] “the `fly scale` command for quick horizontal and vertical scaling”
- [claimed-docs] “Roll back to any checkpoint in seconds”
- [claimed-docs] “The Machine restart policy defines whether and how flyd restarts a Machine after its main process exits.”
Railway's docs mention that settings can be modified via YAML ("YAML optional") alongside a visual canvas, and rollback/versioning of deployments is supported, hinting at config-as-code support, but there is no evidence of a single manifest file that defines and version-controls an entire multi-service architecture (e.g., a documented railway.json/toml schema covering networking, environments, and services in one file). missing for 10: explicit single-file architecture schema/manifest documentation, examples of defining multi-service topology in one file, and confirmation that this file is the primary/versioned source of truth rather than the canvas.
- [claimed-docs] “Modify settings directly from the canvas. YAML optional.”
- [claimed-docs] “Rollback to any previous version instantly when something breaks.”
- [claimed-docs] “Unlimited environments mean everyone can ship simultaneously.”
Manual deploy
developerDeploy a project by uploading a folder of files directly, with no Git repo required and the new deployment kept private until I choose to publish it
weight 2 · round to Fly.ioFly.io's flyctl/fly launch and fly deploy explicitly work from a local project directory without requiring a Git repository (fly-io-docs-1, fly-io-docs-16), satisfying the 'no Git repo required, folder upload' part of the story. However, there is no evidence of a staging/private mode where a new deployment is kept private until the developer chooses to publish it — Fly apps get a live URL immediately upon deploy. Missing for 10: documentation of a private/draft deployment state, a publish/promote step, or preview-URL gating before public exposure.
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
Railwaynone0/10Evidence shows Railway's primary deploy paths are Git-repo connection and CLI/local deploy, but nothing describes a folder-upload deploy flow or a private-until-published toggle for a new deployment.
- [claimed-docs] “Connect your repo, Railway handles the rest. Auto-config, instant previews, no new tools to learn.”
- [claimed-docs] “Develop locally and deploy from your terminal with the Railway CLI.”
- [probe] “official CLI documented at https://docs.railway.com/guides/cli”
Preview envs
developerGet ephemeral full-stack preview environments for every pull request
weight 2 · round to RailwayFly.ionone0/10Evidence shows Fly.io provides general app deployment (fly launch, Machines, scaling) but no dedicated feature or integration for automatic ephemeral preview environments tied to pull requests (e.g., GitHub PR bot, per-PR subdomains, auto-teardown). Missing for 10: PR-triggered environment creation, GitHub Actions/PR integration, automatic teardown on merge/close, full-stack (app+DB) preview provisioning per PR.
Railway's docs explicitly state every PR gets its own preview environment with instant, auto-configured full-stack previews and unlimited environments for parallel work. missing for 10: independent/hands-on verification of PR preview behavior at scale and details on environment teardown/ephemerality lifecycle.
- [claimed-docs] “Connect your repo, Railway handles the rest. Auto-config, instant previews, no new tools to learn.”
- [claimed-docs] “Unlimited environments mean everyone can ship simultaneously.”
- [claimed-docs] “Every pull request gets its own preview. No surprises after merge.”
developerGet an automatic preview URL for every pull request
weight 3 · round to RailwayFly.ionone0/10No evidence pack mentions PR-preview environments, GitHub integration, or automatic preview URLs for pull requests; only deployment via flyctl/fly launch is documented.
Railway docs explicitly state "Every pull request gets its own preview. No surprises after merge" and marketing highlights "instant previews" upon connecting a repo, directly matching the story. Missing for 10: independent/hands-on confirmation of PR preview URLs working in practice and details on configuration/limitations.
- [claimed-docs] “Every pull request gets its own preview. No surprises after merge.”
- [claimed-docs] “Connect your repo, Railway handles the rest. Auto-config, instant previews, no new tools to learn.”
Rollback recovery
developerRoll back a deployed service to any previous version in one click
weight 3 · round to RailwayFly.io's marketing copy claims 'Roll back to any checkpoint in seconds' (fly-io-docs-9), which suggests rollback capability, but this appears tied to Sprites (AI-agent sandboxes) checkpointing rather than a documented one-click rollback for standard deployed services/releases. No CLI docs (e.g., 'fly releases rollback') or hands-on/community confirmation of this exact deploy-workflow capability are present. Missing for 10: explicit docs on rolling back a deployed app/service version via CLI or dashboard, a one-click UI flow, and independent verification that it works as described for standard services.
- [claimed-docs] “Roll back to any checkpoint in seconds”
- [probe] “official CLI documented at https://fly.io/docs/flyctl/”
Railway's own docs explicitly claim 'Rollback to any previous version instantly when something breaks,' directly matching the story, but this is a single marketing-style claim with no independent/hands-on corroboration or detail on the one-click UX flow. Community evidence discusses platform stability incidents but does not specifically test or contradict rollback functionality. missing for 10: independent/hands-on verification of the rollback flow, documentation detail on how rollback is triggered (UI steps), confirmation it works across all deploy types.
- [claimed-docs] “Rollback to any previous version instantly when something breaks.”
Templates
developerDeploy one-click templates and starters for common use cases
weight 1 · round to RailwayFly.ionone0/10Evidence shows `fly launch` scaffolding/deploying from an existing source directory and generic tech-specific docs, but there is no evidence of a curated one-click template/starter gallery for common use cases (e.g., a marketplace of pre-built app templates).
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “Check out the docs specific to your tech so you can move faster.”
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
Docs explicitly state "One-click deployable applications and starters for common use cases" and the platform overall emphasizes quick repo-to-deploy workflows, supporting a template/starter deploy story. Missing for 10: independent hands-on verification of the template gallery experience and details on breadth/quality of available templates.
- [claimed-docs] “One-click deployable applications and starters for common use cases.”
- [claimed-docs] “Railway is an all-in-one intelligent cloud provider that makes it easy to provision infrastructure, develop locally, and deploy to the cloud…”
Dev experience — day-to-day developer experience — setup friction, docs, debugging, iteration speedDev experience
Day-to-day developer experience — setup friction, docs, debugging, iteration speed
Environments
developerSpin up unlimited isolated environments so my whole team can ship simultaneously
weight 1 · round to RailwayFly.io's Machines/Sprites provide fast-launching, hardware-isolated VMs that can scale out to thousands of instances, and features like per-agent Sprites (docs-4/5/6) and fast `fly machine run`/`fly launch` (docs-16/27) support spinning up many isolated environments for parallel work. However, there's no explicit 'unlimited' guarantee, and multiple independent hands-on reports describe reliability problems (stuck/unreachable machines, flaky deploys, outages affecting staging/production) that would undermine a team's ability to reliably ship simultaneously at scale. Missing for 10: an explicit 'unlimited' or team-collaboration framing, and stronger independent corroboration that concurrent environment creation is reliable at scale.
- [claimed-docs] “Every agent gets its own Linux box. A Sprite checkpoints itself while the agent works, so harnesses like Claude and Codex find everything wh…”
- [claimed-docs] “Give it a Sprite that sleeps until it's needed and wakes up remembering everything.”
- [claimed-docs] “Run every MCP server in its own Sprite, with persistent disk, and egress policy. Pay only for the tool calls you use.”
- [claimed-docs] “Run it in a hardware-isolated VM. A clean baseline per request, so no two users ever share state. Egress-locked and disposable.”
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
- [claimed-docs] “The `fly machine run` command is a tool to configure, build, and start a new Machine in one line.”
- [community] “We tried to migrate all our staging environments to Fly last year but it was the flakiest experience I've experienced on any PaaS. Pushing s…”
- [community] “Fly has a nice DX but unfortunately the platform still has major issues with reliability and networking. There are often problems with deplo…”
Railway docs explicitly claim unlimited environments enabling simultaneous team shipping, backed by per-PR preview environments and instant rollback, directly matching the story. Missing for 10: independent/hands-on corroboration of environment isolation at scale and no detail on limits/costs for large teams.
- [claimed-docs] “Unlimited environments mean everyone can ship simultaneously.”
- [claimed-docs] “Every pull request gets its own preview. No surprises after merge.”
- [claimed-docs] “Rollback to any previous version instantly when something breaks.”
Scaling continuity
developerRely on the same core APIs and primitives from prototype through massive scale, without rearchitecting or migrating platforms as usage grows
weight 2 · round to Fly.ioFly.io explicitly claims consistent primitives across scale ("Keep the same primitives at every scale... The API you start with is the API you keep") and backs this with Fly Machines' fast-boot VM API, `fly scale`, and a unified REST API for Apps/Machines/Volumes usable from prototype to production. However, this is a first-party marketing claim with no independent case study or benchmark showing a real app scaling from prototype to 'massive scale' on the same primitives without changes. Missing for 10: independent verification of large-scale deployments retaining the original API/architecture, and case studies documenting zero-rearchitecture growth.
- [claimed-docs] “Keep the same primitives at every scale: no rearchitecting when you grow, and no platform migration when you get big. The API you start with…”
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “the `fly scale` command for quick horizontal and vertical scaling”
- [claimed-docs] “The Fly Machines REST API provides resources to provision and manage Fly Apps, Fly Machines, and Fly Volumes.”
- [claimed-docs] “Fly Machines are fast-launching VMs with a simple REST API.”
Railwaydisputedcontradicted4/10Railway's docs claim seamless scaling via the same primitives (scale CPU/RAM, add replicas, multi-region deploy) without new tooling, but independent community reports describe a platform-wide outage that took down the entire service and raised doubts about stability and trust at scale, directly undercutting the 'same core APIs from prototype to massive scale' promise. Missing for 10: first-party case studies of large-scale customers not hitting limits, and resolution of the stability concerns raised in the outage thread.
- [claimed-docs] “Scale CPU and RAM or add replicas with load balancing.”
- [claimed-docs] “Run your application closer to where your users are.”
- [claimed-docs] “Railway is an all-in-one intelligent cloud provider that makes it easy to provision infrastructure, develop locally, and deploy to the cloud…”
- [community] “Sorry, I have a hard time blaming Google for this, when Railway seems to be having increasing trouble keeping the platform stable. Something…”
- [community] “This is bad. Even their own website is down at railway.com. Looks like total dependency on google cloud. Surprising for a company of their s…”
- [community] “Railway is back, but I'm not sure if I can trust keeping my projects there, so I'm going to migrate to another company.”
Secrets management
devops-leadManage team-level access, secrets, and environment variables centrally
weight 2 · round to Fly.ioFly.io docs show centralized secrets management (encrypted vault) and org-wide connection/credential rotation with granular access grants by Sprite/label, which covers secrets and env-var centralization. However there is no evidence of team-member role management, invites, or permission tiers for a devops-lead to administer team access explicitly. Missing for 10: team/user role management (RBAC), member invite/permission workflows, and independent corroboration of the org-wide credential features actually working at scale.
- [claimed-docs] “Secrets allow sensitive values, such as credentials, to be passed securely to your Fly App. The secret is encrypted and stored in a vault.”
- [claimed-docs] “Connect a service once for your whole org, and every Sprite can reach it. Nobody pastes a token.”
- [claimed-docs] “Grant access on purpose: by Sprite name, by label, or down to a single endpoint, so a Slack bot can post messages and never touch admin.”
- [claimed-docs] “Rotate once and every Sprite has the new credential. Cut off a single Sprite and the rest never notice.”
Evidence only hints at environment management (unlimited environments, 'pull variables' via MCP) but never documents team roles/permissions, secrets vaults, or centralized access control for a devops-lead. Missing for 10: explicit team/role-based access control docs, shared secrets management features, and independent confirmation of these enterprise controls.
- [claimed-docs] “Unlimited environments mean everyone can ship simultaneously.”
- [claimed-docs] “It lets AI assistants create projects, deploy templates, manage environments, pull variables, and redeploy services.”
Tool continuity
developerKeep using my existing Git, CI, editor, and framework workflows without adopting proprietary tooling
weight 2 · round to RailwayFly.io explicitly supports deploying arbitrary Docker images and offers framework-specific guides, implying existing frameworks and build tooling continue to work rather than requiring a proprietary runtime (fly-io-docs-32, fly-io-docs-2, fly-io-docs-25). However, developers must still adopt the flyctl CLI and fly.toml config to deploy (fly-io-docs-1, fly-io-docs-16), and there is no evidence of native Git-push deploy, first-class CI/CD templates, or editor integrations. missing for 10: explicit CI/CD pipeline integration evidence, git-based deploy workflow, editor/IDE integration, confirmation that no Fly-specific code changes are needed for common frameworks.
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “Check out the docs specific to your tech so you can move faster.”
- [claimed-docs] “Run `fly launch` from inside your project source directory to create, configure, and (for most apps) deploy a new application.”
- [claimed-docs] “Fly.io makes it easier to run an Elixir cluster on multiple servers around the world and deploy to those regions with a single command.”
- [claimed-docs] “Fly.io converts Docker images to fast-booting VMs and runs them globally. This is great or ingesting incoming data!”
Railway's docs show it connects directly to existing Git repos with auto-config, PR-based preview environments, and a CLI for local development/deploy — i.e., no proprietary editor or framework required (railway-docs-1, railway-docs-12, railway-docs-15, railway-docs-17). However, config is presented as best done via a proprietary visual 'canvas' (YAML optional but canvas-first), and there's no explicit evidence of compatibility with external CI pipelines (e.g., GitHub Actions) or independent developer confirmation of a friction-free existing-workflow experience. missing for 10: evidence of CI-tool interoperability, independent/hands-on confirmation that no Railway-specific tooling is required, editor-specific integration details.
- [claimed-docs] “Connect your repo, Railway handles the rest. Auto-config, instant previews, no new tools to learn.”
- [claimed-docs] “Every pull request gets its own preview. No surprises after merge.”
- [claimed-docs] “Develop locally and deploy from your terminal with the Railway CLI.”
- [claimed-docs] “One-click deployable applications and starters for common use cases.”
- [claimed-docs] “Modify settings directly from the canvas. YAML optional.”
Edge compute — running code at the edge — regions, cold starts, runtime limitsEdge compute
Running code at the edge — regions, cold starts, runtime limits
Agent runtime
ai-native builderGive my coding agent a persistent runtime environment that checkpoints its state automatically while working
weight 2 · round to Fly.ioFly.io's docs explicitly describe 'Sprites' as persistent agent runtimes that checkpoint themselves automatically and can be rolled back to any checkpoint, directly matching the story (fly-io-docs-4, fly-io-docs-5, fly-io-docs-9, fly-io-docs-28). This is a first-party marketing description with no independent hands-on verification of the Sprite checkpointing mechanism specifically, and general community reliability complaints (unrelated to Sprites) raise some doubt about production robustness. Missing for 10: independent/hands-on corroboration of Sprite checkpoint/restore behavior, and detail on failure modes or reliability of the checkpointing feature itself.
- [claimed-docs] “Every agent gets its own Linux box. A Sprite checkpoints itself while the agent works, so harnesses like Claude and Codex find everything wh…”
- [claimed-docs] “Give it a Sprite that sleeps until it's needed and wakes up remembering everything.”
- [claimed-docs] “Roll back to any checkpoint in seconds”
- [claimed-docs] “Give your agent a real computer and get back to building.”
Railway's cloud-agents feature explicitly provides a persistent runtime: the agent's machine 'stays running when you disconnect' and files persist on a dedicated disk, with coding-agent credentials carried over (railway-docs-22, 23, 24). However, there is no evidence of automatic state checkpointing/snapshotting of the agent's work — Railway's rollback feature (railway-docs-13) applies to deployment versions, not agent runtime state. Missing for 10: explicit automatic checkpoint/snapshot mechanism for agent state, evidence of restore-from-checkpoint after crash or disconnect, and independent corroboration of persistence reliability.
- [claimed-docs] “Railway prepares the environment and carries available coding-agent credentials from your machine. Your files stay on the agent's disk, and …”
- [claimed-docs] “Use Claude Desktop to work on a Railway cloud agent. Claude Code runs on the remote machine, with its development tools and persistent disk,…”
- [claimed-docs] “Give Codex Desktop a project on a Railway computer. The app connects over SSH, and Codex reads files, runs commands, and makes changes on yo…”
- [claimed-docs] “Rollback to any previous version instantly when something breaks.”
Placement latency
devops-leadDeploy my application to specific regions closer to my users
weight 2 · round to Fly.ioFly.io explicitly documents deploying apps to specific regions worldwide via its global Anycast network so users connect to the nearest server, and other docs reference region-based multi-server deployment with a single command. missing for 10: independent/hands-on corroboration specifically validating regional deployment behavior (community evidence focuses on outages/reliability, not region selection).
- [claimed-docs] “You can deploy apps in regions worldwide, so your users in Tokyo, São Paulo, or Amsterdam connect to the nearest server through our global A…”
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “Fly.io makes it easier to run an Elixir cluster on multiple servers around the world and deploy to those regions with a single command.”
Railway's marketing docs mention 'Run your application closer to where your users are' (railway-docs-7), implying multi-region deployment, but there is no documentation on selecting specific regions, listing available regions, or configuring region-based deployment, and no independent/hands-on corroboration of this capability. Missing for 10: detailed region list/selection docs, technical configuration guidance, independent verification of region-based deployment working as described.
- [claimed-docs] “Run your application closer to where your users are.”
devops-leadRely on automatic compute placement near backend data to optimize latency
weight 2 · round to Fly.ioFly.io documents global region deployment and Anycast routing so user traffic reaches the nearest server (fly-io-docs-20), and offers a managed Postgres service (fly-io-docs-19), but region placement is manual (fly launch/region flags) rather than an automatic system that senses backend-data location and relocates compute accordingly. Missing for 10: explicit automatic/backend-data-aware placement logic, documentation of latency-optimizing scheduling relative to a database's region, and independent verification that this placement actually reduces latency in production.
- [claimed-docs] “You can deploy apps in regions worldwide, so your users in Tokyo, São Paulo, or Amsterdam connect to the nearest server through our global A…”
- [claimed-docs] “Fly.io's Managed Postgres is our fully-managed database service that handles all aspects of running production PostgreSQL databases”
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
Railway's marketing states 'Run your application closer to where your users are' (docs-7), implying region selection for latency, but there is no evidence of automatic compute placement decisions based on backend data location, no documentation of an algorithm or feature that colocates compute with data stores. missing for 10: technical detail on automatic placement logic, evidence of data-aware region selection, and any independent corroboration of latency optimization near data.
- [claimed-docs] “Run your application closer to where your users are.”
Sandboxing
developerRun untrusted code in hardware-isolated, disposable sandboxes
weight 2 · round to Fly.ioFly.io explicitly documents hardware-isolated, disposable Fly Machines/Sprites with 'clean baseline per request', egress-locked, no shared state, and per-agent isolated Linux boxes for untrusted/agent workloads (fly-io-docs-7, fly-io-docs-14, fly-io-docs-4, fly-io-docs-6, fly-io-docs-17). This directly matches the sandboxed untrusted-code execution story. Missing for 10: independent/hands-on verification of the isolation guarantees and no third-party security audit evidence is present.
- [claimed-docs] “Run it in a hardware-isolated VM. A clean baseline per request, so no two users ever share state. Egress-locked and disposable.”
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “Every agent gets its own Linux box. A Sprite checkpoints itself while the agent works, so harnesses like Claude and Codex find everything wh…”
- [claimed-docs] “Run every MCP server in its own Sprite, with persistent disk, and egress policy. Pay only for the tool calls you use.”
- [claimed-docs] “Fly Machines are tightly isolated VMs that are perfect for running MCP servers.”
Railwaynone0/10Railway's evidence covers deployment, CI/CD, cloud agents with persistent disks, and AI integrations, but nothing addresses hardware-isolated, disposable sandboxes for running untrusted code — in fact cloud agents are described as persistent ('machine stays running when you disconnect'), the opposite of disposable isolation. Missing for 10: any mention of sandbox isolation technology (VM/hypervisor-level), disposability guarantees, or untrusted-code execution security model.
- [claimed-docs] “Railway prepares the environment and carries available coding-agent credentials from your machine. Your files stay on the agent's disk, and …”
Scaling
devops-leadDeploy instances that boot in under a second and scale to tens of thousands under load
weight 2 · round to Fly.ioFly.iodisputedcontradicted5/10Fly.io's own docs explicitly claim Machines 'boot in under a second... and scale out to tens of thousands when the traffic arrives' (fly-io-docs-14), backed by fast-launch API and scale commands (fly-io-docs-18, fly-io-docs-24). However, multiple independent community reports describe real production reliability problems under load — frequent machine failures, stuck/unreachable instances, flaky deploys (70-80% failure rate), and multi-machine outages with no status-page acknowledgment (fly-io-comm-1, fly-io-comm-5, fly-io-comm-7, fly-io-comm-19) — directly undercutting the claim of dependable scaling at volume. missing for 10: independent benchmark data on actual boot latency and large-scale load tests, and resolution/acknowledgment of the reliability complaints.
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “Fly Machines are fast-launching VMs with a simple REST API.”
- [claimed-docs] “the `fly scale` command for quick horizontal and vertical scaling”
- [community] “This is the second time I've had this kind of issue with Fly, where my service just goes down, Fly reports everything healthy, and there's l…”
- [community] “We tried to migrate all our staging environments to Fly last year but it was the flakiest experience I've experienced on any PaaS. Pushing s…”
- [community] “I really want to love Fly.io... but to be honest I don't think anyone should be building mission critical applications on their service. I e…”
- [community] “Fly has a nice DX but unfortunately the platform still has major issues with reliability and networking. There are often problems with deplo…”
Railwaynone0/10Railway's docs mention general scaling (CPU/RAM/replicas, load balancing) and running closer to users, but there is no evidence of sub-second instance boot times or scaling to tens of thousands of instances under load, which is the specific edge-compute claim in the story. Community evidence even highlights platform-wide stability issues rather than proven massive-scale elasticity.
- [claimed-docs] “Scale CPU and RAM or add replicas with load balancing.”
- [claimed-docs] “Run your application closer to where your users are.”
- [community] “Sorry, I have a hard time blaming Google for this, when Railway seems to be having increasing trouble keeping the platform stable. Something…”
- [community] “This is bad. Even their own website is down at railway.com. Looks like total dependency on google cloud. Surprising for a company of their s…”
devops-leadRely on load-based autoscaling to absorb large traffic bursts automatically
weight 2 · round to Fly.ioFly.io markets that 'Fly Machines ... scale out to tens of thousands when the traffic arrives' and offers a 'fly scale' command for horizontal/vertical scaling, suggesting scaling capability exists, but the evidence never documents concrete load-based autoscaling policies (e.g., CPU/request-triggered auto-scale-up/down) — 'fly scale' reads as a manual operator command rather than automatic burst absorption. Community reports also describe outages and unreliable behavior under production load, undercutting confidence that bursts are absorbed smoothly. Missing for 10: documented automatic autoscaling triggers/policies, hands-on evidence of successful burst handling, and resolution of the reliability complaints during high load.
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “the `fly scale` command for quick horizontal and vertical scaling”
- [community] “We had four machines (app + Postgres for staging and production) running yesterday, and three of the four (including both databases) are sti…”
- [community] “I really want to love Fly.io... but to be honest I don't think anyone should be building mission critical applications on their service. I e…”
- [community] “Fly response: 'We had a single host in the syd region go down, hard, with multiple issues... Ultimately it required a significant amount of …”
Railwaynone0/10Docs mention manual scaling (adjusting CPU/RAM or adding replicas with load balancing) but there is no evidence of automatic, load-triggered autoscaling that reacts to traffic bursts without manual intervention.
- [claimed-docs] “Scale CPU and RAM or add replicas with load balancing.”
- [claimed-docs] “Private connections, public endpoints, SSL, and load balancing live from the moment you deploy.”
Serverless runtime
developerRun my code on a global edge network close to end users
weight 3 · round to Fly.ioFly.io explicitly documents deploying apps to multiple regions worldwide via a global Anycast network so users connect to the nearest server, backed by fast-booting, hardware-isolated Fly Machines that scale globally (fly-io-docs-14, fly-io-docs-20, fly-io-docs-18). This directly matches the edge-compute story. Missing for 10: independent latency benchmarks confirming real-world edge performance, and community reports of intermittent outages/reliability issues (fly-io-comm-1, fly-io-comm-7, fly-io-comm-19) temper confidence in production robustness even though they don't contradict the core global-deployment capability.
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “You can deploy apps in regions worldwide, so your users in Tokyo, São Paulo, or Amsterdam connect to the nearest server through our global A…”
- [claimed-docs] “Fly Machines are fast-launching VMs with a simple REST API.”
- [community] “Fly has a nice DX but unfortunately the platform still has major issues with reliability and networking. There are often problems with deplo…”
Railwaydisputedcontradicted4/10Railway's marketing states only a single vague line — 'Run your application closer to where your users are' (railway-docs-7) — with no detail on edge regions, PoPs, or CDN architecture. Community reports of a major outage explicitly attribute it to 'total dependency on google cloud' and note the entire service (including the marketing site) went down together, which contradicts the premise of a distributed, resilient global edge network (railway-comm-4, railway-comm-6, railway-comm-1). missing for 10: documented edge-region list, latency/PoP architecture details, evidence of multi-region failover, and confirmation the claim survives independent scrutiny.
- [claimed-docs] “Run your application closer to where your users are.”
- [community] “This is bad. Even their own website is down at railway.com. Looks like total dependency on google cloud. Surprising for a company of their s…”
- [community] “Wait… railway runs on GCP? Didn't they make a whole thing about not 'building a cloud on top of another cloud'?”
- [community] “Sorry, I have a hard time blaming Google for this, when Railway seems to be having increasing trouble keeping the platform stable. Something…”
developerDeploy serverless functions and full-stack apps with no servers to manage
weight 3 · round to RailwayFly.iodisputedcontradicted5/10Fly.io's docs describe a workflow (`fly launch`, Fly Machines that boot in under a second and scale to zero, Managed Postgres, global regions, pay-as-you-go pricing) that strongly emphasizes not needing to provision or manage servers for full-stack apps [fly-io-docs-1][fly-io-docs-14][fly-io-docs-19][fly-io-docs-20][fly-io-docs-21]. However, multiple independent hands-on reports describe machines/Postgres going down with no self-healing, requiring manual recovery work, and Fly's own support response recommending users 'run multiple instances to mitigate the impact of single-host failures' — directly undercutting the 'no servers to manage' promise [fly-io-comm-2][fly-io-comm-4][fly-io-comm-9][fly-io-comm-15][fly-io-comm-19]. Missing for 10: true FaaS/serverless-functions primitive (Fly is VM/container-based, not function-as-a-service), and resolution of the reliability complaints that contradict hands-off operation.
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “Fly.io's Managed Postgres is our fully-managed database service that handles all aspects of running production PostgreSQL databases”
- [claimed-docs] “You can deploy apps in regions worldwide, so your users in Tokyo, São Paulo, or Amsterdam connect to the nearest server through our global A…”
- [claimed-docs] “Unlike Heroku's dyno-based tiers, Fly.io uses pay-as-you-go pricing - you only pay for the compute, storage, and bandwidth you actually use.”
- [community] “We had four machines (app + Postgres for staging and production) running yesterday, and three of the four (including both databases) are sti…”
- [community] “Don't get me started with Fly — especially postgres machines. In my experience, a really nice idea with poor support and unreliable infrastr…”
- [community] “Fly response: 'We had a single host in the syd region go down, hard, with multiple issues... Ultimately it required a significant amount of …”
- [community] “I was an early adopter of Fly.io. It is not production-ready. They should fix their basic features before adding new ones.”
- [community] “Fly has a nice DX but unfortunately the platform still has major issues with reliability and networking. There are often problems with deplo…”
Railway's docs strongly evidence full-stack app deployment with no server management (connect repo, auto-config, scaling, networking, previews), fitting the 'full-stack apps' half of the story. However there is no evidence of a serverless/FaaS functions offering (e.g., function-level deploys, event-triggered execution) — Railway's model is service/container-based rather than classic serverless functions. missing for 10: evidence of serverless function deployment/FaaS primitives, independent corroboration of zero-ops scaling under load.
- [claimed-docs] “Connect your repo, Railway handles the rest. Auto-config, instant previews, no new tools to learn.”
- [claimed-docs] “Railway is an all-in-one intelligent cloud provider that makes it easy to provision infrastructure, develop locally, and deploy to the cloud…”
- [claimed-docs] “Private connections, public endpoints, SSL, and load balancing live from the moment you deploy.”
- [claimed-docs] “Scale CPU and RAM or add replicas with load balancing.”
- [claimed-docs] “Every pull request gets its own preview. No surprises after merge.”
Workflows
developerDeploy durable, long-running workflows as code without manually wiring queues and retries
weight 2 · round drawnFly.ionone0/10Fly.io evidence covers VM provisioning, restart policies, and infrastructure primitives (Machines, checkpoints, Postgres) but nothing about a durable workflow-as-code abstraction, built-in queueing, or automatic retry orchestration for long-running business workflows. The restart policy docs (fly-io-docs-26) only cover process-level VM restarts, not workflow state/retry semantics.
- [claimed-docs] “The Machine restart policy defines whether and how flyd restarts a Machine after its main process exits.”
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [community] “For me the missing bit in Fly is scheduled tasks. I know how to solve this by spinning up an app that runs permanently as a scheduler, but b…”
Railwaynone0/10Railway's evidence pack shows it as a deployment/hosting platform (repo-to-deploy, networking, scaling, environments, previews) with no mention of a durable execution engine, workflow-as-code framework, or built-in queue/retry orchestration primitives. The story describes a specific workflow-orchestration capability (like Temporal/Inngest) that Railway does not evidence.
- [claimed-docs] “Railway is an all-in-one intelligent cloud provider that makes it easy to provision infrastructure, develop locally, and deploy to the cloud…”
- [claimed-docs] “Scale CPU and RAM or add replicas with load balancing.”
- [claimed-docs] “Develop locally and deploy from your terminal with the Railway CLI.”
Networking security — network controls and isolation — private access, firewalls, encryptionNetworking security
Network controls and isolation — private access, firewalls, encryption
Access control
developerPut simple password protection on a deployed site without standing up a custom auth system
weight 2 · round drawnFly.ionone0/10The evidence pack contains no mention of any built-in basic-auth/password-protection feature for deployed apps (e.g., no proxy-level HTTP auth, no dashboard toggle). While secrets management is documented (fly-io-docs-23), that only covers passing credentials to the app, not adding password gating without custom auth code — this axis is plausible for a deployment platform but unevidenced here.
- [claimed-docs] “Secrets allow sensitive values, such as credentials, to be passed securely to your Fly App. The secret is encrypted and stored in a vault.”
Ddos waf
devops-leadHave DDoS protection and a web application firewall applied to every request by default
weight 3 · round drawnFly.ionone0/10No evidence pack item mentions DDoS protection or a web application firewall being applied by default (or at all); docs cover secrets, scaling, and Machine isolation but never security perimeter features like WAF/DDoS mitigation.
Railwaynone0/10Evidence covers SSL, load balancing, and general networking but no mention of DDoS protection or a web application firewall being applied by default; the outage-related community items discuss general stability, not DDoS/WAF specifically. missing for 10: any mention of DDoS mitigation, WAF, or default request-level security filtering.
- [claimed-docs] “Private connections, public endpoints, SSL, and load balancing live from the moment you deploy.”
Domains
developerAttach custom domains to my deployed services
weight 2 · round drawnFly.ionone0/10The evidence pack contains no mention of custom domain attachment, certificate management, or DNS configuration (e.g., no reference to `fly certs add` or similar). While Fly.io is a PaaS where this axis clearly applies, no documentation or community evidence in the pack demonstrates this capability.
Railwaynone0/10The evidence pack never explicitly mentions custom domain attachment; docs only reference generic 'public endpoints, SSL' (railway-docs-4) without confirming a custom-domain feature. Missing for 10: any documentation or community proof of attaching/verifying custom domains on deployed services.
- [claimed-docs] “Private connections, public endpoints, SSL, and load balancing live from the moment you deploy.”
developerProgrammatically provision and route custom domains for my own multi-tenant SaaS customers
weight 2 · round drawnFly.ionone0/10The evidence pack contains no mention of Fly.io's certificate/custom-domain APIs (e.g. `fly certs add`, automated TLS issuance, per-app domain routing) needed for multi-tenant SaaS customer domains — only general Machines/Apps provisioning and networking primitives are described. Missing for 10: any documentation of programmatic custom-domain/cert provisioning, per-tenant routing, or multi-tenant domain API, and any independent confirmation it works reliably.
Railwaynone0/10Railway's docs mention custom domains, SSL, and public endpoints being available at deploy time, but there is no evidence of a programmatic API/CLI capability for provisioning and routing custom domains on behalf of a multi-tenant SaaS's own customers, and a probe for an OpenAPI/domains API returned 404s.
- [claimed-docs] “Private connections, public endpoints, SSL, and load balancing live from the moment you deploy.”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.railway.com/openapi.json, https://docs.railway.com/swagger.json, https://docs.railway.c…”
Env isolation
devops-leadIsolate non-production environments from accessing production services
weight 1 · round drawnFly.ionone0/10The evidence covers Fly Machines, Sprites (AI-agent sandboxes with egress policies), and general platform features, but nothing addresses network segmentation or access control between staging/non-production and production environments for a devops team. Docs-11/12/13 describe org-wide service connectivity and per-Sprite access grants for agent workloads, not environment isolation for prod vs non-prod apps.
Railwaynone0/10Evidence mentions 'unlimited environments' and private networking/endpoints generally, but there is no documentation or claim about network-level isolation preventing non-production environments from reaching production services (e.g., separate private networks, environment-scoped service discovery restrictions). Missing for 10: explicit environment-to-environment network isolation policy, private networking scoped per environment, access-control documentation preventing staging/dev from calling prod services.
- [claimed-docs] “Private connections, public endpoints, SSL, and load balancing live from the moment you deploy.”
- [claimed-docs] “Unlimited environments mean everyone can ship simultaneously.”
Identity
devops-leadSecure internal apps, deployments, and agents using my organization's identity provider
weight 2 · round drawnFly.ionone0/10No evidence of SSO/SAML/OIDC integration with an external identity provider for securing Fly apps, deployments, or agents. The pack mentions secrets vaults, WireGuard networking, and service-to-service credential rotation, but nothing about enterprise IdP-based authentication or access control.
Private networking
devops-leadKeep internal service-to-service traffic off the public internet without VPC complexity
weight 2 · round to RailwayEvidence hints at private networking (WireGuard checks via `fly doctor`, org-wide service connectivity and egress policies for Sprites) but never explicitly documents a general internal DNS/private network (e.g. 6PN/.internal) mechanism for ordinary Fly Apps talking to each other off the public internet. Missing for 10: explicit docs on private networking/internal DNS for app-to-app traffic, VPC-less peering details, and independent confirmation it works reliably at scale.
- [claimed-docs] “Run diagnostics: `fly doctor` — checks WireGuard, IPs, and Docker.”
- [claimed-docs] “Connect a service once for your whole org, and every Sprite can reach it. Nobody pastes a token.”
- [claimed-docs] “Run every MCP server in its own Sprite, with persistent disk, and egress policy. Pay only for the tool calls you use.”
- [claimed-docs] “Run it in a hardware-isolated VM. A clean baseline per request, so no two users ever share state. Egress-locked and disposable.”
Railway's docs claim built-in "Private connections" alongside public endpoints, SSL, and load balancing "live from the moment you deploy," implying internal service traffic can stay off the public internet without manual VPC setup. However, the evidence pack gives only a single high-level marketing line with no architectural detail, no explicit VPC-avoidance framing, and no independent corroboration of how private networking actually isolates traffic. Missing for 10: detailed technical docs on private networking implementation, explicit no-VPC-needed messaging, and independent/hands-on verification of traffic isolation.
- [claimed-docs] “Private connections, public endpoints, SSL, and load balancing live from the moment you deploy.”
Protocols
developerServe HTTP, TCP, gRPC, and WebSocket traffic from the same deployed service without separate protocol configuration
weight 1 · round to RailwayFly.ionone0/10The evidence pack contains no documentation or community discussion describing Fly.io's ability to serve HTTP, TCP, gRPC, and WebSocket traffic from a single deployed service without separate protocol configuration; docs cover deployment, scaling, secrets, and Machines but not multi-protocol service configuration.
Railway's own docs explicitly state 'HTTP, TCP, gRPC, WebSockets handled automatically' alongside private connections, public endpoints, SSL, and load balancing available from deploy time, directly matching the story. Missing for 10: independent/hands-on verification of multi-protocol serving from a single service and technical detail on how protocol detection/config works under the hood.
- [claimed-docs] “HTTP, TCP, gRPC, WebSockets handled automatically.”
- [claimed-docs] “Private connections, public endpoints, SSL, and load balancing live from the moment you deploy.”
Tls certs
devops-leadGet free, automatically renewing TLS certificates for every domain including wildcards
weight 3 · round to RailwayFly.ionone0/10No evidence in the pack mentions TLS certificate provisioning, automatic renewal, or wildcard domain support; the docs excerpts focus on Machines, Sprites, MCP servers, and pricing, not certificate management.
Docs mention SSL is provided automatically for endpoints from the moment of deployment, implying free TLS certs, but there is no evidence about wildcard domain certificates or explicit auto-renewal guarantees. missing for 10: wildcard certificate support, explicit auto-renewal details, independent confirmation of custom domain TLS coverage.
- [claimed-docs] “Private connections, public endpoints, SSL, and load balancing live from the moment you deploy.”
Observability — seeing what the system is doing — logs, metrics, traces, alertsObservability
Seeing what the system is doing — logs, metrics, traces, alerts
Alerting
devops-leadSet up alerts that notify me via chat or email when specific conditions are met
weight 2 · round to RailwayFly.ionone0/10No evidence in the pack of Fly.io offering alerting/notification features (chat or email) tied to conditions or metrics; documentation covers deployment, compute, and infrastructure but not alerting/observability notification setup.
Railway docs explicitly state it sends Slack, Discord, or email notifications the moment specified conditions are met, directly matching the story. missing for 10: no independent/hands-on corroboration of alert configuration UX, no detail on what conditions/metrics can trigger alerts.
- [claimed-docs] “Slack, Discord, or emails the moment conditions you specify are met.”
- [claimed-docs] “Create custom dashboards with metrics from any service.”
Metrics dashboards
devops-leadBuild custom dashboards combining metrics from any service
weight 2 · round to RailwayFly.ionone0/10No evidence of a custom dashboarding capability or metrics/observability aggregation feature across services; the evidence pack covers deployment, compute, VMs, and agent 'Sprites' but nothing about building dashboards from metrics.
Direct first-party claim confirms custom dashboards combining metrics from any service, but the pack offers no independent/hands-on corroboration or detail on dashboard flexibility (e.g., cross-project aggregation, widget types, sharing). missing for 10: independent verification, detail on dashboard customization scope, evidence of multi-service/cross-project metric combination in practice.
- [claimed-docs] “Create custom dashboards with metrics from any service.”
Reliability sla
devops-leadCheck a public status page with historical incident postmortems before committing
weight 2 · round drawnFly.ionone0/10The evidence pack contains no vendor/docs-tier claim of a public status page with historical incident postmortems, and the only related mentions are community complaints that Fly's status page shows 'no incidents'/green even during real outages (fly-io-comm-3, fly-io-comm-6, fly-io-comm-9), which undercuts rather than confirms the existence of a trustworthy postmortem history. Since disputed requires a documented vendor claim contradicted by hands-on evidence from a distinct tier, and no such first-party claim is present here, this falls to 'none' rather than 'disputed'.
- [community] “As an aside, it's kind of a kick in the teeth to see the status page for our organization reporting no incidents - the same page that lists …”
- [community] “The worst thing about Fly is, when something goes wrong, it's not just one thing, there's bunch of things broken at the same time and their …”
- [community] “Fly response: 'We had a single host in the syd region go down, hard, with multiple issues... Ultimately it required a significant amount of …”
Railwaynone0/10No evidence pack item references a public status page or historical incident postmortems; only community complaints about an outage exist, with no mention of a status.railway.app or postmortem documentation. missing for 10: public status page, incident history, postmortem write-ups.
devops-leadReview a published uptime SLA with defined credits for downtime
weight 3 · round drawnFly.ionone0/10No evidence of a published SLA document with defined uptime percentage or downtime credits; community evidence even shows Fly's status page failing to reflect actual outages, but this is not a documented SLA dispute, just absence of one.
Railwaynone0/10No evidence in the pack of a published uptime SLA or defined downtime credits; docs focus on features like logs, alerts, and scaling, not contractual guarantees. Community evidence even describes a major outage with no mention of any credit/SLA process, reinforcing the absence.
- [community] “Sorry, I have a hard time blaming Google for this, when Railway seems to be having increasing trouble keeping the platform stable. Something…”
- [community] “This is bad. Even their own website is down at railway.com. Looks like total dependency on google cloud. Surprising for a company of their s…”
- [community] “Railway is back, but I'm not sure if I can trust keeping my projects there, so I'm going to migrate to another company.”
devops-leadThe guaranteed support response times and escalation paths for my plan tier
weight 2 · round drawnFly.ionone0/10No evidence of documented support-tier SLAs, guaranteed response times, or escalation paths for different plan tiers; docs only vaguely claim support is 'run by Fly.io engineers,' with no timing commitments. Community reports (fly-io-comm-4, -5, -7, -14) even describe unresponsive or non-existent support, but since no vendor claim of guaranteed SLAs exists to contradict, this is an unmet axis rather than a dispute.
- [claimed-docs] “Run by Fly.io engineers, not chat bots”
- [community] “Don't get me started with Fly — especially postgres machines. In my experience, a really nice idea with poor support and unreliable infrastr…”
- [community] “We tried to migrate all our staging environments to Fly last year but it was the flakiest experience I've experienced on any PaaS. Pushing s…”
- [community] “I really want to love Fly.io... but to be honest I don't think anyone should be building mission critical applications on their service. I e…”
- [community] “Does it have basic functioning other stuff? I am shocked at how our production usage of Fly has gone. Even basic stuff as support not being …”
Unified logging
developerView all my build, deploy, and runtime logs in one place
weight 3 · round to RailwayFly.ionone0/10The evidence pack contains no documentation or mention of a unified logging view for build, deploy, and runtime logs (e.g., no reference to `fly logs`, log streaming, or a logs dashboard). Community evidence even suggests logging/observability gaps ('Fly reports everything healthy, and there's literally no information'), but this doesn't confirm or deny a unified log feature — it's simply absent from the pack.
- [community] “This is the second time I've had this kind of issue with Fly, where my service just goes down, Fly reports everything healthy, and there's l…”
- [community] “The worst thing about Fly is, when something goes wrong, it's not just one thing, there's bunch of things broken at the same time and their …”
Railway explicitly claims a unified log view ('All logs in one place. Spot issues without switching tools') plus supporting observability features like custom dashboards and metrics, matching the story directly. missing for 10: independent/hands-on verification of unified log UX, and detail on separating build vs deploy vs runtime log views.
- [claimed-docs] “All logs in one place. Spot issues without switching tools.”
- [claimed-docs] “Create custom dashboards with metrics from any service.”
Openness — open source, data portability, and self-hosting storiesOpenness
Open source, data portability, and self-hosting stories
ai-native userDo everything through the API that I can do in the UI
weight 2 · round to Fly.ioFly.io documents a Machines REST API that can provision and manage Apps, Machines, and Volumes (fly-io-docs-31, fly-io-docs-18), and flyctl itself is built atop this API, suggesting broad API/CLI-UI parity typical of Fly's API-first design. However, no evidence explicitly confirms full feature parity with the web dashboard, and a probe found no discoverable OpenAPI spec (fly-io-probe-3), making it hard to verify completeness. Missing for 10: explicit parity claim/doc mapping UI features to API endpoints, published OpenAPI/swagger spec, and confirmation that dashboard-only features (billing UI, status page, etc.) are also API-accessible.
- [claimed-docs] “The Fly Machines REST API provides resources to provision and manage Fly Apps, Fly Machines, and Fly Volumes.”
- [claimed-docs] “Fly Machines are fast-launching VMs with a simple REST API.”
- [probe] “PROBE openapi: all candidate paths 404 (https://fly.io/openapi.json, https://fly.io/swagger.json, https://fly.io/api/openapi.json, https://f…”
- [probe] “official CLI documented at https://fly.io/docs/flyctl/”
Railway ships a CLI and an MCP server that let AI assistants create projects, deploy templates, manage environments, pull variables, and redeploy services (railway-docs-20, railway-docs-19), showing partial programmatic parity with the UI, but there is no documented OpenAPI/GraphQL spec (probe for openapi paths returned 404s) and no claim that canvas-level configuration, dashboards, or alerting can be fully replicated via API. Missing for 10: explicit full API/UI parity documentation, discoverable OpenAPI/GraphQL schema, and evidence that canvas/dashboard/alert features are API-accessible.
- [claimed-docs] “Install the Railway CLI and configure agent support — skills, MCP, and authentication — in one command.”
- [claimed-docs] “It lets AI assistants create projects, deploy templates, manage environments, pull variables, and redeploy services.”
- [claimed-docs] “The Railway Agent is a chat-based AI assistant built into the Railway dashboard. It can operate the platform on your behalf: creating and co…”
- [probe] “PROBE openapi: all candidate paths 404 (https://docs.railway.com/openapi.json, https://docs.railway.com/swagger.json, https://docs.railway.c…”
ai-native userExport all of my data in open formats and leave
weight 3 · round drawnFly.ionone0/10No evidence describes data export tooling, open-format export, or an account/data portability process for Fly.io apps, volumes, or databases; the pack covers deployment, scaling, and infrastructure features but nothing about exporting user data to leave the platform.
Railwaynone0/10Railway's docs cover migrating INTO Railway (e.g., from Heroku) and CLI/local dev workflows, but there is no evidence of any data export feature, open-format export of configs/logs/variables, or documented offboarding path for leaving the platform with your data intact.
- [claimed-docs] “Develop locally and deploy from your terminal with the Railway CLI.”
- [claimed-docs] “Migrate from Heroku”
ai-native userRead the product's source under an open license
weight 2 · round drawnFly.ionone0/10Fly.io is a proprietary cloud platform; the evidence pack contains no mention of an open-source license or public source repository for the core platform (flyctl is open source per outside knowledge, but no evidence here confirms this), and no docs claim open-source availability of the product's source.
ai-native userSelf-host the core product
weight 3 · round drawnFly.ionone0/10Fly.io is presented purely as a managed public cloud (flyctl + fly launch, hosted Sprites/Machines, hosted secrets, hosted networking) with no evidence anywhere of an open-source or self-hostable core product; all docs and probes describe a proprietary hosted service.
- [claimed-docs] “**Step 1:** Install `flyctl` ... **Step 2:** Run `fly launch`”
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [probe] “PROBE llms.txt: HTTP 200 at https://fly.io/llms.txt # Fly.io Documentation > Fly.io is a global public cloud for running full-stack and AI …”
- [probe] “PROBE openapi: all candidate paths 404 (https://fly.io/openapi.json, https://fly.io/swagger.json, https://fly.io/api/openapi.json, https://f…”
Railwaynone0/10Railway is a managed cloud PaaS; nothing in the evidence pack mentions an open-source or self-hostable version of the core Railway platform — all docs describe using Railway's own hosted service. missing for 10: any open-source repo/license for the core platform, self-hosting instructions or Docker/K8s deployment guide for running Railway itself.
- [claimed-docs] “Railway is an all-in-one intelligent cloud provider that makes it easy to provision infrastructure, develop locally, and deploy to the cloud…”
Pricing limits — free-tier ceilings, usage caps, and rate limits before you have to payPricing limits
Free-tier ceilings, usage caps, and rate limits before you have to pay
Cost controls
devops-leadSet hard spending limits to control my cloud costs
weight 2 · round to RailwayFly.ionone0/10No evidence describes hard spending limits, budget caps, or cost-control controls; only pay-as-you-go pricing and cost visibility are mentioned, with community comments criticizing pricing/network costs but no dispute of a spending-limit feature.
A community tweet explicitly praises Railway for offering hard spending limits, confirming the capability exists and is well-received, but there is no first-party documentation detailing configuration options, scope (project vs org-level), or enforcement behavior. Missing for 10: official docs on setting/configuring hard spending limits, independent hands-on verification, and details on limit granularity.
- [claimed-docs] “really gotta commend @Railway for allowing hard spending limits, I have never, ever seen this with any cloud provider.”
devops-leadReceive migration credits when moving my production infrastructure to a new platform
weight 1 · round drawnFly.ionone0/10No evidence in the pack mentions migration credits or any financial incentive program for switching from another platform; pricing docs only describe pay-as-you-go billing.
Railwaynone0/10Railway has migration guides (e.g., 'Migrate from Heroku', 'Compare to Render') but no evidence of any migration credit program, discount, or reimbursement offered to customers moving production infrastructure to the platform.
- [claimed-docs] “Migrate from Heroku”
- [claimed-docs] “Compare to Render”
Execution limits
devops-leadThe CPU and wall-clock execution time limits for functions so I can tell whether long-running or agentic workloads will fit
weight 3 · round drawnFly.ionone0/10Fly.io's evidence pack markets Machines/Sprites explicitly for long-running and agentic workloads (e.g., fly-io-docs-4, fly-io-docs-5, fly-io-docs-28) but never states any CPU or wall-clock execution time limit — Fly Machines are persistent VMs, not time-boxed functions, and no documentation here quantifies compute/runtime ceilings a devops-lead could compare against. Missing for 10: any documented CPU quota, timeout, or wall-clock limit per Machine/Sprite invocation, and any comparison table or pricing-limits doc addressing execution duration.
- [claimed-docs] “Every agent gets its own Linux box. A Sprite checkpoints itself while the agent works, so harnesses like Claude and Codex find everything wh…”
- [claimed-docs] “Give it a Sprite that sleeps until it's needed and wakes up remembering everything.”
- [claimed-docs] “Give your agent a real computer and get back to building.”
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
Railwaynone0/10Railway's evidence pack describes general CPU/RAM scaling (railway-docs-6) but contains no documentation of specific CPU or wall-clock execution time limits for functions/workloads, which is the core of this story. Missing for 10: any explicit stated time limits, timeout thresholds, or documentation addressing how long-running or agentic workloads are bounded.
- [claimed-docs] “Scale CPU and RAM or add replicas with load balancing.”
Overage fees
devops-leadThe exact overage fee schedule for exceeding plan usage limits (bandwidth, invocations, build minutes) before I get billed
weight 3 · round drawnFly.ionone0/10Evidence shows only generic pay-as-you-go pricing claims (compute/storage/bandwidth billed by usage) but no documented overage fee schedule, no specifics on invocation or build-minute overage costs, and no billing rate table. There is also community commentary calling billing UI 'cobbled together' but nothing detailing exact overage fees.
- [claimed-docs] “Unlike Heroku's dyno-based tiers, Fly.io uses pay-as-you-go pricing - you only pay for the compute, storage, and bandwidth you actually use.”
- [community] “My experience with Fly.io has not been a good one... Following their guide you get postgres not spinning up and linking to your app correctl…”
Railwaynone0/10The evidence pack contains no documentation of an overage fee schedule for bandwidth, invocations, or build minutes — only a community tweet praising hard spending limits, which is a cap mechanism, not a fee schedule disclosure. missing for 10: published overage rate/pricing table, per-resource overage unit costs, and billing documentation describing what happens when limits are exceeded.
- [claimed-docs] “really gotta commend @Railway for allowing hard spending limits, I have never, ever seen this with any cloud provider.”
devops-leadWhether pricing is per-seat with minimum seat counts that increase cost as my team grows
weight 2 · round drawnFly.ionone0/10Fly.io's pricing is explicitly usage-based (pay-as-you-go for compute/storage/bandwidth), not per-seat, and no evidence mentions seat counts or minimums; missing for 10: any pricing page or docs referencing seat-based billing or per-user minimums since the model is consumption-based, not applicable in the seat-pricing sense the story asks about.
- [claimed-docs] “Unlike Heroku's dyno-based tiers, Fly.io uses pay-as-you-go pricing - you only pay for the compute, storage, and bandwidth you actually use.”
Usage billing
devops-leadPay only for active compute time rather than idle wall-clock time
weight 2 · round to Fly.ioFly.io's docs describe pay-as-you-go pricing tied to actual compute/storage/bandwidth usage rather than fixed dyno-tier billing, plus machines that sleep/suspend when idle and only bill for active compute (e.g., 'Pay only for the tool calls you use', disk billed only for bytes used, Sprites that 'sleep until needed'). This directly matches the devops-lead story of paying for active time, not idle wall-clock. missing for 10: explicit per-second billing granularity documentation, and independent/third-party confirmation that idle machines are not billed.
- [claimed-docs] “Unlike Heroku's dyno-based tiers, Fly.io uses pay-as-you-go pricing - you only pay for the compute, storage, and bandwidth you actually use.”
- [claimed-docs] “Your disk grows as you write it, up to 100 GB per Sprite, and you're only billed for the bytes you actually put there.”
- [claimed-docs] “Run every MCP server in its own Sprite, with persistent disk, and egress policy. Pay only for the tool calls you use.”
- [claimed-docs] “Give it a Sprite that sleeps until it's needed and wakes up remembering everything.”
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
Railwaynone0/10The evidence pack contains no documentation or claims about usage-based/active-compute billing versus idle time; only a community tweet about 'hard spending limits' is present, which addresses spend caps, not billing granularity. Missing for 10: any doc on per-second/active-compute billing model, idle-scale-to-zero mechanics, or pricing page evidence.
- [claimed-docs] “really gotta commend @Railway for allowing hard spending limits, I have never, ever seen this with any cloud provider.”
devops-leadAttached disk/storage auto-grows with usage and I'm billed only for bytes actually written rather than pre-provisioned capacity
weight 2 · round to Fly.ioFly.io's own docs explicitly state disk auto-grows with usage up to 100GB per Sprite and billing is only for bytes actually written, matching the story precisely, and general pay-as-you-go storage pricing is reiterated elsewhere. Missing for 10: independent/hands-on verification of this billing behavior, clarity on whether it applies to standard Fly Volumes beyond the newer 'Sprites' product, and confirmation of no minimum/rounding charges.
- [claimed-docs] “Your disk grows as you write it, up to 100 GB per Sprite, and you're only billed for the bytes you actually put there.”
- [claimed-docs] “Unlike Heroku's dyno-based tiers, Fly.io uses pay-as-you-go pricing - you only pay for the compute, storage, and bandwidth you actually use.”
Railwaynone0/10The evidence pack contains no mention of persistent volumes, disk auto-scaling behavior, or usage-based storage billing (only general platform, deployment, and AI-agent features). This is a fair question for a cloud infra provider, but nothing in the pack confirms auto-grow disks or byte-metered storage pricing.
Privacy posture — data-handling and privacy storiesPrivacy posture
Data-handling and privacy stories
ai-native userChoose where my data is stored (region/residency)
weight 2 · round to Fly.ioFly.io documents global region deployment allowing users to place apps/data near specific regions (e.g. Tokyo, São Paulo, Amsterdam) via its Anycast network, which supports choosing where compute/data runs. However, there's no explicit documentation of data residency guarantees, compliance certifications, or region-locking specifically for data storage/backups (e.g. Postgres/volumes pinned to a region for residency purposes) rather than just latency-driven deployment. Missing for 10: explicit data residency/compliance documentation, volume/database region-pinning guarantees, independent verification of residency enforcement.
- [claimed-docs] “You can deploy apps in regions worldwide, so your users in Tokyo, São Paulo, or Amsterdam connect to the nearest server through our global A…”
- [claimed-docs] “Fly.io's Managed Postgres is our fully-managed database service that handles all aspects of running production PostgreSQL databases”
- [claimed-docs] “Fly.io makes it easier to run an Elixir cluster on multiple servers around the world and deploy to those regions with a single command.”
Railway docs mention running applications 'closer to where your users are' (railway-docs-7), implying some region choice for deployments, but there is no explicit documentation of selectable regions, data storage location controls, or compliance/residency certifications (e.g., GDPR, SOC2) that would let an AI-native user confirm where their data is stored. Missing for 10: explicit region list/selector for compute and storage, data residency guarantees, and compliance documentation.
- [claimed-docs] “Run your application closer to where your users are.”
ai-native userControl data retention and deletion
weight 2 · round drawnFly.ionone0/10The evidence pack contains no documentation of data retention policies, deletion tooling, or export/erasure controls for Fly.io's platform. The only relevant community evidence (fly-io-comm-8) describes a user whose account-deletion request was ignored, further indicating no working deletion mechanism was demonstrated.
- [community] “They had my account on some sort of shadow ban with no communication whatsoever after asking them to delete my account from their systems. I…”
Railwaynone0/10No evidence in the pack addresses data retention policies, deletion controls, or privacy settings for AI workflows on Railway; the only tangential item (railway-comm-8) is a community jab about backup security, not a documented retention/deletion control. missing for 10: documentation of data retention policy, explicit deletion/opt-out controls, AI-specific data handling disclosures.
- [community] “Isn't Railway the 'the API key to delete the backups is in the prod database, because that's where the backups live duh' guys?”
ai-native userOpt out of telemetry and usage tracking
weight 2 · round drawnFly.ionone0/10No evidence in the pack mentions telemetry collection, usage tracking, or any opt-out mechanism/settings for Fly.io products or CLI; the docs cover deployment, scaling, and infrastructure features but never address analytics/telemetry controls.
Storage data — storing and moving data — persistence, formats, durabilityStorage data
Storing and moving data — persistence, formats, durability
Cdn assets
developerServe static assets faster using a global CDN with edge caching
weight 2 · round drawnFly.ionone0/10Evidence shows Fly.io provides global compute deployment across regions via Anycast (fly-io-docs-20) and fast-booting VMs, but there is no mention of a CDN, edge caching, or static asset acceleration feature anywhere in the pack.
- [claimed-docs] “You can deploy apps in regions worldwide, so your users in Tokyo, São Paulo, or Amsterdam connect to the nearest server through our global A…”
Railwaynone0/10No evidence of a CDN or edge-caching feature for static assets; Railway's evidence covers deployment, scaling, monitoring, and AI agent tooling but nothing about a global CDN or edge caching. 'Run your application closer to where your users are' refers to regional deployment, not CDN/static asset caching.
- [claimed-docs] “Run your application closer to where your users are.”
Integrations
developerConnect my code to external services like databases, APIs, and storage through native bindings with minimal boilerplate instead of hand-wiring SDKs and credentials
weight 2 · round drawnFly.io offers secrets management (fly-io-docs-23), org-wide service connections without manual token-pasting (fly-io-docs-11, fly-io-docs-12, fly-io-docs-13), and its own Managed Postgres (fly-io-docs-19), which reduce some credential-wiring boilerplate. However, these are infrastructure-level networking/secrets primitives rather than native code-level bindings/SDKs for external databases, third-party APIs, or storage services akin to Workers-style bindings. Missing for 10: explicit SDK/binding integrations for common third-party services (e.g., S3, Stripe, external APIs), documentation showing reduced application code boilerplate, and independent corroboration that this actually simplifies service wiring in practice.
- [claimed-docs] “Connect a service once for your whole org, and every Sprite can reach it. Nobody pastes a token.”
- [claimed-docs] “Grant access on purpose: by Sprite name, by label, or down to a single endpoint, so a Slack bot can post messages and never touch admin.”
- [claimed-docs] “Rotate once and every Sprite has the new credential. Cut off a single Sprite and the rest never notice.”
- [claimed-docs] “Fly.io's Managed Postgres is our fully-managed database service that handles all aspects of running production PostgreSQL databases”
- [claimed-docs] “Secrets allow sensitive values, such as credentials, to be passed securely to your Fly App. The secret is encrypted and stored in a vault.”
Railway offers private networking, one-click databases/starters, and infrastructure provisioning with auto-config, which reduces some manual wiring, but the evidence pack lacks specifics on native SDK-free bindings for external third-party services (e.g., managed connectors to Stripe, S3, external APIs) or credential-injection features beyond internal service linking. Missing for 10: concrete documentation of native bindings/connectors to external third-party services and APIs, evidence of automatic credential/env-var injection without manual SDK setup, and independent corroboration of this workflow in practice.
- [claimed-docs] “Connect your repo, Railway handles the rest. Auto-config, instant previews, no new tools to learn.”
- [claimed-docs] “Private connections, public endpoints, SSL, and load balancing live from the moment you deploy.”
- [claimed-docs] “One-click deployable applications and starters for common use cases.”
- [claimed-docs] “Railway is an all-in-one intelligent cloud provider that makes it easy to provision infrastructure, develop locally, and deploy to the cloud…”
Kv cache
developerUse a Redis-compatible key-value store for low-latency caching and job queues
weight 2 · round drawnFly.ionone0/10The evidence pack contains no mention of Redis, Upstash, or any managed key-value/caching service offered by Fly.io; only generic Machines, Postgres, and Sprite-related docs are present. Missing for 10: any documentation or claim of a Redis-compatible store, caching service, or job-queue integration.
Railwaynone0/10The evidence pack contains no mention of Redis, a key-value store, or caching/queue database offerings on Railway—only generic deployment, templates, and AI-agent features. While Railway as a cloud platform could plausibly offer this (axis applies), there is no documentation or community evidence confirming a Redis-compatible database is available.
Object storage
developerPersist user-generated uploads using built-in object storage
weight 2 · round drawnFly.ionone0/10Evidence shows Fly.io offers persistent disks/volumes per Machine (fly-io-docs-10) but no built-in object storage service; a community report explicitly states Fly lacks an S3-equivalent and users had to leave Fly to store user assets (fly-io-comm-13), confirming absence of this capability.
- [claimed-docs] “Your disk grows as you write it, up to 100 GB per Sprite, and you're only billed for the bytes you actually put there.”
- [community] “About Fly but not about the GPU announcement, I wish they had a S3 replacement, they suggest a GNU Affero project that is a dealbreaker for …”
Relational db
developerProvision a fully-managed Postgres database with point-in-time recovery, replicas, and high availability
weight 3 · round to Fly.ioFly.iodisputedcontradicted3/10Fly.io docs mention a 'fully-managed Postgres' service (Managed Postgres) but provide no detail on point-in-time recovery, replica configuration, or HA topology in the evidence pack. Community reports concretely contradict reliability of Postgres on Fly, including databases going down with no info (fly-io-comm-2), explicit complaints about 'postgres machines' unreliability (fly-io-comm-4), and Postgres failing to spin up/link during setup (fly-io-comm-18). missing for 10: explicit documentation of PITR, replica setup, HA failover mechanics, and independent confirmation these work as advertised.
- [claimed-docs] “Fly.io's Managed Postgres is our fully-managed database service that handles all aspects of running production PostgreSQL databases”
- [community] “We had four machines (app + Postgres for staging and production) running yesterday, and three of the four (including both databases) are sti…”
- [community] “Don't get me started with Fly — especially postgres machines. In my experience, a really nice idea with poor support and unreliable infrastr…”
- [community] “My experience with Fly.io has not been a good one... Following their guide you get postgres not spinning up and linking to your app correctl…”
Railwaynone0/10The evidence pack contains no documentation of Railway provisioning managed Postgres with point-in-time recovery, replicas, or high availability — only generic claims about scaling CPU/RAM and adding replicas for services in general (railway-docs-6), with no database-specific backup/PITR/HA feature descriptions. A community comment even suggests backup handling has been criticized (railway-comm-8), but nothing confirms the capability exists as described.
- [claimed-docs] “Scale CPU and RAM or add replicas with load balancing.”
- [community] “Isn't Railway the 'the API key to delete the backups is in the prod database, because that's where the backups live duh' guys?”
developerQuery a serverless SQL database directly from my code with no connection management
weight 2 · round drawnFly.ionone0/10Fly.io offers Managed Postgres, a fully-managed traditional PostgreSQL service (fly-io-docs-19), but there's no evidence of a serverless SQL query interface with no connection management (e.g., HTTP-based query API like Neon's serverless driver); Postgres still requires standard connection handling.
- [claimed-docs] “Fly.io's Managed Postgres is our fully-managed database service that handles all aspects of running production PostgreSQL databases”
Railwaynone0/10Railway's evidence pack shows it as a general deployment/hosting platform with infra provisioning, CLI, and observability features, but nothing describes a serverless SQL database with an HTTP/driverless query interface (no connection pooling/management) like Neon or PlanetScale offer. Railway databases are provisioned as long-running services requiring standard connection strings, so the specific 'no connection management' serverless query capability is unevidenced.
Not comparable on these axes
ai-native userPlug MCP servers into this product so it can use their tools
weight 3 · not comparableFly.ion/aFly.io is a cloud infrastructure/hosting platform, not an AI agent or assistant that itself consumes tools; the evidence shows Fly.io as infrastructure for *running* MCP servers (docs-6, docs-17) or *exposing* an MCP server via flyctl (docs-29), not as a client that plugs in MCP servers to gain tool-use capabilities. This story's axis (product acting as an MCP client) doesn't apply to a compute/hosting platform.
- [claimed-docs] “Run every MCP server in its own Sprite, with persistent disk, and egress policy. Pay only for the tool calls you use.”
- [claimed-docs] “Fly Machines are tightly isolated VMs that are perfect for running MCP servers.”
- [claimed-docs] “[experimental] Start a flyctl MCP server”
Railway's docs explicitly describe connecting external MCP servers (Notion, Linear, Sentry, or any custom remote MCP server) to the Railway Agent so it can read from and use their tools while operating on projects, directly matching the story. Missing for 10: independent/hands-on corroboration beyond first-party docs, and detail on tool-use scope/limitations when using third-party MCP servers.
- [claimed-docs] “Connect Notion, Linear, or Sentry from the catalog, or point Railway at any remote MCP server you run, and the agent can read from it while …”
- [claimed-docs] “The Railway Agent is a chat-based AI assistant built into the Railway dashboard. It can operate the platform on your behalf: creating and co…”
- [claimed-docs] “Build with Railway using Agent Skills and the MCP server for AI-powered workflows.”
ai-native userGet AI-generated insights and suggestions from my data inside the product
weight 2 · not comparableFly.ion/aFly.io is an infrastructure/compute platform (VMs, deployment, scaling) rather than a data product with an interface for surfacing AI-generated insights from a user's own data; this axis targets analytics/SaaS-type products, not a cloud hosting platform.
The Railway Agent (dashboard chat assistant) can inspect deployments, diagnose failures, and surface insights like opening PRs to fix broken builds, which is a form of AI-generated insight/suggestion within the product; connectors to Notion/Linear/Sentry extend this. However, this is framed as an operational/agentic assistant rather than data-analytics insights (e.g., no evidence of AI-generated insights over metrics, logs analytics, or cost/usage trends beyond failure diagnosis). Missing for 10: evidence of proactive AI-driven analytics/insights over metrics or usage data, independent user validation of insight quality, and broader scope beyond deployment troubleshooting.
- [claimed-docs] “The Railway Agent is a chat-based AI assistant built into the Railway dashboard. It can operate the platform on your behalf: creating and co…”
- [claimed-docs] “If the fix is in your code, the agent can open a pull request with the change so you can review and merge it.”
- [claimed-docs] “Connect Notion, Linear, or Sentry from the catalog, or point Railway at any remote MCP server you run, and the agent can read from it while …”
ai-native userDelegate tasks to a built-in AI assistant inside the product
weight 3 · not comparableFly.ion/aFly.io is an infrastructure/PaaS platform for hosting apps and agent sandboxes (Sprites, Machines); it has no built-in AI assistant that end-users delegate tasks to inside the product itself. This axis is a category error for a cloud infra provider, not a gap in an AI-product feature set.
Railway ships a built-in chat-based AI assistant ("Railway Agent") directly in the dashboard that can create/configure services, inspect deployments, diagnose failures, and open PRs to fix issues, plus Slack integration for delegating tasks conversationally. Missing for 10: independent/hands-on corroboration of the Railway Agent's real-world reliability and depth beyond vendor docs.
- [claimed-docs] “The Railway Agent is a chat-based AI assistant built into the Railway dashboard. It can operate the platform on your behalf: creating and co…”
- [claimed-docs] “If the fix is in your code, the agent can open a pull request with the change so you can review and merge it.”
- [claimed-docs] “Connect Notion, Linear, or Sentry from the catalog, or point Railway at any remote MCP server you run, and the agent can read from it while …”
- [claimed-docs] “Mention @Railway in Slack to ask the Railway Agent about your projects, inspect deployments, read logs, and make changes without leaving the…”
developerUse an in-browser toolbar to leave feedback, manage feature flags, and inspect performance directly on my live site
weight 1 · not comparableFly.ion/aFly.io is a cloud/VM infrastructure platform, not a product with an in-browser feedback/feature-flag/performance toolbar for end-user apps; this story is a category error for this kind of product.
Railwayn/aThis story describes an in-browser feedback/feature-flag/performance widget embedded on a live website (a product like a customer feedback/observability toolbar), which is not a category fit for Railway, a cloud infrastructure/deployment platform. Railway's evidence covers deployment, scaling, logs, and dashboards but nothing about an embeddable end-user-facing toolbar for feedback or feature flags on a live site.
developerLet teammates and collaborators comment directly on preview and production environments
weight 1 · not comparableFly.ion/aFly.io is an infrastructure/compute platform for deploying apps and VMs, not a collaboration or commenting tool; there is no concept of teammates commenting on environments in its product category.
developerWrite code against standard runtime APIs so it runs unmodified across multiple edge platforms instead of being locked into proprietary storage/runtime APIs
weight 2 · not comparableFly.io runs standard Docker images and VMs rather than a proprietary edge-runtime API, and docs explicitly state 'the API you start with is the API you keep' with no rearchitecting or migration needed, plus standard Postgres rather than proprietary storage APIs. Missing for 10: independent third-party confirmation that code written for Fly ports unmodified to other platforms, and explicit contrast with proprietary edge-runtime APIs (e.g. Workers/Deno) rather than just general platform-migration claims.
- [claimed-docs] “Fly.io converts Docker images to fast-booting VMs and runs them globally. This is great or ingesting incoming data!”
- [claimed-docs] “Keep the same primitives at every scale: no rearchitecting when you grow, and no platform migration when you get big. The API you start with…”
- [claimed-docs] “Fly Machines are fast-launching VMs with a simple REST API.”
- [claimed-docs] “Fly.io's Managed Postgres is our fully-managed database service that handles all aspects of running production PostgreSQL databases”
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
Railwayn/aRailway is a general-purpose PaaS/cloud provider for deploying containers and standard app runtimes, not an edge-compute platform (like Cloudflare Workers/Deno Deploy) with proprietary storage/runtime APIs that this story is contrasting against. Nothing in the evidence discusses edge-platform portability or standard runtime API compliance across edge providers, so this axis is a category mismatch for Railway's product type.
devops-leadVisualize and edit my entire infrastructure stack on a visual canvas without writing config files
weight 2 · not comparableFly.ion/aFly.io is a CLI/API-driven infrastructure platform; no visual canvas or GUI-based infra editing is part of its product category, making this a wrong-axis question.
Railway's docs explicitly describe a visual canvas showing the entire stack, with settings editable directly on canvas and YAML marked optional, matching the story closely. Missing for 10: independent hands-on review/screenshots demonstrating full editing depth on canvas beyond marketing copy, and no community corroboration of the canvas editing experience specifically.
- [claimed-docs] “Craft on a visual canvas that makes your entire stack visible at a glance.”
- [claimed-docs] “Modify settings directly from the canvas. YAML optional.”
- [claimed-docs] “Connect your repo, Railway handles the rest. Auto-config, instant previews, no new tools to learn.”
ai-native builderRun AI model inference globally with a single call without managing GPUs or capacity planning
weight 3 · not comparableFly.ionone0/10The evidence pack documents Fly Machines/Sprites as general-purpose global compute VMs, agent sandboxes, and Postgres, but contains no mention of a managed AI inference/model-serving API, GPU provisioning, or 'single call' inference capability abstracting away GPU/capacity management. Absence of evidence for this applicable capability yields none.
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “Fly Machines are fast-launching VMs with a simple REST API.”
- [claimed-docs] “You can deploy apps in regions worldwide, so your users in Tokyo, São Paulo, or Amsterdam connect to the nearest server through our global A…”
ai-native builderAccess hundreds of AI models through a single unified gateway endpoint
weight 3 · not comparableFly.ion/aFly.io is an edge-compute/VM hosting platform, not an AI model gateway/router; nothing in the evidence pack indicates a unified model-access endpoint for hundreds of AI models. This capability is a category error for this product type.
Railwayn/aRailway is a deployment/infrastructure platform, not an AI model gateway; nothing in the evidence describes a unified endpoint for accessing hundreds of AI models. This story targets a fundamentally different product category (LLM gateway/router), making the axis a category error for Railway.
developerHandle persistent WebSocket connections from within edge/serverless functions for real-time apps
weight 2 · not comparableFly Machines are persistent, hardware-isolated VMs (not typical ephemeral serverless functions) that boot fast and run close to users globally, which architecturally supports long-lived connections like WebSockets, but no evidence explicitly documents WebSocket handling or real-time connection APIs. missing for 10: explicit WebSocket/real-time API documentation, hands-on example of persistent socket handling in a Fly app, confirmation of behavior under Fly's edge/global routing for stateful connections.
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “Fly Machines are fast-launching VMs with a simple REST API.”
- [claimed-docs] “You can deploy apps in regions worldwide, so your users in Tokyo, São Paulo, or Amsterdam connect to the nearest server through our global A…”
Railwayn/aRailway is a persistent-container/service hosting platform, not an edge/serverless-functions runtime (e.g., Cloudflare Workers, Vercel Edge Functions); the story specifically asks about WebSocket handling within short-lived edge/serverless function execution models, which is a different product category than what Railway offers.
developerRun unmodified Node.js code and npm packages in edge/serverless functions instead of a restricted runtime subset
weight 3 · not comparableFly.io's architecture (Fly Machines are hardware-isolated VMs running arbitrary Docker images, not a restricted sandboxed runtime) implies full support for running unmodified Node.js/npm code rather than a limited edge-function subset, and docs mention converting Docker images to fast-booting global VMs. However, the evidence pack never explicitly confirms Node.js/npm support, unmodified code compatibility, or names it as an edge-function alternative. Missing for 10: explicit Node.js/npm runtime documentation, confirmation that npm native modules/unsupported APIs work unmodified, and independent developer reports of running standard Node apps without modification.
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “Fly Machines are fast-launching VMs with a simple REST API.”
- [claimed-docs] “The Fly Machines REST API provides resources to provision and manage Fly Apps, Fly Machines, and Fly Volumes.”
- [claimed-docs] “Fly.io converts Docker images to fast-booting VMs and runs them globally. This is great or ingesting incoming data!”
Railwayn/aRailway is a general container/service deployment platform (build from repo, deploy full services with CPU/RAM scaling), not an edge/serverless-functions product with a restricted runtime that this story presupposes; no evidence describes an edge-function runtime at all, restricted or otherwise, so the axis is a category mismatch rather than an unmet capability.
- [claimed-docs] “Railway is an all-in-one intelligent cloud provider that makes it easy to provision infrastructure, develop locally, and deploy to the cloud…”
- [claimed-docs] “Scale CPU and RAM or add replicas with load balancing.”
- [claimed-docs] “Run your application closer to where your users are.”
developerWrite edge functions in languages other than JavaScript such as Python, Go, or Rust
weight 2 · not comparableFly.io runs any Docker container as a Fly Machine deployed globally close to users, which inherently supports any language (Python, Go, Rust, etc.) rather than restricting to JavaScript, and docs highlight global regional deployment via Anycast for low-latency edge-like compute. However, there is no explicit 'edge functions' framework or per-language quickstart tutorials cited, so this is inferred from the general VM/container model rather than dedicated edge-function tooling. Missing for 10: explicit per-language edge-function guides/examples for Python, Go, Rust, and independent corroboration of using Fly for edge-function-style workloads.
- [claimed-docs] “Fly Machines are hardware-isolated VMs that boot in under a second, run close to your users, and scale out to tens of thousands when the tra…”
- [claimed-docs] “You can deploy apps in regions worldwide, so your users in Tokyo, São Paulo, or Amsterdam connect to the nearest server through our global A…”
- [claimed-docs] “Fly.io converts Docker images to fast-booting VMs and runs them globally. This is great or ingesting incoming data!”
- [claimed-docs] “Fly Machines are fast-launching VMs with a simple REST API.”
Railwayn/aRailway is a general-purpose cloud deployment/hosting platform for full applications and services, not an edge-function/edge-compute product with a language-restricted runtime; the story's premise (writing edge functions in specific languages) is a category error for this product type.
developerProtect my app from automated bot traffic with invisible challenge-based protection
weight 1 · not comparableFly.ion/aFly.io is an infrastructure/compute platform (VMs, apps, machines); bot-challenge/CAPTCHA-style traffic protection is a WAF/CDN-layer feature outside its product category, and no evidence pack item discusses such a capability.
Railwaynone0/10No evidence in the pack mentions bot protection, challenge pages, CAPTCHA, or invisible bot-mitigation of any kind; Railway's docs focus on deployment, scaling, and networking basics like SSL/load balancing, not bot/WAF-style traffic protection. missing for 10: any mention of bot detection, challenge-based traffic filtering, or WAF/CDN-style bot mitigation features.
devops-leadTrack site performance and visitor behavior with built-in analytics
weight 1 · not comparableFly.ion/aFly.io is an infrastructure/compute platform (VMs, deploys, scaling), not a web-analytics or site-behavior-tracking product; built-in visitor analytics is outside its category (though it offers metrics/logging for ops, not visitor behavior analytics).
Railwaynone0/10Railway's docs show infrastructure-level metrics dashboards, logs, and alerting (railway-docs-8, railway-docs-9, railway-docs-10) but nothing about tracking visitor behavior or site-level analytics (page views, user sessions, traffic patterns) as a built-in feature — the evidence only covers service/CPU/RAM/log observability, not visitor analytics.
- [claimed-docs] “Create custom dashboards with metrics from any service.”
- [claimed-docs] “Slack, Discord, or emails the moment conditions you specify are met.”
- [claimed-docs] “All logs in one place. Spot issues without switching tools.”
ai-native userPrevent my data from being used to train AI models
weight 3 · not comparableFly.ion/aFly.io is an infrastructure/compute platform (VMs, deployment, databases) and not an AI model provider that trains on user data or offers an opt-out; this privacy-posture axis about AI training data is a category error for this product type.
Railwaynone0/10No evidence pack items mention data usage for AI model training or any opt-out/privacy controls regarding AI training; this is a cloud infrastructure provider and the axis of preventing AI training on user data is applicable (as with any SaaS handling customer data/code) but wholly unaddressed. Missing for 10: any privacy policy statement on AI training, opt-out settings, or data usage terms.
developerResize and optimize images on demand via a built-in image CDN
weight 1 · not comparableFly.ion/aFly.io is a compute/infrastructure platform (VMs, Postgres, deploy tooling); it does not offer an image CDN/transformation product, and this is a wrong-axis question for its category rather than a missing feature.