# napkin.l3.ai — 2026 System Design & Hardware Reality Observatory

> Revisit system-design defaults using dated hardware specifications, scoped measurements, and worked estimates with explicit assumptions.

## Key 2026 Hardware & System-Design Findings
- PCIe Gen6 NVMe (Samsung PM1763): up to 28.4 GB/s sequential read, 21 GB/s write, 6.8M random-read IOPS [S03].
- Latency examples mix a rounded SSD reference, a 2024 fio mean, advertised io2 average targets, and a specific 2026 us-west-2 RTT experiment. They are not universal p50/p99 values.
- The calculator uses 730 hours, N. Virginia Linux on-demand rates, explicit payload sizes and assumed capacity margins. It excludes storage, standby capacity and many other cloud meters. Public egress above 500 TB/month needs a custom quote for actual budgeting.

## 24 System-Design Dogmas Audited
- **1. Measure Before Adding a Remote Cache**: Dogma: "Put a distributed Redis/Memcached cluster in front of PostgreSQL on day one because disk reads take 10 ms." -> 2026 guidance: "Compare the complete cached and uncached paths, including miss behavior, serialization, invalidation, and failure recovery. A remote cache is not automatically faster than a local indexed read." [S30, S32, S57] Worked scenario: 95% hits: 5% reach origin -> Cold cache: 100% reach origin. Assumptions: Assume every miss reaches the origin, no request coalescing, and unchanged incoming traffic. 100% / 5% = 20× origin requests; this does not predict database capacity.
- **2. Size Up Before Committing to Capacity Shards**: Dogma: "Horizontally shard your relational database at 500 GB or 10,000 QPS using application-level shard routers or Vitess." -> 2026 guidance: "Check available single-instance capacity and benchmark the data model before adding shards. Larger hosts do not eliminate availability, contention, or failure-domain requirements." [S10, S38, S53] Worked scenario: 8xlarge: 32 vCPU / 128 GiB -> 48xlarge: 192 vCPU / 768 GiB. Assumptions: Same instance family: 6× CPU and memory, not a 6× application throughput claim. M9g is EBS-only. Costs, storage, and failover are separate.
- **3. A Broker Is a Requirement Decision**: Dogma: "Deploy a multi-broker Kafka or RabbitMQ cluster for every background job queue, webhook processor, or transactional outbox." -> 2026 guidance: "For a local job queue, a PostgreSQL transaction can store business state and job intent together. SKIP LOCKED can support workers; it does not replace every broker feature." [S59, S60, S69, S70] Worked scenario: Two independent writes can diverge -> One database transaction commits both. Assumptions: State and queue row share one PostgreSQL database. External side effects still need idempotency; pollers and queue tables add database load.
- **4. Search Does Not Always Need a Separate Database**: Dogma: "Run Postgres (OLTP) + Elasticsearch (BM25 search) + Pinecone/Milvus (vectors) + MongoDB (JSON) + Redis (cache) with CDC glue." -> 2026 guidance: "PostgreSQL offers text search and JSONB; pgvector adds exact, HNSW, and IVFFlat vector search. Evaluate recall, filtering, ranking, and write overhead before selecting a separate engine." [S67] Worked scenario: 20M × 1,536 × 4 bytes -> 122.88 GB before index overhead. Assumptions: Float32 payload only. Excludes tuple headers, HNSW graph, metadata, replicas, and build space. No QPS, recall, or RAM-fit guarantee.
- **5. Price the Actual Cross-Zone Path**: Dogma: "Decompose into 40 microservices calling each other round-robin across 3 Availability Zones over REST/JSON." -> 2026 guidance: "Use zonal affinity where it preserves the required fault tolerance. Price each endpoint and service-specific path; do not remove durability replication to save transfer fees." [S30, S34, S35, S45, S57] Worked scenario: 2 GB/s × 730 h -> 4,895,031 GiB × $0.02 ≈ $97,901. Assumptions: Decimal GB/s; binary billed GiB; 730-hour month; $0.01/GiB at each ordinary EC2 endpoint. One-way payload only, no return payload, exemptions, discounts, or protocol overhead.
- **6. Separate Log Ingestion from Retention Cost**: Dogma: "Ship 100% of raw un-sampled application logs to a SaaS observability vendor or a 30-node stateful Elasticsearch hot cluster." -> 2026 guidance: "Choose sampling, retention, storage, and query latency separately. Object storage plus an analytical engine can fit some workloads, but storage price is not the total cost of observability." [S39, S43] Worked scenario: 100k lines/s × 1,000 B × 730 h -> 262.8 TB raw / 5 = 52.56 TB stored. Assumptions: Assumed constant rate and 5:1 compression; 52.56 decimal TB = 48,950.31 GiB. One full retained model month at $0.023/GiB-month is $1,125.86. Excludes ingestion, queries, requests, indexing, replication, and transfer.
- **7. Choose Database Semantics, Not a SQL/NoSQL Label**: Dogma: "Assume every SQL database has identical consistency, and every NoSQL database sacrifices transactions for write scale." -> 2026 guidance: "Specify isolation, conflict handling, durability acknowledgments, and partition behavior for the selected product and configuration. Neither SQL nor NoSQL branding defines these guarantees." [S50, S60] Worked scenario: Read Committed -> Serializable when the invariant needs it. Assumptions: PostgreSQL 18 defaults to Read Committed. Serializable can abort transactions; retry the complete transaction and handle external effects safely.
- **8. Provision EBS for the Required Performance**: Dogma: "Provision maxed-out enterprise SAN / io2 volumes for every database, or assume cloud block storage can deliver 2,000 MiB/s at 4-KiB random I/O." -> 2026 guidance: "Price volume capacity and provisioned performance separately. gp3 and io2 have different performance and durability characteristics; neither price proves the other meets the workload." [S26, S27, S41, S42] Worked scenario: io2: 256k IOPS ≈ $9,824/month -> gp3: 80k IOPS + 2,000 MiB/s ≈ $542/month. Assumptions: gp3: 1024×$0.08 + (80000−3000)×$0.005 + (2000−125)×$0.04 = $541.92. io2: 1024×$0.125 + 32000×$0.065 + 32000×$0.046 + 192000×$0.032 = $9,824. Unequal provisioned performance; excludes snapshots and compute.
- **9. A Lease Needs Protection at the Resource**: Dogma: "Run a 5-node Redis Redlock cluster with wall-clock TTL leases to guarantee mutually exclusive distributed locks." -> 2026 guidance: "A paused lease holder can resume after its lease expires. Where correctness depends on ownership, the resource must reject stale fencing tokens, or use an atomic conditional update suited to the invariant." [S60, S66, S71] Worked scenario: Writer with epoch 41 pauses -> Resource accepts 42, then rejects 41. Assumptions: A trusted authority issues monotonically increasing epochs. Every protected operation checks the epoch at the resource. An ETag or expected-version comparison is per-object compare-and-swap, not automatically a fencing protocol.
- **10. Start a URL Shortener with Workload Arithmetic**: Dogma: "Start every URL shortener with an ID service, a sharded database, a cache cluster, and a broker before estimating its traffic." -> 2026 guidance: "Translate monthly totals into average rates, then measure peak demand, skew, redirects, durability, and abuse controls before adding shards or caches." [S10, S56, S57] Worked scenario: 730 h = 2,628,000 s -> 38.05 writes/s; 3,805.18 reads/s. Assumptions: Uniform arrival assumption for averages only. Burst factors, hot keys, query cost, payloads, replication, and failure reserve need separate estimates.
- **11. Retries Need a Budget, Not Another Layer**: Dogma: "Retry every timeout three times at the gateway, service, and database client; more retries mean higher availability." -> 2026 guidance: "Retries multiply across layers. Assign retry ownership, propagate deadlines, add jitter, and make mutations idempotent before retrying." [S56] Worked scenario: 4 × 4 × 4 = 64 -> 1 retry owner: at most 4. Assumptions: Three layers; each permits an initial attempt plus three retries. All attempts fail. Single-owner budget also depends on the remaining deadline.
- **12. Autoscaling Is Not Your AZ-Failure Plan**: Dogma: "Run all three zones near full utilization; the autoscaler will replace capacity after a zone fails." -> 2026 guidance: "Pre-provision surviving capacity for the failure you promise to tolerate. Keep the data plane working while scaling APIs or schedulers are unavailable." [S57] Worked scenario: 1.5D → 100% after AZ loss -> 2.143D → 70% after AZ loss. Assumptions: Three equal zones; one lost; survivors must stay at or below an assumed 70% utilization. C ≥ D / ((2/3) × 0.70). This is not a universal utilization target.
- **13. Low CPU Does Not Mean Spare Capacity**: Dogma: "Scale on CPU alone and keep an unbounded request queue so every request eventually succeeds." -> 2026 guidance: "Bound in-flight work, shed excess load, propagate cancellation, and measure successful responses completed before their deadlines." [S56, S58] Worked scenario: 2,000/s × 5 ms = 10 -> 2,000/s × 500 ms = 1,000. Assumptions: Steady-state illustration using mean arrival rate and mean time in the system. It predicts required concurrency, not a p99 latency.
- **14. Queue Depth Is Not a Recovery-Time Estimate**: Dogma: "Add a durable queue and let workers catch up later; backlog size alone tells us whether the service is healthy." -> 2026 guidance: "Track oldest-message age and net drain rate. Recovery requires service rate greater than incoming work, including retries." [S58] Worked scenario: 1M jobs at μ = λ: never drains -> 1M / (12k − 10k) = 500 s. Assumptions: A fixed initial backlog of one million jobs; sustained arrival 10,000/s; sustained completion 12,000/s; no new failures or capacity changes.
- **15. Exactly-Once Delivery Does Not Make Payments Exactly Once**: Dogma: "Select an exactly-once broker and remove duplicate handling from consumers and external side effects." -> 2026 guidance: "Define the guarantee boundary. Store a business idempotency key atomically with the state change; propagate a stable key to external providers and reconcile unknown outcomes." [S59, S60] Worked scenario: 2 deliveries can charge twice -> 1 key → 1 committed local effect. Assumptions: Local database transaction with a unique (tenant_id, idempotency_key) constraint. External payment effects need provider idempotency or reconciliation too.
- **16. Most IDs Do Not Need a Snowflake Service**: Dogma: "Every scalable service needs a dedicated ID fleet, coordinated worker IDs, and ZooKeeper." -> 2026 guidance: "Use a database sequence or UUIDv7 when those semantics fit. PostgreSQL 18 provides uuidv7(); timestamp ordering is not a global commit-order guarantee." [S61, S62] Worked scenario: Worker-ID coordination -> 48 timestamp bits + 74 remaining bits. Assumptions: UUIDv7 also reserves six version/variant bits. The remaining 74 bits can encode randomness and optional sub-millisecond or counter fields; collision behavior depends on implementation.
- **17. Deep Pagination Needs an Index and a Cursor**: Dogma: "LIMIT 50 OFFSET 1000000 costs roughly the same as page one because the database has an index." -> 2026 guidance: "Skipped rows still require work. Seek from a stable composite key such as (created_at, id), with a matching index and explicit ordering." [S63] Worked scenario: 1,000,050 for OFFSET + LIMIT -> About 50 after a cursor seek. Assumptions: Matching B-tree access path with no additional filtering; excludes tree traversal, MVCC checks, and heap fetches. Work estimate, not a latency benchmark.
- **18. ACID Is Not an Isolation-Level Specification**: Dogma: "A transaction in an ACID database automatically protects every cross-row business invariant." -> 2026 guidance: "Choose constraints, row locks, or serializable isolation for the actual invariant. PostgreSQL defaults to Read Committed; serializable transactions can abort and require a whole-transaction retry." [S60] Worked scenario: 2 doctors read 2 on call; both leave -> Enforce at least 1 on call. Assumptions: Each transaction updates a different doctor after checking the count. No row is shared for locking; use a shared guard row or serializable isolation and retry.
- **19. Three Replicas Are Not Three Backups**: Dogma: "Replicate the database three ways and skip backup restores; any replica can recover lost data." -> 2026 guidance: "Replication also propagates bad writes and deletes. Keep independent recovery history, define RPO/RTO, and rehearse point-in-time restores." [S51, S64] Worked scenario: 10 TB / 1 GB/s = 2 h 47 min -> 10 TB / 5 GB/s = 33 min 20 s. Assumptions: Decimal TB and GB/s; sustained end-to-end throughput. Excludes provisioning, WAL replay, validation, index work, and application recovery.
- **20. Fast Local NVMe Does Not Imply Durable Storage**: Dogma: "A local NVMe fsync and a large SSD are enough to replace durable replicated database storage." -> 2026 guidance: "An fsync protects a local write within the device contract. Instance-store data can disappear on stop, termination, or hardware loss; replication acknowledgments and tested backups define recovery." [S50, S51, S52, S53] Worked scenario: Local-only: host loss can lose data -> RPO depends on replication policy. Assumptions: RPO is the amount of accepted data you may lose. Synchronous durability also depends on replica placement, acknowledgment configuration, fencing, and failover selection.
- **21. S3 Is Not Eventually Consistent for Every Operation**: Dogma: "Object storage always needs sleeps and a separate metadata database to hide stale reads after every write." -> 2026 guidance: "S3 documents strong consistency for object PUT/DELETE and subsequent GET/LIST. That does not provide a transaction across multiple keys or make cross-region replication synchronous." [S65, S66] Worked scenario: Many objects in a dataset -> Conditional update of one object. Assumptions: General-purpose S3 object semantics; use documented If-Match/If-None-Match support for each operation. A table format needs its own metadata commit protocol.
- **22. Vector Count Is Not a Memory Budget**: Dogma: "Ten million embeddings always require a dedicated vector database, and approximate indexes preserve exact recall." -> 2026 guidance: "Estimate dimensions, precision, index overhead, filters, and replicas first. pgvector supports exact and approximate search; HNSW trades memory and build cost for query speed and recall." [S67] Worked scenario: Float32: 61.44 GB (57.22 GiB) -> Float16: 30.72 GB (28.61 GiB). Assumptions: Payload only at 4 versus 2 bytes per dimension. Excludes row headers, HNSW graph, metadata, replicas, and temporary build space; reduced precision needs quality evaluation.
- **23. Token Throughput Is Not an LLM Latency SLO**: Dogma: "Choose an inference service by aggregate tokens per second; that number predicts every user experience." -> 2026 guidance: "Measure time to first token, inter-token latency, end-to-end completion time, and concurrent sequence memory at your prompt/output lengths and arrival rate." [S68] Worked scenario: 20 req/s × 2 s = 40 -> 20 req/s × 20 s = 400. Assumptions: Steady-state mean arrival rate and mean generation residency time; includes waiting if measured end to end. This is concurrency math, not a GPU throughput benchmark.
- **24. A p99 Dependency Does Not Make a p99 Endpoint**: Dogma: "If each of 100 parallel dependencies meets its latency budget 99% of the time, the endpoint does too." -> 2026 guidance: "All-required fan-out compounds tail risk. Budget the entire critical path, reduce synchronous dependencies, and make partial results explicit where the product permits them." [S56] Worked scenario: 100 independent calls: 0.99^100 -> Only 36.6% meet it together. Assumptions: All 100 calls are required and have independent, identical 1% threshold-exceedance probability. Correlated failures require a different model; this does not predict the endpoint p99.

## Machine-Readable Dataset
- JSON API: https://napkin.l3.ai/api/napkin.json