Systems Nobody Asked Us to Derive

A method validated only against its author’s own examples has passed a take-home exam it also wrote. The remedy is systems this book had no hand in: derive them from their operators’ published commitments alone, register predictions before checking outcomes, grade in the open, and keep the misses. Four systems take the exam here — Stack Overflow in its 2016 configuration, Shopify in the pod era, Discord’s message path across 2017–2023, and a British government registrar whose place in this chapter needs its own explanation when its turn comes.

Honesty about the protocol first. True blindness is impossible for famous systems — their architectures are folklore. Defensibility comes from the same discipline the whole book runs on: every derivation step cites a published answer, never the known outcome, with sources and eras pinned so inputs and outcomes match in time. Predictions go on the record before the answer sheets are assembled, which creates the possibility this chapter values most: the registered prediction and the derivation can disagree, and when they do, one of them is about to be graded against reality. It happens twice below.

Because the four runs were not equally protected, their evidence carries explicit grades, worst protection first:

Three more disclosures belong in the protocol note, because a careful reviewer will ask and the answers favor honesty over polish. Verifiability: the registrations for these four runs are internally documented (dated working files, session records) and attested here rather than externally provable; the artifact set — answer sheets, registered predictions, grading rubrics, and for the fourth run the derivation transcript and operator protocols verbatim — is published as a replication kit at the repository the references cite, and every run from the repository’s creation onward registers in public, by commit, where timestamps verify themselves. Selection: all four systems are famous publishers of engineering writing — organizations selected, by the method’s own sourcing needs, for having coherent documentable architectures; the protocol requires published answers, and publication is not a random sample of the industry. Scoring: graded positions are not independent (a single-deployable position makes direct calls nearly automatic), so raw hit counts flatter any method, including this one. The honest baseline question is sharper and is answered per run below: what would “always predict the boring monolith” have scored, and where did the derivation beat it? Three of the four systems derive to mostly-null vectors, so the naive default scores respectably on position counts. It misses precisely the calls that matter: it predicts read replicas at Stack Overflow where the answers refuse them; it has no way to reach Shopify’s full-stack cells or the edge admission that saves the flash sale; and it keeps Companies House’s register reads unified where the statutes force the split. The method’s value over the default is concentrated in exactly the positions where getting it wrong is expensive — which is what a selection procedure is for. The baseline a critic would rightly prefer is sharper still: not a naive default but an experienced architect’s registered judgment on the same sheets. That comparison exists exactly once in this set — the author’s own Stack Overflow prediction, registered and then beaten by the derivation — and once is not a corpus; running it properly, with architects who are not this book’s author, is standing work the replication kit invites.

Stack Overflow, 2016: the answers beat the prior

The published sheet, from the operators’ own engineering posts: ~209 million HTTP requests a day against a 50-millisecond render budget; a store-level read/write ratio of 40:60 — write-majority at the database under a read-dominated HTTP tier; the entire database cached in memory; a relaxed availability stance stated outright, with deploys taking servers out routinely; ~15 committers deploying 5–10 times a day as one train; a loud cost stance — “hardware is cheaper than developers and efficient code”; no compliance, audit, or replay statement anywhere; and two workloads with genuinely divergent shapes: tag matching (a giant in-memory index with a two-minute reload cycle) and full-text search (inverted-index access, with a published cost citation for offloading it).

The registered prediction: a monolith with read replicas, plus two scope splits. The derivation, run against the sheet, refused half of it.

Deployment: fifteen committers on one uniform train, ops-cost ceiling explicit, availability stance tolerant — nothing presses; single deployable, N identical instances as capacity. Persistence — the step that killed the prediction: the reflex says read-heavy site, read replicas. The answers say the store is 40:60 write-majority and the whole database lives in RAM: the primary contains the read demand outright. What the sheet does press is redundancy: failover drills, a disaster-recovery site. Cheapest containing mechanism: an HA replica that serves failover rather than traffic, asynchronous because the accepted loss budget permits it. The hardware rung, as company policy: the 50-millisecond budget at three thousand requests a second is contained with zero axis moves because they sized the null position: RAM for the working set, web tier idling at 5–15% CPU. Chapter 3’s rung zero, operated as an explicit value statement — and the engineering culture Chapter 3 left unnamed is this one. Two splits, exactly: tag matching’s shape (divergent state, divergent lifecycle) and search’s shape (divergent access, plus their own cost citation) each escape the web tier’s containment — two components, at those two scopes, nothing else diverging. Unified reads with anonymous-scope caching (the cache rung, one below replicas). Current-state storage, roll-forward operational posture, direct calls with one scoped pub/sub for cache invalidation.

Graded against the documented 2016 architecture: a .NET monolith on nine web servers; a tag engine on three dedicated servers and an Elasticsearch cluster — two splits, those two; SQL clusters described by their own operator as one primary taking almost all load with an AlwaysOn replica — the replica is an availability mechanism, exactly as derived and exactly against the registered prediction; Redis caching with no separated read models; roll-forward migrations. Every axis matches, including the splits’ count and location, and including the detail the prediction got wrong.

That miss is the finding. The prediction was a taste-shaped prior — read-heavy site, therefore read replicas — and the derivation, forced to cite answers, refused the taste and was right. The thesis ate its own author, first, in public, and the lesson generalizes: the reflexes this method replaces include the author’s.

Shopify, pod era: the expensive values, earned

The sheet: 80 thousand requests a second at peak across 600 thousand merchants, growing to holiday peaks of 11 million queries a second and 11 terabytes a second of read I/O; flash sales described by the operator as Black-Friday-scale “every seven or eight days,” selling out in seconds; a 2016 incident stated as a demand — one celebrity sale took down every store sharing the database shard; merchants isolated from each other with no cross-shop actions; over a thousand developers, 400 commits a day, deployed as one train around forty times a day; payment-card compliance handled, in their words, by never letting the monolith see card data; checkout write-concentrated on single records, storefront read-dominant and staleness-tolerant.

The derivation’s spine: read volume past any single store’s economics, with a natural partition key the multi-X question hands over (shops never interact), gives sharded on the shop. The 2016 incident plus per-merchant stakes demand that one tenant’s burst cannot reach another; a store-only shard leaves shared caches and queues as cross-tenant failure paths, so the shard boundary widens to the full stack: cells, Chapter 3’s escalation, derived rather than defined. A thousand developers press ownership; the uniform train presses against fragmentation; same scope, decompose. Ownership is real, release independence is explicitly not wanted: modular monolith, the same resolution as the regional ticketing platform at twenty times the headcount. Flash-sale contention, every checkout step hitting the same records, is the second-copy diagnostic saying no: admission control at the edge, queueing and fair ordering before the app tier. Storefront reads at 11 TB/s escape the primary (unlike Stack Overflow’s RAM-contained reads, same chain, different citations, different rung): cache plus replicas at the storefront scope; checkout stays unified and strict. Card compliance: scope exclusion — split the card path out; the monolith exits the mandate’s reach entirely. Oversell: reservation during payment — a hold, design-out. Money: defined inverses, compensation.

Graded: “We chose to evolve Shopify into a modular monolith” — their words; pods as “a fully isolated instance of Shopify with its own datastores,” no actions across pods, a hundred-plus of them; a leaky-bucket edge throttle with queue pages and fair ordering; storefront reading from dedicated replicas with full-page caching; the monolith kept out of card scope. Every registered prediction graded a hit, one under-prediction noted: the derivation’s design-out via reservation was sharper than the registered “compensation for checkout money.”

Discord, 2017–2023: the revision that beat the fashion

The sheet: forty million messages a day growing to four billion, trillions stored under a product commitment made early and kept — “store all chat history forever”; reads “extremely random” at a 50/50 store ratio, drifting read-heavy at the disk; sub-millisecond write and five-millisecond read ambitions with an 80-millisecond P95 alert; five million concurrent connections fanning out millions of events a second; availability preferred explicitly over strong consistency, last-write-wins, per-channel ordering by sortable IDs; hot channels concentrating “unbounded concurrency” on single partitions; and the loudest bound-mode answer in this chapter — four backend engineers, no dedicated operations team.

Two predictions were registered, one of them against this book’s own earlier claim. Queuing Discord for derivation, this author had described its read path as “genuine read-model separation.” The registered revision said that prior was wrong — expect a coalescing layer over a unified store, not CQRS.

The derivation: trillions of rows, node-loss tolerance, linear scaling, four engineers — the single store fails on volume and on operations at once; the partition key is again a gift (messages belong to one channel, reads are per-channel recency), sharded on (channel, time bucket), in a wide-column store shaped to the read (polyglot at the message scope). The real-time fan-out workload diverges from request/response on every dimension: one split, the gateway tier; the API itself stays a monolith — four engineers press hard against anything more. Hot partitions are read-side contention, one channel has one home, so more sharding cannot help: coalesce identical in-flight reads, bound the concurrency; the read model stays unified. Writes reshape (upserts, last-write-wins, sortable IDs), so write conflicts stop existing: design-out. Presence is ephemeral and re-derivable: degrade-and-continue, decay as a feature.

Graded: an Elixir gateway tier beside what the operator calls “our API monolith”; the exact key — ((channel_id, bucket), message_id); Rust data services with “no business logic,” one endpoint per query, coalescing routed by channel ID over the store — the registered revision graded a hit, the fashion (call it CQRS and separate the reads) graded as what the answers never demanded. An honesty note stands in the record: that revision drew on memory of the same public sources, so it counts as a registered-and-confirmed call, a weaker species of evidence than Stack Overflow’s clean answers-beat-prior.

One more Discord finding pays for the whole exercise. Their 2017 store migration (document store to wide-column) moved an axis — architecture-grade, hard, slow. Their 2022 migration (one wide-column product to another) moved no axis — same position, different product. The vector classifies migrations into architectural and procurement before they start, and the two migrations’ documented difficulty matches the classification. Part III builds on exactly this.

Companies House: the boring one, and the strictest protocol

Three consumer-scale celebrities prove range but leave two gaps. The method’s home turf (enterprise backends: statutory deadlines, compliance, moderate scale, long lives) was unrepresented. And one axis had never moved: across three blind runs, the read/write model stayed unified every time, raising a real question of whether the axis deserved its seat at all or should fold into storage. The fourth run was chosen to close both gaps: the UK’s company registrar — 5.4 million companies, statutory filings, a public register — dull by design, and instrumented to be decisive about the read axis either way.

The protocol also tightened, for an honest reason: during target scouting, outcome details leaked into the orchestrating context: research summaries described the real architecture before any derivation ran. The author was contaminated, and rather than pretend otherwise, the run was restructured so contamination could not matter: three isolated operators — an assembler that gathered the answer sheet from demand-side sources only (statutes, audit-office reports, annual accounts: technology venues forbidden); a deriver that received only the answer sheet and Part I’s instruments, with no ability to browse and a standing quarantine rule (any conclusion untraceable to a cited answer plus a named rule is forbidden); and a grader that assembled the documented architecture independently, with citations, after the derivation was registered. One disclosure completes the methods note, foregrounded rather than buried: the three operators were AI agents, each a language-model instance executing its written protocol, with the prompts preserved for the replication kit. Far from weakening the run, this is the mechanizability thesis demonstrated at its strongest — a derivation procedural enough that a quarantined machine can execute it from the sheet and the rulebook alone is a derivation that has genuinely left the realm of taste; the operator could not browse to the outcome — and, a machine executing a written protocol, had no stake to defend and no prediction to protect — while every conclusion it drew traces to a cited answer and a named rule. One limit stays on the record: Companies House is a well-documented public institution whose engineering writing plausibly sits in any modern model’s training corpus, so the protocol controlled contextual contamination, not parametric memory — procedural blindness, not informational blindness, which no language-model operator can currently promise. The evidence that the two came apart here is the miss itself: the derivation predicted the forced minimum and the real registrar had decomposed further; whatever the model had absorbed, it did not quietly copy the documented answer. A run steered by memory of the outcome fails toward the outcome; this one failed away from it. A second distinction the framing owes the reader: the target’s selection was not blind — the run was aimed, deliberately, at a system likely to be decisive about the read axis. The strong claim is about how the derivation ran, not how its subject was chosen; that a statutory registrar reads heavier than it writes surprises nobody, and the surprise the run earns is the axis moving only under cited convergence, not the direction it moved. The methods note stays in the record because that is what methods notes are for.

The demand side is an elicitation showcase. The register was accessed 16.3 billion times in a year against 14.7 million filings: a thousand-to-one read/write asymmetry, from the annual report. December concentrates an accounts-filing peak (28,874 companies once missed the deadline outright; 503 filed in the final hour). And the statutes answer the consistency contract better than most product owners do: a filing accepted in error is not edited: a correcting filing layers on top, the original remaining “properly delivered”; true removal takes a court order; a restored company “is deemed to have continued in existence” — a retroactive legal fiction the storage design must honor; and by statute certain fields (a director’s day of birth) must be withheld from public inspection while remaining on the record — the served shape differs from the stored shape, by law.

The derived vector’s spine: a modest modular core (release structure UNKNOWN, criticality tiers UNKNOWN, and honest UNKNOWNs derive null positions with low confidence, per the method); event-based filing intake (queued, deadline-shaped); the read/write axis moved — public search separated from the filing path, and the forcing was cross-question convergence: the thousand-to-one ratio (a volume answer) alone would have stopped at replicas; the statutory redaction duty (an external-constraint answer) alone is a mechanism note; together they clear the bar neither clears alone — the read shape diverges from the stored truth by law, at register scale. Current-state storage with corrections as layered records (the statutes’ own append semantics); no event sourcing: nothing demands replay; the registrar’s new query-and-annotate powers derive as flagged-pending states; recovery mixed: design-out for the append-shaped register, degrade-and-continue for backfills gated on each company’s own filing anniversary, compensation only where courts order removal.

Graded — ten hits, two partials, two misses, nothing ungradeable. The hits include documentary confirmations of almost embarrassing precision: the public API schema carries a literal annotations field on filing-history items; the operator’s engineering blog describes the pipeline that keeps a search index current from the filing-side stores by name. The read-path prediction graded a hit against first-party documentation — the axis moved in a blind run for the first time, and reality confirmed the move. The watch item died; the axis kept its seat by earning it.

The instructive miss is topology. The deriver predicted a modest modular core and flagged that prediction, in advance, as its most exposed claim: resting on a rule from the first three runs (“headcount presses cadence, never topology”) plus three UNKNOWNs staying inert. Reality: the registrar’s engineering organization describes itself as having adopted “the microservice pattern,” with several hundred public repositories of fine-grained services. The grader suggested the answer sheet should have included that self-description; the suggestion is declined on principle: “we adopted microservices” is the outcome, and feeding it to the deriver makes the experiment circular. The legitimate gap was the UNKNOWN release-structure answer. And the miss rescopes the rule rather than breaking it: nothing in the published demands forces the decomposition — the rule speaks about what forces topology, and it held; what it cannot do is predict what organizations choose beyond the forced minimum. A blind derivation predicts the demanded architecture; the gap between that and the observed one is either an unpublished demand or an unforced position — and detecting unforced positions is, from Chapter 9’s angle, exactly what the audit is for. The method under-predicts fashion. It is supposed to.

To keep that rescoping honest, name what would break the rule rather than bend it. One case would: topology fragmentation with no forcing demand and no organizational home — no separate teams, no divergent cadence, no blast-radius line — structure the unforced-position ledger cannot file anywhere. And one trend would, more decisively: if blind runs keep under-predicting topology — if the forced minimum sits systematically below what disciplined organizations build — then something real is pressing that the sheet does not carry, and “unforced” has become this method’s word for its own blind spot. One run does not make that trend; the commitment is that the ledger of misses stays open to both.

Cross-findings

Four runs, one instrument panel, and the patterns that recur are the ledger validating itself in the field. Contention is side-symmetric: write-side admission control at Shopify’s flash sales, read-side coalescing at Discord’s hot channels (the chapter’s C-grade run, discounted accordingly): the same refusal to shard a one-home problem, at two of the industry’s most-watched systems. The containment chain stops where the citations stop: RAM at Stack Overflow, replicas at Shopify’s storefront, coalescing at Discord, projections at Companies House: four different rungs, each forced by that system’s answers, none by fashion in either direction. Team size never pressed topology: four engineers and a thousand-plus both derived to monoliths from opposite ends, and the one microservices outcome in the set arrived unforced by any published demand — the strongest public evidence this book can offer that “we’re big now, so services” is not a derivation. Scale shape is per-tier: the same system graded read-heavy at HTTP and write-majority at the store; every bare adjective would have been wrong twice. And product identity presses like any SLO: “store all chat history forever” was voluntary — and it pressed the persistence axis as hard as any regulator, priced in the operational record of the company that said it.

Two misses in four runs, both kept, both converted into rules — one the author’s registered prior, which the derivation overrode; one the derivation’s own, the boundary between forced and chosen. A method that can say this is where I stop predicting has a shape; the next chapter shows the sharper version of that property — the derivation refusing to produce an architecture at all, and what it hands back instead.

Rules exercised: citation-or-nothing · pre-registration and open grading · the hardware rung as policy · natural partition keys from multi-X · cells from compounded demands · scope exclusion · the coalescing rung · cross-question convergence · UNKNOWN-derives-null · the forced/chosen boundary.