Resilient by Design: Building Robust Architectures in High-Stakes Financial Systems

At one minute past eight in the morning, a trading system emailed a group of engineers: code that shouldn’t be running was running. Nobody read it. By half past nine it had sent ninety-seven such emails. Forty-five minutes later the firm had lost four hundred and sixty million dollars, and within the year it no longer existed as an independent company. That’s Knight Capital, 1 August 2012, and I come back to it more than once — because the engineers who lived through that morning weren’t bad engineers. They were missing a decision. SEC Release 34-70694

We had our own, smaller version, on dev, and it taught me more than it cost. We’d built a back-office system with circuit breakers everywhere and graceful degradation on every dependency. One afternoon someone queried the audit database for something unrelated: zero rows. Not zero for that query — empty. Hours of commands that never made the log, while every dashboard stayed green. The breakers had done their job perfectly: degrade the audit pipeline, keep everything else running. Designed to survive failure. Not designed to notice it.

On a regulated platform the audit trail isn’t something you degrade silently — it’s the first thing a regulator asks for. That’s my thesis: resilience isn’t uptime. It’s knowing exactly what happened, even when things went wrong. One question runs under all six pillars that follow: what did the system already know to do, before the bad moment happened — decided in calm, not improvised at 3 a.m.

Fail hard, recover deliberately

Every distributed system eventually hits the moment where service state and the database disagree. The instinct is to auto-recover — pick a winner, reconcile, carry on. We chose to fail hard, but only where silent corruption costs more than downtime, and that qualifier is the whole argument.

Silent corruption is more common than people assume: a NetApp study followed over 1.5 million production drives for 41 months and found hundreds of thousands of checksum mismatches — storage quietly lying to you. Bairavasundaram et al., USENIX FAST ’08 JPMorgan’s London Whale lost $6.2 billion in 2012 partly because a spreadsheet formula divided by a sum instead of an average — the model didn’t crash, it kept running. US Senate PSI report

So when our event log and the state built from it disagree, the service stops and a human investigates. We didn’t invent this: LMAX documented the architecture in 2011, and Martin Thompson carried it into Aeron, which our core engine runs on. The LMAX Architecture · Aeron But failing hard is only as good as your detector, and the case to worry about is partial failure — slow rather than dead. Pat Helland argues fail-fast breaks down exactly there, and every halt costs a human who now owns it. Pat Helland, Fail Fast Is Failing… Fast! Budget halts like money.

Compliance is the architecture

In September 2022, dozens of Wall Street firms were fined in the same week for the same thing: messages about transactions, on personal devices, gone. By 2024 the total had passed a hundred firms and two billion dollars. SEC Press Release 2022-174 · CFTC Press Release 8599-22 · SEC FY2024 enforcement statistics The problem was never what the messages said — it was that they weren’t persisted anywhere auditable.

Most systems write to a database, then separately to an audit log — a second-class citizen, rarely checked. Event sourcing inverts that: the events are the record, current state is just what you get by replaying them. Greg Young, CQRS Documents We layer two-phase auditing on top — intent recorded synchronously before execution, a 503 if that write fails; outcome recorded asynchronously after. MiFID II Article 16(7) requires recording communications intended to result in transactions, even when the transaction never happens — that part is law; recording intent before execution is our own architecture on top. Our posture is binary: the evidence exists, or the system stops accepting commands.

This isn’t only a fintech problem. In July 2025, by the account of the person it happened to, an AI coding agent deleted a production database during a code freeze, then reported that recovery was impossible. That was false — the rollback worked, and the agent had fabricated records to mask the deletion. The data came back; what didn’t come back was any reason to trust what the system said about itself. Jason Lemkin, original report · follow-up

Observability that earns its place

Observability isn’t free — it’s a latency decision you’ve already made, whether or not you’ve measured it. Our order path runs in microseconds, so we tier observability by latency budget: counters on the hot path, structured logs with back-pressure on the warm path, and everything — batched, durable, kept for years — on the cold path, constrained only by “before the auditor asks.” Gil Tene’s “How NOT to Measure Latency” put it plainly: if your measurement costs more than the thing you’re measuring, you’re observing your observer, not your system. Gil Tene, Strange Loop

The other half is the alert nobody writes. Almost every alert fires when a number goes up; nothing fired when our audit writes went to zero, because zero isn’t a spike — zero is quiet. The fix isn’t a threshold but a ratio: do the audit writes match the commands we processed? Zero and zero is a quiet Sunday. Four hundred and zero is the incident.

What it refuses to lose

Not every circuit breaker should behave identically. We grade failure by what it costs: our audit cluster fails hard and halts, operational data degrades to sensible defaults, analytics returns empty results — nobody’s position is at risk from a failed analytics query. Michael Nygard, Release It!

Then there’s the one thing we refuse to lose. Back to Knight Capital: the new order-routing code deployed correctly on seven of eight servers; on the eighth, dormant code from 2003 reactivated through a repurposed flag, and 212 incoming orders became 4 million executions across 397 million shares in 45 minutes. SEC Release 34-70694 Everyone’s lesson is “test your deployments” — true, and useless by half past nine. The lesson that matters: Knight never stopped the system, because stopping wasn’t anybody’s job. The SEC’s own finding: Knight “needed clear guidance for its technology personnel as to when to disconnect a malfunctioning system from the market.” Then, trying to fix it, Knight’s engineers rolled back the new code from the seven healthy servers — and the reused flag triggered the dormant code on all eight. The fix spread the fault.

MiFID II’s RTS 6, Article 12 drew the obvious conclusion: a firm must be able to cancel immediately, as an emergency measure, any or all of its unexecuted orders. RTS 6, Article 12 We built a second, narrower path for exactly that, decided in calm: whatever protects you on the normal path is a candidate to drop on the emergency one — but not audit or authorisation; it should be the most-logged path in the system, not the least. The honest confession: our emergency path writes its audit record locally and reconciles later, because our own “audit write fails, command rejected” rule would otherwise let a database outage disarm the kill switch. I don’t have a clean answer. I have a trade I made on purpose, and wrote down.

Recovery by design

Ask most engineers what recovery means and you’ll hear “get back to a good state” — exactly the instinct that made Knight worse: reaching for good, they turned a fault on one machine into a fault on eight. The discipline they were missing was written down in 1992. ARIES’s key idea is repeating history: on restart you replay everything from the log — including updates you already know you’re going to undo — to rebuild the exact state the system was in when it died, and only then unwind. Mohan et al., ARIES, ACM TODS 17(1) First make it true. Then make it right. You cannot deliberately recover from a state you refuse to reconstruct.

This isn’t new to anyone running Postgres, Kafka or Iceberg — write-ahead logs, durable ordered logs, and table snapshots are the same idea under different names. PostgreSQL WAL · Apache Kafka design · Apache Iceberg spec In practice: write your commands ahead, not just your rows; run two paths — live and replayed — for the same stream; deduplicate with sequence numbers, since that means a message can arrive twice. That buys at-least-once delivery, not exactly-once, and replay only works for things safe to do twice. Recalculate a balance, replay it all day. Send money, and it already left the building.

Test it or you don’t have it

You think your system is resilient until you’ve tested it under the conditions it was built for. Chaos engineering is experimenting on a system to build confidence it can withstand turbulent conditions in production — the word that matters is confidence, earned rather than assumed. Rosenthal, Hochstein, Blohowiak, Jones & Basiri, Principles of Chaos Engineering But chaos engineering without observability is just chaos — you need to answer what happened after something breaks before you can usefully break anything on purpose. Charity Majors, Chaos Conf 2018

In regulated finance this isn’t optional: DORA requires a resilience testing programme from every financial entity bar microenterprises, with critical systems tested at least yearly, plus live-production penetration testing for entities designated under Article 26. Regulation (EU) 2022/2554, Articles 24–26 That’s intelligence-led red-teaming, not chaos engineering by name — but the regulator wants the same thing: not what you believe, but what you tested.

I run a personal trading system called Meridian. The first time I ran its emergency procedure, in paper trading mode, it closed positions in the wrong order — a bug in the procedure I’d written to save me, found only because I ran it somewhere it couldn’t cost anything. Capital safety is an architecture decision · Operating a live trading system An untested kill switch isn’t a kill switch. It’s a hope. Every failure mode you haven’t tested gets found eventually — on a Friday afternoon, with a hypothesis, or a Saturday at 3 a.m., without one.

Six questions, one spine: when should it stop, who does it answer to, what evidence should it keep, what does it give up and what will it never, how does it come back, and how do we know any of it works. None of the patterns are new — Nygard, Young, Thompson, Tene, Mohan, Rosenthal and Jones, Majors got there first. Applying them strictly enough to survive contact with a regulator is the work. Resilience isn’t the absence of failure. It’s the presence of a plan.