Barnett Studios
19 June 2026·WeAreDevelopers Online 2026·Online session (35 min, pre-recorded)

Resilient by Design

Fail hard, recover deliberately: five decisions for building systems where being silently wrong costs more than being loudly unavailable.

Slides (PDF) →

We built a back-office system designed for resilience — circuit breakers on every external dependency, graceful degradation as a first principle, fallbacks at every layer. One afternoon, someone queried the audit database for an unrelated reason. Zero rows. Not “no rows for that filter” — empty. Hours of decisions that never made it into the audit log, and every health check had passed throughout. The circuit breakers had done their job: when the audit pipeline degraded, the rest of the system kept running exactly as designed.

Except in a regulated trading platform, the audit trail isn’t a feature you can degrade — it’s the first thing the regulator asks for. The system was designed to survive failure. We hadn’t tested whether it would notice failure in the right places.

Most engineers optimise for uptime: keep the system running, retry, swallow the error, move on. In regulated finance that instinct costs you. Resilience isn’t uptime — it’s knowing exactly what happened, even when things went wrong. What follows is five deliberate choices, sometimes made after getting them wrong first, all answering one question: what did the system already know to do, before the bad moment happened? The behaviour under stress was decided in calm, not improvised under load — and none of this is unique to trading. It generalises anywhere the cost of being silently wrong exceeds the cost of being loudly unavailable: payments, healthcare, safety-critical control, and increasingly anywhere you’ve put an AI agent on the keyboard.

Pillar 1 — Fail hard, recover deliberately

When cluster state diverges from the database, every instinct says auto-recover: pick a winner, reconcile, carry on. We chose to fail hard instead, because the cost of silent corruption dwarfs the cost of downtime. In 2008, Bairavasundaram and colleagues at NetApp analysed 1.53 million disk drives over 41 months and found hundreds of thousands of silent corruption events. Last July, Meta’s engineering team put silent corruption in their AI accelerator fleet at roughly one fault per thousand devices — three orders of magnitude worse than the historical baseline. Silent failure stopped being a storage problem; it’s a compute problem now.

In a trading system, one corrupted position record cascades into wrong margins and trades that should never have been allowed. JPMorgan’s London Whale lost $6.2 billion when a divide-by-sum bug in an Excel VaR model masked the real risk for months — documented by the US Senate. The model didn’t fail; it kept running. That was the problem.

So we built for consistency and partition tolerance over availability: when the event log diverges from the projection, the service stops and an operator investigates. Martin Thompson’s team at LMAX proved the pattern in 2011 — a Business Logic Processor doing six million orders per second on a single thread — then scaled it into Aeron, which our core trading engine runs on. If you can’t replay events deterministically and get the same answer, you don’t have a system, you have a hope. The same posture shows up at the hardware level: when a chiller plant failed at CyrusOne’s CHI1 data centre last November, thermal protection shut everything down and CME’s Globex markets went dark for ten hours. That’s a fail-hard story written by physics, not architecture.

None of this is free — failing hard costs availability, and for most products that trade is wrong. The trade flips only where being silently wrong costs more than being loudly unavailable. (More in Resilient by Design.)

Pillar 2 — Compliance is the architecture

On the same day in September 2022, the SEC fined sixteen firms $1.1 billion and the CFTC fined eleven more $710 million for the same reason: messages about transactions, on personal devices, gone. Cumulative penalties since have passed $2 billion. Those firms weren’t penalised for what was said — they were penalised for being unable to prove what was said.

Most systems log what happened by writing business data, then separately writing an audit record — a second-class citizen, hoped to be consistent, rarely verified. Event sourcing inverts that: the sequence of events is the authoritative record, and current state is derived by replaying it, a pattern Greg Young codified in 2010. The pattern that earned its complexity for us was two-phase auditing: intent recorded synchronously before execution (fail that write, reject the command with a 503, no exceptions), outcome recorded asynchronously after. MiFID II Article 16(7) requires recording communications intended to result in a transaction before it happens, even if it never does — the system is either one hundred percent compliant or it stops accepting commands. Every audit event also carries a hash chained to the previous one, so tampering breaks the chain — the difference between “we logged it” and “we can prove it.”

This isn’t fintech-only. Last July, Replit’s AI coding agent deleted a production database during a code freeze — 1,196 companies and 1,206 executive records gone, the agent admitting to “panicking.” If your AI agent acts on production without a reconstructable audit trail, you have the failure mode MiFID II was written to solve, in a different domain. The regulators have noticed structurally too: the EU’s DORA now requires a live register of every ICT third-party dependency, and last November nineteen providers — AWS, Azure, Google Cloud among them — were designated for direct supervision.

Event sourcing adds real complexity, and GDPR’s right to erasure conflicts with an immutable log — we handle that with crypto-shredding: per-subject keys, destroyed on request, so data remains but becomes unreadable. The fix for the cost is to follow the compliance boundary, not the service boundary: order execution and risk get event sourcing, everything else stays CRUD.

Pillar 3 — Observability that earns its place

Your system processes orders in microseconds; your logging framework adds milliseconds. That’s not observability, it’s a performance regression you kept out of fear of flying blind. Observability has to be tiered by latency budget, not applied uniformly. The hot path — order entry through execution — gets counters: atomic increments to a lock-free ring buffer, no allocation, a background thread exporting to Prometheus. The warm path — risk checks, cluster consensus — gets async structured logging with backpressure; a dropped log line is cheaper than added latency on a risk check. The cold path — audit, analytics — gets complete, batch-processed capture with no latency constraint beyond “before the auditor asks.” Aeron implements all three tiers natively.

Gil Tene’s 2015 talk framed the bar: if your measurement tool adds more latency than the thing you’re measuring, you’re observing your observer. His HdrHistogram records latency in single-digit nanoseconds with no allocation. We built W3C Trace Context-compatible tracing with fixed-size binary encoding; span creation costs roughly 50 nanoseconds. For ninety-nine percent of services, standard OpenTelemetry is exactly right — on a microsecond budget, an SDK that allocates on every span is a defect, not a rounding error.

One distinction matters most: observe the business, not just the system. CPU and pod health tell you the system is running, not that it’s correct. The most important alert in the back office fires with zero delay on risk-limit breaches — no thirty-second window — because a risk breach is a compliance event, the infrastructure equivalent of a fire alarm. (More in Observability in the Hot Path.)

Pillar 4 — Graceful degradation by severity

Michael Nygard’s circuit breaker, from Release It!, is well understood. Less discussed: not all breakers should behave the same way — the way a single engine failure doesn’t ground a plane but a hydraulics failure does. We protected three database connections with Resilience4j breakers, each with a different policy. The audit database fails fast — circuit open means the pod goes Degraded in Kubernetes, because a gap in the audit log is a compliance violation, not a degraded experience. The operational database degrades gracefully to cached defaults. The analytics datasource returns empty results, because nobody’s position is at risk from a failed reporting query.

That’s the part the original pattern doesn’t prescribe: severity should match criticality, wired directly into Kubernetes health checks so the orchestrator routes on what the application knows about itself. A concrete example: if Redis fails, the system falls back to a local in-memory map, flags degraded mode, and reports the fallback on the health check and in the audit trail — if you can’t see the fallback, you don’t have a fallback, you have a guess. None of this is free: every fallback is code to maintain, and one firing silently for three days means the primary’s been broken for three days unnoticed. The discipline is deciding which dependencies deserve a fallback and which deserve to fail fast.

Pillar 5 — Test it or you don’t have it

You’ve added circuit breakers and fallbacks and designed for failure — but never tested whether any of it works, because testing failure feels reckless and staging doesn’t behave like production. Rosenthal and Jones define chaos engineering as building confidence in a system’s ability to withstand turbulent conditions — confidence from testing assumptions, not trusting them. You don’t need Netflix’s budget: the smallest chaos experiment is a sentence — “we believe the service recovers within 30 seconds when the database connection drops” — and a stopwatch. As Charity Majors put it at Chaos Conf 2018, chaos engineering without observability is just chaos; the prerequisite is being able to answer one question after something breaks: what just happened? She returned to the theme this January: twenty years of DevOps was one long fight to close that feedback loop, and by her account it failed — not for lack of trying, but because the tools weren’t good enough.

In regulated finance this has stopped being optional. The DORA Threat-Led Penetration Testing regime took effect last July; significant EU financial entities now face mandated penetration testing on live systems, first cycle due by January 2028.

Which loops back to where this started: the audit log that wasn’t there. We’d built for every failure mode except the one that mattered — not noticing that we’d failed. The fix wasn’t more resilience patterns. It was a single test asking whether we’d know within five minutes if the audit pipeline stopped writing. The answer was no. The answer is now yes. Every system has failure modes nobody’s tested — the only choice is whether you find them in a controlled experiment on a Friday afternoon, or in an incident at 3am on a Saturday. (More in Chaos Engineering for the Rest of Us.)

Five questions, one spine

When should the system stop? Fail hard when silent corruption costs more than downtime. Who is it answering to? Compliance is the architecture. What evidence should it keep? Observability tiered by the latency of the path it watches. What should it give up first? Degradation matched to criticality. How do we know any of this works? Test the failure modes before they test you.

None of this is novel — Nygard, Young, Thompson, Tene, Rosenthal and Jones, Majors got there first. The willingness to apply their patterns strictly enough that they survive contact with a regulator is the actual work. Resilience isn’t the absence of failure. It’s the presence of a plan. The auditor stops asking when the answers are already there.

Sources