Barnett Studios
29 September 2026·KCD Sofia 2026·Session (30 min)

Building Observability for Microsecond Systems

Observability isn't free — it's a latency decision already made by your framework, your substrate, or nobody at all. Tier it by what each layer can actually afford.

Slides (PDF) →

How long does it take to create a span? opentelemetry-java publishes a continuous benchmark, so anyone can check: a small span costs about three hundred nanoseconds; a larger one, with attributes and data, about twenty microseconds. Most services create thousands of these a second without a guess at that number.

The question I asked the room at KCD Sofia to hold for half an hour was: what if you couldn’t afford that? Not “what if it’s slow” — what if the instrument costs more than what it measures, and nobody noticed, because nothing forced anyone to check. I used a trading platform I’ve worked on as the example, where the hot path has a latency budget measured in microseconds. It isn’t a trading talk — it’s about what happens to your assumptions once the budget shrinks, and at scale it shrinks on more systems than you’d think.

A number is not a measurement

Elastic benchmarked their own OpenTelemetry Java agent against the same service with it switched off: p95 went from 1.96 milliseconds to 2.06 — one tenth of a millisecond of overhead. Is that a lot? It isn’t a property of the number alone. Against a hundred-millisecond web request it’s a tenth of one percent — free. Against a one-millisecond order-processing call it’s ten percent of the whole budget. Against a fifty-microsecond matching engine, that same tenth of a millisecond is twice the entire budget — the observability is now slower than the thing it observes.

The number didn’t change. The budget did. Which is the thesis: observability isn’t free. It’s a latency decision you’ve already made — whether or not you measured it. Not an accusation — the decision got made for you, by something. A default, a framework author, a vendor, an execution model. This talk is about finding out what.

Your framework decided

Logback’s AsyncAppender is the standard way to keep logging off a request thread, but the defaults are a 256-event queue, dropping TRACE/DEBUG/INFO once it’s 20% full, and neverBlock set to false — meaning, in the manual’s own words, the appender blocks on a full queue rather than lose the message. The request thread stops until a background thread drains room. Nobody chose that for your incident; a framework author weighed “lose a log line” against “add latency” with no idea what your budget was. Every stack has its own version of this at every level — log shipper buffer, sidecar queue, ingress timeout, CNI rate limit. Somebody picked those numbers. Probably not you, and even if it was, have you measured what happens when they’re hit?

Your substrate decided

If a framework that doesn’t know your budget is the problem, the fix isn’t less observability — it’s different observability at each layer, chosen by what that layer can pay for. A trading platform’s three tiers: hot (order entry through execution, microseconds, counters only), warm (risk checks, health, cluster consensus, milliseconds, structured logs that drop rather than block under back-pressure), cold (compliance and audit, kept for years). The analogy: a Formula 1 car in testing carries an aero rake, a scaffold of pressure probes mapping the airflow — test equipment, not race equipment, because when peak performance matters the instrument cannot be big enough to change what it’s measuring.

This tiering wasn’t invented for the talk. Aeron, the messaging transport behind a modern trading platform, decided it at the substrate level years ago. It tracks internal state as counters in a memory-mapped file that “can be read by any external process with no significant impact on performance,” and on the error path records distinct, deduplicated error counts with first/last timestamps instead of log files. They didn’t make error logging faster — they replaced it. Counters move observation’s cost entirely off the observed thread; Aeron spends ninety-six of every hundred and twenty-eight bytes in each counter slot on padding, purely so two counters never share a cache line.

Our own counter on top is a plain long increment into a map — safe with a single writer because Aeron Cluster executes each service on one thread, guaranteed by the execution model rather than our discipline. The cheapest concurrency primitive is the one you delete by choosing an execution model that never needed it — though our counters still sit in an on-heap map, and snapshotting allocates and walks every series. Zero-cost observability doesn’t exist.

What a hard decision costs

A label is not free: an untagged counter costs single-digit nanoseconds, one dimension takes it to roughly fifty — ten times the cost, before cardinality even enters the story. Breaching a capped metric map means terminating the unstable node rather than carrying on, because a replica that diverges from its peers is worse than one that’s simply dead. Fail hard and recover deliberately, rather than slowly leaking memory until an out-of-memory crash at three in the morning.

The opposite failure sits next to it: a cache of tag objects that, when full, logs a warning and returns empty tags while the metric keeps recording — it silently stops having dimensions. Nothing breaks, the dashboard stays green, and it quietly becomes less useful. Two caps, opposite philosophies, and nobody decided the asymmetry on purpose.

The decision we didn’t make

The obvious objection: why not Chronicle-Logger, a genuinely good sub-microsecond Java logger writing binary records into a memory-mapped queue? Its own benchmark gives 785 nanoseconds for a simple message against Log4j2’s file appender at 2,400 — about three times faster than what most people use. Put that next to what the hot path needs: a counter increment, single-digit nanoseconds. Chronicle still costs two orders of magnitude more than the primitive required. Not a criticism of Chronicle — a statement about the category. The fastest possible log line is still a log line; we didn’t need a faster logger, we needed none. One level up, the warm path does log, with Logback, mostly at defaults — and there Chronicle would have been defensible. We never evaluated it. Logback was good enough. Hold that thought.

Something decided what “healthy” means

That’s the nanoseconds half. The half that saves you at three in the morning has little to do with performance. Something also decided what “healthy” means on your system — probably whatever your monitoring stack shipped with: CPU, memory, pod health. Those tell you the system is running, not that it’s correct. A cluster node can fail over exactly as designed, every health check passing, dashboard green throughout, while a misconfiguration opens a window where incoming data is silently dropped. Healthy and incorrect — nobody finds out until someone queries the output weeks later.

The alerts that matter most are business alerts wearing infrastructure clothes. The most important one on a trading system fires on a risk-limit breach with zero delay — no for: window — because a risk breach is a fire alarm, and a fire alarm doesn’t wait to confirm the fire is sustained. For you it’s checkout completion or payment success; the metric changes, the principle doesn’t.

My favourite alert watches for an absence rather than an event: a recurring timer that does work, then reschedules itself. If the chain breaks silently, every metric it published simply stops changing — which on most dashboards looks identical to nothing happening. It watches the age of the last reschedule, not the timer’s presence, which is different from Prometheus’s absent(): that catches the series that disappeared, not the one still scraping fine but no longer moving. Check every CronJob, every queue consumer, every heartbeat: how many alert when they fail, versus when they simply stop?

One more from the same family: clamping a histogram value to avoid a crash is the right call only if every clamp is counted separately. Exporters drop on a full queue, samplers discard by design, rate limiters throw data away under pressure — if your instrument has to discard data to survive, count the discards. A silent lossy path is worse than no path, because you’ll trust the number it hands you.

When nothing decides

I went back and read our warm-path configuration properly. The cluster launcher has a well-configured async appender — good queue size, discarding threshold zero, neverBlock true. The same file also wires a synchronous file appender with immediateFlush enabled into the same root logger, so every INFO line performs a synchronous, immediately-flushed disk write on the calling thread — precisely what the async appender exists to prevent. I’d written documentation describing the good configuration without checking the config actually matched it.

I found that by pointing an agent, with read-only access to the repository and our GitOps state, at where the two disagreed — it found this in minutes. Give an agent read-only access to what you declared and what’s actually running, and the gap between them is where incidents live. Keep it off the request path: an agent scoring traces in flight is one more thing deciding your latency budget. It didn’t tell me what to do about the finding. It told me where to look.

The hot path is disciplined because Aeron forces it to be — single-threaded execution, no allocation budget, counters in shared memory. You cannot be sloppy there; the foundation won’t let you. The warm path drifted to framework defaults because nothing stopped it. No test failed, no node died. “Good enough” was true, even under load — it just was never a decision, and nothing ever made us check whether it stayed true. Same engineers, same repository. The difference was never discipline. It was what was doing the deciding. Architectural discipline that nothing enforces decays. Not because anyone got careless — because nobody had to think about it, so nobody did.

Choose what decides

You can’t personally make every one of these decisions. The idea is choosing what makes them for you:

  1. Find out what your observability costs — your own service, agent on and off, your own hardware. If you don’t know the number, you inherited the decision.
  2. Open your logging config and look at neverBlock, or whatever your stack calls it.
  3. Stop instrumenting every tier the same way. You have a hot path even if it’s a checkout. Spend less than it can afford.
  4. Find one thing that fails by stopping rather than erroring, and write the alert that notices its silence.
  5. Check the claims in your own documentation against your own config.

Gil Tene built HdrHistogram for exactly this over a decade ago: a histogram for latency- and performance-sensitive applications, with recording times measured at 3–6 nanoseconds. The tool that measures the outliers must not become one. Logback still blocks by default. Something already decided what your observability costs you. Go and find out what it was.

Sources

Slides: PDF