The Epistemology of Production Incidents. What post-mortems actually tell you — and what they systematically cannot
Post-mortems are the closest thing software engineering has to a scientific method. Yet every structured inquiry into failure carries a quiet epistemological trap — one that most teams never name, let alone address.
The blameless post-mortem, popularised by Google’s Site Reliability Engineering culture and documented in their widely-read SRE Book, is now standard practice at most mature engineering organisations. The idea is simple and genuinely good: when things go wrong, investigate without assigning blame, understand the system, and prevent recurrence. Furthermore, the practice encourages psychological safety — engineers can speak honestly without fear of becoming the scapegoat.
But here is the problem nobody talks about: post-mortems reconstruct causality backwards from a known outcome. That single structural fact shapes — and distorts — everything they can discover. In other words, they are biased by design, not by incompetence.
1. The Hindsight Machine
When your database falls over at 2 a.m. on a Tuesday, the post-mortem that follows is a story told in reverse. You start with the alarm, then trace backward: the query plan changed, the index was missing, the schema migration ran without a review. It all seems, in retrospect, obvious.
This is precisely what psychologists call the hindsight bias — the tendency, once an outcome is known, to overestimate how predictable it was. Gary Klein and Daniel Kahneman explored this dynamic extensively in their joint work, most accessibly summarised in Kahneman’s Thinking, Fast and Slow. The bias is not a moral failure; it is a feature of how human cognition reconstructs narrative. And post-mortems, being narratives by nature, amplify it.
Consequently, the timeline you produce is clean, logical, and seductive. It suggests a world where incidents are comprehensible chains of events. However, that tidiness is partly an artefact of the reconstruction, not a property of the incident itself.
“The system did exactly what it was designed to do. We just didn’t understand what it was designed to do until it failed.”— Paraphrased from Richard Cook, How Complex Systems Fail
2. Proximate vs. Distal Causation
Philosophy of science distinguishes between proximate causes — the immediate trigger of an event — and distal causes — the deeper structural conditions that made the trigger dangerous in the first place. A post-mortem is exceptionally good at finding proximate causes. It is structurally weak at distal ones.
Consider a real-world pattern. A service goes down because a dependency times out. The proximate cause: no circuit breaker was configured. But why was there no circuit breaker? Because the architecture grew organically over three years, the original authors left, no one owned the service holistically, and the team’s on-call rotation had never had the time or mandate to review resilience patterns. That is the distal cause. The post-mortem will likely capture the first; it rarely reaches the second.
This is not laziness. It is structural. A post-mortem is bounded by the incident: what happened, when, and how. By contrast, the distal causes are unbounded — they live in hiring decisions, in technical debt, in the organisational pressures that discourage saying “we need two weeks to harden this before shipping.”
| Cause Type | Example | Visible in Post-Mortem? | Addressed By |
|---|---|---|---|
| Proximate | Missing circuit breaker | Usually yes | Action item in post-mortem |
| Distal (technical) | No resilience review process | Occasionally | Architecture reviews, chaos engineering |
| Distal (organisational) | On-call team too overloaded to harden services | Rarely | Engineering leadership + process change |
| Systemic / Latent | Decade of architectural decisions favouring velocity over resilience | Almost never | Longitudinal system studies, game days |
3. The Observer Effect in Complex Systems
Richard Cook’s landmark paper How Complex Systems Fail — required reading for any SRE or platform engineer — articulates something that the philosophy of science community had long established in other domains: complex systems are never in a “safe” state. They are always running with some combination of latent failures, workarounds, and compensating mechanisms. The incident is simply the moment when enough of those latent conditions aligned.
This creates what we might call an observer effect problem for post-mortems. The moment an incident occurs, the system’s normal operating state has already been disrupted. You can observe the failure mode. But the pre-failure equilibrium — all the subtle adaptations that engineers had quietly put in place — is partially overwritten by the incident itself. Additionally, the humans involved recall the incident timeline, not their ambient mental model of the system on a normal Tuesday.
Where Post-Mortem Action Items Actually Land
As the chart above illustrates, the overwhelming majority of post-mortem action items cluster around proximate, technical fixes — configuration changes, alerting improvements, runbook updates. Fewer than one in five result in structural or process changes, and organisational or architectural remediation is remarkably rare. This is not because engineering teams lack ambition; it is because the post-mortem format makes those deeper causes harder to articulate and harder to assign ownership.
4. The Narrative Trap
Post-mortems are, fundamentally, stories. They have a beginning (normal operations), a middle (something went wrong), and an end (resolution and lessons). That narrative shape is what makes them readable and shareable — and it is also what limits them.
Stories require protagonists, causes, and resolutions. Therefore, when writing a post-mortem, teams unconsciously select facts that fit this structure. A configuration flag that nobody noticed for six months becomes “the root cause.” The five other configuration flags in a similar state — none of which triggered an incident — go unmentioned, because they have no narrative role. However, they are part of the true causal picture.
This is closely related to what philosopher of science Bas van Fraassen called contrastive explanation: we don’t just ask “why did X happen?” but “why did X happen rather than Y?” Post-mortems rarely ask the contrastive question. They ask why the incident happened; they almost never ask why similar conditions elsewhere in the system did not yet produce an incident — and what that difference reveals.
The Silence Problem: The most dangerous failures in complex systems are often the ones that didn’t happen yet. A post-mortem can only observe what occurred; it has no native mechanism for surfacing the near-misses, the workarounds, or the quietly accumulating brittleness that has not yet crossed the incident threshold. Those signals require a fundamentally different kind of observation.
5. What Post-Mortems Are Genuinely Good At
It would be wrong — and uncharitable — to read this as an argument against post-mortems. They remain indispensable for several important reasons. For one thing, they create a shared record of what happened, which prevents the organisation from re-learning the same lessons. Moreover, the act of writing one forces clarity: teams that cannot produce a coherent post-mortem often discover, in the process, that they do not actually understand what happened.
Post-mortems also serve a social function that their technical critics tend to underweight. They are a ritualised space for acknowledging that something went wrong without assigning blame, which is essential for psychological safety. As Amy Edmondson’s research at Harvard Business School shows, teams with higher psychological safety report more errors — not because they make more errors, but because they are willing to surface them. A good post-mortem culture is part of that infrastructure.
MTTR vs. Post-Mortem Quality Score
The data from DORA’s research shows a meaningful relationship: organisations with higher post-mortem quality scores — covering timeline, contributing factors, and follow-through on action items — show notably lower MTTR in subsequent related incidents. Notably, however, the correlation weakens substantially for novel incident types, which is precisely the class of failure that distal causes produce.
6. What They Systematically Cannot Do
Given everything above, it is worth being precise about the structural limitations. Post-mortems cannot, by design, surface:
Silent failures. If something has been broken in a non-disruptive way for months — a queue that processes more slowly, a backup that quietly fails, a cache that never invalidates — it will not appear in an incident timeline. It produces no incident until, eventually, it does.
Organisational causation. The decision to ship without load testing, the on-call team that was too exhausted to review the design doc, the product pressure that shortened a hardening sprint — these are rarely captured with the specificity needed to produce actionable change.
The normal work that prevented worse incidents. Resilience researchers like Woods, Dekker, and Cook emphasise that most complex systems stay running because of the invisible adaptive work engineers do continuously. Post-mortems study failure; they almost never study the skilled recovery and workaround behaviour that keeps things running the rest of the time. That asymmetry distorts the picture.
7. Practices That Surface the Deeper Picture
If post-mortems are necessary but insufficient, what else does a mature engineering organisation need? Several complementary practices address the gaps, though none of them are as tidy or as well-institutionalised as the post-mortem format.
Game Days and Chaos Engineering
Chaos engineering — deliberately injecting failures in a controlled environment — is one of the few practices that surfaces latent failures before they produce an incident. Netflix’s Chaos Monkey is the famous example. The important insight is that chaos engineering asks the contrastive question: under what conditions does the system fail, versus not? That is precisely the question post-mortems cannot ask.
Service Level Objectives as Leading Indicators
SLOs (Service Level Objectives) are a mechanism for making slow degradation visible before it becomes an incident. An error budget that is burning down faster than expected is a signal of systemic brittleness — not a single proximate cause, but an accumulation of them. Teams that review SLO burn rates weekly are, in effect, conducting a continuous low-grade version of the deeper investigation that post-mortems rarely reach.
Resilience Reviews and Architectural Retros
Separate from incident-driven reviews, a periodic architectural retrospective asks: where in this system are we most likely to be surprised? These are qualitative, speculative, and hard to make rigorous — but they are specifically designed to surface the distal, structural causes that incident timelines obscure. Teams at organisations like Spotify, Slack, and Shopify have published thoughtful accounts of how they run these; the InfoQ archive contains several worth reading.
Near-Miss Reporting
Aviation adopted near-miss reporting decades before software engineering considered it. The NASA Aviation Safety Reporting System (ASRS) has collected voluntary, anonymous near-miss reports since 1976 — and the safety improvements that followed were substantial. The equivalent in software engineering — a lightweight, voluntary, psychologically safe mechanism for reporting “this almost went wrong in a way we didn’t understand” — remains rare. Most organisations have no formal channel for it at all.
| Practice | Proximate Causes | Distal / Systemic Causes | Silent Failures |
|---|---|---|---|
| Post-mortem | ✓ Strong | △ Weak | ✗ None |
| Chaos engineering | ✓ Strong | ✓ Good | ✓ Good |
| SLO burn-rate review | △ Partial | ✓ Good | ✓ Good |
| Architectural retro | ✗ Not focus | ✓ Strong | △ Partial |
| Near-miss reporting | △ Partial | ✓ Good | ✓ Strong |
8. A Different Kind of Attention
The deeper problem — and the honest conclusion — is that surfacing distal causes requires a fundamentally different kind of attention than incident review. Post-mortems are reactive, bounded, and narrative. The practices that surface structural brittleness are proactive, unbounded, and statistical. They require sustained, unglamorous investment: reviewing burn rates no one is alerting on, running game days when nothing is broken, maintaining near-miss channels when nobody is sure they are worth the overhead.
That kind of attention is hard to sustain because it produces no visible return until, eventually, it prevents a catastrophic incident that therefore never appears in anyone’s post-mortem queue. It is the fundamental economics of prevention: the benefits are invisible, and the costs are immediate.
Furthermore, organisations that invest in it tend to be the ones that have already experienced a sufficiently painful incident to justify the spend. In other words, even the decision to build better observability into your failure-learning process is often driven by a post-mortem. The epistemological limitations of the format, in a rather circular way, are part of what perpetuates them.
“A post-mortem is a theory of the past. What you actually need is a theory of your system.”— Inference drawn from Cook, Woods & Rasmussen’s work on resilience engineering
9. What We Have Learned
Over the course of this article, we have traced the epistemological limits of the post-mortem format. We started with the hindsight bias — the way knowing an outcome makes causality seem obvious in retrospect. We then moved through the distinction between proximate causes (what the post-mortem captures well) and distal, structural causes (what it largely cannot). We examined the observer effect: how the act of investigating failure partially overwrites the pre-failure operating state that most needs understanding. And we looked at the narrative trap — the way the story format selects for facts that fit a timeline, excluding the silent failures and near-misses that don’t.
Importantly, none of this makes post-mortems less valuable. They remain the best-institutionalised practice for learning from failure in software engineering, and their social function — normalising honest conversation about what went wrong — is not trivial. However, treating them as sufficient is the real epistemological error. The practices that surface the deeper picture — chaos engineering, SLO burn-rate reviews, architectural retrospectives, and near-miss reporting — require a different kind of attention: proactive, sustained, and largely invisible until it works.
The honest summary: post-mortems tell you a lot about the last fire. They tell you surprisingly little about the fire that has not yet started.
