Why the world’s most reliable systems embrace failure instead of preventing it
Picture this: You’re building a system that must not go down. Lives might depend on it. Your instinct says: add more error handling, more validation, more checks. Catch every exception. Handle every edge case. Build walls of defensive code.
Now picture doing the exact opposite: Let it crash. When something goes wrong, don’t handle it—just die and restart.
This sounds insane. Yet this philosophy, inherited from Erlang and embraced by Elixir, powers systems with uptime measured in nines you can count on both hands. We’re talking about 99.9999999% uptime—about 31 milliseconds of downtime per year.
1. The Problem with Defensive Programming
Most programming languages teach you to be defensive. In Java, you write try-catch blocks. In Go, you check every error. In Python, you add exception handlers everywhere. The goal is noble: prevent crashes at all costs.
But here’s what actually happens:
1.1 The Hidden Cost of Error Handling
Research from Stanford found that error handling code contains bugs at rates 2-10 times higher than normal code. Why? Because error paths are rarely tested, often written as afterthoughts, and create exponentially complex state spaces.
Consider this scenario: You make a database call. It might succeed, fail immediately, timeout, return partial data, or throw an exception. For each outcome, you have more code—and each of those branches can fail. You’ve just created a tree of failure states that’s impossible to test comprehensively.
| Code Type | Bug Rate | Test Coverage | Complexity Factor |
|---|---|---|---|
| Happy path logic | 1x (baseline) | 85-95% | 1x |
| Error handling code | 2-10x | 20-40% | 3-5x |
| Nested error handlers | 5-15x | 5-15% | 10-20x |
Elixir’s philosophy says: Don’t write that error handling code. Just crash. Let something else worry about recovery.
2. The Supervisor Tree: Architecture of Resilience
The magic that makes “let it crash” work is the supervisor tree. Think of it like a management hierarchy at a company, but one that actually works.
At the bottom, you have worker processes doing actual work—handling requests, processing data, managing connections. These workers are allowed to be fragile. They focus on the happy path and crash when things go wrong.
Above them, you have supervisors whose only job is to watch workers and restart them when they crash. Supervisors don’t do real work—they just supervise. And above those supervisors? More supervisors, creating a tree structure.
2.1 Restart Strategies That Match Reality
Supervisors can restart processes with different strategies, each matching a real-world failure pattern:
One-for-one: If a worker crashes, restart just that worker. This is like having a team where if one person gets sick, you bring in their backup, but everyone else keeps working. Perfect for independent workers like web request handlers.
One-for-all: If any worker crashes, restart all workers under this supervisor. This is like restarting your entire web server when the database connection dies, because all workers share that connection. Sometimes a clean slate for everyone is simpler than managing partial states.
Rest-for-one: If a worker crashes, restart that worker and all workers started after it. This handles dependencies—if process B depends on process A, and A crashes, you need to restart B too.
Key Insight: These strategies aren’t trying to prevent crashes—they’re managing crashes as a normal operational event. The system is designed assuming things will fail, not hoping they won’t.
3. The Nine Nines: Real-World Uptime
Here’s where theory meets reality. Ericsson reported that their AXD301 telecommunications switch, built on Erlang’s “let it crash” philosophy, achieved 99.9999999% availability. That’s nine nines.
Compare this to typical uptime targets:
| Uptime % | Downtime per Year | Typical For |
|---|---|---|
| 99% (“two nines”) | 3.65 days | Consumer web apps |
| 99.9% (“three nines”) | 8.77 hours | Standard SaaS |
| 99.99% (“four nines”) | 52.6 minutes | Premium SaaS |
| 99.999% (“five nines”) | 5.26 minutes | Cloud infrastructure |
| 99.9999999% (“nine nines”) | 31.5 milliseconds | Telecom switches (Erlang) |
How? Because when a component fails, it fails fast, gets restarted clean, and the rest of the system keeps running. The alternative—systems limping along in corrupted states—causes outages that cascade and take down everything.
3.1 WhatsApp’s Billion Users
WhatsApp famously scaled to 900 million users with just 50 engineers, largely thanks to Erlang’s architecture. Their servers handled 2 million connections each—a number that sounds impossible in languages built on defensive programming.
The secret wasn’t that their code never crashed. It’s that crashes were isolated, automatic recovery was built-in, and engineers could focus on features instead of defensive coding.
4. Why This Feels Wrong to Most Developers
If you come from Java, C#, Python, or most mainstream languages, “let it crash” feels deeply wrong. Your instincts scream at you. Here’s why the culture clash is so profound:
4.1 Different Mental Models
4.2 The Cost of Context Switching
In Java, creating a thread costs ~1MB of memory. Crashing and restarting is expensive. Your instinct to prevent crashes makes sense.
In Elixir, creating a process costs ~2KB of memory. You can have millions of processes running simultaneously. Restarting one takes microseconds. Crashes are cheap.
When crashes are expensive, you write defensive code. When crashes are cheap, you embrace them. The economics drives the philosophy.
5. What “Let It Crash” Actually Means
This philosophy is often misunderstood. It doesn’t mean “write buggy code and hope for the best.” Here’s what it actually means:
1. Handle Expected Errors, Crash on Unexpected Ones
User submitted invalid email format? Handle it gracefully—that’s expected. Database returned malformed data? Crash. Something you didn’t anticipate? Crash. The system knows how to recover from crashes. It doesn’t know how to recover from corrupted state.
2. Isolate Failures
Each process is isolated. When one crashes, it doesn’t take others with it. This is fundamentally different from a shared-memory system where one null pointer dereference can corrupt everything.
Dave Thomas, author of “Programming Elixir,” describes it: “In Elixir, processes share nothing. When something goes wrong in one process, it’s isolated. The rest of the system continues normally.”
3. Design for Recovery, Not Prevention
Instead of asking “how do we prevent this error?” ask “how quickly can we recover from this error?” Recovery is often simpler, faster, and more reliable than prevention.
| Scenario | Prevention Approach | Recovery Approach |
|---|---|---|
| Database timeout | Add connection pooling, retry logic, circuit breakers, fallback caches (500 lines) | Crash worker, supervisor restarts it with fresh connection (5 lines) |
| Memory leak | Find and fix leak, add monitoring, tune garbage collection (weeks) | Restart process periodically, memory is reclaimed (minutes) |
| Deadlock | Analyze locking patterns, add timeouts, restructure code (days) | Process crashes on timeout, restarts clean (already built-in) |
6. Adopting the Mindset
You don’t need to use Elixir to benefit from this philosophy. Here’s how to apply it elsewhere:
6.1 Design for Crash Isolation
Use containers, microservices, or separate processes to isolate failures. When one component crashes, it shouldn’t take the whole system down. Companies like Netflix apply this with their Chaos Engineering approach—they deliberately crash components to ensure isolation works.
6.2 Make Restarts Fast and Automatic
Whether using Kubernetes, systemd, or cloud orchestration, ensure crashed processes restart automatically and quickly. The faster restart time, the more viable “let it crash” becomes.
6.3 Externalize Critical State
Don’t keep critical state in memory. Use databases, message queues, or distributed caches. When a process crashes and restarts, it should be able to reconstruct necessary state from external sources.
6.4 Write Simple Code
This is the hardest lesson. Every try-catch block you don’t write is complexity you don’t have. Every error handler you skip is a bug you can’t introduce. Joel Spolsky once noted that the best code is the code you don’t write. “Let it crash” takes this seriously.
7. When Not to Let It Crash
This philosophy has boundaries. Don’t use it when:
- State can’t be reconstructed: If losing process state means losing critical data (and you can’t recover it), you need different strategies
- Restart cost is high: If spinning up a new process takes minutes, crashing isn’t viable
- External resources are scarce: If you’re managing hardware, file handles, or expensive external connections, you can’t just restart
- Errors are expected business logic: User validation errors aren’t crashes—handle them normally
The philosophy works best for distributed systems handling many concurrent operations where isolation is possible and restarts are cheap.
8. What We’ve Learned
The “let it crash” philosophy inverts traditional thinking about reliability. Instead of preventing failures through ever-more-complex defensive code, it embraces failures as normal events and focuses on fast, automatic recovery.
This approach achieves something remarkable: systems become simpler and more reliable simultaneously. By writing code for the happy path and letting supervisors handle failures, you reduce complexity, eliminate bug-prone error handling code, and create systems that automatically recover to known-good states.
The numbers speak for themselves. Nine nines of uptime. Millions of concurrent connections per server. Billion-user systems managed by small teams. These aren’t accidents—they’re the natural result of a philosophy that works with failure instead of against it.
For developers coming from traditional languages, this requires a fundamental mindset shift. You must overcome years of instinct telling you to catch every exception and handle every edge case. You must trust that sometimes the best error handling code is no error handling code at all.
Whether you adopt Elixir or not, the lessons apply: Isolate failures. Make recovery fast and automatic. Write simple code. And when something goes wrong in a way you didn’t anticipate, sometimes the smartest thing to do is admit defeat, crash cleanly, and start fresh.
Because in complex systems, a clean restart is often more reliable than limping along in an unknown state. The telecom switches, messaging apps, and distributed systems that keep our world connected have proven this. Maybe it’s time to let it crash.
