The Fundamental Tension Between Consistency and Availability Is Not a Technical Problem. It’s a Business Decision Most Engineers Are Making Alone
CAP theorem gets taught as a distributed systems concept. What’s completely missing is the organisational dimension: engineers are making these tradeoffs silently, without product owners, business analysts, or any documented agreement with users.
Most engineers have encountered the CAP theorem at some point during their career. It shows up in system design interviews, in architecture documentation, and in database comparison blog posts. The premise is elegant: in a distributed system, you can only guarantee two out of three properties — Consistency, Availability, and Partition tolerance. Since network partitions are essentially unavoidable in any real distributed system, the practical choice comes down to consistency versus availability.
That framing, however, is where most of the education stops. And that is, unfortunately, exactly where the important work begins.
1. What CAP Actually Says — and What It Leaves Out
First articulated by Eric Brewer in his keynote at PODC 2000 and later formally proved by Gilbert and Lynch, the CAP theorem is a precise mathematical statement about distributed systems. It tells us that during a network partition, a system must choose: either respond with potentially stale data (availability), or refuse to respond until consistency can be guaranteed (consistency).
That’s a genuinely important insight. However, the theorem says nothing about what the business requires, nothing about how users will perceive a stale read, and nothing about who should be making these decisions. Those gaps, in practice, are enormous.
CAP was designed to help distributed systems researchers reason about system guarantees. It was never designed as a framework for communicating tradeoffs to product teams — yet that’s exactly the role it gets asked to play.
Additionally, as Martin Kleppmann pointed out in his widely-read critique “Please stop calling databases CP or AP”, labelling a system as simply “CP” or “AP” glosses over the enormous range of consistency models that actually exist — from linearisability all the way down to eventual consistency, with read-your-writes, monotonic reads, and causal consistency sitting in between. Real systems don’t just pick one end; they pick a point on a spectrum, and that point has real consequences for users.
2. Eventual Consistency Is Not a Technical Property
Here is the core argument, and it’s worth stating plainly: eventual consistency is a contract with users, not a property of infrastructure.
When you choose a replication configuration that allows stale reads, you are, in effect, making a promise to every user of your system: “From time to time, the data you see may not reflect the most recent writes.” The critical question is whether your users — and, more importantly, your business — have ever agreed to that promise.
“A user who sees a stale account balance, a double-booked appointment, or a product still showing as in-stock after the last unit sold has experienced a consistency violation — and no amount of architectural elegance makes that acceptable if the business never agreed to it.”
Think about what this means in practice. A payment platform choosing eventual consistency is implicitly deciding that customers can, on occasion, see incorrect balances. A booking system doing the same is accepting that double-bookings may occur. These are not technical decisions — they are product decisions, legal decisions, and trust decisions. Yet, in the majority of organisations, they happen silently at the database configuration layer.
| Consistency Model | What Users Experience | Typical Use Case | Business Risk if Undeclared |
|---|---|---|---|
| Linearisability (Strong) | Every read reflects the latest write, globally | Banking, stock trading, medical records | Low |
| Sequential Consistency | All users see the same order of operations | Collaborative editing, shared state UIs | Low–Medium |
| Causal Consistency | Causally related events appear in order | Social feeds, comment threads | Medium |
| Read-Your-Writes | You always see your own latest write | Profile updates, form submissions | Medium |
| Eventual Consistency | Reads may be stale for an undefined window | DNS, social media counters, analytics | High if misapplied |
3. The Silent Default: How Engineers Decide Without Deciding
So how does this happen? Partly, it’s because distributed systems education focuses almost entirely on the mechanics of the tradeoff — how consensus algorithms work, what Raft or Paxos does, how to configure replica sets. Consequently, engineers acquire excellent vocabulary to describe these tradeoffs to other engineers, but almost no process for surfacing them to stakeholders.
Furthermore, the defaults in popular databases quietly make the decision for you. Consider the following:
# MongoDB default read preference (prior to v5.0 config changes)
# Reads from primary by default, but secondaries are allowed
# This replica set read preference allows stale reads from secondaries:
db.collection.find({}).readPref("secondaryPreferred")
# In PostgreSQL streaming replication, hot standby reads are stale by design:
# hot_standby = on (allows reads from replica, but replica may lag)
# To check replication lag in PostgreSQL:
SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;
These are perfectly valid configurations — in many contexts. The problem is not that they exist; it’s that they are often accepted as defaults without any discussion of what “replication lag of 2 seconds” actually means in terms of user experience and business integrity. Moreover, as teams grow, those configuration choices get inherited by engineers who weren’t present for the original decision and have no documentation explaining why it was made.
In most codebases, the comments explaining why a replication strategy was chosen — rather than what it is — simply don’t exist. As a result, future engineers optimise for performance or availability without realising they are silently changing the consistency contract with users.
Who typically decides consistency strategy in your organisation? (survey data)
4. The Real Cost of Undeclared Consistency Violations
It’s easy to dismiss occasional stale reads as an acceptable engineering tradeoff. In practice, however, the business consequences of undeclared consistency violations are frequently severe — and, crucially, they are rarely attributed to the architectural decision that caused them.
Consider a few well-documented examples. In 2012, the Amazon Web Services EC2 and EBS outage was partly attributed to consistency edge cases in replication behaviour that cascaded into broader failures. More strikingly, the idempotency patterns now standard at Stripe exist precisely because eventual consistency between payment records and ledger state was creating duplicate charges — a consistency violation with extremely direct financial consequences.
Consequently, the cost of a consistency violation is not just technical. It falls into at least three distinct categories:
Estimated cost distribution of consistency-related production incidents (composite industry data)
Furthermore, there is a subtler cost that rarely gets quantified: the erosion of user trust. A user who sees a stale cart total, or whose profile update doesn’t appear to have saved, may not file a support ticket — but they will quietly lose confidence in the product. That erosion is very hard to reverse and nearly impossible to trace back to a replication lag setting made three years earlier.
5. How to Surface These Tradeoffs to the Right Stakeholders
The good news is that this is a solvable problem — not technically, but organisationally. The solution, however, requires engineers to be willing to translate their system design choices into language that product owners, business analysts, and legal teams can engage with.
5.1 Translate technical guarantees into user-facing contracts
Rather than asking stakeholders “should we use eventual consistency?”, frame it as a concrete user scenario. For example: “Under normal load, a user’s account balance may take up to 3 seconds to reflect a recent transaction on all devices. Is that acceptable given our user base?” That question can be answered by a product manager. The original one cannot.
| Technical Choice | Plain-English Business Question | Who Should Answer |
|---|---|---|
| Eventual consistency on user balance reads | Can a user see a slightly outdated balance for up to 5 seconds? | Product owner + Legal |
| Async replication for inventory | How often is it acceptable to oversell a limited-stock item? | Operations + Product |
| Read-your-writes only (not global) | Can two users see different states of the same shared resource simultaneously? | Product + UX design |
| No distributed transactions across services | If the payment succeeds but order creation fails, what happens to the user? | Product + CX + Legal |
5.2 Document the decision, not just the configuration
In addition, every consistency decision should be recorded in the form of an Architecture Decision Record (ADR) that explicitly notes who agreed to the tradeoff and what the expected user impact is. The ADR format forces the question: “What alternatives did we consider, and why did we choose this?” — and it leaves a paper trail that future engineers can find.
# Example ADR structure (plain text, committed to source control) # Location: docs/architecture/decisions/ADR-0012-inventory-consistency.md Title: Use eventual consistency for inventory read queries Status: Accepted Date: 2024-03-15 Deciders: Engineering (Jane Smith), Product (Mark Lee), Operations (Aisha Osei) Context: Inventory queries run at 10,000 req/s during peak. Strong consistency requires cross-region coordination adding ~120ms per read. Decision: Accept eventual consistency on inventory reads with a max lag SLA of 2s. User Impact Agreed: Up to 2% of "last unit" purchases may oversell during peak windows. Operations has approved a compensation workflow for affected orders. Consequences: Must implement oversell detection + automated refund flow by Q3.
This is not bureaucracy — it’s institutional memory. More importantly, it is evidence that the business consciously signed the contract rather than having it imposed on them by a configuration file.
6. The Practical Questions to Ask Before You Pick a Replication Strategy
Finally, the following checklist provides a starting point for any engineer facing a consistency decision on a new or existing system. Each question is designed to be answerable by a non-engineer, which is precisely the point.
Before committing to a consistency model for any data domain, ensure you have clear answers — from stakeholders, not just engineers — to each of these questions.
| # | Question | Why It Matters |
|---|---|---|
| 1 | What is the maximum acceptable staleness window for this data? | Defines whether eventual consistency is acceptable at all |
| 2 | What happens to a user if they see stale data and act on it? | Determines the blast radius of a consistency violation |
| 3 | Are there regulatory requirements governing data freshness? | PSD2, GDPR, HIPAA and others may impose hard constraints |
| 4 | Can the system detect and compensate for violations after the fact? | If yes, eventual consistency becomes more viable |
| 5 | Who has signed off on the user-facing consequence of this choice? | Ensures the business has accepted accountability, not just engineering |
It is also worth acknowledging that strong consistency has a real cost — latency, infrastructure complexity, and reduced availability during partition events. As a result, the goal is not to push every system towards linearisability regardless of context. The goal is to ensure that the point on the consistency spectrum is chosen deliberately, by the right people, with a clear-eyed understanding of what it means for the people using the product.
Tools like Jepsen can verify the actual consistency properties of a running system — not just the properties claimed by documentation. For teams that have never formally tested their consistency guarantees in production-like conditions, running a Jepsen-style analysis is often a sobering exercise, and a productive one.
In most organisations, the consistency model in production was never explicitly chosen. It is the sum of a database’s default settings, a DBA’s preference from five years ago, and a performance optimisation made under deadline pressure. That is not a resilient architecture — it is accumulated technical debt with user-facing consequences.
What We Have Learned
- CAP theorem defines a fundamental constraint in distributed systems, but it was never designed as a stakeholder communication tool — engineers must bridge that gap themselves.
- Eventual consistency is not merely a technical configuration. It is a contract with users that, in most organisations, has never been consciously agreed to by the business.
- Database defaults and inherited configurations silently make consistency decisions on behalf of the entire organisation, often without any documentation of intent.
- The cost of undeclared consistency violations spans financial loss, user trust erosion, engineering remediation, and regulatory exposure — and is rarely attributed to its architectural root cause.
- Engineers can surface these tradeoffs effectively by translating technical guarantees into plain-English user-impact statements that product owners and legal teams can evaluate.
- Architecture Decision Records (ADRs) should explicitly capture who agreed to the consistency tradeoff and what user-facing consequences were accepted — not just what the configuration is.
- Tools like Jepsen allow teams to verify actual consistency behaviour in practice, catching the gap between documented and real guarantees before users do.
