The Illusion of Idempotency: Why “Safe to Retry” Is Harder to Guarantee Than It Looks
A deep dive into idempotency keys, at-least-once delivery, and the subtle scenarios where retry logic silently corrupts state.
“Just make it idempotent and retry” is one of the most reassuring sentences in distributed systems, and one of the most quietly overloaded. It’s treated as a switch a developer can flip, when it’s actually a property that has to be deliberately engineered into every layer a request passes through — the client, the network, the queue, the handler, and every side effect that handler triggers. Get any one of those layers wrong, and “safe to retry” becomes “safe to retry, except during the one hour a month it silently double-charges someone.”
What “Idempotent” Actually Means (and What It Doesn’t)
An operation is idempotent if applying it multiple times produces the same result as applying it once. HTTP already labels certain methods this way — PUT and DELETE are specified as idempotent, while POST is not. But this specification-level idempotency describes intent, not implementation. A PUT request is only truly idempotent if the server code backing it is written to actually behave that way; nothing about the HTTP method itself enforces it. This gap between semantic idempotency and implemented idempotency is where most real incidents live.
At-Least-Once Delivery: The Root Cause of Every Retry Problem
Retries exist because networks are unreliable, and the systems built to compensate for that unreliability almost universally choose at-least-once delivery over the alternative. Apache Kafka’s documentation is candid about this trade-off: guaranteeing a message is delivered and processed exactly once, across an arbitrary failure, is extraordinarily expensive, so most systems settle for guaranteeing at least once and pushing deduplication responsibility onto the consumer.
This means duplicate delivery isn’t a rare edge case to defend against — it’s the expected, designed-for behavior of the messaging layer. A message can be delivered twice for entirely mundane reasons: the producer times out waiting for an acknowledgment that was actually sent successfully, or a consumer crashes after processing a message but before committing its offset, causing the broker to redeliver it to the next available consumer.
Idempotency Keys: The Standard Fix and Its Limits
The most widely adopted solution is the idempotency key: the client generates a unique identifier for a specific logical operation and attaches it to every retry attempt, so the server can recognize repeats and return the original result instead of executing the operation again. Stripe’s implementation is the reference example most engineers have encountered directly, and it illustrates both the pattern and its subtleties well.
The server has to store the key, the resulting response, and a record that the operation is either in progress or complete — and that storage itself has to be written atomically with the side effect it’s protecting, or the mechanism designed to prevent duplication becomes a second source of duplication. The key also needs a retention window: keep it too briefly and a legitimately delayed retry looks like a brand-new request; keep it too long and storage grows without bound.
| Approach | How it works | Main limitation |
|---|---|---|
| Client-supplied idempotency key | Client generates a unique ID per logical operation; server deduplicates on it | Requires atomic storage of key + result alongside the side effect |
| Naturally idempotent operations | Design the operation itself to be safe to repeat (e.g. “set balance to X” instead of “add X”) | Not always expressible for the actual business operation |
| Unique constraint deduplication | Database rejects a second insert with the same natural key | Only protects the write it’s attached to, not downstream side effects |
| Distributed locks | Lock on a resource ID prevents concurrent duplicate processing | Adds latency and a new failure mode if the lock itself is unavailable |
Where Retry Logic Silently Corrupts State
The scenarios that actually cause incidents are rarely the obvious ones. They tend to hide in the gap between what looks idempotent and what’s actually guaranteed to be.
Scenario 1: The Response Is Lost, Not the Request
A payment is charged successfully, but the network drops before the client receives the confirmation. The client, seeing a timeout rather than a success, retries. If the idempotency key wasn’t durably recorded before the charge was confirmed, or if the retry arrives after the key’s retention window expired, the second request executes as if it were new — and the customer is charged twice for something that only appears once in their own logs as a timeout.
Scenario 2: The Idempotent Write Hides a Non-Idempotent Side Effect
An order-creation endpoint might be carefully built so that inserting the order row is idempotent, protected by a unique constraint on the order ID. But if that same request handler also sends a confirmation email or calls an external shipping API, those side effects often aren’t covered by the same protection. The database write looks perfectly safe to retry; the email that goes out three times tells a different story.
Scenario 3: Two Retries Arrive Concurrently
A client library retries aggressively after a slow response, and both the original request and the retry end up in flight at the same time, racing each other. If the idempotency key lookup and the “mark as in progress” write aren’t atomic, both requests can pass the deduplication check before either has recorded that it’s processing, and the operation executes twice despite an idempotency key being present the entire time.
Scenario 4: Key Reuse Across Logically Different Operations
A key generated from something that isn’t actually unique to the operation — a timestamp with insufficient precision, or a client-side counter that resets on restart — can collide across two genuinely different requests. The second, unrelated operation gets silently treated as a duplicate of the first and never executes at all, which is arguably worse than a duplicate, since nothing about the failure is visible from the outside.
| Scenario | Root cause | Mitigation |
|---|---|---|
| Lost response, retried request | Key not durably stored before confirming success | Write key + result atomically with the side effect itself |
| Non-idempotent side effect inside an idempotent write | Deduplication scoped to the database write only | Wrap all side effects in the same idempotency boundary, or make them idempotent independently |
| Concurrent retries racing each other | Key check-and-set isn’t atomic | Use a single atomic compare-and-set or unique constraint as the gate |
| Key collision across unrelated operations | Key isn’t derived from something truly unique to the operation | Generate keys from a UUID or a value guaranteed unique per logical request |
Designing Truly Safe Retries
Before trusting a retry path in production, check:
- Is the idempotency key generated from something guaranteed unique to this specific logical operation, not a timestamp or a resettable counter?
- Is the key stored, checked, and set atomically with the operation it protects, rather than as a separate step that can race?
- Do every side effect the operation triggers — emails, webhooks, downstream API calls — fall inside the same idempotency boundary, or do they need their own?
- Is the retention window for keys long enough to cover realistic retry delays, including client-side backoff?
- Has this actually been tested under concurrent retries, not just sequential ones?
What We Learned
Idempotency looks like a simple property until you trace it through every layer a request actually passes through: at-least-once delivery guarantees duplicates will happen, idempotency keys are the standard defense, and nearly every real incident comes from a gap between where the protection was assumed to apply and where it actually does. Lost responses, side effects hiding outside the protected boundary, races between concurrent retries, and poorly chosen keys are the four recurring ways “safe to retry” quietly stops being true. None of them require an exotic failure — they require exactly the ordinary network hiccups and timing coincidences that retries were built to survive in the first place.
