Idempotency Is Not a Nice-to-Have

Sending a request twice should be fine. Add a retry with exponential backoff and move on. Network blips happen: the request times out, the client retries, the server handles it. Standard stuff. Easy.

Except the server doesn’t know the request timed out. It just knows it received a request, then received the same request again, and processed both. The user gets charged twice. The email goes out twice. The database row gets written twice. The downstream service gets called twice, and it doesn’t know either. That’s not a network problem. It’s a design problem.

The operation is the unit, not the request

Most systems are built around requests. A request comes in, a handler runs, a response goes out, and if the request fails partway through, the client retries and a new request comes in. But the work inside that handler isn’t necessarily safe to run more than once. Writing to a database isn’t. Sending a message to a queue isn’t. Charging a payment method definitely isn’t.

An idempotent operation is one that produces the same result whether it runs once or ten times. Not the same HTTP response, the same effect on the world. That’s a much stronger guarantee than “we handled the request.”

At-least-once delivery is the real contract

Picture a payment flow:

client  →  POST /charge { user: 42, amount: 100 }
server  →  calls payment processor
           charge succeeds
           ...timeout before response reaches client
client  →  POST /charge { user: 42, amount: 100 }   (retry)
server  →  calls payment processor again
           charge succeeds again
           user is now charged $200

The client never got a response, so it retried. The server had no way to know the first charge had already succeeded. The payment processor doesn’t deduplicate by default. Everyone did exactly what they were supposed to do, and the user still got charged twice.

Same pattern with a webhook:

upstream service  →  POST /webhook { event: order.created, order_id: 99 }
your server       →  processes event, fulfills order
                     ...crashes before acknowledging
upstream service  →  POST /webhook { event: order.created, order_id: 99 }   (retry)
your server       →  processes event again, fulfills order again

Webhook delivery systems assume at-least-once delivery. They retry until they get a 200. If your handler isn’t idempotent, every retry is a fresh side effect.

This shows up everywhere: message queues, job schedulers, distributed crons, API integrations. Any system that guarantees delivery can’t also guarantee exactly-once delivery. The guarantee is always at-least-once, and what you do with that is your problem.

Idempotency keys are the standard answer

The common solution is an idempotency key: a unique value the client generates and sends with the request. The server stores it, and on retry it recognizes the key and returns the same response without running the operation again.

client  →  POST /charge
           idempotency-key: abc-123
           { user: 42, amount: 100 }

server  →  checks: have we seen abc-123 before?
           no  →  run charge, store result under abc-123, return result
           yes →  return stored result, do nothing

Stripe does this. Most well-designed payment APIs do this. It works. But it also pushes responsibility onto the caller. Every client has to generate a unique key, store it, and reuse the same one on retry instead of generating a fresh one. If the client generates a new key on retry, you’re right back where you started.

And storing the result raises its own questions. Where does it live? For how long? What happens if the operation failed but the key was already stored, do you retry the operation or return the failure?

The check-then-act trap

A common implementation mistake is to check for the key first, then act:

1. read idempotency key from request
2. look up key in database
3. if found  → return cached response
4. if not    → run operation
5.            → store key and response

The problem sits between steps 4 and 5. If the server crashes after the operation runs but before the key is stored, the next retry runs the operation all over again. The check didn’t help at all.

attempt 1:
  key not found  →  charge runs  →  crash before key stored

attempt 2:
  key not found  →  charge runs again  →  double charge

The fix is to store the key before the operation, not after. Better still, make the key storage and the operation atomic, so they either both happen or neither does.

attempt 1:
  store key (status: in_progress)
  charge runs  →  update key (status: complete, result: ...)

attempt 2:
  key found (status: in_progress or complete)
  if in_progress  →  wait or return a 202
  if complete     →  return cached result

Now a crash mid-operation leaves the key in a state you can detect. The retry decides what to do based on that state instead of treating every attempt as the first one.

Expiry is not the last line of defense

Idempotency keys don’t last forever. Most systems expire them after 24 hours, or a week, or some other window, and that’s reasonable. Storing every key forever is expensive, and clients don’t retry after a week anyway.

But expiry creates its own edge case. What if a legitimate duplicate arrives after the key has expired? The server treats it as new and the operation runs again. For most use cases that’s fine. For payments it isn’t.

The answer isn’t to stretch the window out indefinitely. It’s to let the data model carry the guarantee instead. A charge record tied to an order ID is idempotent by design: you can’t create a second charge for the same order, because the data model won’t let you. The idempotency key becomes a performance optimization on top, not the only thing standing between you and a double charge.

table: charges
  order_id     UNIQUE   ← enforced at the database level
  user_id
  amount
  status

second attempt to insert charge for order 42
  → unique constraint violation
  → return existing charge record
  → no double charge, regardless of whether the key expired

That constraint is the real guarantee. Everything else is just convenience.

The retry is not the problem

A common instinct is to reduce retries. If retries cause duplicates, do fewer of them. That fixes the symptom and creates a worse problem. Retries exist because things fail: networks drop packets, servers restart mid-request, responses disappear without warning. Removing retries makes your system brittle in exactly the conditions where it needs to be most reliable.

The retry isn’t the enemy. An operation that isn’t safe to retry is the enemy. What you want is operations that don’t care how many times they run.

State transitions beat creation operations

Idempotency isn’t a feature you bolt on afterwards. It’s a property of how you model the operation in the first place. An endpoint that creates a resource is harder to make idempotent than one that sets a state. “Create a subscription” is harder than “set subscription status to active.” The first is only safe to run once. The second can run a hundred times and land in the same place.

That isn’t always an option. Sometimes you genuinely need to create a new thing, and when you do, the idempotency key is the right tool. But whenever you can model the operation as a state transition instead of a creation, you get idempotency almost for free.

So before building anything that touches money, writes shared state, or sends a message downstream, ask one question: what happens if this runs twice? If the answer is “bad things,” the design isn’t finished.