I think Exactly-Once Processing(EOP) is a Myth

On retries, idempotency, and the illusion of reliable execution.
I was working on an integration recently, connecting our application to an external service that sends updates whenever something happens on its end.
The idea was pretty straightforward.
The external service sends a webhook, our backend receives it, puts it into a queue, and a background worker picks it up to update the relevant record in our system.
Nothing particularly complicated.
Until I started thinking about what happens when the same event arrives more than once.
Let's use a shipment update as an example.
An external logistics service sends a webhook notifying us that a shipment has been picked up. Our application receives the event, processes it and updates the shipment status.
But for one reason or another, the external service doesn't receive our acknowledgement.
What does it do?
It retries.
Our application receives the same event again. The worker processes it again. And depending on what that operation does, we might end up creating duplicate activities, sending multiple notifications or triggering the same downstream action twice.
The interesting part is that neither system necessarily did anything wrong.
The external service is retrying because it doesn't know whether we received the event. Our application is processing it because that's what it was designed to do.
And this got me thinking about a fundamental problem in distributed systems: exactly-once processing (EOP).
So, why is exactly once difficult?
When two systems communicate over a network, there is always some uncertainty around what actually happened.
A request can reach its destination, get processed successfully, and still fail to return an acknowledgement.
A worker can finish updating the database and crash before acknowledging the message.
A queue can redeliver a message because it never received confirmation that processing was completed.
This is why we commonly talk about three delivery guarantees:
- At-most-once: A message is delivered zero or one time. Duplicates are avoided, but messages can be lost.
- At-least-once: Messages are retried until acknowledged. Delivery is more reliable, but duplicates are possible.
- Exactly-once: The intended effect happens once, even when retries and failures occur.
In practice, at-least-once delivery is common because losing an event can be worse than receiving it twice.
But it means your application needs to be prepared for repetition.
Perhaps idempotency solves the problem?
This is where idempotency comes in.
An operation is idempotent when executing it multiple times produces the same intended result as executing it once.
Going back to our shipment example, updating a shipment status from in_transit to delivered multiple times should leave the shipment in the same state.
But what if processing that event also creates an activity record, sends a notification or triggers another external action?
Those side effects can still happen multiple times.
One way to handle this is through event identifiers.
Every incoming event has a unique ID. Before processing it, the application checks whether that event has already been handled.
If it has, we skip the duplicate. If it hasn't, we process it and record that it was handled.
Of course, that check and the actual database operation need to be coordinated properly. Otherwise, two workers could receive the same event concurrently, both check that it hasn't been processed, and both proceed.
So there's a little more to it than adding an if statement.
Idempotency makes repeated requests safer, but it doesn't automatically guarantee that every part of a distributed workflow happens exactly once.
The bigger picture
Idempotency is one part of building reliable integrations. You also have to think about retries, transaction boundaries, dead-letter queues and reconciliation.
For example, the transactional outbox pattern helps when an application needs to update its database and publish an event without risking that one succeeds while the other silently fails.
And even with all these patterns, exactly-once guarantees have boundaries. Guaranteeing an operation within one transactional system is not the same as guaranteeing an entire workflow across multiple independent services.
I think this is one of the things that makes backend engineering interesting.
We spend a lot of time designing the happy path. The request comes in, the operation succeeds, the response goes out.
But real systems don't always follow that sequence.
Events get delivered twice. Workers crash. Acknowledgements disappear. External services become unavailable.
You can't always prevent these things from happening.
You can, however, build systems that are prepared for them.
Maybe the goal isn't to make sure an operation happens exactly once. It's to make sure that when it happens more than once, your system still gets the right result.
Randomly thinking out loud.