Same message, different outcome: determinism in event-driven architecture
Why does the same message produce different outcomes in an event-driven system? The hidden inputs that break determinism — and four moves to win it back.
Book II of III The Unreliable Systems Trilogy
Messaging, retries, idempotency, outbox, ordering and schema evolution
A practical guide to the patterns that keep event-driven systems correct when networks, brokers, deploys and people misbehave.
Page of the Türkçe edition: /tr/kitaplar/designing-reliable-event-driven-systems/
No sign-up, no email. License: © 2026 Muhammet Şafak — free to download and read
message brokers
failure semantics
Define what happens when a message is late, lost, duplicated or out of order.
Treat at-least-once as a fact and make handlers safe to run twice.
Plan how the system finds its way back after things went wrong.
One instance committed an order and shut down before it published the event; no alert fired. By the next morning a buyer was looking at an order marked as placed, while Payments had authorized no card and Inventory had reserved nothing. The scene is set at Parcelly, a fictional marketplace, and nobody wrote a bug in the usual sense: every line did what it said, and the system failed between a database commit and a network call.
The book’s claim is that the problem is not message brokers but failure semantics: deciding in advance what happens when a message is late, lost, duplicated or out of order. The gap in the opening scene is the dual-write problem, which the book closes with the transactional outbox before moving on to delivery, duplication, recovery and replay.
It is broker-agnostic and does not promise that following it will make your system correct. Reliability is not a property you add; it is a set of failures you have decided how to handle and a set you have decided to accept. Each numbered chapter states the pattern’s costs, its common mistakes and when not to use it.
An excerpt from one chapter; the full text is in the downloadable edition.
The inbox is the table of processed message ids on the consumer side. Chapter 4 gave the producer a transactional outbox so that a state change and its event commit together. The inbox is the mirror image: the consumer commits its state change and its record of having handled the message together, in one local transaction in its own database.
table inbox:
consumer text not null
event_id uuid not null
processed_at timestamp not null
primary key (consumer, event_id)
The consumer column lets one service run several independent consumers against one table, so that the Ledger’s posting consumer and its reporting consumer each remember their own history. That is why the primary key is the pair (consumer, event_id). With event_id alone, the first consumer to handle a message would make the second consumer’s insert fail, and the second consumer’s effect would be skipped as if it were a duplicate. A service with a single consumer per table can drop the column and key on event_id, which is the simpler form used elsewhere in the book.
The handler writes the inbox row and the business change in the same transaction.
function handlePaymentCaptured(message):
if inboxContains("ledger-poster", message.event_id):
// fast path only; this check is not the safety net
ack(message)
return
transaction:
postLedgerEntry(message.payload)
// primary key (consumer, event_id): a concurrent duplicate
// fails here, the transaction rolls back, and the
// message is redelivered and finds the committed row
inboxInsert("ledger-poster", message.event_id)
// commit point: effect and inbox row exist together or not at all
ack(message)
Walk through the failures. If the process dies before the commit, neither the ledger entry nor the inbox row exists, so redelivery processes the message from scratch. If it dies after the commit and before the ack, both exist, and redelivery takes the fast path: the check finds the row, the message is acknowledged, and nothing is written. The effect and its memory can never disagree, because they commit as one.
The check at the top is an optimization. It saves a transaction in the common duplicate case, and it can be removed without losing correctness. The safety net is the unique constraint inside the transaction, which is where the real arbitration happens.
Inside a transaction, the order of the writes does not change atomicity, so the warning about reversing the two writes applies only to the unsafe version above. It does change cost. In this version a concurrent duplicate does all the work before it fails on the insert, and in some databases a violation aborts the whole transaction, so the handler cannot recover inside it. I would claim first: insert the inbox row with “on conflict do nothing” semantics, look at whether a row was inserted, and skip the effect if not. The duplicate then waits on the key, does no work, and never raises. […]
A word on ack, because the term hides a difference between the two families. On a queue-based broker it acknowledges one message. On a log-based broker it is usually a commit of an offset, which means “everything up to here”, and a consumer that processes a partition in parallel can commit past a message it has not finished, and lose it. The inbox protects against duplicates, not against that, and Chapter 9 looks at it where ordering and parallelism meet.
The key terms used in the book, with short definitions.
8 questions
Backend and platform engineers and tech leads who build or operate systems that communicate through events.
None in particular. The book is not a tutorial for any specific broker; it is broker-agnostic and uses language-independent pseudocode and Mermaid diagrams.
The book does not promise that. It treats reliability as a set of failures you have decided how to handle and a set you have decided to accept, and it asks you to treat at-least-once delivery as a fact and make handlers safe to run twice.
You do not need to be an expert in messaging. The preface asks you to be comfortable with relational databases and transactions (what a commit is, what a unique constraint does and, roughly, what isolation means), with services that talk over a network and the vocabulary of timeouts, and with the general idea of a message broker.
Yes. The license line reads "© 2026 Muhammet Şafak — free to download and read". PDF and EPUB files are offered in the download panel on this page. This edition is 333 pages.
Seven languages: English, Turkish, Spanish, Simplified Chinese, Portuguese, German and French. Each is offered as a PDF and an EPUB.
It contains pseudocode. The code is language-independent, will not compile and calls no real library. Blocks are short, at most 30 lines; table schemas use SQL-like notation and diagrams use Mermaid.
It is the second of three books in The Unreliable Systems Trilogy: I Access (The Production Database Access Guide), II Events (this book) and III Agents (Engineering Reliable AI Agents). The prefaces ask for general engineering knowledge, not for another book, so you can start with the one that matches the problem in front of you.
Blog posts that touch on this book's subject.
Why does the same message produce different outcomes in an event-driven system? The hidden inputs that break determinism — and four moves to win it back.
How to set up asynchronous job queues in Laravel using the database driver, and keep workers running reliably in production with Supervisor.