Skip to content
Muhammet Şafak
tr

Book II of III The Unreliable Systems Trilogy

Designing Reliable Event-Driven Systems

Messaging, retries, idempotency, outbox, ordering and schema evolution

A practical guide to the patterns that keep event-driven systems correct when networks, brokers, deploys and people misbehave.

  • 333 pages
  • 1st edition
  • Free

The book's thesis

Not

message brokers

But

failure semantics

  • 01

    Failure semantics

    Define what happens when a message is late, lost, duplicated or out of order.

  • 02

    Delivery & duplication

    Treat at-least-once as a fact and make handlers safe to run twice.

  • 03

    Recovery & replay

    Plan how the system finds its way back after things went wrong.

About the book

One instance committed an order and shut down before it published the event; no alert fired. By the next morning a buyer was looking at an order marked as placed, while Payments had authorized no card and Inventory had reserved nothing. The scene is set at Parcelly, a fictional marketplace, and nobody wrote a bug in the usual sense: every line did what it said, and the system failed between a database commit and a network call.

The book’s claim is that the problem is not message brokers but failure semantics: deciding in advance what happens when a message is late, lost, duplicated or out of order. The gap in the opening scene is the dual-write problem, which the book closes with the transactional outbox before moving on to delivery, duplication, recovery and replay.

It is broker-agnostic and does not promise that following it will make your system correct. Reliability is not a property you add; it is a set of failures you have decided how to handle and a set you have decided to accept. Each numbered chapter states the pattern’s costs, its common mistakes and when not to use it.

Contents

  1. 01 Preface Opens with the Parcelly scene in which an order is committed but its event is never published, and states the book's thesis: reliability comes from failure semantics, not from the choice of broker. Also covers who the book is for, how it is organized and how to read its pseudocode.
  2. 02 The Lie of the Happy Path Shows that a flow tested only against the happy path has been shown to work in the one case that rarely holds. Moving to events changes the coupling instead of removing it, so each asynchronous hop needs a bound, an alert and a defined waiting state.
  3. 03 Messaging Semantics Explains what a broker promises across the three handoffs and where the acknowledgment or offset commit decides between at-most-once and at-least-once. Exactly-once is an effect built from at-least-once delivery and an idempotent consumer.
  4. 04 A Practical Failure Model Builds a practical failure model: a timeout means "I stopped waiting", not "it failed", and every seam has crash windows that fail silently. It covers slow consumers, clocks, restores and the people who operate the system.
  5. 05 The Dual-Write Problem and the Transactional Outbox Shows why no ordering of commit and publish makes a dual write safe, and introduces the transactional outbox, which turns it into one local transaction plus a repeatable move. The outbox guarantees at-least-once publication, not exactly-once.
  6. 06 Relays, Polling and Change Data Capture Covers the relay that moves outbox rows to the broker: publish first and mark second, polling by state, several relays with skip-locked claims, and change data capture. Start with polling unless latency, load or an existing platform says otherwise.
  7. 07 Retries Done Right Treats a retry as a promise to try again, not to succeed. Classify failures, use exponential backoff with jitter under a deadline and a shared retry budget, and rely on circuit breakers and backpressure instead of retrying harder.
  8. 08 Idempotency Explains why every consumer should assume any message can arrive twice, and builds the inbox: the inbox row and the business change commit in one local transaction. Covers the dedup window, external side effects and idempotency keys at the API boundary.
  9. 09 Poison Messages and Dead Letters Describes the poison message that fails every time and the dead-letter queue that sets it aside with context. A redrive is safe only when the cause is fixed, the original identifiers are kept and the consumer is idempotent.
  10. 10 Ordering Argues that per-aggregate ordering is the order worth designing for. Covers partition keys, keyed parallelism, aggregate version numbers, stale-event handling and hold-back buffers with timeouts and owners.
  11. 11 Sagas and Long-Running Workflows Replaces the distributed transaction with local transactions and compensations. Covers choreography against orchestration, ordering steps by how hard they are to undo, deadlines on every wait and idempotent compensations.
  12. 12 Event Design Separates events, commands and documents, domain events from integration events, and thin events from fat ones. Covers naming, the envelope and its identifiers, what must stay out of payloads and why every event needs an owning service.
  13. 13 Schema Evolution Treats an event schema as a contract with readers you do not control. Covers backward and forward compatibility, which changes break, the limits of a schema registry, versioning strategies, upcasters, tolerant readers and migrating a breaking change in safe steps.
  14. 14 Observability and Reconciliation Covers the identifiers that must survive every hop, message age as a better signal than lag, alerts tied to deadlines (including alerts for silence) and reconciliation: comparing records across two views of the truth, then repairing through the normal path.
  15. 15 Recovery, Disasters and Broker Migration Covers what survives when a store is lost, restored or replaced: the outbox, the inbox, stable event identifiers and reconciliation. Addresses replay as a recovery tool, restore and region failover, and a broker migration with the relay as the single switch point.
  16. 16 Testing Event-Driven Systems Explains why tests that assume a broker delivering each message once and in order test a system that does not exist. Covers handler tests on a real database, perturbed streams, consumer-driven contracts, fault injection and recovery drills.
  17. 17 A Reliability Checklist and Closing Thoughts Gathers the book into a reliability checklist built on three questions: what can fail, how you would observe it and which guarantee survives it. A completed checklist shows that someone thought about each item, not that the system is correct.
  18. 18 Appendix: Glossary and Pattern Quick Reference Reference material: a glossary of the book's vocabulary and a quick reference to the patterns, so that a term or pattern can be looked up without rereading a chapter.

Sample chapter: Idempotency

An excerpt from one chapter; the full text is in the downloadable edition.

The inbox pattern

The inbox is the table of processed message ids on the consumer side. Chapter 4 gave the producer a transactional outbox so that a state change and its event commit together. The inbox is the mirror image: the consumer commits its state change and its record of having handled the message together, in one local transaction in its own database.

table inbox:
    consumer      text not null
    event_id      uuid not null
    processed_at  timestamp not null
    primary key (consumer, event_id)

The consumer column lets one service run several independent consumers against one table, so that the Ledger’s posting consumer and its reporting consumer each remember their own history. That is why the primary key is the pair (consumer, event_id). With event_id alone, the first consumer to handle a message would make the second consumer’s insert fail, and the second consumer’s effect would be skipped as if it were a duplicate. A service with a single consumer per table can drop the column and key on event_id, which is the simpler form used elsewhere in the book.

The handler writes the inbox row and the business change in the same transaction.

function handlePaymentCaptured(message):
    if inboxContains("ledger-poster", message.event_id):
        // fast path only; this check is not the safety net
        ack(message)
        return

    transaction:
        postLedgerEntry(message.payload)
        // primary key (consumer, event_id): a concurrent duplicate
        // fails here, the transaction rolls back, and the
        // message is redelivered and finds the committed row
        inboxInsert("ledger-poster", message.event_id)
    // commit point: effect and inbox row exist together or not at all

    ack(message)

Walk through the failures. If the process dies before the commit, neither the ledger entry nor the inbox row exists, so redelivery processes the message from scratch. If it dies after the commit and before the ack, both exist, and redelivery takes the fast path: the check finds the row, the message is acknowledged, and nothing is written. The effect and its memory can never disagree, because they commit as one.

The check at the top is an optimization. It saves a transaction in the common duplicate case, and it can be removed without losing correctness. The safety net is the unique constraint inside the transaction, which is where the real arbitration happens.

Inside a transaction, the order of the writes does not change atomicity, so the warning about reversing the two writes applies only to the unsafe version above. It does change cost. In this version a concurrent duplicate does all the work before it fails on the insert, and in some databases a violation aborts the whole transaction, so the handler cannot recover inside it. I would claim first: insert the inbox row with “on conflict do nothing” semantics, look at whether a row was inserted, and skip the effect if not. The duplicate then waits on the key, does no work, and never raises. […]

A word on ack, because the term hides a difference between the two families. On a queue-based broker it acknowledges one message. On a log-based broker it is usually a commit of an offset, which means “everything up to here”, and a consumer that processes a partition in parallel can commit past a message it has not finished, and lose it. The inbox protects against duplicates, not against that, and Chapter 9 looks at it where ordering and parallelism meet.

Glossary

The key terms used in the book, with short definitions.

at-least-once
A delivery semantic in which a message is delivered one or more times, so duplicates are possible. It holds while the message is still retained and the consumer acknowledges only after the effect. A consumer that lags beyond retention still loses messages silently. It is the book's default.
dual write
Updating two systems, such as a database and a broker, without a shared transaction.
outbox
The table written in the same local transaction as the state change, from which a relay publishes events. The full term is **transactional outbox**.
relay
The process that moves outbox rows to the broker.
inbox
The table of processed message ids on the consumer side, written in the same transaction as the handler's effects.
idempotency key
The key a consumer or API uses to recognize repeated work. For event consumption it is normally the `event_id`.
dedup window
The period during which a consumer remembers processed ids. A message whose inbox row has been deleted can be processed again.
poison message
A message that fails every time it is processed. On a log-based broker one that crashes the process leaves the consumer stuck on one position, so it is detected by lag with no throughput and by a counter per position.
dead-letter queue (DLQ)
A separate destination where a consumer or broker places messages it has given up on. On a log-based broker the same idea is a **dead-letter topic**.
redrive
Giving dead letters back to the consumer that failed on them, to its own input and never to the shared topic they came from. The key and `event_id` are kept, the attempt count starts again, and `redrive_count` records the round.
saga
A long-running workflow made of local transactions with compensations, either orchestrated by a coordinator or choreographed through events.
failure semantics
For a given seam, what can fail, how you would observe it and which guarantee survives it. It is the book's thesis.
reconciliation
Comparing two systems' views of truth and repairing the differences.

Frequently asked

8 questions

  • Who is this book for?

    Backend and platform engineers and tech leads who build or operate systems that communicate through events.

  • Which message broker or language do I need?

    None in particular. The book is not a tutorial for any specific broker; it is broker-agnostic and uses language-independent pseudocode and Mermaid diagrams.

  • Will following the book make my system correct?

    The book does not promise that. It treats reliability as a set of failures you have decided how to handle and a set you have decided to accept, and it asks you to treat at-least-once delivery as a fact and make handlers safe to run twice.

  • What do I need to know before reading it?

    You do not need to be an expert in messaging. The preface asks you to be comfortable with relational databases and transactions (what a commit is, what a unique constraint does and, roughly, what isolation means), with services that talk over a network and the vocabulary of timeouts, and with the general idea of a message broker.

  • Is the book free?

    Yes. The license line reads "© 2026 Muhammet Şafak — free to download and read". PDF and EPUB files are offered in the download panel on this page. This edition is 333 pages.

  • Which languages and formats is it available in?

    Seven languages: English, Turkish, Spanish, Simplified Chinese, Portuguese, German and French. Each is offered as a PDF and an EPUB.

  • Does the book contain code examples?

    It contains pseudocode. The code is language-independent, will not compile and calls no real library. Blocks are short, at most 30 lines; table schemas use SQL-like notation and diagrams use Mermaid.

  • Is this a standalone book or part of a series, and in what order should I read the series?

    It is the second of three books in The Unreliable Systems Trilogy: I Access (The Production Database Access Guide), II Events (this book) and III Agents (Engineering Reliable AI Agents). The prefaces ask for general engineering knowledge, not for another book, so you can start with the one that matches the problem in front of you.

Blog posts that touch on this book's subject.

Search the site

Start typing to search posts, projects and pages.

Esc to close Powered by Pagefind