# The life of a message

> From schema to delivery, from idempotency to DLQ replay: what to check at every stop so a message survives the queue intact.

- Kind: Presentation
- Published: 2026-10-03
- Slides: 16
- Based on these articles: https://muhammetsafak.com/engineering/notes/validate-schema-at-the-edge/ · https://muhammetsafak.com/engineering/essays/schema-evolution-without-breaking/ · https://muhammetsafak.com/engineering/notes/idempotency-duplicate-delivery/ · https://muhammetsafak.com/engineering/notes/replaying-a-dead-letter-queue/
- Source: https://muhammetsafak.com/visuals/d/the-life-of-a-message/
- Language: en-US
- Author: Muhammet Şafak

---
## Slide 1 — The life of a message

Muhammet Şafak — Visuals

Schema, delivery, repeat, replay: a message is tested four times in the queue.

Muhammet Şafak October 3, 2026

## Slide 2 — Four stops

Agenda

1.   Schema validate at the edge
2.   Evolution without breaking the contract
3.   Idempotency the same message twice
4.   DLQ replay reset, select, test

## Slide 3 — Schema

Section 01

The queue doesn't look inside the payload; you add the shape guarantee at the edge.

## Slide 4 — Where is the schema validated?

The queue carries bytes, not types

An upstream service dropped a field from the payload; the worker threw a KeyError in production.

1.  Producer side primary defense
2.  Consumer side safety net → DLQ
3.  Schema registry per message type

## Slide 5

> The consumer side is the seatbelt, not the steering.

Muhammet Şafak Validating the schema at the edge of the queue · Engineering

## Slide 6 — Changing a schema without breaking it

Loosening is safe, tightening is not

#### Safe

-   Add an optional field
-   Drop the required constraint
-   Widen an enum, relax a minimum

#### Breaking

-   Add a required field
-   Remove, rename or retype a field
-   Narrow an enum, tighten a rule

## Slide 7 — A new identity, parallel lives

Consumers upgrade first

1.   Step 1: Publish v2 `orders:created.v2`
2.   Step 2: v1 continues `producers emit both`
3.   Step 3: Consumer migration `on their own schedule`
4.   Step 4: Retire v1 `once it has no consumers`

## Slide 8 — Repeat

Section 02

The broker keeps its promise: a message arrives at least once, sometimes twice.

## Slide 9

The same message, two charges

The same message can arrive twice . The charge must not happen twice.

The payment worker charged the card and crashed before it could ack the message; the broker redelivered it, and the customer was charged twice.

## Slide 10 — Idempotent or not?

f(f(x)) = f(x)

|  | Not idempotent | Idempotent |
| --- | --- | --- |
| Example | Increase the balance by 10 | Set the order status to paid |
| Twice | Adds 20, or two emails | Same as the effect applied once |
| Needs | A key + dedupe | Possibly nothing |

## Slide 11 — When do you record the key?

The side effect decides

Concurrent duplicates need a constraint, not a check: a unique constraint on the key column turns the second insert into a conflict.

| Shape | When recorded | Window | Fits |
| --- | --- | --- | --- |
| Seen-set | After the handler returns successfully | Yes | Third-party calls: email, charge |
| Transactional | In the same DB transaction as the change | None | A row you own |

## Slide 12 — Key and constraint

Keeping the key honest

-   **Sender** The sender produces the key: the message id or the Idempotency-Key header.
-   **Payload** Don't derive it from the payload alone: two legitimately identical requests collide.
-   **Constraint** A constraint, not a check: a unique constraint turns a concurrent second insert into a conflict.

## Slide 13 — Replay

Section 03

Replay isn't moving bytes back: it's resetting a message and safely reprocessing it.

## Slide 14

20

Example scenario

minute outage

The downstream API was down for twenty minutes; four thousand messages used up their retries and landed in the DLQ.

## Slide 15 — Replaying a DLQ safely

A naive replay poisons again and bills twice

1.   Step 1: Reset `id, payload, trace_id stay`
2.   Step 2: Select the set `reason: timeout`
3.   Step 3: Dry-run `count without touching`
4.   Step 4: Sandbox `side effects stubbed`
5.   Step 5: Production `if the first two are boring`

## Slide 16 — Thank you

[muhammetsafak.com/engineering/](https://muhammetsafak.com/engineering/)

Validate the schema at the edge, suppress repeats, replay by testing first.
