The life of a message
Keyboard: ← → to move, F for full screen, O for overview.
Muhammet Şafak — Visuals
The life of a message
Schema, delivery, repeat, replay: a message is tested four times in the queue.
Muhammet Şafak
Agenda
Four stops
- 01 Schema validate at the edge
- 02 Evolution without breaking the contract
- 03 Idempotency the same message twice
- 04 DLQ replay reset, select, test
Section 01
Schema
The queue doesn't look inside the payload; you add the shape guarantee at the edge.
The queue carries bytes, not types
Where is the schema validated?
An upstream service dropped a field from the payload; the worker threw a KeyError in production.
- Producer side primary defense
- Consumer side safety net → DLQ
- Schema registry per message type
The consumer side is the seatbelt, not the steering.
Loosening is safe, tightening is not
Changing a schema without breaking it
Safe
- Add an optional field
- Drop the required constraint
- Widen an enum, relax a minimum
Breaking
- Add a required field
- Remove, rename or retype a field
- Narrow an enum, tighten a rule
Consumers upgrade first
A new identity, parallel lives
- 1 Step 1: Publish v2
orders:created.v2 - 2 Step 2: v1 continues
producers emit both - 3 Step 3: Consumer migration
on their own schedule - 4 Step 4: Retire v1
once it has no consumers
Section 02
Repeat
The broker keeps its promise: a message arrives at least once, sometimes twice.
The same message, two charges
The same message can arrive twice . The charge must not happen twice.
The payment worker charged the card and crashed before it could ack the message; the broker redelivered it, and the customer was charged twice.
f(f(x)) = f(x)
Idempotent or not?
| Not idempotent | Idempotent | |
|---|---|---|
| Example | Increase the balance by 10 | Set the order status to paid |
| Twice | Adds 20, or two emails | Same as the effect applied once |
| Needs | A key + dedupe | Possibly nothing |
The side effect decides
When do you record the key?
| Shape | When recorded | Window | Fits |
|---|---|---|---|
| Seen-set | After the handler returns successfully | Yes | Third-party calls: email, charge |
| Transactional | In the same DB transaction as the change | None | A row you own |
Keeping the key honest
Key and constraint
- Sender The sender produces the key: the message id or the Idempotency-Key header.
- Payload Don't derive it from the payload alone: two legitimately identical requests collide.
- Constraint A constraint, not a check: a unique constraint turns a concurrent second insert into a conflict.
Section 03
Replay
Replay isn't moving bytes back: it's resetting a message and safely reprocessing it.
20
Example scenario
minute outage
The downstream API was down for twenty minutes; four thousand messages used up their retries and landed in the DLQ.
A naive replay poisons again and bills twice
Replaying a DLQ safely
- 1 Step 1: Reset
id, payload, trace_id stay - 2 Step 2: Select the set
reason: timeout - 3 Step 3: Dry-run
count without touching - 4 Step 4: Sandbox
side effects stubbed - 5 Step 5: Production
if the first two are boring
Thank you
Validate the schema at the edge, suppress repeats, replay by testing first.