The life of a message
Keyboard: ← → to move, F for full screen, O for overview.
Muhammet Şafak — Presentations
The life of a message
Schema, delivery, repeat, replay: a message is tested four times in the queue.
Muhammet Şafak
Agenda
Four stops
- 01Schemavalidate at the edge
- 02Evolutionwithout breaking the contract
- 03Idempotencythe same message twice
- 04DLQ replayreset, select, test
Section 01
Schema
The queue doesn't look inside the payload; you add the shape guarantee at the edge.
The queue carries bytes, not types
Where is the schema validated?
An upstream service dropped a field from the payload; the worker threw a KeyError in production.
- Producer sideprimary defense
- Consumer sidesafety net → DLQ
- Schema registryper message type
The consumer side is the seatbelt, not the steering.
Loosening is safe, tightening is not
Changing a schema without breaking it
Safe
- Add an optional field
- Drop the required constraint
- Widen an enum, relax a minimum
Breaking
- Add a required field
- Remove, rename or retype a field
- Narrow an enum, tighten a rule
Consumers upgrade first
A new identity, parallel lives
- 1Step 1: Publish v2
orders:created.v2 - 2Step 2: v1 continues
producers emit both - 3Step 3: Consumer migration
on their own schedule - 4Step 4: Retire v1
once it has no consumers
Section 02
Repeat
The broker keeps its promise: a message arrives at least once, sometimes twice.
The same message, two charges
The same message can arrive twice. The charge must not happen twice.
The payment worker charged the card and crashed before it could ack the message; the broker redelivered it, and the customer was charged twice.
f(f(x)) = f(x)
Idempotent or not?
| Not idempotent | Idempotent | |
|---|---|---|
| Example | Increase the balance by 10 | Set the order status to paid |
| Twice | Adds 20, or two emails | Same as the effect applied once |
| Needs | A key + dedupe | Possibly nothing |
The side effect decides
When do you record the key?
| Shape | When recorded | Window | Fits |
|---|---|---|---|
| Seen-set | After the handler returns successfully | Yes | Third-party calls: email, charge |
| Transactional | In the same DB transaction as the change | None | A row you own |
Keeping the key honest
Key and constraint
- SenderThe sender produces the key: the message id or the Idempotency-Key header.
- PayloadDon't derive it from the payload alone: two legitimately identical requests collide.
- ConstraintA constraint, not a check: a unique constraint turns a concurrent second insert into a conflict.
Section 03
Replay
Replay isn't moving bytes back: it's resetting a message and safely reprocessing it.
20
Example scenario
minute outage
The downstream API was down for twenty minutes; four thousand messages used up their retries and landed in the DLQ.
A naive replay poisons again and bills twice
Replaying a DLQ safely
- 1Step 1: Reset
id, payload, trace_id stay - 2Step 2: Select the set
reason: timeout - 3Step 3: Dry-run
count without touching - 4Step 4: Sandbox
side effects stubbed - 5Step 5: Production
if the first two are boring
Thank you
Validate the schema at the edge, suppress repeats, replay by testing first.