The life of a message

Keyboard: ← → to move, F for full screen, O for overview.

Tan focused on the work

Muhammet Şafak — Visuals

The life of a message

Schema, delivery, repeat, replay: a message is tested four times in the queue.

Muhammet Şafak

Tan pointing

Agenda

Four stops

  1. Schema validate at the edge
  2. Evolution without breaking the contract
  3. Idempotency the same message twice
  4. DLQ replay reset, select, test

Section 01

Schema

The queue doesn't look inside the payload; you add the shape guarantee at the edge.

The queue carries bytes, not types

Where is the schema validated?

An upstream service dropped a field from the payload; the worker threw a KeyError in production.

  1. Producer side primary defense
  2. Consumer side safety net → DLQ
  3. Schema registry per message type

The consumer side is the seatbelt, not the steering.

Muhammet Şafak Validating the schema at the edge of the queue · Engineering

Loosening is safe, tightening is not

Changing a schema without breaking it

Safe

  • Add an optional field
  • Drop the required constraint
  • Widen an enum, relax a minimum

Breaking

  • Add a required field
  • Remove, rename or retype a field
  • Narrow an enum, tighten a rule

Consumers upgrade first

A new identity, parallel lives

  1. Step 1: Publish v2 orders:created.v2
  2. Step 2: v1 continues producers emit both
  3. Step 3: Consumer migration on their own schedule
  4. Step 4: Retire v1 once it has no consumers

Section 02

Repeat

The broker keeps its promise: a message arrives at least once, sometimes twice.

The same message, two charges

The same message can arrive twice . The charge must not happen twice.

The payment worker charged the card and crashed before it could ack the message; the broker redelivered it, and the customer was charged twice.

f(f(x)) = f(x)

Idempotent or not?

Not idempotent Idempotent
Example Increase the balance by 10 Set the order status to paid
Twice Adds 20, or two emails Same as the effect applied once
Needs A key + dedupe Possibly nothing

The side effect decides

When do you record the key?

Concurrent duplicates need a constraint, not a check: a unique constraint on the key column turns the second insert into a conflict.
Shape When recorded Window Fits
Seen-set After the handler returns successfully Yes Third-party calls: email, charge
Transactional In the same DB transaction as the change None A row you own

Keeping the key honest

Key and constraint

  • Sender The sender produces the key: the message id or the Idempotency-Key header.
  • Payload Don't derive it from the payload alone: two legitimately identical requests collide.
  • Constraint A constraint, not a check: a unique constraint turns a concurrent second insert into a conflict.

Section 03

Replay

Replay isn't moving bytes back: it's resetting a message and safely reprocessing it.

20

Example scenario

minute outage

The downstream API was down for twenty minutes; four thousand messages used up their retries and landed in the DLQ.

A naive replay poisons again and bills twice

Replaying a DLQ safely

  1. Step 1: Reset id, payload, trace_id stay
  2. Step 2: Select the set reason: timeout
  3. Step 3: Dry-run count without touching
  4. Step 4: Sandbox side effects stubbed
  5. Step 5: Production if the first two are boring
Tan waving

Thank you

muhammetsafak.com/engineering/

Validate the schema at the edge, suppress repeats, replay by testing first.

Share, embed, download