An invoice's reliability

Keyboard: ← → to move, F for full screen, O for overview.

Tan with a serious expression

Muhammet Şafak — Visuals

An invoice's reliability

Number, write, repeat and outcome: an invoice can go wrong four times on its way.

Muhammet Şafak

Section 01

Number

In e-invoicing, a number must be both free of duplicates and free of gaps.

Generate too late and you get a race; generate too early and you get a gap.

Muhammet Şafak Number collision in an e-invoice integration

Two constraints

Four approaches, two constraints

The database's own sequence (SEQUENCE, AUTO_INCREMENT) does not give back a value consumed on rollback; it produces gaps.
Approach No duplicates No gaps Failure mode
MAX()+1 at send time No Yes Collision with parallel workers
Generate at enqueue Yes No Failed or cancelled → gap
DB sequence Yes No Rollback doesn't return the value → gap
JIT reservation Yes Yes Cost: an early commit is mandatory

JIT reservation

The number, right before sending

  1. Step 1: Lock lockForUpdate
  2. Step 2: Reserve if reserved_no is empty, counter++
  3. Step 3: Early commit status = SENDING
  4. Step 4: Integrator outside the lock
  5. Step 5: Retry same reserved_no

A lock is held for DB state, not for external I/O.

Muhammet Şafak The integrator call takes 200 ms to 2 seconds · Engineering

Section 02

Delivery

The invoice goes to the database, the event to RabbitMQ: two systems, no single atomic step.

Dual-write

Writing to two systems

Dual-write save() and publish() are separate
  • Publish fails → lost event
  • Rollback happens → phantom event
  • DB::transaction does not cover RabbitMQ
Transactional outbox The write drops to a single system
  • Invoice and outbox row in the same commit
  • A separate process publishes to the broker
  • Dual-write shrinks to a single write

Repeat

Repeat delivery must be harmless

At-least-once only Idempotent consumer
Delivery At least once, sometimes twice At least once, sometimes twice
Record Processed ids are not kept processed_messages, unique constraint
Transaction Work and stamp are separate Work and stamp in the same transaction
Outcome A repeat repeats the effect Effectively-once

Write the data to a single system, suppress repeats at the consumer, seal the sensitive field before it leaves.

Muhammet Şafak Outbox, idempotent consumption and field encryption

Section 03

Outcome

The outbox guaranteed delivery; it did not guarantee the same outcome.

The real equation

outcome = f(message, hidden inputs )

Between production and consumption there can be milliseconds, twenty minutes or three days.

Hidden inputs

Six hidden inputs

Six hidden inputs encountered in an event-driven invoice flow.
Input In practice
Time Reading the current clock for the document date
Database Prefix X at creation, Y at send time
Configuration The same allow-list in two separate services
External system "This already exists", but what exists is unclear
Environment One framework swallows the warning, another turns it into an exception
Worker Parallel replicas, locks, repeat delivery

A noisy failure beats a silent wrong outcome

Four moves

  1. Step 1: Freeze the context at production time
  2. Step 2: Time is an input a retry does not change it
  3. Step 3: Name the fork write the reason to the log
  4. Step 4: Don't resolve silently stop on ambiguity
Tan waving

Thank you

muhammetsafak.com

What enters the decision is frozen; what doesn't is carried by reference.

Share, embed, download