Health check design: how do I separate Liveness and Readiness?
Keep liveness dependency-free and check dependencies in readiness with a cached probe, since liveness restarts the pod while readiness only sheds traffic.
Just Ask
Ask me anything about software architecture, careers, PHP, Go and the craft of building software; I answer here, in the open, for everyone. (Page 7/10)
The questions here are compiled from what I have run into in the field and from what people have asked me; I write the answers. No record carries the name of whoever asked.
Keep liveness dependency-free and check dependencies in readiness with a cached probe, since liveness restarts the pod while readiness only sheds traffic.
Stand up monthly RANGE declarative partitioning on created_at, backfill history in batches while the app writes, then swap names in a single transaction.
Acquire locks in one global order such as ascending PK, keep transactions short, target FOR UPDATE at the fewest rows, and retry the rest with backoff.
Wrap requests in a defer recover() middleware at the outermost layer, log the stack with the request id, return a generic 500 and emit a metric.
RPO 5min buys continuous cross-region replication and PITR, RTO 30min an IaC-provisioned warm standby behind Route 53 failover, rehearsed on a game-day.
Collapse repeated errors into one counted line, back off retries, cap logrotate by size and count, and give /var/log a volume off the root disk.
Per-IP limits lose to rotating proxies, so key the Redis sliding-window on account + IP + ASN, add account lockout and leave volumetric floods to the edge.
Split the critical path (browse, cart, checkout) from the nice-to-haves, put those behind feature flags with short timeouts, and rehearse the kill-switch.
Split the transaction into local steps with compensating actions, driven by one orchestrator over a queue, each idempotent and published through an outbox.
Don't hand-roll failover: put promotion behind a Patroni + etcd quorum, and let a primary cut off from the majority demote itself by fencing.