Health check design: how do I separate Liveness and Readiness?
Question
We're going to write `/health` endpoints for containers behind Kubernetes / a Load Balancer. But just returning `{status:ok}` doesn't show whether the app is actually working — if the DB connection is down but HTTP is up, it's misleading. Architecturally, how do I separate Liveness and Readiness? And what should I watch out for so these endpoints don't overload the DB?
Answer
Short answer
A static {status:ok} only proves the HTTP server is up, not that the app can actually serve. The fix is to separate two different questions: “is it alive?” and “is it ready?”.
Context
The real issue is this: Liveness and Readiness measure different things, and if you mix them up you shoot yourself in the foot — especially if you put the dependency check on the wrong probe. Readiness’ “pull me out of rotation but don’t kill me” behaviour is the same traffic-draining logic you need when a pod shuts down during autoscaling; I covered that side in the SIGTERM graceful shutdown answer.
Why
- The two probes map to different consequences. If liveness fails, Kubernetes restarts the pod; if readiness fails, it just stops traffic. This distinction is critical: on a temporary dependency issue you don’t want to restart the pod, you only want to stop traffic.
- Putting a dependency on liveness creates a restart loop. A momentary DB blip throws all your healthy pods into an endless restart loop — even though nothing is wrong with the processes themselves.
- Probe cost multiplies by replica count. If every replica’s probe runs a real query against Postgres every few seconds, 50 pods × constant probes = you take down your own database.
What to do
- Keep liveness cheap and dependency-free. Don’t check the DB, cache or queue here. Liveness should only ask “is the process responding?”.
- Check the critical dependencies in readiness. Check the dependencies you need to serve here, like the DB, cache and queue. When a dependency goes down, let the Load Balancer pull the pod out of rotation — but without killing it. When the dependency comes back, the pod re-enters traffic.
- Run readiness with a short timeout and cache its result. Cache the result for a few seconds, and check only what you truly need to serve.
- Add a startup probe for slow boots. If your app boots slowly, use a separate startup probe so liveness doesn’t kill the pod while the app is still coming up.
Bottom line: I’d make liveness cheap and dependency-free, and readiness dependency-aware but cached. The golden rule: never let your probes DDoS the database. I separately discuss whether you really need Kubernetes in the Engineering section; but if you don’t make this distinction, the smallest DB tremor will shake your whole cluster.
Compiled from questions I have run into in the field or been asked.
Related Reading
Comments
Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.