When an external service goes down
Timeout, retry and circuit breaker: how three lines of defense stop a slow external service from locking up your whole application.
Not down, just slow
An SMS provider slowed down. The whole application stopped.
Response time climbed to 30 seconds; within half an hour even endpoints with nothing to do with SMS were timing out.
Cascading failure
How slowness spreads
- Step 1: The process locks up
worker / PHP-FPM process tied up - Step 2: Time disappears
30 s call = 30 s busy - Step 3: The pool fills
new requests wait in line - Step 4: The healthy die
unrelated endpoints time out
5 s
First defense: timeout
total timeout
Not the default 30 seconds, but 5: Http::timeout(5)->connectTimeout(2). The values are the source's suggested example; this number should be a decision.
Second defense: bounded retry
Three attempts, two waits
wait, ms
- Wait 1200
- Wait 2400
~3.4 min
Put a ceiling on backoff
12th wait without a cap
delay = min(cap, base × factorⁿ⁻¹). Without a cap, base 100 ms × 2 reaches about 3.4 minutes by the 12th attempt.
Thundering herd
Jitter types
| Type | Range | Behavior |
|---|---|---|
| Full | [0, delay] | Widest spread; one client may retry very early |
| Equal | [delay/2, delay] | Keeps a floor, still spreads the herd |
| Decorrelated | [base, previous × 3] | Capped; follows the last wait, a strong default |
| None | no jitter | Fine for a single client, dangerous for a fleet |
Third defense: circuit breaker
Stop making the call
Even a 5-second timeout is 5 seconds per process; in the open state that drops to 0.
Fail-fast
- 01Closedcalls pass, errors are counted
- 02Openthe external service isn't called, instant error
- 03Half-opena single trial: if it succeeds, closed
Fail-fast beats slow-fail
What you do while the circuit is open
- Queue itIf the SMS doesn't have to go out right away, leave it to be processed once the circuit closes.
- Serve stale dataIf the exchange-rate service is down, show the last known rate.
- Degrade gracefullySaying "Recommendations can't load right now" beats crashing the page.
