The Backup Didn't Come Up: An Hour-by-Hour Log of an Outage
Our whole system went down in four minutes and the Ankara backup never came up. What we prioritized at the crisis table, and what was left behind.
Book I of III The Unreliable Systems Trilogy
Identities, credentials, just-in-time access, mediated execution and audit
A practical guide to deciding who and what can reach a production database, how, for how long, and with what record left behind.
Page of the Türkçe edition: /tr/kitaplar/production-database-access-guide/
No sign-up, no email. License: © 2026 Muhammet Şafak — free to download and read
a credential-distribution problem
an authorization, execution and accountability problem
Decide who may do what to which data, where and for how long — decided per request.
Inspect each statement before it runs and bound its effect (transaction, row guard, timeout) before it commits.
Keep evidence of who asked, who approved and what actually ran.
On a Thursday evening, a one-row fix changed a batch of invoices, and the log named only a shared admin role. The scene takes place at Ledgerly, a fictional invoicing and payments company, and no step in it was unreasonable: the password was shared to get the first service running, the client auto-committed by default, and the role could change every table because that was easier than deciding which tables each service needed.
From that scene the book draws one claim: production database access is not a credential-distribution problem but an authorization, execution and accountability problem. The credential in the story was legitimate and did exactly what it was issued to do; what was missing was everything around it. Across five parts and sixteen chapters, the book follows Ledgerly from a shared admin password to an access model that was designed on purpose.
Every control asks something of the people who build and operate it, so each chapter states the cost, the common mistakes and when not to use the control. The book is written from the defender’s side: it does not teach attack techniques, does not interpret compliance regimes and is not a database administration book.
An excerpt from one chapter; the full text is in the downloadable edition.
Break-glass is emergency access with a defined logging set and mandatory review. The name comes from the glass-fronted fire switch: reachable by anyone who needs it, and breaking it is visible to everyone.
In the book’s model it is the role human_breakglass, and it is the only human role that reads Restricted columns. It reads every table and writes INSERT and UPDATE on every table except the two append-only ones; on bank_accounts and payment_tokens it writes UPDATE only, and only on non-Restricted columns. It never writes ledger_entries or audit_events, because no human role does. It is not a route for routine fixes, and it is not a data fix path: a data fix is planned, reviewed and bounded (Chapter 10), and break-glass is none of those things by design.
A break-glass request is self-declared. It has no approver. The person in the session says an emergency exists, gives a reason, and the gateway marks the request approved and grants the role to the same session login until expires_at. The same revoke job and the same session timer take it back.
The approver is the whole control for JIT, so an emergency role without one may seem odd. Break-glass exists for the cases where the approver pool has not answered, and an approval step would reproduce that failure. The control moves from before the access to around it: an alert at declaration, the logging set below, a short fixed window, and a review that follows without exception.
function declareBreakGlass(session, reason):
// a hijacked token alone must not be enough to declare
requireFreshMultiFactor(session.principal)
// one live request per session: a break-glass declaration supersedes it
supersedeLiveRequest(session.login)
// the window is the gateway's fixed maximum; there is no argument for it
window = breakGlassMaxWindow
request = newRequest(kind: "breakglass", requester: session.principal,
requester_login: session.login, approver: null,
reason: reason, expires_at: now() + window)
markApproved(request.id)
try:
auditLogAppend({ actor: session.principal, action: "breakglass",
subject: session.login, request: request.id,
expires_at: request.expires_at })
on AuditLogUnavailable:
markRefused(request.id)
raise GrantRefused
setSessionEnd(session.login, request.expires_at)
grant(human_breakglass, to: session.login)
// paged after the grant: a paging outage must not block an emergency;
// the review queue below is the net for a page that never arrived
queueReview(request.id, reviewer: "not-the-requester")
notify("security", "break-glass declared: " + request.id)
notify("platform", "break-glass declared: " + request.id)
Three decisions in that block need a comment.
The window is fixed by the gateway, at two hours in Ledgerly’s illustration. No longer window is available on request, because “I need it a little longer” turns an exception into a habit. If the emergency outlasts the window, the person declares again and the second declaration alerts again; that is the intended friction.
The audit-log check is the same as for JIT. If the log is down, the declaration is refused. I accept that cost, because a break-glass session with no record is the one thing a review cannot reconstruct, and the review is what makes the role tolerable. What on-call does when the normal path and the log are both down belongs to Chapter 15.
The alert is a notification to the security contacts and the platform team that someone has opened the broadest access Ledgerly has, not a request for permission. In a healthy system it is rare enough to be read; if it fires weekly it will stop being read, and break-glass is being used for something that deserves a designed path.
The key terms used in the book, with short definitions.
8 questions
Backend and platform engineers who build services that talk to a production database, SRE and DBA roles who operate it, tech leads who decide how access should work, and engineers who have newly become responsible for security. No security background is required.
No. Familiarity with an engine helps with the examples but is not required. The book is engine-independent and uses pseudocode, an engine-neutral SQL-like notation and Mermaid diagrams.
No. Regimes such as GDPR, KVKK, PCI DSS, SOC 2 and HIPAA come up only to explain what auditors and customers tend to ask. The book does not interpret any regime, cite its articles or say what is required of you; it aims to give your legal and compliance colleagues an architecture they can reason about.
No security background is required. The preface asks you to be comfortable with relational databases (tables, roles and grants, transactions, replicas), with services that connect to a database over a network (connection pools and timeouts), and with on-call work: being paged, repairing something under pressure and writing down afterward what happened.
Yes. The license line reads "© 2026 Muhammet Şafak — free to download and read". PDF and EPUB files are offered in the download panel on this page. This edition is 329 pages.
Seven languages: English, Turkish, Spanish, Simplified Chinese, Portuguese, German and French. Each is offered as a PDF and an EPUB.
No example is written in a real programming language, and nothing in the book is meant to be pasted into a terminal. Procedures are pseudocode; roles, grants and tables use a SQL-like notation that belongs to no particular engine. There is no configuration syntax for a specific proxy, vault or cloud product, and diagrams use Mermaid.
It is the first of three books in The Unreliable Systems Trilogy: I Access (this book), II Events (Designing Reliable Event-Driven Systems) and III Agents (Engineering Reliable AI Agents). The preface asks for general engineering knowledge, not for another book, so you can also start with whichever of the three matches your current problem.
Blog posts that touch on this book's subject.
Our whole system went down in four minutes and the Ankara backup never came up. What we prioritized at the crisis table, and what was left behind.
The only person with production database access went on holiday and diagnosis stopped. The need came down to four sentences; each became a design decision.
What is HTTP Basic Authentication, how does it work, and when should you use it? A practical breakdown of the oldest auth mechanism in the web stack.
Continue the trilogy