Most of the PRs Dependabot opens are not obligations, they are reminders.
\nFour kinds of signal get mixed together in the queue: Dependabot PRs, deprecation warnings, EOL announcements and CVE bulletins. Each comes from a different source for a different reason, but all of them land on the same list in the same tone. The screen looks the same; the work is not the same size.
\nWhat is mandatory comes from somewhere else: an EOL date, a CVE, or a dependency that locks you to an old version. A team that reacts to all three the same way is either late or running for nothing.
\nHow urgent a PR is doesn’t say in its title; it says in which clock opened it: one is tied to an advisory, the other to the schedule in dependabot.yml.
According to GitHub’s documentation, when security updates are enabled, a new PR is opened when a dependency with a vulnerability is found. Version updates, on the other hand, are opened when a new version is released and run at the frequency you define in dependabot.yml. The first is triggered by something in the outside world, the second by a schedule you wrote.
One more detail: the cooldown setting delays PR creation for version updates only; it doesn’t apply to security updates. So the setting you put in place to cut PR noise doesn’t silence a security PR.
The practical consequence: a version update PR that arrives on a weekly schedule sits in the same queue as a PR born from an advisory, but it is not the same clock’s job. You can deal with one this week and the other this quarter.
\nBefore you open a PR, look at two things: who opened it, and which label it came with. If it’s a security update, the question is no longer the version difference but whether you are affected. If it’s a version update, the question is whether that version fixes anything. The answer to the second will most often be “no, a new version came out, that’s all.”
\nEOL is known in advance. php.net publishes the end date of security support for each branch, and the policy is simple: two years of active support from release, then two years of security fixes only. PHP 8.2’s security support ends on 31 December 2026. That is about 9.5 weeks after this post’s publication date.
\nThe framework layer keeps its own clock too. Laravel’s 12.x release notes say: bug fixes are provided for 18 months, security fixes for 2 years. For Laravel 12, that means bug fixes ended on 13 August 2026 and security fixes continue until 24 February 2027. So a team running Laravel 12 on the day this post is published has already lost bug fix support, but has not yet lost security support.
\nA CVE doesn’t keep to a calendar. It can show up on a Tuesday morning, and it doesn’t care which version you run.
\nThe aftermath of EOL shouldn’t be exaggerated either. Upstream support ending doesn’t mean no fix will ever arrive; it means whether one arrives is no longer in your control. Ubuntu’s USN-6305-2 bulletin is an example: upstream support for PHP 7.4 had ended in November 2022; on 27 February 2024, Ubuntu released fixes to its PHP packages for CVE-2023-3823 and CVE-2023-3824. php7.4 on 20.04 got it from the standard repository; php7.2 on 18.04 and php7.0 on 16.04 required ESM and Ubuntu Pro.
That can be a way out. But the price is staying tied to a vendor, and leaving the question of which package the fix comes in and how fast to their schedule.
\nThe value of an EOL date is that it is predictable. You can count the date backwards: set the line by which you want to be in production before the end date, and count back from there to staging, and from there to the test work. If I applied my recommendation below to 8.2, the line would be 30 September 2026. If you are still running 8.2 as you read this, that line has passed; this is not an accusation, it’s the reason to write the plan today.
\nA locking trigger carries neither a date nor a bulletin. When a package stops supporting the old runtime, its new version becomes impossible to install, and nobody tells you.
\nIn the PHP 7.1 case, the trigger wasn’t EOL, it was a dependency lock; the details are in The Cost of Deferring an Upgrade.
\nSince a lock has no date like an EOL, you have to measure it: of the pending package updates, how many can’t be installed because of your runtime version? Every number above zero shows that a door has closed.
\nThe lock grows quietly. Because everything keeps working on the old runtime, it raises no alarm; the first visible signal is the day the package you want to update can’t be installed. On that day the type is no longer opportunity but locking, and the way out is likely to be a single piece.
\nI sort signals into three types:
\nThe response times below are my recommendation. They are not an industry standard; shift them to your own team’s capacity, but write a time down. A signal with no time sits in the queue, and a signal that sits in the queue eventually comes back as “urgent.”
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n| Signal | Type | Response time (my recommendation) |
|---|---|---|
| A CVE that affects you | Mandatory | Triage the same business day, patch within days |
| A CVE that doesn’t affect you | Opportunity | Normal cycle; the record that you’re not affected is kept |
| EOL date | Mandatory | The date goes on the calendar; in production at least 3 months before the end date |
| Dependency lock | Locking | Goes into next quarter’s plan |
| Deprecation warning | Locking | Before the next major |
| Feature or performance | Opportunity | Backlog; no date unless there’s a measured gain |
Two rows of the table deserve a separate pause. First, the CVE: the same bulletin is mandatory if it touches a code path you use, and an opportunity if it doesn’t. The urgency is set not by the bulletin’s headline but by your system. Two questions are enough to settle it: am I using the affected component, and can the affected code path be reached from outside? If you answer “yes” to both, you are on the first row of the table; if the fix hasn’t reached your version branch yet, this doesn’t extend the time, it turns the job from patching into a temporary mitigation. If one of the answers is “no,” put that on record; the record keeps you from debating the same bulletin again a month later. Second, performance: “the new version is faster” is an opportunity, not an obligation. Don’t open a calendar slot on that sentence without measuring.
\nWhy is the time a quarter rather than days on the locking rows? Because a lock breaks nothing today; but if one more package drops the old runtime every quarter, the cost of the way out grows. A quarter is short enough to notice that and write it into the next planning.
\n\nTan: Write the type and the response time where the signal lives, not in a separate doc: a label on the issue or one line in the PR description is enough. Whoever triages the queue then reads the decision from the list itself instead of opening a wiki. The \"not affected\" note lives there too, so when the same advisory comes back a month later, the answer is already attached.
This classification doesn’t replace a decision matrix. The type tells you the upper bound of urgency; it doesn’t tell you in which system, with what risk, in what order the work will be done.
\nThe same CVE can be an opportunity in an internal tool and mandatory in a payment flow. What makes that distinction is how critical the system is. The decision matrix is in part 3 of the series.
\nThe type is also not fixed. An opportunity, as it is deferred, turns into a locking one; a lock, as the EOL date approaches, turns into a mandatory one. Don’t classify once and bury it in the queue; look again in every planning round.
\nBack to the start: that Dependabot PR. Which clock opened it? An advisory, or a schedule? Is it releasing a lock, or only telling you a new version came out? Three questions send most PRs to the opportunity or reminder class, and what remains is a few jobs that really need looking at this week.
\nThe aim of classification is not to do fewer upgrades; it’s to do the right one at the right time. A team that meets a reminder as if it were an obligation runs for nothing; a team that takes a lock for a reminder is late. The price of both comes out of the same queue.
\nHere I classified the signals. In part 2 of the series I’ll cover which systems to track in which layers, in part 4 how to choose the target version, and in part 8 when not upgrading is the right decision.
\nTake the last 10 upgrade signals in your queue, sort them into mandatory, locking and opportunity, and write a response time for each.
", "content_text": "> **TL;DR:** A Dependabot PR, an EOL notice, a deprecation warning and a CVE bulletin land in the same queue but not with the same urgency. EOL and a CVE that affects you are mandatory, a dependency lock and a deprecation are locking, and features and performance are opportunity. Writing a response time for each type keeps you from rushing a reminder and from being late to a lock.\n\nMost of the PRs Dependabot opens are not obligations, they are reminders.\n\nFour kinds of signal get mixed together in the queue: Dependabot PRs, deprecation warnings, EOL announcements and CVE bulletins. Each comes from a different source for a different reason, but all of them land on the same list in the same tone. The screen looks the same; the work is not the same size.\n\nWhat is mandatory comes from somewhere else: an EOL date, a CVE, or a dependency that locks you to an old version. A team that reacts to all three the same way is either late or running for nothing.\n\n## Two Different Clocks Drop Into the Queue\n\nHow urgent a PR is doesn't say in its title; it says in which clock opened it: one is tied to an advisory, the other to the schedule in `dependabot.yml`.\n\nAccording to GitHub's documentation, when security updates are enabled, a new PR is opened when a dependency with a vulnerability is found. Version updates, on the other hand, are opened when a new version is released and run at the frequency you define in `dependabot.yml`. The first is triggered by something in the outside world, the second by a schedule you wrote.\n\nOne more detail: the `cooldown` setting delays PR creation for version updates only; it doesn't apply to security updates. So the setting you put in place to cut PR noise doesn't silence a security PR.\n\nThe practical consequence: a version update PR that arrives on a weekly schedule sits in the same queue as a PR born from an advisory, but it is not the same clock's job. You can deal with one this week and the other this quarter.\n\nBefore you open a PR, look at two things: who opened it, and which label it came with. If it's a security update, the question is no longer the version difference but whether you are affected. If it's a version update, the question is whether that version fixes anything. The answer to the second will most often be \"no, a new version came out, that's all.\"\n\n## EOL Is a Date, a CVE Is an Event\n\nEOL is known in advance. [php.net](https://www.php.net/supported-versions.php) publishes the end date of security support for each branch, and the policy is simple: two years of active support from release, then two years of security fixes only. PHP 8.2's security support ends on **31 December 2026**. That is about 9.5 weeks after this post's publication date.\n\nThe framework layer keeps its own clock too. Laravel's [12.x release notes](https://laravel.com/docs/12.x/releases) say: bug fixes are provided for 18 months, security fixes for 2 years. For Laravel 12, that means bug fixes ended on 13 August 2026 and security fixes continue until 24 February 2027. So a team running Laravel 12 on the day this post is published has already lost bug fix support, but has not yet lost security support.\n\nA CVE doesn't keep to a calendar. It can show up on a Tuesday morning, and it doesn't care which version you run.\n\nThe aftermath of EOL shouldn't be exaggerated either. Upstream support ending doesn't mean no fix will ever arrive; it means whether one arrives is no longer in your control. Ubuntu's [USN-6305-2](https://ubuntu.com/security/notices/USN-6305-2) bulletin is an example: upstream support for PHP 7.4 had ended in November 2022; on 27 February 2024, Ubuntu released fixes to its PHP packages for CVE-2023-3823 and CVE-2023-3824. `php7.4` on 20.04 got it from the standard repository; `php7.2` on 18.04 and `php7.0` on 16.04 required ESM and Ubuntu Pro.\n\nThat can be a way out. But the price is staying tied to a vendor, and leaving the question of which package the fix comes in and how fast to their schedule.\n\nThe value of an EOL date is that it is predictable. You can count the date backwards: set the line by which you want to be in production before the end date, and count back from there to staging, and from there to the test work. If I applied my recommendation below to 8.2, the line would be 30 September 2026. If you are still running 8.2 as you read this, that line has passed; this is not an accusation, it's the reason to write the plan today.\n\n## The Locking Trigger Is Invisible\n\nA locking trigger carries neither a date nor a bulletin. When a package stops supporting the old runtime, its new version becomes impossible to install, and nobody tells you.\n\nIn the PHP 7.1 case, the trigger wasn't EOL, it was a dependency lock; the details are in [The Cost of Deferring an Upgrade](/engineering/essays/the-cost-of-deferring-an-upgrade/).\n\nSince a lock has no date like an EOL, you have to measure it: of the pending package updates, how many can't be installed because of your runtime version? Every number above zero shows that a door has closed.\n\nThe lock grows quietly. Because everything keeps working on the old runtime, it raises no alarm; the first visible signal is the day the package you want to update can't be installed. On that day the type is no longer opportunity but locking, and the way out is likely to be a single piece.\n\n## Three Types, Three Response Times\n\nI sort signals into three types:\n\n- **Mandatory:** a signal that comes from outside and whose date or risk is independent of your preference (EOL, a CVE that affects you).\n- **Locking:** a signal that breaks nothing today but narrows your options tomorrow.\n- **Opportunity:** a signal where nothing breaks if you don't act, and you gain something if you do.\n\nThe response times below are my recommendation. They are not an industry standard; shift them to your own team's capacity, but write a time down. A signal with no time sits in the queue, and a signal that sits in the queue eventually comes back as \"urgent.\"\n\n| Signal | Type | Response time (my recommendation) |\n|---|---|---|\n| A CVE that affects you | Mandatory | Triage the same business day, patch within days |\n| A CVE that doesn't affect you | Opportunity | Normal cycle; the record that you're not affected is kept |\n| EOL date | Mandatory | The date goes on the calendar; in production at least 3 months before the end date |\n| Dependency lock | Locking | Goes into next quarter's plan |\n| Deprecation warning | Locking | Before the next major |\n| Feature or performance | Opportunity | Backlog; no date unless there's a measured gain |\n\nTwo rows of the table deserve a separate pause. First, the CVE: the same bulletin is mandatory if it touches a code path you use, and an opportunity if it doesn't. The urgency is set not by the bulletin's headline but by your system. Two questions are enough to settle it: am I using the affected component, and can the affected code path be reached from outside? If you answer \"yes\" to both, you are on the first row of the table; if the fix hasn't reached your version branch yet, this doesn't extend the time, it turns the job from patching into a temporary mitigation. If one of the answers is \"no,\" put that on record; the record keeps you from debating the same bulletin again a month later. Second, performance: \"the new version is faster\" is an opportunity, not an obligation. Don't open a calendar slot on that sentence without measuring.\n\nWhy is the time a quarter rather than days on the locking rows? Because a lock breaks nothing today; but if one more package drops the old runtime every quarter, the cost of the way out grows. A quarter is short enough to notice that and write it into the next planning.\n\n\n> **Tan:** Write the type and the response time where the signal lives, not in a separate doc: a label on the issue or one line in the PR description is enough. Whoever triages the queue then reads the decision from the list itself instead of opening a wiki. The \"not affected\" note lives there too, so when the same advisory comes back a month later, the answer is already attached.\n\n\n## The Limit of This Framework\n\nThis classification doesn't replace a decision matrix. The type tells you the upper bound of urgency; it doesn't tell you in which system, with what risk, in what order the work will be done.\n\nThe same CVE can be an opportunity in an internal tool and mandatory in a payment flow. What makes that distinction is how critical the system is. The decision matrix is in part 3 of the series.\n\nThe type is also not fixed. An opportunity, as it is deferred, turns into a locking one; a lock, as the EOL date approaches, turns into a mandatory one. Don't classify once and bury it in the queue; look again in every planning round.\n\n## Back to the Queue\n\nBack to the start: that Dependabot PR. Which clock opened it? An advisory, or a schedule? Is it releasing a lock, or only telling you a new version came out? Three questions send most PRs to the opportunity or reminder class, and what remains is a few jobs that really need looking at this week.\n\nThe aim of classification is not to do fewer upgrades; it's to do the right one at the right time. A team that meets a reminder as if it were an obligation runs for nothing; a team that takes a lock for a reminder is late. The price of both comes out of the same queue.\n\nHere I classified the signals. In part 2 of the series I'll cover which systems to track in which layers, in part 4 how to choose the target version, and in part 8 when not upgrading is the right decision.\n\nTake the last 10 upgrade signals in your queue, sort them into mandatory, locking and opportunity, and write a response time for each.\n\n## Sources\n\n- [Supported Versions](https://www.php.net/supported-versions.php) — PHP\n- [Laravel 12.x Release Notes](https://laravel.com/docs/12.x/releases) — Laravel\n- [Optimizing PR creation for version updates](https://docs.github.com/en/code-security/tutorials/secure-your-dependencies/optimizing-pr-creation-version-updates) — GitHub\n- [USN-6305-2: PHP vulnerabilities](https://ubuntu.com/security/notices/USN-6305-2) — Canonical", "date_published": "2026-10-25T00:00:00.000Z", "tags": [ "dependencies", "decisions", "security", "production", "Upgrading in Production" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/essays/integration-vs-unit-tests/", "url": "https://muhammetsafak.com/engineering/essays/integration-vs-unit-tests/", "title": "Which Test Catches the Production Bug?", "summary": "In practice, many production bugs originate between components, and an isolated unit test mocks exactly the connection that breaks. Unit tests shine on pure, logic-heavy cores; integration tests check glue code against a real database and HTTP layer. Spend the test budget where bugs are born, and keep mocks for external dependencies such as a payment provider.", "content_html": "A feature with 100% unit test coverage broke in production on day one. The bug was inside no single unit; it was in the way two components talked to each other. Every unit had been tested individually, in isolation, flawlessly. The seam between the two was the one thing no test had ever seen.
\n“Which test would have caught this bug?” is the real question behind any testing strategy.
\nTrace your production bugs back honestly and many of them land on the same address: the space between components. A payload serialized wrong, a contract that doesn’t line up, a transaction boundary drawn in the wrong place, a config difference across environments.
\nThe isolated unit test mocks out exactly the piece that breaks. A test that mocks the database can’t see a query written wrong. A test that mocks the HTTP layer can’t catch a wrong status code. A test is only as good as the fake it puts in place of the real thing.
\nThis is not an argument against unit tests. Unit tests have a clear value: fast feedback on complex, pure logic. A calculation, an algorithm, a state machine, a pile of edge cases — testing those with isolated, fast tests is the right move. Hundreds of scenarios in seconds.
\nUnit tests shine when the logic is rich.
\nIntegration tests answer a different question: do the components actually work together? With a real database, a real HTTP layer, a real queue. Hitting an endpoint and verifying that the right row appears in the database exposes everything mocks would have hidden.
\nA typical web application is mostly glue code: take data, validate it, save it, return it. There’s little logic and a lot of wiring in that code. Only an integration test exercises the wiring.
\nThe classic test pyramid puts a large number of unit tests at the bottom. But for a typical web application, integration tests give the highest payoff in bug prevention — because the bugs are born in the wiring.
\nA practical split: integration tests for the glue code; unit tests for the logic-heavy core. Piling isolated unit tests onto glue code raises the coverage number but doesn’t protect production.
\n\nTan: When a bug escapes to production, write down which test would have caught it: unit or integration. After a few incidents, where the test budget should go stops being an argument and becomes a record. The question at the top of the piece could be the last item of every post-mortem.
The most common mistake is mocking the thing that breaks. A test that mocks the database can never tell you a query is wrong — because that test has replaced the place the query goes with a fake. A query bug like an N+1 in data-intensive systems is only caught by a test that hits a real database.
\nSave mocks for the external things that you don’t control and that are slow — a payment provider, say. Mocking your own database is, more often than not, cutting the test loose from reality.
\n“All tests are green” does not mean “production is safe.” If you don’t know which test prevents which bug, your test count isn’t a source of confidence, it’s a feeling.
\nSpend your testing budget where the bugs are actually born — in the space between components.
", "content_text": "> **TL;DR:** In practice, many production bugs originate between components, and an isolated unit test mocks exactly the connection that breaks. Unit tests shine on pure, logic-heavy cores; integration tests check glue code against a real database and HTTP layer. Spend the test budget where bugs are born, and keep mocks for external dependencies such as a payment provider.\n\nA feature with 100% unit test coverage broke in production on day one. The bug was inside no single unit; it was in the way two components talked to each other. Every unit had been tested individually, in isolation, flawlessly. The seam between the two was the one thing no test had ever seen.\n\n\"Which test would have caught this bug?\" is the real question behind any testing strategy.\n\n## Where real bugs are born\n\nTrace your production bugs back honestly and many of them land on the same address: the space **between** components. A payload serialized wrong, a contract that doesn't line up, a transaction boundary drawn in the wrong place, a config difference across environments.\n\nThe isolated unit test mocks out exactly the piece that breaks. A test that mocks the database can't see a query written wrong. A test that mocks the HTTP layer can't catch a wrong status code. A test is only as good as the fake it puts in place of the real thing.\n\n## What unit tests are for\n\nThis is not an argument against unit tests. Unit tests have a clear value: fast feedback on complex, pure logic. A calculation, an algorithm, a state machine, a pile of edge cases — testing those with isolated, fast tests is the right move. Hundreds of scenarios in seconds.\n\nUnit tests shine when the **logic** is rich.\n\n## What integration tests are for\n\nIntegration tests answer a different question: do the components actually work together? With a real database, a real HTTP layer, a real queue. Hitting an endpoint and verifying that the right row appears in the database exposes everything mocks would have hidden.\n\nA typical web application is mostly **glue code**: take data, validate it, save it, return it. There's little logic and a lot of wiring in that code. Only an integration test exercises the wiring.\n\n## Where the testing budget goes\n\nThe classic test pyramid puts a large number of unit tests at the bottom. But for a typical web application, integration tests give the highest payoff in bug prevention — because the bugs are born in the wiring.\n\nA practical split: integration tests for the glue code; unit tests for the logic-heavy core. Piling isolated unit tests onto glue code raises [the coverage number](/engineering/essays/a-lean-testing-culture/) but doesn't protect production.\n\n\n> **Tan:** When a bug escapes to production, write down which test would have caught it: unit or integration. After a few incidents, where the test budget should go stops being an argument and becomes a record. The question at the top of the piece could be the last item of every post-mortem.\n\n\n## The mock trap\n\nThe most common mistake is mocking the thing that breaks. A test that mocks the database can never tell you a query is wrong — because that test has replaced the place the query goes with a fake. A query bug like an N+1 in [data-intensive systems](/engineering/systems/data-intensive-systems-breaking-points/) is only caught by a test that hits a real database.\n\nSave mocks for the external things that you don't control and that are slow — a payment provider, say. Mocking your own database is, more often than not, cutting the test loose from reality.\n\n---\n\n\"All tests are green\" does not mean \"production is safe.\" If you don't know which test prevents which bug, your test count isn't a source of confidence, it's a feeling.\n\nSpend your testing budget where the bugs are actually born — in the space between components.\n\n## Sources\n\n- [TestPyramid](https://martinfowler.com/bliki/TestPyramid.html) — martinfowler.com", "date_published": "2026-10-24T00:00:00.000Z", "tags": [ "testing", "quality", "opinion" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/essays/the-cost-of-deferring-an-upgrade/", "url": "https://muhammetsafak.com/engineering/essays/the-cost-of-deferring-an-upgrade/", "title": "The Cost of Deferring an Upgrade", "summary": "Deferring an upgrade is not avoiding risk; it is accumulating risk where no one can see it. As packages drop support for the old runtime one by one, a dependency lock gets set quietly, and a job of 1–2 days turns into dozens of breaking changes to fix at once and a bug queue that stretches over months. A deferral is only a decision once its date and its reason are written down.", "content_html": "When I proposed moving from PHP 7.1 to a stable 8.2, the answer was short: “It works, so why bother?” The answer was right — nothing was broken that day. What had started to break was invisible: packages were dropping support for old PHP one by one, and the application was getting locked to 7.1.
\n“It works” describes today. The bill is issued by the days that follow.
\nThe objection isn’t illogical, so I don’t wave it away. Spending time on something that isn’t broken looks like gaining nothing. You can’t refute a sentence that is right by arguing with the sentence; you have to come with a number. What follows is the accounting of what that sentence cost me: by which mechanism, in which line items, and how much of it I was able to measure.
\nThe lock wasn’t set in a single day; with every package update, one more door closed.
\nThe mechanism is ordinary. A package drops support for the old runtime in its new version. Since the runtime hasn’t been upgraded, that version can’t be installed, and you stay on the old one. The same thing happens with the next package, then the one after. None of them raises an alarm on its own: the old version keeps working.
\nAt some point no package is left that can be updated. I call this a dependency lock: the application is tied to the runtime, the packages to the application, and all of it together to 7.1. The way out is a single piece. I couldn’t move any package forward without upgrading the runtime; and without moving the packages forward, I couldn’t see what the upgrade would cost.
\nThat’s why the lock is set unnoticed. Each door that closes is, on its own, a detail that can be deferred; no single package update looks big enough to make you say “we have to do the upgrade today.” Deferring the decision is reasonable every single time. The sum of reasonable decisions is being backed into a corner.
\nThe worst thing about the lock is that it is silent. No error message says “another door closed today.” You notice it only on the day you try to update a package, while reading why the update won’t install.
\nThe sign to look for in your own system is simple. Of the pending package updates, how many can’t be installed because of the runtime version? Every number above zero shows that a door has closed.
\nOn the day I proposed it, the job was a 1–2 day upgrade. While the lock was being set, each package update took a few hours. Both looked like small jobs, and that’s why deferring was easy.
\nA small job is a good candidate for deferral; “let’s finish the urgent thing first” fits into every calendar.
\nOnce deferred, the job didn’t just grow; because it had piled up, it became a single piece. I had to skip versions: not from 7.1 to 8.2 step by step, but passing everything in between in one go. Dozens of breaking changes and deprecated methods lay ahead of me, and all of them had to be solved at the same time. A debt that could have been paid in installments turned into a single collection.
\nWhat the number really taught me isn’t the amount itself but the change in the kind of work. A few hours of package updates were piecemeal jobs that could be done in any order, on any day. Dozens of breaking changes are one indivisible job that has to be finished once started. Deferring didn’t only make the job bigger; it took away its flexibility.
\nThis is the only number I have: a 1–2 day job, plus a few hours per package update. I have no measurement to set next to it for the total cost of deferring. I won’t set one, either; writing down a number I didn’t measure would be a worse argument than “it works, doesn’t it.”
\nEven so, this one number is enough to start a conversation. Against “it works, doesn’t it,” you can go not with “we’re gaining nothing,” but with “1–2 days now, dozens of changes in one go later.” To explain the difference between the two to a manager, a measurement isn’t essential; the direction of the comparison is enough.
\nHow to produce the number is in part 7 of the series; which upgrade comes first is in part 3. Here I’ll say only this: if I could have put this comparison on the table that day, “it works” alone wouldn’t have been enough of an answer.
\nWe cleaned up the breaking changes; what we didn’t expect was the bugs that came after.
\nBreaking changes have a good property: the error message points to them. You draw up the list, close them one by one, and it’s over. This item is long but known: you can roughly estimate its size, place it on the calendar, and tell your manager.
\nThe bugs weren’t like that. Some of them showed up in only a single case — a path that rarely gets triggered, a particular piece of data, a particular order. For a bug like that you first have to see it, then reproduce it, then find its cause. Each one is a separate investigation. Their fixes spread over months. One bug would close, and another would show up in a different case.
\nFor planning, the two have to be written separately. Breaking changes are a predictable item; bugs are not. If you build the budget only for the first, the second lives outside the budget and inside the calendar.
\nThat’s where the real cost is: the long tail. The price of the batch migration fits the calendar; the price of the tail doesn’t. On top of that, because I was carrying dozens of changes together, the source of every bug was one of dozens of candidates; telling which bug came from which change created a job of its own.
\nSaying “there’s technical debt” isn’t enough against the “it works, doesn’t it” objection; the person across the table hears it as an opinion. To answer in business language, you have to break the risk into line items. Every deferred upgrade accumulates four:
\nNone of these produces an error today, and none shows on the monitoring dashboard; all of them will be paid tomorrow. For a manager the difference is this too: asking for budget for something broken is easy, explaining why something that isn’t broken will break is hard. That’s why naming the items works, because each one can be tied to a date, a counter, or a number.
\nThe summary of the four items: “it works” describes today; risk is tomorrow’s balance-sheet item. The cost of upgrading is known, and you schedule the payment. The cost of deferring is unknown, and you don’t get to pick when it is collected.
\n\nTan: The post makes the cost of waiting plain. What bothers me is that the burden of proof always sits with whoever proposes the upgrade. People ask for a reason to upgrade; almost nobody asks for a reason to wait. If both questions were on the table, would “it works” still win so easily?
Not upgrading can also be a legitimate decision; in which cases, I’m leaving to part 8 of the series. My objection here isn’t to deferring, but to the deferral not being written down as a decision.
\nDeferring can be a conscious decision; but a decision that isn’t written down comes back one day as a surprise. A deferral with its date and reason written down is risk management; one without is just forgetting.
\nThere is one thing you can do today: look up the end-of-support dates of the runtime and framework versions you use, and write them down somewhere. Even if the system has no inventory, two lines are enough: what’s running, and when its support ends. The next time someone asks “why?”, you’ll at least have a date in hand. If you also write next to the date how many versions behind the system will be by then, the question stops being “does it work” and becomes “when will we pay”; and the conversation starts there.
\nAsk yourself: if upgrading this system today were a 1–2 day job, would you do it? If the answer is yes, there’s nothing to gain by waiting; the job only grows.
\nThe series continues from here: in part 1, the situations that trigger an upgrade; in part 2, the inventory and layers for seeing what will be upgraded; in part 3, the decision matrix; and up to part 8, the target version, prerequisites, rollout, and organization.
\nWhen you defer an upgrade, you don’t close the risk; you write the bill to the next day.
", "content_text": "> **TL;DR:** Deferring an upgrade is not avoiding risk; it is accumulating risk where no one can see it. As packages drop support for the old runtime one by one, a dependency lock gets set quietly, and a job of 1–2 days turns into dozens of breaking changes to fix at once and a bug queue that stretches over months. A deferral is only a decision once its date and its reason are written down.\n\nWhen I proposed moving from PHP 7.1 to a stable 8.2, the answer was short: \"It works, so why bother?\" The answer was right — nothing was broken that day. What had started to break was invisible: packages were dropping support for old PHP one by one, and the application was getting locked to 7.1.\n\n\"It works\" describes today. The bill is issued by the days that follow.\n\nThe objection isn't illogical, so I don't wave it away. Spending time on something that isn't broken looks like gaining nothing. You can't refute a sentence that is right by arguing with the sentence; you have to come with a number. What follows is the accounting of what that sentence cost me: by which mechanism, in which line items, and how much of it I was able to measure.\n\n## How the Lock Gets Set\n\nThe lock wasn't set in a single day; with every package update, one more door closed.\n\nThe mechanism is ordinary. A package drops support for the old runtime in its new version. Since the runtime hasn't been upgraded, that version can't be installed, and you stay on the old one. The same thing happens with the next package, then the one after. None of them raises an alarm on its own: the old version keeps working.\n\nAt some point no package is left that can be updated. I call this a **dependency lock**: the application is tied to the runtime, the packages to the application, and all of it together to 7.1. The way out is a single piece. I couldn't move any package forward without upgrading the runtime; and without moving the packages forward, I couldn't see what the upgrade would cost.\n\nThat's why the lock is set unnoticed. Each door that closes is, on its own, a detail that can be deferred; no single package update looks big enough to make you say \"we have to do the upgrade today.\" Deferring the decision is reasonable every single time. The sum of reasonable decisions is being backed into a corner.\n\nThe worst thing about the lock is that it is silent. No error message says \"another door closed today.\" You notice it only on the day you try to update a package, while reading why the update won't install.\n\nThe sign to look for in your own system is simple. Of the pending package updates, how many can't be installed because of the runtime version? Every number above zero shows that a door has closed.\n\n## The Only Number I Have\n\nOn the day I proposed it, the job was a 1–2 day upgrade. While the lock was being set, each package update took a few hours. Both looked like small jobs, and that's why deferring was easy.\n\nA small job is a good candidate for deferral; \"let's finish the urgent thing first\" fits into every calendar.\n\nOnce deferred, the job didn't just grow; because it had piled up, it became a single piece. I had to skip versions: not from 7.1 to 8.2 step by step, but passing everything in between in one go. Dozens of breaking changes and deprecated methods lay ahead of me, and all of them had to be solved at the same time. A debt that could have been paid in installments turned into a single collection.\n\nWhat the number really taught me isn't the amount itself but the change in the kind of work. A few hours of package updates were piecemeal jobs that could be done in any order, on any day. Dozens of breaking changes are one indivisible job that has to be finished once started. Deferring didn't only make the job bigger; it took away its flexibility.\n\nThis is the only number I have: a 1–2 day job, plus a few hours per package update. I have no measurement to set next to it for the total cost of deferring. I won't set one, either; writing down a number I didn't measure would be a worse argument than \"it works, doesn't it.\"\n\nEven so, this one number is enough to start a conversation. Against \"it works, doesn't it,\" you can go not with \"we're gaining nothing,\" but with \"1–2 days now, dozens of changes in one go later.\" To explain the difference between the two to a manager, a measurement isn't essential; the direction of the comparison is enough.\n\nHow to produce the number is in part 7 of the series; which upgrade comes first is in part 3. Here I'll say only this: if I could have put this comparison on the table that day, \"it works\" alone wouldn't have been enough of an answer.\n\n## The Real Line Item on the Bill\n\nWe cleaned up the breaking changes; what we didn't expect was the bugs that came after.\n\nBreaking changes have a good property: the error message points to them. You draw up the list, close them one by one, and it's over. This item is long but known: you can roughly estimate its size, place it on the calendar, and tell your manager.\n\nThe bugs weren't like that. Some of them showed up in only a single case — a path that rarely gets triggered, a particular piece of data, a particular order. For a bug like that you first have to see it, then reproduce it, then find its cause. Each one is a separate investigation. Their fixes spread over months. One bug would close, and another would show up in a different case.\n\nFor planning, the two have to be written separately. Breaking changes are a predictable item; bugs are not. If you build the budget only for the first, the second lives outside the budget and inside the calendar.\n\nThat's where the real cost is: the **long tail**. The price of the batch migration fits the calendar; the price of the tail doesn't. On top of that, because I was carrying dozens of changes together, the source of every bug was one of dozens of candidates; telling which bug came from which change created a job of its own.\n\n## \"It Works\" Describes Today\n\nSaying \"there's technical debt\" isn't enough against the \"it works, doesn't it\" objection; the person across the table hears it as an opinion. To answer in business language, you have to break the risk into line items. Every deferred upgrade accumulates four:\n\n- **End of support:** once the runtime or framework is no longer maintained, no new fixes arrive. A set day on the calendar, a known date.\n- **Time exposed to CVEs:** for every CVE announced after the day support ends, no official patch arrives; unless your distribution or a paid extended support patches it, the hole stays in the system until you upgrade. The counter starts the day support ends and stops on the day of the upgrade.\n- **Dependency lock:** as packages are pinned to old versions, your update options narrow. Once the lock is set, the only way out is to move everything together.\n- **Growing version gap:** as the number of versions in between grows, so does the number of breaking changes and deprecated methods to solve in one go. In my case, that was dozens of items.\n\nNone of these produces an error today, and none shows on the monitoring dashboard; all of them will be paid tomorrow. For a manager the difference is this too: asking for budget for something broken is easy, explaining why something that isn't broken will break is hard. That's why naming the items works, because each one can be tied to a date, a counter, or a number.\n\nThe summary of the four items: **\"it works\" describes today; risk is tomorrow's balance-sheet item.** The cost of upgrading is known, and you schedule the payment. The cost of deferring is unknown, and you don't get to pick when it is collected.\n\n\n> **Tan:** The post makes the cost of waiting plain. What bothers me is that the burden of proof always sits with whoever proposes the upgrade. People ask for a reason to upgrade; almost nobody asks for a reason to wait. If both questions were on the table, would “it works” still win so easily?\n\n\n## When Do You Let It Go?\n\nNot upgrading can also be a legitimate decision; in which cases, I'm leaving to part 8 of the series. My objection here isn't to deferring, but to the deferral not being written down as a decision.\n\nDeferring can be a conscious decision; but a decision that isn't written down comes back one day as a surprise. A deferral with its date and reason written down is risk management; one without is just forgetting.\n\nThere is one thing you can do today: look up the end-of-support dates of the runtime and framework versions you use, and write them down somewhere. Even if the system has no inventory, two lines are enough: what's running, and when its support ends. The next time someone asks \"why?\", you'll at least have a date in hand. If you also write next to the date how many versions behind the system will be by then, the question stops being \"does it work\" and becomes \"when will we pay\"; and the conversation starts there.\n\nAsk yourself: if upgrading this system today were a 1–2 day job, would you do it? If the answer is yes, there's nothing to gain by waiting; the job only grows.\n\nThe series continues from here: in part 1, [the situations that trigger an upgrade](/engineering/notes/upgrade-triggers/); in part 2, the inventory and layers for seeing what will be upgraded; in part 3, the decision matrix; and up to part 8, the target version, prerequisites, rollout, and organization.\n\nWhen you defer an upgrade, you don't close the risk; you write the bill to the next day.\n\n## Frequently asked\n\n### It works, so why bother?\n\n\"It works\" describes today; risk is tomorrow's balance-sheet item. Every deferred upgrade accumulates four line items: end of support, time exposed to CVEs, a dependency lock, and a growing version gap. None of them produces an error today, and all of them are paid tomorrow. The cost of upgrading is known and you schedule the payment; the cost of deferring is unknown and you don't get to pick when it is collected.\n\n### Of the pending package updates, how many can't be installed because of the runtime version?\n\nThis number is the sign that shows whether a dependency lock has been set. Packages drop support for the old runtime one by one; none of them raises an alarm, and the old version keeps working. Every number above zero means a door has closed. At some point no package is left that can be updated, and the only way out is to move everything together.", "date_published": "2026-10-18T00:00:00.000Z", "tags": [ "technical-debt", "decisions", "dependencies", "opinion", "Upgrading in Production" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/essays/a-lean-testing-culture/", "url": "https://muhammetsafak.com/engineering/essays/a-lean-testing-culture/", "title": "Chasing 100% Coverage With Tests That Do Nothing", "summary": "Coverage measures only that a line ran, not that it was verified; once it becomes a target, people write the cheapest tests that raise the number and catch no bugs. A test is worth the bugs it catches: test the behavior that matters and let coverage arrive as a by-product. Spend the test budget on complex logic, code that handles money and places that have failed before.", "content_html": "I saw a pull request: forty new tests, all serving a single purpose — pushing coverage to 100%. Most of them called a getter and verified that the value just set came back. Coverage turned green. The number of bugs those forty tests would catch: zero.
\nTest coverage is a useful number — until you turn it into a target.
\nCoverage measures one thing: which lines of code run while the tests execute. What it doesn’t measure matters far more: whether those lines are actually verified.
\nA line counts as “covered” even by a test with no meaningful assert. The code runs, the test is green, but nobody has checked whether the code behaves correctly. Coverage can’t see the difference between code that runs and code that’s verified.
Once coverage becomes a target, human behavior is predictable: people write the tests that raise the number by the cheapest route.
\nassert is meaningless, which only say “it didn’t blow up”.These raise the coverage number and catch no bugs. Worse: a green 100% gives the team false confidence. Code that’s assumed to be tested has merely been executed.
\nA test’s value lies not in the line it covers but in the bug it catches. A simple check: deliberately break the logic of the code — turn a + into a -, invert a condition. A good test goes red. If it doesn’t, that test isn’t verifying anything.
The systematic form of this idea is mutation testing: the tool breaks the code in small ways and measures whether your tests catch it. Coverage says “did the line run”; mutation testing says “was the line actually tested”.
\nThe right order is the reverse: test the behavior that matters first, and let coverage rise as a byproduct. Chase the number and you write tests that do nothing; chase the behavior and coverage comes anyway — and the coverage you get is real.
\n100% is not a target. A critical path that’s never tested can exist even in a 100% codebase; conversely, a system with 75% coverage may be solidly testing every risky path.
\nWriting tests is also writing code; it has a limited budget. Spend that budget according to risk:
\nTesting trivial, logicless code wastes the budget. The “the test setup is longer than the test itself” sign I mentioned in signals that a system has grown too complex connects here too: if a test is very hard to write, the problem isn’t the test, it’s that the code under test is too tightly coupled.
\nA coverage percentage is not a source of confidence; it’s only an observation. Confidence comes from the bugs your tests catch — not the number they chase.
\nAim not for the number, but for the test that goes red when the code is wrong.
", "content_text": "> **TL;DR:** Coverage measures only that a line ran, not that it was verified; once it becomes a target, people write the cheapest tests that raise the number and catch no bugs. A test is worth the bugs it catches: test the behavior that matters and let coverage arrive as a by-product. Spend the test budget on complex logic, code that handles money and places that have failed before.\n\nI saw a pull request: forty new tests, all serving a single purpose — pushing coverage to 100%. Most of them called a getter and verified that the value just set came back. Coverage turned green. The number of bugs those forty tests would catch: zero.\n\nTest coverage is a useful number — until you turn it into a target.\n\n## What coverage measures, and what it doesn't\n\nCoverage measures one thing: which lines of code **run** while the tests execute. What it doesn't measure matters far more: whether those lines are actually **verified**.\n\nA line counts as \"covered\" even by a test with no meaningful `assert`. The code runs, the test is green, but nobody has checked whether the code behaves correctly. Coverage can't see the difference between code that runs and code that's verified.\n\n## The wrong metric rewards the wrong test\n\nOnce coverage becomes a target, human behavior is predictable: people write the tests that raise the number by the cheapest route.\n\n- Getter/setter tests — testing code that has no logic.\n- Tests that verify a mock was called — an interaction, not a real result.\n- Tests whose `assert` is meaningless, which only say \"it didn't blow up\".\n\nThese raise the coverage number and catch no bugs. Worse: a green 100% gives the team false confidence. Code that's assumed to be tested has merely been executed.\n\n## The measure of a good test\n\nA test's value lies not in the line it covers but in the bug it catches. A simple check: deliberately break the logic of the code — turn a `+` into a `-`, invert a condition. A good test goes red. If it doesn't, that test isn't verifying anything.\n\nThe systematic form of this idea is mutation testing: the tool breaks the code in small ways and measures whether your tests catch it. Coverage says \"did the line run\"; mutation testing says \"was the line actually tested\".\n\n## Coverage is a byproduct\n\nThe right order is the reverse: test the behavior that matters first, and let coverage rise as a **byproduct**. Chase the number and you write tests that do nothing; chase the behavior and coverage comes anyway — and the coverage you get is real.\n\n100% is not a target. A critical path that's never tested can exist even in a 100% codebase; conversely, a system with 75% coverage may be solidly testing every risky path.\n\n## Where to spend the testing budget\n\nWriting tests is also writing code; it has a limited budget. Spend that budget according to risk:\n\n- Complex logic, code that handles money, calculations.\n- Edge cases — boundary values, empty input, concurrency.\n- Places that have broken before — if a bug came out of there once, it will again.\n\nTesting trivial, logicless code wastes the budget. The \"the test setup is longer than the test itself\" sign I mentioned in [signals that a system has grown too complex](/engineering/essays/signals-of-an-over-complex-system/) connects here too: if a test is very hard to write, the problem isn't the test, it's that the code under test is too tightly coupled.\n\n---\n\nA coverage percentage is not a source of confidence; it's only an observation. Confidence comes from the bugs your tests catch — not the number they chase.\n\nAim not for the number, but for the test that goes red when the code is wrong.\n\n## Sources\n\n- [Test Coverage](https://martinfowler.com/bliki/TestCoverage.html) — martinfowler.com", "date_published": "2026-10-17T00:00:00.000Z", "tags": [ "testing", "quality", "opinion" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/essays/third-party-package-dependency/", "url": "https://muhammetsafak.com/engineering/essays/third-party-package-dependency/", "title": "The Cost of Installing a Package for Everything", "summary": "Installing a package for a small function puts someone else's code into your codebase, running with your own privileges: security surface, maintenance and upgrade friction add up to a bill that does not end at installation. A package wins on hard, security-critical work such as cryptography, HTTP clients and database drivers; for small, stable work you understand, writing those few lines yourself is cheaper.", "content_html": "One project’s composer.lock had 180 packages. The ones used directly, counted by hand, were twelve. The rest — over a hundred packages — were mostly dependencies dragged in by packages installed for a small function.
Typing composer require feels free. It is not. Every dependency is a door into your codebase, opening onto someone else’s code — and that door never fully closes again.
When you install a package, that package’s code runs on your server, with your application’s privileges. It can reach your database, see your file system, read your environment variables.
\nThis does not mean the package is malicious — but if it gets compromised one day, your application gets compromised too. This is what a supply-chain attack is: the weak link is not the code you wrote, it is the code you trust. If you trust hundreds of packages, that means there are hundreds of teams you have to trust.
\nA dependency’s cost does not end at installation:
\nThese line items are paid for a small package just as they are for a large one.
\nThe real issue is the asymmetry. A few lines of helper code that generate a slug, format a date — installing a package for these opens a wide door for a small gain. The gain is a few lines of code; the cost is the entire bill above.
\nThere is a famous example: in the JavaScript world, an eleven-line package that left-pads a string was unpublished one day, and the builds of thousands of projects collapsed. For eleven lines, all of them had tied themselves to a door they never audited.
\nThe rule is not ideological, it is economic:
\nThe measure is this: is the problem the package solves bigger than the bill the package brings? If it is bigger, install it. If not, write those few lines and own them.
\nIf you have decided to install a package, look at which one you pick: is it actively maintained, when was its last release, how many transitive dependencies does it drag in, is its license suitable, how broad is its community? Choosing a dependency follows the same logic as the innovation token in the boring architecture piece: pick the one that is mature, predictable, and will not be abandoned.
\nAdding a dependency looks like a way to avoid writing code. In reality it is getting someone else to write the code while you keep the responsibility.
\nBefore opening an entire door for a small job, ask: is writing those few lines really more expensive?
", "content_text": "> **TL;DR:** Installing a package for a small function puts someone else's code into your codebase, running with your own privileges: security surface, maintenance and upgrade friction add up to a bill that does not end at installation. A package wins on hard, security-critical work such as cryptography, HTTP clients and database drivers; for small, stable work you understand, writing those few lines yourself is cheaper.\n\nOne project's `composer.lock` had 180 packages. The ones used directly, counted by hand, were twelve. The rest — over a hundred packages — were mostly dependencies dragged in by packages installed for a small function.\n\nTyping `composer require` feels free. It is not. Every dependency is a door into your codebase, opening onto someone else's code — and that door never fully closes again.\n\n## Every dependency is a door\n\nWhen you install a package, that package's code runs on your server, with your application's privileges. It can reach your database, see your file system, read your environment variables.\n\nThis does not mean the package is malicious — but if it gets compromised one day, your application gets compromised too. This is what a supply-chain attack is: the weak link is not the code you wrote, it is the code you trust. If you trust hundreds of packages, that means there are hundreds of teams you have to trust.\n\n## The line items on the bill\n\nA dependency's cost does not end at installation:\n\n- **Security surface.** Every package is a piece of code you have not audited. With transitive dependencies, no one fully knows what you are actually trusting.\n- **Maintenance.** A package gets abandoned, breaks on a new PHP version, conflicts with another package. Its problem is now your problem.\n- **[Upgrade friction](/engineering/essays/the-cost-of-deferring-an-upgrade/).** When you want to upgrade the framework, you are only as fast as your oldest, least-maintained dependency.\n\nThese line items are paid for a small package just as they are for a large one.\n\n## Small job, big door\n\nThe real issue is the asymmetry. A few lines of helper code that generate a slug, format a date — installing a package for these opens a wide door for a small gain. The gain is a few lines of code; the cost is the entire bill above.\n\nThere is a famous example: in the JavaScript world, an eleven-line package that left-pads a string was unpublished one day, and the builds of thousands of projects collapsed. For eleven lines, all of them had tied themselves to a door they never audited.\n\n## When to use a package, when to write it yourself\n\nThe rule is not ideological, it is economic:\n\n- **Use a package:** for hard, well-tested, security-critical work. Cryptography, HTTP clients, database drivers, the framework itself. **Never** write your own crypto code — this is where the package wins, no argument.\n- **Write it yourself:** for small, stable work you understand well. A few lines of helper code are cheaper than inheriting an entire dependency tree.\n\nThe measure is this: is the problem the package solves bigger than the bill the package brings? If it is bigger, install it. If not, write those few lines and own them.\n\n## Choosing a dependency\n\nIf you have decided to install a package, look at which one you pick: is it actively maintained, when was its last release, how many transitive dependencies does it drag in, is its license suitable, how broad is its community? Choosing a dependency follows the same logic as the innovation token in the [boring architecture](/engineering/essays/why-boring-architecture/) piece: pick the one that is mature, predictable, and will not be abandoned.\n\n---\n\nAdding a dependency looks like a way to avoid writing code. In reality it is getting someone else to write the code while you keep the responsibility.\n\nBefore opening an entire door for a small job, ask: is writing those few lines really more expensive?\n\n## Sources\n\n- [kik, left-pad, and npm](https://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm) — npm", "date_published": "2026-10-10T00:00:00.000Z", "tags": [ "dependencies", "security", "simplicity", "opinion" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/when-is-cqrs-necessary/", "url": "https://muhammetsafak.com/engineering/notes/when-is-cqrs-necessary/", "title": "CQRS: When You Need It, When It's Dead Weight", "summary": "CQRS separates the write model from the read model; that does not necessarily mean two databases or a read replica. It brings clarity in complex domains where the shape of writes and reads really diverges, and in plain CRUD it adds only a second model to keep in sync and twice the code. Lightweight CQRS starts with a separate query class.", "content_html": "I once saw a codebase where a plain CRUD application claimed “we implemented CQRS.” A single “create user” operation had a Command, a CommandHandler, a separate read model, and a projection. All that ceremony ended in one line of INSERT.
CQRS buys real clarity in some systems. In this application, the only thing it bought was twice the code.
\nCQRS — Command Query Responsibility Segregation — rests on a single idea: separate the write model from the read model.
\nIn the classic approach a single model is both written and read — the same Order class both creates an order and renders it on screen. CQRS splits this in two: a write side that is a normalized model enforcing the rules; a read side that is a denormalized model shaped to the screen.
Note: this does not necessarily mean two databases. Most of the time it means two models — they can live in the same database.
\nA common confusion: people think CQRS and a read replica are the same thing. They are not.
\nYou can use either one without the other. You can have a read replica but a single model; or do CQRS but on a single server.
\nCQRS adds value when the write model and the read model genuinely diverge. That happens in complex domains: the shape you write drifts away from the shape you read.
\nThe write side is a normalized model that enforces invariants and carries business rules. The read side is a model flattened to the needs of the screens, precomputed, fast to query. Forcing them into a single model makes both mediocre.
\nHere’s the problem: in a plain CRUD application the write model and the read model are already the same — both are that row itself. Applying CQRS there is modeling a distinction that doesn’t exist.
\nThe result: two models, two structures that must be kept in sync, twice the code — and zero gain in return. This is speculative generality at the scale of a pattern: paying today for a complexity that doesn’t exist today.
\nTo “do” CQRS you don’t need event sourcing, separate databases, or projection infrastructure. Those are the heaviest form of CQRS, not its mandatory form.
\nA lightweight start: leave the write side on plain Eloquent models, and for the read side write a separate query class that returns a DTO shaped to the screen. The write and read paths are separated in code — which is the essence of CQRS already. Move to the heavy infrastructure only when you actually need it.
\n\nTan: If the read model is updated through a separate flow after the write, a user may not see what they just saved. Before settling on CQRS, ask whether the product side can accept that delay. The lightweight form often takes this question off the table.
To take CQRS seriously, you should see these signals:
\nWithout these signals, a single model with a separate read query when needed is more than enough.
\nCQRS is not a maturity badge; it’s the solution to a specific problem — reads and writes genuinely diverging. If you don’t have that problem, the only thing it buys you is the burden of keeping a second model in sync.
\nBefore you split the model, look at whether your reads and writes are truly separate.
", "content_text": "> **TL;DR:** CQRS separates the write model from the read model; that does not necessarily mean two databases or a read replica. It brings clarity in complex domains where the shape of writes and reads really diverges, and in plain CRUD it adds only a second model to keep in sync and twice the code. Lightweight CQRS starts with a separate query class.\n\nI once saw a codebase where a plain CRUD application claimed \"we implemented CQRS.\" A single \"create user\" operation had a Command, a CommandHandler, a separate read model, and a projection. All that ceremony ended in one line of `INSERT`.\n\nCQRS buys real clarity in some systems. In this application, the only thing it bought was twice the code.\n\n## What CQRS is, briefly\n\nCQRS — Command Query Responsibility Segregation — rests on a single idea: separate the **write model** from the **read model**.\n\nIn the classic approach a single model is both written and read — the same `Order` class both creates an order and renders it on screen. CQRS splits this in two: a write side that is a normalized model enforcing the rules; a read side that is a denormalized model shaped to the screen.\n\nNote: this does not necessarily mean two databases. Most of the time it means two **models** — they can live in the same database.\n\n## Don't confuse it with a read replica\n\nA common confusion: people think CQRS and a read replica are the same thing. They are not.\n\n- A **read replica** ([read/write splitting](/engineering/notes/read-write-splitting/)) is an infrastructure decision: same model, different server. To scale read load.\n- **CQRS** is a modeling decision: different model, same or different location. Because the **shape** of reads and writes has diverged.\n\nYou can use either one without the other. You can have a read replica but a single model; or do CQRS but on a single server.\n\n## What CQRS buys you\n\nCQRS adds value when the write model and the read model **genuinely** diverge. That happens in complex domains: the shape you write drifts away from the shape you read.\n\nThe write side is a normalized model that enforces invariants and carries business rules. The read side is a model flattened to the needs of the screens, precomputed, fast to query. Forcing them into a single model makes both mediocre.\n\n## Dead weight in plain CRUD\n\nHere's the problem: in a plain CRUD application the write model and the read model are **already the same** — both are that row itself. Applying CQRS there is modeling a distinction that doesn't exist.\n\nThe result: two models, two structures that must be kept in sync, twice the code — and zero gain in return. This is [speculative generality](/engineering/essays/the-cost-of-just-in-case-code/) at the scale of a pattern: paying today for a complexity that doesn't exist today.\n\n## Lightweight CQRS\n\nTo \"do\" CQRS you don't need event sourcing, separate databases, or projection infrastructure. Those are the heaviest form of CQRS, not its mandatory form.\n\nA lightweight start: leave the write side on plain Eloquent models, and for the read side write a separate query class that returns a DTO shaped to the screen. The write and read paths are separated in code — which is the essence of CQRS already. Move to the heavy infrastructure only when you actually need it.\n\n\n> **Tan:** If the read model is updated through a separate flow after the write, a user may not see what they just saved. Before settling on CQRS, ask whether the product side can accept that delay. The lightweight form often takes this question off the table.\n\n\n## When you really need it\n\nTo take CQRS seriously, you should see these signals:\n\n- High domain complexity — the write side carries serious business rules (the kind of domain where [Clean Architecture](/engineering/essays/layered-vs-clean-architecture/) adds value).\n- The shape and load of reads and writes are markedly different.\n- Most often alongside [event-driven architecture](/engineering/notes/when-event-driven-architecture/) — the read model is updated by events emitted from the write side.\n\nWithout these signals, a single model with a separate read query when needed is more than enough.\n\n---\n\nCQRS is not a maturity badge; it's the solution to a specific problem — reads and writes genuinely diverging. If you don't have that problem, the only thing it buys you is the burden of keeping a second model in sync.\n\nBefore you split the model, look at whether your reads and writes are truly separate.\n\n## Sources\n\n- [CQRS](https://martinfowler.com/bliki/CQRS.html) — martinfowler.com", "date_published": "2026-10-03T00:00:00.000Z", "tags": [ "architecture", "cqrs", "patterns" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/archiving-strategy/", "url": "https://muhammetsafak.com/engineering/notes/archiving-strategy/", "title": "An Archiving Strategy: Old Data Without the Slowdown", "summary": "When hot and cold data share one table, the cold rows slow every query: indexes bloat and autovacuum falls behind. Archiving is not a cleanup job but a lifecycle designed on day one. In a table partitioned by time, an old partition is detached with DETACH, moved to colder storage, and the door to it stays open.", "content_html": "I once saw an orders table that had accumulated eight years of data. 95% of the queries looked at the last 90 days — but every query paid the price of the remaining eight years in index size and vacuum time. The hot data was carrying the weight of the cold data.
No table can grow forever. Archiving is accepting that and planning for it from the start.
\nMost tables have a natural hot/cold split. Active data — this week’s orders, open tickets — is read and written constantly. Historical data — records closed three years ago — almost never changes and is rarely read. But when the two live in the same table, the cold data slows down every query against the hot data: indexes bloat, autovacuum can’t keep up, the working set grows artificially.
\nArchiving isn’t a cleanup chore you think of once the table has grown large enough to cause panic. It’s a lifecycle for the data, and it should be designed on day one: data is born hot, cools over time, then turns cold. The question isn’t “will we archive,” it’s “where does the cooling data go.”
\nThe cleanest method is to partition the table by time — the structure described at the third breaking point of data-intensive systems. In append-heavy data the natural key is time, and partitions make archiving nearly free.
\nInstead of deleting an old partition, you detach it:
\nALTER TABLE orders DETACH PARTITION orders_2023_01;With DETACH, the partition breaks away from the main table but continues to exist in the database as a separate table. Now you can move it however you like — without deleting it, without freezing it.
Detached partitions can go to different places depending on cost and access needs:
\narchive schema; queryable, just out of the hot path.This is the hard part: archived data is often not entirely dead. An audit, a support request, a legal obligation can demand it again. Don’t confuse archiving with “making it inaccessible.”
\nPractical balances: keeping archive tables queryable (just on cheap storage); reaching data in object storage via a foreign data wrapper; or leaving a separate path that is explicitly known as the “archive query,” slow but working. Whichever you choose, leave a door to the cold data — as long as that door is separate from the hot path.
\nSometimes the right archive is deletion. Not every piece of data needs to be kept; a retention policy answers the question “how long does this data live” from the start. Regulations like KVKK/GDPR already make deletion mandatory for some data. Design deletion as a step in the lifecycle too — not as an afterthought.
\nAn archiving strategy starts with accepting on day one that the table will grow. The way to keep hot data fast is to stop cold data from being a burden on it.
\nYour table will grow; the only question is whether you planned for it or production reminded you.
", "content_text": "> **TL;DR:** When hot and cold data share one table, the cold rows slow every query: indexes bloat and autovacuum falls behind. Archiving is not a cleanup job but a lifecycle designed on day one. In a table partitioned by time, an old partition is detached with DETACH, moved to colder storage, and the door to it stays open.\n\nI once saw an `orders` table that had accumulated eight years of data. 95% of the queries looked at the last 90 days — but every query paid the price of the remaining eight years in index size and vacuum time. The hot data was carrying the weight of the cold data.\n\nNo table can grow forever. Archiving is accepting that and planning for it from the start.\n\n## A table can't grow forever\n\nMost tables have a natural hot/cold split. Active data — this week's orders, open tickets — is read and written constantly. Historical data — records closed three years ago — almost never changes and is rarely read. But when the two live in the same table, the cold data slows down every query against the hot data: indexes bloat, autovacuum can't keep up, the working set grows artificially.\n\n## Archiving is a lifecycle\n\nArchiving isn't a cleanup chore you think of once the table has grown large enough to cause panic. It's a lifecycle for the data, and it should be designed on day one: data is born hot, cools over time, then turns cold. The question isn't \"will we archive,\" it's \"where does the cooling data go.\"\n\n## Partition-based archiving\n\nThe cleanest method is to partition the table by time — the structure described at the third breaking point of [data-intensive systems](/engineering/systems/data-intensive-systems-breaking-points/). In append-heavy data the natural key is time, and partitions make archiving nearly free.\n\nInstead of deleting an old partition, you **detach** it:\n\n```sql\nALTER TABLE orders DETACH PARTITION orders_2023_01;\n```\n\nWith `DETACH`, the partition breaks away from the main table but continues to exist in the database as a separate table. Now you can move it however you like — without deleting it, without freezing it.\n\n## Cold storage options\n\nDetached partitions can go to different places depending on cost and access needs:\n\n- **A separate archive schema** — same database but in an `archive` schema; queryable, just out of the hot path.\n- **Cheaper disk** — moving the partition's files to a different tablespace.\n- **Object storage** — moving the partition into an S3-like store as a compressed dump or as Parquet; the cheapest, the slowest to access.\n- **A separate archive database** — fully independent of the live system's capacity.\n\n## Keeping access while you archive\n\nThis is the hard part: archived data is often not entirely dead. An audit, a support request, a legal obligation can demand it again. Don't confuse archiving with \"making it inaccessible.\"\n\nPractical balances: keeping archive tables queryable (just on cheap storage); reaching data in object storage via a foreign data wrapper; or leaving a separate path that is explicitly known as the \"archive query,\" slow but working. Whichever you choose, leave a door to the cold data — as long as that door is separate from the hot path.\n\n## Deleting is an archive decision too\n\nSometimes the right archive is deletion. Not every piece of data needs to be kept; a retention policy answers the question \"how long does this data live\" from the start. Regulations like KVKK/GDPR already make deletion mandatory for some data. Design deletion as a step in the lifecycle too — not as an afterthought.\n\n---\n\nAn archiving strategy starts with accepting on day one that the table will grow. The way to keep hot data fast is to stop cold data from being a burden on it.\n\nYour table will grow; the only question is whether you planned for it or production reminded you.\n\n## Sources\n\n- [PostgreSQL 18: Table Partitioning](https://www.postgresql.org/docs/18/ddl-partitioning.html) — PostgreSQL", "date_published": "2026-09-26T00:00:00.000Z", "tags": [ "postgresql", "database", "performance", "production" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/essays/big-data-syndrome/", "url": "https://muhammetsafak.com/engineering/essays/big-data-syndrome/", "title": "Is Your Data Really Big, or Just Badly Modeled?", "summary": "A team that says our data is big usually rests on a feeling of slowness, not a measurement; a 12 GB table is small for a modern PostgreSQL. Moving to a distributed system before the data is genuinely big is the most expensive wrong turn; the tax is paid up front and nothing gets solved in return. If there is no number, what you have is not big data but unmeasured data.", "content_html": "A sentence came up in a meeting: “Our data has grown so much, we need to move to a distributed database now.” Then we looked at the actual size: 12 GB. For PostgreSQL on a modern server, 12 GB isn’t big — it’s small.
\n“Big data” is said most often about data that isn’t big at all. And that sentence isn’t a diagnosis; more often it’s an excuse.
\nThe sentence “our data is big” almost never rests on a number. It rests on a sense of slowness, a fear, a conference talk. Yet a modern PostgreSQL comfortably carries hundreds of GB on an ordinary server, and low TB if it’s set up carefully.
\nWhether data is big isn’t felt; it’s measured. A “big” verdict reached without measuring is the first step toward the wrong architecture.
\nThe real cause of the slowness mistaken for big data is almost always one of three:
\nAll three are solved by discipline, not by size — and none of the three requires a distributed database.
\nTo answer the “is it big” question, ask concrete questions:
\npg_total_relation_size — not by guessing.pg_stat_database dropped below 99%?EXPLAIN ANALYZE output say?If the answers to these questions reveal a real breaking point, what to do next — index, read replica, partitioning — is in the breaking points of data-intensive systems. This piece is about the measurement that comes before those steps.
\nMoving to a distributed system before the data is genuinely big is the most expensive wrong turn. Because you pay the costs up front — with manual sharding or distributed NoSQL, losing the JOIN across tables and eventual consistency; even with distributed SQL, twice the operational load and distributed debugging — and nothing gets solved in return. A real tax for a problem that doesn’t exist.
The rule from is PostgreSQL enough for everything applies here too: most teams use a small fraction of PostgreSQL’s jsonb, partitioning, BRIN, and indexing capabilities and then conclude it “isn’t enough.”
Let’s be honest: real big data exists. Throughput that sustainably exceeds the write capacity of a single large machine, or a genuinely multi-TB hot working set — these are real limits. A system that gets there has also reached a scale that can finance that complexity; at that point a distributed architecture isn’t a debt, it’s an earned cost.
\nThe point isn’t to be against distributed systems; it’s to avoid moving to them before they’re truly needed — that is, before measuring.
\nWhen you hear “our data is big,” the only thing to ask for is a number. If there’s no number, what you have isn’t big data, it’s unmeasured data.
\nReal size is measured; felt size is, most of the time, a missing index.
", "content_text": "> **TL;DR:** A team that says our data is big usually rests on a feeling of slowness, not a measurement; a 12 GB table is small for a modern PostgreSQL. Moving to a distributed system before the data is genuinely big is the most expensive wrong turn; the tax is paid up front and nothing gets solved in return. If there is no number, what you have is not big data but unmeasured data.\n\nA sentence came up in a meeting: \"Our data has grown so much, we need to move to a distributed database now.\" Then we looked at the actual size: 12 GB. For PostgreSQL on a modern server, 12 GB isn't big — it's small.\n\n\"Big data\" is said most often about data that isn't big at all. And that sentence isn't a diagnosis; more often it's an excuse.\n\n## \"Big\" is a feeling, not a measurement\n\nThe sentence \"our data is big\" almost never rests on a number. It rests on a sense of slowness, a fear, a conference talk. Yet a modern PostgreSQL comfortably carries hundreds of GB on an ordinary server, and low TB if it's set up carefully.\n\nWhether data is big isn't felt; it's measured. A \"big\" verdict reached without measuring is the first step toward the wrong architecture.\n\n## What most \"big data\" problems actually are\n\nThe real cause of the slowness mistaken for big data is almost always one of three:\n\n- **Missing index.** A query doing a sequential scan makes a 5 GB table feel like 500 GB. The problem isn't the size, it's the scanned size.\n- **Bad modeling.** Unnormalized tables, giant JSON blobs embedded in every row, an append-heavy table that was never partitioned. The data isn't \"big,\" it's badly placed.\n- **Wrong access pattern.** N+1 queries, pulling the whole row when one column would do, queries inside a loop. The application uses small data in a big way.\n\nAll three are solved by **discipline**, not by size — and none of the three requires a distributed database.\n\n## Measuring the real size\n\nTo answer the \"is it big\" question, ask concrete questions:\n\n- How much space does the table actually take? Look with `pg_total_relation_size` — not by guessing.\n- Does the frequently accessed data (working set) fit in RAM? Has the cache hit ratio from `pg_stat_database` dropped below 99%?\n- What's the growth rate? 1 GB per month and 100 GB per month are entirely different decisions.\n- Which query is slow, and **why**? What does the `EXPLAIN ANALYZE` output say?\n\nIf the answers to these questions reveal a real breaking point, what to do next — index, read replica, partitioning — is in [the breaking points of data-intensive systems](/engineering/systems/data-intensive-systems-breaking-points/). This piece is about the measurement that comes before those steps.\n\n## The distributed-system tax\n\nMoving to a distributed system before the data is genuinely big is the most expensive wrong turn. Because you pay the costs **up front** — with manual sharding or [distributed NoSQL](/engineering/essays/the-nosql-trap/), losing the `JOIN` across tables and eventual consistency; even with distributed SQL, twice the operational load and distributed debugging — and nothing gets solved in return. A real tax for a problem that doesn't exist.\n\nThe rule from [is PostgreSQL enough for everything](/engineering/essays/is-postgresql-enough-for-everything/) applies here too: most teams use a small fraction of PostgreSQL's `jsonb`, partitioning, BRIN, and indexing capabilities and then conclude it \"isn't enough.\"\n\n## If it really is big\n\nLet's be honest: real big data exists. Throughput that sustainably exceeds the write capacity of a single large machine, or a genuinely multi-TB hot working set — these are real limits. A system that gets there has also reached a scale that can finance that complexity; at that point a distributed architecture isn't a debt, it's an earned cost.\n\nThe point isn't to be against distributed systems; it's to avoid moving to them before they're truly needed — that is, before measuring.\n\n---\n\nWhen you hear \"our data is big,\" the only thing to ask for is a number. If there's no number, what you have isn't big data, it's unmeasured data.\n\nReal size is measured; felt size is, most of the time, a missing index.", "date_published": "2026-09-19T00:00:00.000Z", "tags": [ "database", "scaling", "architecture", "opinion" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/sizing-a-postgres-queue-table-by-its-live-set/", "url": "https://muhammetsafak.com/engineering/notes/sizing-a-postgres-queue-table-by-its-live-set/", "title": "Sizing a PostgreSQL Queue Table by Its Live Set", "summary": "On a queue table the set doing the work is the pending rows, while the table grows forever with finished ones. Anything that scales with the table scales wrong: at 10 million rows a composite index reached 310 MB while a partial index stayed at 7.6 MB, and the default autovacuum threshold never caught the 1.75 million dead tuples that piled up in fifteen minutes.", "content_html": "On a ten-million-row jobs table, 5,000 jobs were waiting in the queue, and the index the claim query used took up 310 MB. A 7.6 MB partial index did the same job, and the speed difference between the two was 6.9 percent.
So the question was never speed. It was what grows along with the table.
\nTwo separate sets live in a queue table:
\npending. The part that workers look for, that claims lock, and that actually does the work. Its size depends on the balance between throughput and the arrival rate; in a healthy system it stays roughly constant.done. Nobody queries them, but as long as they are not deleted the table grows with them forever.PostgreSQL does not know about this split. From MVCC’s point of view a finished row is a live tuple: pg_class.reltuples counts it, the planner statistics count it, the autovacuum threshold counts it. In the table I measured there were 5,000 pending rows and an n_live_tup of 10,005,726.
That is why two wrong scalings happen on a queue table by themselves: the index grows with the table, and so does the vacuum threshold. Both look at the archive, not at the set doing the work.
\nThe numbers below come from pg-queue-bench: PostgreSQL 17, one container, shared_buffers 1 GB, 8 clients, claims with FOR UPDATE SKIP LOCKED. Every pgbench transcript is in the repository.
The live set was held at 5,000 rows at every size; the only thing that changed was how many finished rows surrounded it. Claims per second, median of three repeats:
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n| Strategy | 100k | 1M | 10M | Index size (10M) |
|---|---|---|---|---|
| no index | 2,001 | 248 | 8 | — |
(status) | 6,500 | 6,417 | 6,426 | 66.1 MB |
(status, created_at) | 12,415 | 11,203 | 10,796 | 310.4 MB |
partial (created_at) WHERE status = 'pending' | 13,041 | 11,708 | 11,538 | 7.6 MB |
The no-index row shows why the question exists at all: finding 5,000 rows among 10 million with a sequential scan can be done only 7.6 times a second.
\nThe throughput difference between composite and partial is 6.9 percent. That is not a reason to pick a partial index. The reason is size:
\nAt ten million rows the difference is 41 times. Keeping a 310 MB index in shared_buffers and keeping a 7.6 MB one are not the same decision.
CREATE INDEX CONCURRENTLY jobs_pending_created_at\n ON jobs (created_at)\n WHERE status = 'pending';ORDER BY created_at gets its ordering from the index for free; the claim query must write the predicate exactly as the index does. Building the index with CONCURRENTLY on a live table is how you avoid blocking writes; the rest of zero-downtime schema change is covered in its own piece.
PostgreSQL uses a partial index only if it can prove, at planning time, that the query’s WHERE clause implies the index predicate. A generic plan is built without knowing what $1 is; it cannot prove that status = $1 means status = 'pending', so it has to drop the partial index.
Claims that ran at 11,752 per second under force_custom_plan fell to 7.2 when I forced force_generic_plan, and mean latency went from 0.68 ms to 1,113 ms — roughly 1,635 times. The index’s scan counter stayed at zero. Under the same condition the composite index was untouched (11,417 per second), because it has no predicate to prove; status is a column inside the index.
The cliff is real but fenced. In the default auto mode PostgreSQL runs the first five executions on custom plans, then compares the generic plan’s estimated cost against the average of the custom ones. In a single session, 40 out of 40 executions used a custom plan; the planner never switched to a generic one. I did not record plan costs; the likeliest explanation is that a generic plan, unable to use the partial index, comes out with a high estimated cost. It may not be the only reason, though: the (status) index, which has no predicate, also stayed on custom plans for all 40 executions. The composite index switched to a generic plan at the sixth execution and lost nothing by it.
That protection has a price: a query that reaches the partial index is re-planned on every execution. At this scale the difference is too small to measure (averaged over the last five executions: partial 0.20 ms, composite 0.21 ms). The same cannot be said for a many-table join or a long IN list.
The rule: do not reach a predicated index through a parameter. Write status = 'pending' as a literal, and leave plan_cache_mode alone.
In thirty-second runs the two indexes were nearly equal. Under fifteen minutes of sustained load the picture changed. In this run the claimed row was set to done, and a producer ran alongside at 2,000 jobs per second.
The partial index went from 0.125 MB to 38.2 MB — roughly 305 times. The live set stayed between 3,000 and 5,000 rows for most of the run; what grew was the dead entries in the index.
\nThe mechanism fits in one sentence: the pending → done transition takes the row out of the partial index’s scope, but its entry stays in the index until vacuum arrives. The partial index’s smallness comes from the live set, its bloat rate comes from throughput, and nothing connects the two. A small index makes vacuum cheaper, not rarer.
In fifteen minutes 1,753,949 dead tuples piled up. The autovacuum counter: 0.
\nThe default threshold is computed like this:
\nvacuum threshold = autovacuum_vacuum_threshold + autovacuum_vacuum_scale_factor × reltuples\n = 50 + 0.2 × ~10,000,000\n ≈ 2,000,000 dead tuplesBecause reltuples counts finished rows too, the threshold grows with the archive. The set doing the work stays constant and dead tuples arrive at the rate of throughput, but the number vacuum waits for drifts a little further away with every new done row. The bigger the table, the later vacuum comes.
The composite index paid for it. In the same fifteen minutes it went from 301 MB to 427 MB and fell behind the target:
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n| Strategy | At the start | At 900 s | Pending jobs |
|---|---|---|---|
| partial | 2,024 tps · 1.7 ms | 2,016 tps · 4,906 ms | 5,003 → 14,364 |
| composite | 2,000 tps · 0.52 ms | 1,204 tps · 61,533 ms | 5,000 → 126,024 |
Latency includes pgbench’s scheduling lag; a client that cannot keep to the target rate books its debt there. The partial index slipped too in the last 45 seconds (1,740 tps at 855 s, pending jobs from 4,549 to 14,364), but it finished the run at the target rate. The composite index never recovered after 660 s.
\nThe global default was not designed for this access pattern. Cut the threshold loose from the table’s size and pin it to a fixed number:
\nALTER TABLE jobs SET (\n autovacuum_vacuum_scale_factor = 0,\n autovacuum_vacuum_threshold = 50000\n);scale_factor = 0 makes the threshold independent of reltuples; only the fixed base remains. The right number is set by throughput, not table size: a queue processing 2,000 claims per second produces 50,000 dead tuples in about 25 seconds.
I did not measure this setting — the run above used the defaults, and because autovacuum never ran, the cost of vacuuming was not measured either. 50,000 is a starting point; tune it to your own throughput and to how long vacuum takes on your own table.
\nFirst, look at the ratio between the two sets:
\nSELECT status, count(*)\n FROM jobs\n GROUP BY status;Then at the size of the indexes, and whether they are used at all:
\nSELECT indexrelname,\n pg_size_pretty(pg_relation_size(indexrelid)) AS size,\n idx_scan\n FROM pg_stat_user_indexes\n WHERE relname = 'jobs';Finally, put the dead tuple count next to the global threshold:
\nSELECT s.n_dead_tup,\n s.autovacuum_count,\n s.last_autovacuum,\n current_setting('autovacuum_vacuum_threshold')::int\n + current_setting('autovacuum_vacuum_scale_factor')::float8 * c.reltuples\n AS global_threshold,\n c.reloptions\n FROM pg_stat_user_tables s\n JOIN pg_class c ON c.oid = s.relid\n WHERE s.relname = 'jobs';A reltuples of -1 means the table has not been analyzed yet and the calculation is meaningless. An empty reloptions means the table uses the global threshold. If n_dead_tup keeps growing below global_threshold for hours while autovacuum_count stands still, your vacuum is looking at the archive.
status as a parameter and a generic plan is forced. You cannot use the partial index; a composite index is safer.The numbers come from one machine, PostgreSQL 17 and one access pattern. Thirty-second and fifteen-minute runs do not stand in for months of a real queue; the direction holds, the magnitudes belong to your workload.
\nOn a queue table, table size is noise. Size the index and the vacuum to the set doing the work.
", "content_text": "> **TL;DR:** On a queue table the set doing the work is the pending rows, while the table grows forever with finished ones. Anything that scales with the table scales wrong: at 10 million rows a composite index reached 310 MB while a partial index stayed at 7.6 MB, and the default autovacuum threshold never caught the 1.75 million dead tuples that piled up in fifteen minutes.\n\nOn a ten-million-row `jobs` table, 5,000 jobs were waiting in the queue, and the index the claim query used took up 310 MB. A 7.6 MB partial index did the same job, and the speed difference between the two was 6.9 percent.\n\nSo the question was never speed. It was what grows along with the table.\n\n## A queue table is two tables\n\nTwo separate sets live in a queue table:\n\n- **The live set** — the rows in `pending`. The part that workers look for, that claims lock, and that actually does the work. Its size depends on the balance between throughput and the arrival rate; in a healthy system it stays roughly constant.\n- **The archive** — the rows in `done`. Nobody queries them, but as long as they are not deleted the table grows with them forever.\n\nPostgreSQL does not know about this split. From MVCC's point of view a finished row is a **live tuple**: `pg_class.reltuples` counts it, the planner statistics count it, the autovacuum threshold counts it. In the table I measured there were 5,000 `pending` rows and an `n_live_tup` of 10,005,726.\n\nThat is why two wrong scalings happen on a queue table by themselves: the index grows with the table, and so does the vacuum threshold. Both look at the archive, not at the set doing the work.\n\nThe numbers below come from [pg-queue-bench](https://github.com/muhammetsafak/pg-queue-bench): PostgreSQL 17, one container, `shared_buffers` 1 GB, 8 clients, claims with `FOR UPDATE SKIP LOCKED`. Every pgbench transcript is in the repository.\n\n## Index the queue, not the table\n\nThe live set was held at 5,000 rows at every size; the only thing that changed was how many finished rows surrounded it. Claims per second, median of three repeats:\n\n| Strategy | 100k | 1M | 10M | Index size (10M) |\n| --- | --- | --- | --- | --- |\n| no index | 2,001 | 248 | 8 | — |\n| `(status)` | 6,500 | 6,417 | 6,426 | 66.1 MB |\n| `(status, created_at)` | 12,415 | 11,203 | 10,796 | 310.4 MB |\n| partial `(created_at) WHERE status = 'pending'` | 13,041 | 11,708 | 11,538 | 7.6 MB |\n\nThe no-index row shows why the question exists at all: finding 5,000 rows among 10 million with a sequential scan can be done only 7.6 times a second.\n\nThe throughput difference between composite and partial is 6.9 percent. That is not a reason to pick a partial index. The reason is size:\n\n- The composite index grew with the table: 11.2 → 40.3 → 310.4 MB.\n- The partial index stopped at 6.5 → 7.6 → 7.6 MB. What it indexes is not the table but the **queue**, and the queue is constant, so the index is too.\n\nAt ten million rows the difference is **41 times**. Keeping a 310 MB index in `shared_buffers` and keeping a 7.6 MB one are not the same decision.\n\n```sql\nCREATE INDEX CONCURRENTLY jobs_pending_created_at\n ON jobs (created_at)\n WHERE status = 'pending';\n```\n\n`ORDER BY created_at` gets its ordering from the index for free; the claim query must write the predicate **exactly as the index does**. Building the index with `CONCURRENTLY` on a live table is how you avoid blocking writes; [the rest of zero-downtime schema change](/engineering/notes/zero-downtime-database-migrations/) is covered in its own piece.\n\n### If you ask with a parameter\n\nPostgreSQL uses a partial index only if it can prove, **at planning time**, that the query's `WHERE` clause implies the index predicate. A generic plan is built without knowing what `$1` is; it cannot prove that `status = $1` means `status = 'pending'`, so it has to drop the partial index.\n\nClaims that ran at 11,752 per second under `force_custom_plan` fell to 7.2 when I forced `force_generic_plan`, and mean latency went from 0.68 ms to 1,113 ms — roughly **1,635 times**. The index's scan counter stayed at zero. Under the same condition the composite index was untouched (11,417 per second), because it has no predicate to prove; `status` is a column inside the index.\n\nThe cliff is real but fenced. In the default `auto` mode PostgreSQL runs the first five executions on custom plans, then compares the generic plan's estimated cost against the average of the custom ones. In a single session, **40 out of 40** executions used a custom plan; the planner never switched to a generic one. I did not record plan costs; the likeliest explanation is that a generic plan, unable to use the partial index, comes out with a high estimated cost. It may not be the only reason, though: the `(status)` index, which has no predicate, also stayed on custom plans for all 40 executions. The composite index switched to a generic plan at the sixth execution and lost nothing by it.\n\nThat protection has a price: a query that reaches the partial index is **re-planned on every execution**. At this scale the difference is too small to measure (averaged over the last five executions: partial 0.20 ms, composite 0.21 ms). The same cannot be said for a many-table join or a long `IN` list.\n\nThe rule: **do not reach a predicated index through a parameter.** Write `status = 'pending'` as a literal, and leave `plan_cache_mode` alone.\n\n## A small index does not mean less vacuuming\n\nIn thirty-second runs the two indexes were nearly equal. Under fifteen minutes of sustained load the picture changed. In this run the claimed row was set to `done`, and a producer ran alongside at 2,000 jobs per second.\n\nThe partial index went **from 0.125 MB to 38.2 MB** — roughly 305 times. The live set stayed between 3,000 and 5,000 rows for most of the run; what grew was the dead entries in the index.\n\nThe mechanism fits in one sentence: the `pending → done` transition takes the row out of the partial index's scope, but its entry stays in the index until vacuum arrives. The partial index's **smallness comes from the live set**, its **bloat rate comes from throughput**, and nothing connects the two. A small index makes vacuum cheaper, not rarer.\n\n## The autovacuum threshold looks at the archive\n\nIn fifteen minutes 1,753,949 dead tuples piled up. The autovacuum counter: **0**.\n\nThe default threshold is computed like this:\n\n```\nvacuum threshold = autovacuum_vacuum_threshold + autovacuum_vacuum_scale_factor × reltuples\n = 50 + 0.2 × ~10,000,000\n ≈ 2,000,000 dead tuples\n```\n\nBecause `reltuples` counts finished rows too, the threshold grows with the archive. The set doing the work stays constant and dead tuples arrive at the rate of throughput, but the number vacuum waits for drifts a little further away with every new `done` row. The bigger the table, the later vacuum comes.\n\nThe composite index paid for it. In the same fifteen minutes it went from 301 MB to 427 MB and fell behind the target:\n\n| Strategy | At the start | At 900 s | Pending jobs |\n| --- | --- | --- | --- |\n| partial | 2,024 tps · 1.7 ms | 2,016 tps · 4,906 ms | 5,003 → 14,364 |\n| composite | 2,000 tps · 0.52 ms | 1,204 tps · 61,533 ms | 5,000 → 126,024 |\n\nLatency includes pgbench's scheduling lag; a client that cannot keep to the target rate books its debt there. The partial index slipped too in the last 45 seconds (1,740 tps at 855 s, pending jobs from 4,549 to 14,364), but it finished the run at the target rate. The composite index never recovered after 660 s.\n\n## Cut the threshold loose from the table\n\nThe global default was not designed for this access pattern. Cut the threshold loose from the table's size and pin it to a fixed number:\n\n```sql\nALTER TABLE jobs SET (\n autovacuum_vacuum_scale_factor = 0,\n autovacuum_vacuum_threshold = 50000\n);\n```\n\n`scale_factor = 0` makes the threshold independent of `reltuples`; only the fixed base remains. The right number is set by **throughput**, not table size: a queue processing 2,000 claims per second produces 50,000 dead tuples in about 25 seconds.\n\nI did not measure this setting — the run above used the defaults, and because autovacuum never ran, the **cost** of vacuuming was not measured either. 50,000 is a starting point; tune it to your own throughput and to how long vacuum takes on your own table.\n\n## Measure your own table\n\nFirst, look at the ratio between the two sets:\n\n```sql\nSELECT status, count(*)\n FROM jobs\n GROUP BY status;\n```\n\nThen at the size of the indexes, and whether they are used at all:\n\n```sql\nSELECT indexrelname,\n pg_size_pretty(pg_relation_size(indexrelid)) AS size,\n idx_scan\n FROM pg_stat_user_indexes\n WHERE relname = 'jobs';\n```\n\nFinally, put the dead tuple count next to the global threshold:\n\n```sql\nSELECT s.n_dead_tup,\n s.autovacuum_count,\n s.last_autovacuum,\n current_setting('autovacuum_vacuum_threshold')::int\n + current_setting('autovacuum_vacuum_scale_factor')::float8 * c.reltuples\n AS global_threshold,\n c.reloptions\n FROM pg_stat_user_tables s\n JOIN pg_class c ON c.oid = s.relid\n WHERE s.relname = 'jobs';\n```\n\nA `reltuples` of -1 means the table has not been analyzed yet and the calculation is meaningless. An empty `reloptions` means the table uses the global threshold. If `n_dead_tup` keeps growing below `global_threshold` for hours while `autovacuum_count` stands still, your vacuum is looking at the archive.\n\n## When to drop this pattern\n\n- **You delete finished rows right away, or [move them to an archive table](/engineering/notes/archiving-strategy/).** The table stays close to its live set, so both the composite index's growth and the default threshold's drift stay small.\n- **Your ORM or driver binds `status` as a parameter and a generic plan is forced.** You cannot use the partial index; a composite index is safer.\n- **You partition the table by time.** Dropping old partitions separates both the index and the vacuum load from the table's size; it solves the problem this note solves by another route.\n\nThe numbers come from one machine, PostgreSQL 17 and one access pattern. Thirty-second and fifteen-minute runs do not stand in for months of a real queue; the direction holds, the magnitudes belong to your workload.\n\n---\n\nOn a queue table, table size is noise. Size the index and the vacuum to the set doing the work.\n\n## Sources\n\n- [Routine Vacuuming: the autovacuum daemon and its thresholds](https://www.postgresql.org/docs/17/routine-vacuuming.html) — PostgreSQL\n- [Partial Indexes](https://www.postgresql.org/docs/17/indexes-partial.html) — PostgreSQL\n- [PREPARE: generic and custom plans](https://www.postgresql.org/docs/17/sql-prepare.html) — PostgreSQL\n- [pg-queue-bench: raw pgbench runs behind every number in this note](https://github.com/muhammetsafak/pg-queue-bench) — GitHub\n\n## Frequently asked\n\n### Is a partial index faster on a queue table?\n\nNot meaningfully. On a table with 10 million finished rows the partial index carried 11,538 claims per second and the (status, created_at) composite index 10,796; the difference is 6.9 percent. The real difference is size: the partial index held at 7.6 MB while the composite one reached 310 MB.\n\n### Why does autovacuum reach a queue table so late?\n\nThe default threshold is 50 + 0.2 × reltuples, and reltuples counts finished rows too. On a 10-million-row table that is roughly 2 million dead tuples; in the measurement 1.75 million dead tuples piled up in fifteen minutes and autovacuum did not run once. As the table grows, so does the threshold.", "date_published": "2026-09-16T00:00:00.000Z", "date_modified": "2026-10-07T00:00:00.000Z", "tags": [ "postgresql", "queue", "performance", "indexing", "production" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/essays/orm-vs-native-sql/", "url": "https://muhammetsafak.com/engineering/essays/orm-vs-native-sql/", "title": "ORM or Native SQL?", "summary": "An ORM is the right default for CRUD: it binds parameters and cuts repetition. Its danger is what it hides — N+1 queries, whole rows pulled for one column, a forgotten where. For multi-table reports, aggregations and bulk operations plain SQL is faster and clearer, and what keeps a query safe is parameter binding, not the tool.", "content_html": "A single page of an admin panel was firing 1,400 queries. The cause wasn’t visible on screen: inside a loop, the innocent-looking $order->customer->name call fired a separate query for every order. The ORM hadn’t actually hidden anything — it had only let the developer stop thinking about which query was running.
Is the ORM making us lazy, or protecting us? The answer is both — and the team decides which one dominates.
\nThe ORM has real upsides, and they shouldn’t be dismissed:
\nUser::create($data) is shorter and clearer than a hand-written INSERT.The vast majority of projects are CRUD at their core, and for that work the ORM is the right default. I have no objection to that.
\nThe danger isn’t in what the ORM does, but in what it hides. The ORM keeps SQL out of sight; and when the developer can’t see the SQL, they stop thinking about which query is running.
\nThe result is familiar: the N+1 from the opening of this piece; pulling a whole row into memory when a single column would do; forgetting a where and unknowingly loading an entire table. The ORM doesn’t “do” any of these; they’re all done by the developer who never looks at what the ORM produces.
In the data-intensive systems piece, the first row of the false-breaking-points table is exactly this: what looks like “the database is slow” is, most of the time, the application firing 200 queries.
\nThe ORM shines at transactional CRUD: create a record, update it, read it with its relationships. Where it struggles is just as clear:
\nGROUP BY.UPDATE instead of walking millions of rows one model at a time.For this kind of work, insisting on the ORM produces code that is both slower and less readable. Plain SQL is both faster and clearer here. Picking the right indexes is part of this work too — the index management note covers that side.
\nThe two aren’t rivals. The right use is roughly this:
\nThe mark of seniority isn’t knowing the ORM; it’s knowing where it ends. A team that forces the ORM onto everything turns it into a performance trap; a team that never uses the ORM and writes everything by hand gives up safety and readability for nothing.
\nA common mistake: “plain SQL = unsafe.” What provides safety isn’t the ORM; it’s parameter binding. A plain query written with bound parameters is as closed to injection as the ORM:
\nDB::select('SELECT id, total FROM orders WHERE status = ?', ['pending']);What’s dangerous isn’t plain SQL; it’s embedding user input into the query through string concatenation — and you can do that inside the ORM’s whereRaw too. The habit protects you, not the tool.
The ORM can be a laziness machine or a safety net — the difference is made by how the team looks at it. Use it, but never stop seeing what it produces.
\nA good developer writes with the ORM; they also know when to put it down.
", "content_text": "> **TL;DR:** An ORM is the right default for CRUD: it binds parameters and cuts repetition. Its danger is what it hides — N+1 queries, whole rows pulled for one column, a forgotten where. For multi-table reports, aggregations and bulk operations plain SQL is faster and clearer, and what keeps a query safe is parameter binding, not the tool.\n\nA single page of an admin panel was firing 1,400 queries. The cause wasn't visible on screen: inside a loop, the innocent-looking `$order->customer->name` call fired a separate query for every order. The ORM hadn't actually hidden anything — it had only let the developer stop thinking about which query was running.\n\nIs the ORM making us lazy, or protecting us? The answer is both — and the team decides which one dominates.\n\n## What an ORM buys you\n\nThe ORM has real upsides, and they shouldn't be dismissed:\n\n- **Safety.** Queries are parameterized by default; under ordinary use, the SQL injection surface is closed.\n- **Readability.** For CRUD operations, `User::create($data)` is shorter and clearer than a hand-written `INSERT`.\n- **Less repetition.** Relationships, migrations, model events — all in one place.\n\nThe vast majority of projects are CRUD at their core, and for that work the ORM is the right default. I have no objection to that.\n\n## Where the ORM makes you lazy\n\nThe danger isn't in what the ORM does, but in what it **hides**. The ORM keeps SQL out of sight; and when the developer can't see the SQL, they stop thinking about which query is running.\n\nThe result is familiar: the N+1 from the opening of this piece; pulling a whole row into memory when a single column would do; forgetting a `where` and unknowingly loading an entire table. The ORM doesn't \"do\" any of these; they're all done by the developer who never looks at what the ORM produces.\n\nIn the [data-intensive systems](/engineering/systems/data-intensive-systems-breaking-points/) piece, the first row of the false-breaking-points table is exactly this: what looks like \"the database is slow\" is, most of the time, the application firing 200 queries.\n\n## Where the ORM ends\n\nThe ORM shines at transactional CRUD: create a record, update it, read it with its relationships. Where it struggles is just as clear:\n\n- Multi-table, multi-step reports.\n- Aggregations, window functions, complex conditions on top of `GROUP BY`.\n- Bulk operations — a single `UPDATE` instead of walking millions of rows one model at a time.\n\nFor this kind of work, insisting on the ORM produces code that is both slower and less readable. Plain SQL is both faster and clearer here. Picking the right indexes is part of this work too — the [index management](/engineering/notes/index-management/) note covers that side.\n\n## The call: ORM as default, SQL as a tool\n\nThe two aren't rivals. The right use is roughly this:\n\n- 90% of the work — CRUD, simple lists — with the ORM. Fast, safe, readable.\n- The remaining 10% — heavy reporting, aggregation, bulk operations — with the query builder or plain SQL.\n\nThe mark of seniority isn't knowing the ORM; it's knowing **where it ends**. A team that forces the ORM onto everything turns it into a performance trap; a team that never uses the ORM and writes everything by hand gives up safety and readability for nothing.\n\n## Both can be written safely\n\nA common mistake: \"plain SQL = unsafe.\" What provides safety isn't the ORM; it's **parameter binding**. A plain query written with bound parameters is as closed to injection as the ORM:\n\n```php\nDB::select('SELECT id, total FROM orders WHERE status = ?', ['pending']);\n```\n\nWhat's dangerous isn't plain SQL; it's embedding user input into the query through string concatenation — and you can do that inside the ORM's `whereRaw` too. The habit protects you, not the tool.\n\n---\n\nThe ORM can be a laziness machine or a safety net — the difference is made by how the team looks at it. Use it, but never stop seeing what it produces.\n\nA good developer writes with the ORM; they also know when to put it down.\n\n## Sources\n\n- [Database: Query Builder (Laravel 12.x)](https://laravel.com/docs/12.x/queries) — Laravel", "date_published": "2026-09-12T00:00:00.000Z", "tags": [ "database", "orm", "sql", "opinion" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/index-management/", "url": "https://muhammetsafak.com/engineering/notes/index-management/", "title": "Indexes: Too Few Slow Reads, Too Many Kill Writes", "summary": "An index is not free read speed; it is read speed paid for with writes. Every INSERT writes an entry into every index on the table, and an UPDATE does the same the moment it touches an indexed column. A HOT update and a partial index are the exceptions, not the default. So add an index because a real query plan asked for it, and drop one because pg_stat_user_indexes shows idx_scan = 0.", "content_html": "I’ve seen two opposite teams. One had put no indexes on the table at all — every query a sequential scan, every list page taking seconds. The other had done the exact opposite: “just in case,” it had created an index on every column, and now every INSERT crawled along.
Both are two ends of the same fallacy: thinking an index is a free source of speed.
\nAn index speeds up reads, because instead of scanning the whole table the database looks at an ordered structure. But that ordered structure doesn’t stay current on its own: an INSERT writes an entry into every index on the table, and an UPDATE does the same as soon as it touches an indexed column. There are exceptions — a HOT update that changes no indexed column leaves the indexes alone, a partial index is skipped for rows outside its WHERE clause, and a DELETE leaves its index entries behind for VACUUM to clean up later — but they are exceptions, not the default.
So every index is a trade: you buy read speed with write cost. Inserting a single row into a table with five indexes means updating six structures at once. An index isn’t “free read speed,” it’s “read speed paid for with writes.”
\nFind a missing index from the query plan, not from a hunch:
\nEXPLAIN ANALYZE\nSELECT * FROM orders WHERE status = 'pending';If you see a Seq Scan on a large table in the output and the query returns a small fraction of the rows, an index is probably missing. Scan the pg_stat_user_tables table for large tables with a high seq_scan count — those are your candidates.
Add the index for a real, slow query. An index added “in case we need it later” is the database-layer version of speculative generality.
\nIn the other direction, indexes that are never used make every write more expensive for nothing. Find those by measurement too:
\nSELECT indexrelname, idx_scan\nFROM pg_stat_user_indexes\nWHERE idx_scan = 0;An index with idx_scan = 0 means it’s speeding up no reads at all but adding cost to every write. (Keep the indexes for primary keys and unique constraints, of course.) This is the first cleanup listed at the write breaking point of data-intensive systems.
An index isn’t just “present or absent”; choosing the right type and shape matters:
\n(a, b) index serves equality on a + a range query on b; a lookup on b alone can still use the index, but it doesn’t narrow the portion that gets scanned — before PostgreSQL 18 the whole index is scanned, and from 18 on the skip scan optimization narrows it down. Put the column filtered by equality first, the one filtered by range last.CREATE INDEX ... WHERE status = 'active'. A smaller index, cheaper maintenance.INCLUDE you can add frequently read columns to the index and keep the database from going to the table at all.jsonb and full text; small, cheap BRIN for purely ordered, append-heavy data.An index on (a) is redundant if you already have an (a, b) index — the composite index serves queries starting with a too. Clean up overlaps like these periodically; each one is a silent write tax.
One rule: don’t add an index by guesswork, and don’t drop one by guesswork either. An index is added because a real query plan asks for it; an index is dropped because the statistics show no one is using it. The balance between too few and too many is struck with EXPLAIN ANALYZE and pg_stat_user_indexes, not with gut feeling.
Index management starts with shedding the belief that “more indexes is better.” Every index is a read gain and a write cost; good management is keeping the two in balance by measuring them.
\nA missing index slows the query; too many slow the whole table. Measurement guards against both.
", "content_text": "> **TL;DR:** An index is not free read speed; it is read speed paid for with writes. Every INSERT writes an entry into every index on the table, and an UPDATE does the same the moment it touches an indexed column. A HOT update and a partial index are the exceptions, not the default. So add an index because a real query plan asked for it, and drop one because pg_stat_user_indexes shows idx_scan = 0.\n\nI've seen two opposite teams. One had put no indexes on the table at all — every query a sequential scan, every list page taking seconds. The other had done the exact opposite: \"just in case,\" it had created an index on every column, and now every `INSERT` crawled along.\n\nBoth are two ends of the same fallacy: thinking an index is a free source of speed.\n\n## An index is not free speed\n\nAn index speeds up reads, because instead of scanning the whole table the database looks at an ordered structure. But that ordered structure doesn't stay current on its own: an `INSERT` writes an entry into **every index** on the table, and an `UPDATE` does the same as soon as it touches an indexed column. There are exceptions — a HOT update that changes no indexed column leaves the indexes alone, a partial index is skipped for rows outside its `WHERE` clause, and a `DELETE` leaves its index entries behind for `VACUUM` to clean up later — but they are exceptions, not the default.\n\nSo every index is a trade: you buy read speed with write cost. Inserting a single row into a table with five indexes means updating six structures at once. An index isn't \"free read speed,\" it's \"read speed paid for with writes.\"\n\n## A missing index: measure, don't guess\n\nFind a missing index from the query plan, not from a hunch:\n\n```sql\nEXPLAIN ANALYZE\nSELECT * FROM orders WHERE status = 'pending';\n```\n\nIf you see a `Seq Scan` on a large table in the output and the query returns a small fraction of the rows, an index is probably missing. Scan the `pg_stat_user_tables` table for large tables with a high `seq_scan` count — those are your candidates.\n\nAdd the index for a real, slow query. An index added \"in case we need it later\" is the database-layer version of [speculative generality](/engineering/essays/the-cost-of-just-in-case-code/).\n\n## Too many indexes: find the unused ones\n\nIn the other direction, indexes that are never used make every write more expensive for nothing. Find those by measurement too:\n\n```sql\nSELECT indexrelname, idx_scan\nFROM pg_stat_user_indexes\nWHERE idx_scan = 0;\n```\n\nAn index with `idx_scan = 0` means it's speeding up no reads at all but adding cost to every write. (Keep the indexes for primary keys and unique constraints, of course.) This is the first cleanup listed at the write breaking point of [data-intensive systems](/engineering/systems/data-intensive-systems-breaking-points/).\n\n## How do you choose the right index?\n\nAn index isn't just \"present or absent\"; choosing the right type and shape matters:\n\n- **Column order in a composite index.** An `(a, b)` index serves equality on `a` + a range query on `b`; a lookup on `b` alone can still use the index, but it doesn't narrow the portion that gets scanned — before PostgreSQL 18 the whole index is scanned, and from 18 on the skip scan optimization narrows it down. Put the column filtered by equality first, the one filtered by range last.\n- **Partial index.** If the query always looks at the same subset, limit the index to that subset too: `CREATE INDEX ... WHERE status = 'active'`. A smaller index, cheaper maintenance.\n- **Covering index.** With `INCLUDE` you can add frequently read columns to the index and keep the database from going to the table at all.\n- **Non-B-tree types.** B-tree for equality/ordering; GIN for `jsonb` and full text; small, cheap BRIN for purely ordered, append-heavy data.\n\n## Duplicate and overlapping indexes\n\nAn index on `(a)` is redundant if you already have an `(a, b)` index — the composite index serves queries starting with `a` too. Clean up overlaps like these periodically; each one is a silent write tax.\n\n## Balance is set with the query plan\n\nOne rule: don't add an index by guesswork, and don't drop one by guesswork either. An index is added because a real query plan asks for it; an index is dropped because the statistics show no one is using it. The balance between too few and too many is struck with `EXPLAIN ANALYZE` and `pg_stat_user_indexes`, not with gut feeling.\n\n---\n\nIndex management starts with shedding the belief that \"more indexes is better.\" Every index is a read gain and a write cost; good management is keeping the two in balance by measuring them.\n\nA missing index slows the query; too many slow the whole table. Measurement guards against both.\n\n## Sources\n\n- [CREATE INDEX: CONCURRENTLY, INCLUDE, partial indexes and index methods](https://www.postgresql.org/docs/current/sql-createindex.html) — PostgreSQL\n- [Multicolumn Indexes: column order and the skip scan optimization](https://www.postgresql.org/docs/current/indexes-multicolumn.html) — PostgreSQL\n- [PostgreSQL 18 Release Notes: allow skip scans of btree indexes](https://www.postgresql.org/docs/release/18.0/) — PostgreSQL\n- [Heap-Only Tuples (HOT)](https://www.postgresql.org/docs/current/storage-hot.html) — PostgreSQL\n\n## Frequently asked\n\n### Does every INSERT have to update every index on the table?\n\nFor an INSERT, yes: the new row gets an entry in every index. An UPDATE is different. A HOT update that changes no indexed column leaves the indexes alone, and a partial index is skipped for rows outside its WHERE clause. A DELETE leaves its index entries behind for VACUUM to clean up later.\n\n### How do you find indexes nobody uses?\n\nQuery pg_stat_user_indexes for idx_scan = 0. Those indexes speed up no reads at all while adding cost to every write. Keep the indexes backing primary keys and unique constraints regardless.", "date_published": "2026-09-05T00:00:00.000Z", "date_modified": "2026-09-06T00:00:00.000Z", "tags": [ "postgresql", "database", "performance", "indexing", "PostgreSQL in Production" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/zero-downtime-database-migrations/", "url": "https://muhammetsafak.com/engineering/notes/zero-downtime-database-migrations/", "title": "Changing Schema in Production With Zero Downtime", "summary": "A schema change is not what is dangerous; doing it in one step is. Only operations taking an ACCESS EXCLUSIVE lock queue up reads as well as writes, and on a fifty-million-row table that queue is the outage. Break the change into expand-contract steps: add the constraint NOT VALID and run VALIDATE separately, build indexes CONCURRENTLY, and give the migration session a short lock_timeout.", "content_html": "A deploy ran a single ALTER TABLE on a table with 50 million rows and locked it for four minutes. For four minutes, every request that touched that table waited; the site was effectively down. The migration itself was correct — the problem was doing it in one step.
The dangerous thing is not the schema change. The dangerous thing is making a schema change all at once, without thinking about backward compatibility.
\nNot every schema change costs the same. In modern PostgreSQL, adding a column with a constant DEFAULT is a metadata operation — it’s fast. The real danger is in operations that lock the table for a long time:
CREATE INDEX — without CONCURRENTLY, it closes the table to writes.ALTER COLUMN ... TYPE changes that rewrite the table.NOT NULL, CHECK, or foreign key additions that scan the whole table.These operations take a strong lock; the ones that take it in ACCESS EXCLUSIVE mode — a table rewrite, or adding NOT NULL/CHECK — queue up every query touching the table, reads included. The rest block writes only. If the table is large, the queue grows.
The solution is not to avoid migrations; it’s to break every dangerous migration into small steps, each of which is safe and backward-compatible on its own. This is called the expand-contract pattern.
\nSuppose you want to rename a column. A single-step RENAME COLUMN instantly breaks running code that reads the old column. Instead, three deploys:
Each step works with both the code from the previous release and the new code. At no moment is the running code incompatible with the schema.
\nThe zero-downtime versions of common changes:
\nA new NOT NULL column. Adding NOT NULL in one step scans the table. Split it:
-- 1. Add it as nullable first\nALTER TABLE orders ADD COLUMN status text;\n\n-- 2. Backfill existing rows in batches (on the application side)\n\n-- 3. Add the constraint NOT VALID first, then validate in a separate step\nALTER TABLE orders ADD CONSTRAINT orders_status_not_null\n CHECK (status IS NOT NULL) NOT VALID;\nALTER TABLE orders VALIDATE CONSTRAINT orders_status_not_null;A constraint added with NOT VALID applies immediately to new rows but does not scan existing ones; VALIDATE then scans the table with only a SHARE UPDATE EXCLUSIVE lock — it doesn’t block writes.
An index. Always CONCURRENTLY:
CREATE INDEX CONCURRENTLY idx_orders_status ON orders (status);A foreign key. The same two-step pattern: first ADD CONSTRAINT ... NOT VALID, then VALIDATE CONSTRAINT.
lock_timeout. While a migration waits for a lock behind a long-running query, it blocks everything behind it too. To prevent this, give the migration session a short lock_timeout — if the lock can’t be taken immediately, let the migration fail instead of waiting, and retry it yourself.
\nTan: Expand-contract is paid for in calendar time: a change like the rename example is spread over three deploys. If the contract step is forgotten, the old column becomes permanent. Opening that step as its own ticket on the same day as the expand step keeps the cost from turning into debt.
The essence of expand-contract is one rule: at every intermediate step, both the old code still running and the new code must be able to work with the current schema.
\nThe database schema is what I called a “one-way door” in the cost of just-in-case code — rolling it back is expensive. So design the change not as one big, irreversible step, but as small steps that can each be rolled back individually.
\nZero-downtime migration is not a tool but a discipline: breaking every schema change into steps small enough that the running code never notices.
\nThe dangerous thing is not the change itself, but doing it in one breath.
", "content_text": "> **TL;DR:** A schema change is not what is dangerous; doing it in one step is. Only operations taking an ACCESS EXCLUSIVE lock queue up reads as well as writes, and on a fifty-million-row table that queue is the outage. Break the change into expand-contract steps: add the constraint NOT VALID and run VALIDATE separately, build indexes CONCURRENTLY, and give the migration session a short lock_timeout.\n\nA deploy ran a single `ALTER TABLE` on a table with 50 million rows and locked it for four minutes. For four minutes, every request that touched that table waited; the site was effectively down. The migration itself was correct — the problem was doing it in one step.\n\nThe dangerous thing is not the schema change. The dangerous thing is making a schema change all at once, without thinking about backward compatibility.\n\n## Who takes the lock?\n\nNot every schema change costs the same. In modern PostgreSQL, adding a column with a constant `DEFAULT` is a metadata operation — it's fast. The real danger is in operations that lock the table for a long time:\n\n- `CREATE INDEX` — without `CONCURRENTLY`, it closes the table to writes.\n- `ALTER COLUMN ... TYPE` changes that rewrite the table.\n- `NOT NULL`, `CHECK`, or foreign key additions that scan the whole table.\n\nThese operations take a strong lock; the ones that take it in `ACCESS EXCLUSIVE` mode — a table rewrite, or adding `NOT NULL`/`CHECK` — queue up every query touching the table, reads included. The rest block writes only. If the table is large, the queue grows.\n\n## The dangerous part is the single step\n\nThe solution is not to avoid migrations; it's to break every dangerous migration into small steps, each of which is safe and backward-compatible on its own. This is called the **expand-contract** pattern.\n\nSuppose you want to rename a column. A single-step `RENAME COLUMN` instantly breaks running code that reads the old column. Instead, three deploys:\n\n1. **Expand.** Add the new column. Have the code write to both old and new, still reading from old.\n2. **Migrate.** Move the old data into the new column in batches. Now have the code read from new.\n3. **Contract.** Drop the old column.\n\nEach step works with both the code from the previous release and the new code. At no moment is the running code incompatible with the schema.\n\n## Safe recipes\n\nThe zero-downtime versions of common changes:\n\n**A new `NOT NULL` column.** Adding `NOT NULL` in one step scans the table. Split it:\n\n```sql\n-- 1. Add it as nullable first\nALTER TABLE orders ADD COLUMN status text;\n\n-- 2. Backfill existing rows in batches (on the application side)\n\n-- 3. Add the constraint NOT VALID first, then validate in a separate step\nALTER TABLE orders ADD CONSTRAINT orders_status_not_null\n CHECK (status IS NOT NULL) NOT VALID;\nALTER TABLE orders VALIDATE CONSTRAINT orders_status_not_null;\n```\n\nA constraint added with `NOT VALID` applies immediately to new rows but does not scan existing ones; `VALIDATE` then scans the table with only a `SHARE UPDATE EXCLUSIVE` lock — it doesn't block writes.\n\n**[An index](/engineering/notes/index-management/).** Always `CONCURRENTLY`:\n\n```sql\nCREATE INDEX CONCURRENTLY idx_orders_status ON orders (status);\n```\n\n**A foreign key.** The same two-step pattern: first `ADD CONSTRAINT ... NOT VALID`, then `VALIDATE CONSTRAINT`.\n\n**`lock_timeout`.** While a migration waits for a lock behind a long-running query, it blocks everything behind it too. To prevent this, give the migration session a short `lock_timeout` — if the lock can't be taken immediately, let the migration fail instead of waiting, and retry it yourself.\n\n\n> **Tan:** Expand-contract is paid for in calendar time: a change like the rename example is spread over three deploys. If the contract step is forgotten, the old column becomes permanent. Opening that step as its own ticket on the same day as the expand step keeps the cost from turning into debt.\n\n\n## Code and schema must be compatible together\n\nThe essence of expand-contract is one rule: at every intermediate step, both the **old code still running** and the **new code** must be able to work with the current schema.\n\nThe database schema is what I called a \"one-way door\" in [the cost of just-in-case code](/engineering/essays/the-cost-of-just-in-case-code/) — rolling it back is expensive. So design the change not as one big, irreversible step, but as small steps that can each be rolled back individually.\n\n---\n\nZero-downtime migration is not a tool but a discipline: breaking every schema change into steps small enough that the running code never notices.\n\nThe dangerous thing is not the change itself, but doing it in one breath.\n\n## Steps\n\n**Changing a column without downtime, using expand–contract** — Running migrations without causing downtime in production: backward-compatible steps, multi-phase column changes, and avoiding table locks\n\n1. **Expand** — Add the new column. Have the code write to both old and new, still reading from old.\n2. **Migrate** — Move the old data into the new column in batches; once it is moved, have the code read from new.\n3. **Contract** — Drop the old column. Every step works with both the previous release’s code and the new code, so the running code is never incompatible with the schema.\n\n## Sources\n\n- [ALTER TABLE: NOT VALID constraints, VALIDATE CONSTRAINT and ADD COLUMN with a DEFAULT](https://www.postgresql.org/docs/current/sql-altertable.html) — PostgreSQL\n- [Explicit Locking: table-level lock modes](https://www.postgresql.org/docs/current/explicit-locking.html) — PostgreSQL\n- [CREATE INDEX: building indexes concurrently](https://www.postgresql.org/docs/current/sql-createindex.html) — PostgreSQL\n- [Client Connection Defaults: lock_timeout](https://www.postgresql.org/docs/current/runtime-config-client.html) — PostgreSQL\n\n## Frequently asked\n\n### Which schema changes block reads and not just writes?\n\nThe ones that take an ACCESS EXCLUSIVE lock: a table rewrite, or adding NOT NULL or CHECK. Only that lock mode conflicts with a plain SELECT. A CREATE INDEX without CONCURRENTLY takes a weaker lock that closes the table to writes but lets reads through.\n\n### Why add a constraint NOT VALID first and validate it in a separate step?\n\nA constraint added with NOT VALID applies to new rows immediately without scanning the table, so it commits at once. The separate VALIDATE CONSTRAINT step then scans the table holding only a SHARE UPDATE EXCLUSIVE lock, which does not block writes.", "date_published": "2026-08-29T00:00:00.000Z", "date_modified": "2026-09-30T00:00:00.000Z", "tags": [ "postgresql", "database", "migrations", "production", "PostgreSQL in Production" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/read-write-splitting/", "url": "https://muhammetsafak.com/engineering/notes/read-write-splitting/", "title": "Read/Write Splitting: Separating Read and Write Load", "summary": "A read replica scales reads and adds nothing to write capacity, so the trade is not free: the price is replication lag. Laravel's sticky option repairs read-after-write only inside a single request; across two requests the user still sees the profile they just updated as stale. Do index discipline first, because a replica is often an expensive way to buy what one CREATE INDEX would have solved.", "content_html": "The primary database’s CPU was constantly maxed out. The interesting part: the write rate was low. Almost all of the load was reads — report pages, listing endpoints, search. A single primary was trying to carry a pile of reads that never needed it in the first place.
\nSeparating read and write load — read/write splitting — is the known fix for this picture. But applied at the wrong time, or without awareness of the right traps, it brings more problems than it solves. Separating the read and write model is a different decision; CQRS and a read replica are not the same thing.
\nHow read-heavy the traffic is depends on the workload: in measurements of the standard OLTP benchmarks, TPC-E runs 90.69% reads while TPC-C stays at 65.71% — so measure your own ratio instead of assuming it. Every order is written once but read dozens of times: in the list, in the detail view, in a report, on a dashboard. This asymmetry is what makes read/write splitting appealing — because the side you need to scale is obvious.
\nStop before adding a replica. A full primary doesn’t always mean “out of capacity.” Often a single missing index makes the primary look many times busier than it is.
\nAdding a replica — a new server, replication setup, lag monitoring — can amount to expensively buying your way out of a problem a single CREATE INDEX would have solved. The order in the breaking points of data-intensive systems is clear: index discipline first, then replicas. Don’t skip that order.
Measure your queries with EXPLAIN ANALYZE. If the primary is genuinely saturating under correctly indexed queries — that’s when you reach for a replica.
Let’s be clear: a read replica adds nothing to your write capacity. The same writes are replayed on every replica. A replica solves a read-load problem; if you have a write-load problem, a replica is the wrong tool.
\nLaravel supports read/write connection splitting natively:
\n// config/database.php\n'pgsql' => [\n 'driver' => 'pgsql',\n 'read' => ['host' => ['10.0.0.2']], // replica\n 'write' => ['host' => ['10.0.0.1']], // primary\n 'sticky' => true,\n // ...shared settings\n],SELECTs go to the replica, INSERT/UPDATE/DELETEs go to the primary. The replica gets its own connection pool — separate from the primary’s; if you use pgBouncer, they are two distinct pools.
The replica trails the primary by a few milliseconds — a few seconds under load. This delay is the real cost of read/write splitting, and its name is the read-after-write problem.
\nsticky => true partly addresses it: if you wrote within a request, the subsequent reads in that same request go to the primary. But sticky only works within the boundary of a single request.
Outside that boundary it’s still open: a user updates their profile (request 1, written to the primary), moves to the next page (request 2, read from the replica), and sees their old profile, not yet replayed on the replica. The user sees their own data as stale. This looks like a bug, but it’s actually a tradeoff the architecture accepts — one that has to be accepted deliberately.
\nSetting up read/write splitting requires every read to answer one question: can this query read stale data?
\nThis classification is now part of the architecture, and it needs to be documented. A new developer must have somewhere to look for “where should this query read from” — otherwise the classification quietly rots.
\nRead/write splitting is the right move when these three conditions hold together:
\nIf any one of these is missing, deferring the replica is cheaper.
\nRead/write splitting is a cheaper scaling move than growing vertically — but it isn’t free. The price is replication lag, and the cost of ignoring it is paid by showing users their own data as stale.
\nBefore splitting reads, know which reads can tolerate staleness.
", "content_text": "> **TL;DR:** A read replica scales reads and adds nothing to write capacity, so the trade is not free: the price is replication lag. Laravel's sticky option repairs read-after-write only inside a single request; across two requests the user still sees the profile they just updated as stale. Do index discipline first, because a replica is often an expensive way to buy what one CREATE INDEX would have solved.\n\nThe primary database's CPU was constantly maxed out. The interesting part: the write rate was low. Almost all of the load was reads — report pages, listing endpoints, search. A single primary was trying to carry a pile of reads that never needed it in the first place.\n\nSeparating read and write load — read/write splitting — is the known fix for this picture. But applied at the wrong time, or without awareness of the right traps, it brings more problems than it solves. Separating the read and write *model* is a different decision; [CQRS](/engineering/notes/when-is-cqrs-necessary/) and a read replica are not the same thing.\n\n## Most load is read-heavy\n\nHow read-heavy the traffic is depends on the workload: in measurements of the standard OLTP benchmarks, TPC-E runs 90.69% reads while TPC-C stays at 65.71% — so measure your own ratio instead of assuming it. Every order is written once but read dozens of times: in the list, in the detail view, in a report, on a dashboard. This asymmetry is what makes read/write splitting appealing — because the side you need to scale is obvious.\n\n## First: is this really a capacity problem?\n\nStop before adding a replica. A full primary doesn't always mean \"out of capacity.\" Often a single missing index makes the primary look many times busier than it is.\n\nAdding a replica — a new server, replication setup, lag monitoring — can amount to expensively buying your way out of a problem a single `CREATE INDEX` would have solved. The order in [the breaking points of data-intensive systems](/engineering/systems/data-intensive-systems-breaking-points/) is clear: index discipline first, then replicas. Don't skip that order.\n\nMeasure your queries with `EXPLAIN ANALYZE`. If the primary is genuinely saturating under correctly indexed queries — that's when you reach for a replica.\n\n## A replica scales reads, not writes\n\nLet's be clear: a read replica adds **nothing** to your write capacity. The same writes are replayed on every replica. A replica solves a read-load problem; if you have a write-load problem, a replica is the wrong tool.\n\n## Setting it up in Laravel\n\nLaravel supports read/write connection splitting natively:\n\n```php\n// config/database.php\n'pgsql' => [\n 'driver' => 'pgsql',\n 'read' => ['host' => ['10.0.0.2']], // replica\n 'write' => ['host' => ['10.0.0.1']], // primary\n 'sticky' => true,\n // ...shared settings\n],\n```\n\n`SELECT`s go to the replica, `INSERT/UPDATE/DELETE`s go to the primary. The replica gets its own connection pool — separate from the primary's; if you use [pgBouncer](/engineering/notes/pgbouncer-auth-query/), they are two distinct pools.\n\n## Replication lag: the real bill\n\nThe replica trails the primary by a few milliseconds — a few seconds under load. This delay is the real cost of read/write splitting, and its name is the **read-after-write** problem.\n\n`sticky => true` partly addresses it: if you wrote within a request, the subsequent reads in that same request go to the primary. But `sticky` only works within the boundary of **a single request**.\n\nOutside that boundary it's still open: a user updates their profile (request 1, written to the primary), moves to the next page (request 2, read from the replica), and sees their old profile, not yet replayed on the replica. The user sees their own data as stale. This looks like a bug, but it's actually a tradeoff the architecture accepts — one that has to be accepted deliberately.\n\n## The query classification discipline\n\nSetting up read/write splitting requires every read to answer one question: **can this query read stale data?**\n\n- **Can read from a replica:** lists, reports, search results, dashboards. A few seconds of delay is irrelevant.\n- **Must read from the primary:** account balance, stock count, authorization checks, any read a write decision depends on.\n\nThis classification is now part of the architecture, and it needs to be documented. A new developer must have somewhere to look for \"where should this query read from\" — otherwise the classification quietly rots.\n\n## When do you actually need it?\n\nRead/write splitting is the right move when these three conditions hold together:\n\n1. Index discipline is complete; queries are correctly indexed and the primary is still saturating.\n2. The load is measurably read-heavy.\n3. There's the discipline to do and document the stale-read classification.\n\nIf any one of these is missing, deferring the replica is cheaper.\n\n---\n\nRead/write splitting is a cheaper scaling move than growing vertically — but it isn't free. The price is replication lag, and the cost of ignoring it is paid by showing users their own data as stale.\n\nBefore splitting reads, know which reads can tolerate staleness.\n\n## Sources\n\n- [Log-Shipping Standby Servers: streaming replication and hot standby](https://www.postgresql.org/docs/current/warm-standby.html) — PostgreSQL\n- [TPC-E vs. TPC-C: Characterizing the New TPC-E Benchmark via an I/O Comparison Study](https://www.cs.cmu.edu/~chensm/papers/TPCE-sigmod-record10.pdf) — ACM SIGMOD Record 39(3), 2010\n- [Database: Read and Write Connections](https://laravel.com/docs/12.x/database#read-and-write-connections) — Laravel\n\n## Frequently asked\n\n### Does adding a read replica increase write capacity?\n\nNo. The same writes are replayed on every replica, so a replica adds nothing to write throughput. It solves a read-load problem; if the problem is on the write side, a replica is the wrong tool.\n\n### What does the sticky option solve, and what does it not?\n\nWith sticky enabled, reads that follow a write within the same request go to the primary, so a user never reads their own write as stale inside one request. It does nothing across requests: the next page can still read from a replica that has not replayed the write yet.", "date_published": "2026-08-22T00:00:00.000Z", "date_modified": "2026-10-07T00:00:00.000Z", "tags": [ "postgresql", "database", "scaling", "performance", "PostgreSQL in Production" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/the-right-cache-strategy/", "url": "https://muhammetsafak.com/engineering/notes/the-right-cache-strategy/", "title": "Before You Reach for Redis: The Right Cache Strategy", "summary": "A cache is never a faster database; it is speed bought with accepted staleness, so the first question is not technical: how stale may this data be? TTL is the boring default, because it does not require knowing every write path and it does not break silently when one is forgotten. Explicit invalidation is exact only if you catch every path. And a cache reduces read load, never write load.", "content_html": "A team added a Redis cache because “the site is slow.” A week later the support queue filled up: “I updated the price but the old one still shows.” The cache had fixed the slowness — and quietly created a new problem.
\nAdding a cache is not a free performance win. It trades freshness for speed; and if you didn’t make that trade deliberately, the bill comes back as a bug.
\nThe moment you cache a value, you’ve accepted this: someone, for a while, may see stale data. A cache is never a “faster database”; it’s “speed in exchange for accepted staleness.”
\nSo the first question isn’t technical: how stale can this data be? Data whose answer is “not at all” — an account balance, a stock count — should be thought through twice before it’s cached.
\nBefore you turn on Redis:
\nCREATE INDEX would solve is moving the problem, not solving it — this is exactly the wrong breaking point in data-intensive systems.Cache invalidation gets cited as one of the two hard problems in software — and that’s no exaggeration. Putting the cache in is easy; clearing it at the right moment is hard.
\nBecause a cached value must be cleared on every write path that affects it. A product price changes not only from the “edit product” screen; it also changes from a bulk price update, a discount job, an admin script. Finding all of those paths and clearing the cache — that’s the hard part. Miss one, and you get the support ticket from the top of this post.
\nIn practice there are two roads:
\nMy default is TTL. It’s boring, predictable, and it doesn’t break silently because of a forgotten write path. I move to explicit invalidation only when staleness is genuinely unacceptable and the number of write paths is limited and known. Most of the time the two are used together: a short TTL as a safety net, explicit invalidation for speed.
\nA common mistake: trying to rescue a system under write load by adding a cache. A cache reduces read load; it has no effect on write load — if anything, invalidation itself is extra write work. If your problem is on the write side, a cache is the wrong tool.
\n\nTan: The staleness window should not be an engineer’s solo pick. How long a customer may see the old price after an update is a question for the product side to answer. Get that answer in writing: a sentence like “an update shows up within this long” is the contract you can point to before the support queue fills up.
A cache is not the cure for a slow system; it’s the relief of a measured read bottleneck, bought with a deliberate consistency concession. A cache added without an invalidation plan takes back more as debt than it speeds up.
\nBefore you add a cache, ask: how old can this data be — and who’s going to clear it?
", "content_text": "> **TL;DR:** A cache is never a faster database; it is speed bought with accepted staleness, so the first question is not technical: how stale may this data be? TTL is the boring default, because it does not require knowing every write path and it does not break silently when one is forgotten. Explicit invalidation is exact only if you catch every path. And a cache reduces read load, never write load.\n\nA team added a Redis cache because \"the site is slow.\" A week later the support queue filled up: \"I updated the price but the old one still shows.\" The cache had fixed the slowness — and quietly created a new problem.\n\nAdding a cache is not a free performance win. It trades freshness for speed; and if you didn't make that trade deliberately, the bill comes back as a bug.\n\n## A cache is a consistency concession\n\nThe moment you cache a value, you've accepted this: someone, for a while, may see stale data. A cache is never a \"faster database\"; it's \"speed in exchange for accepted staleness.\"\n\nSo the first question isn't technical: **how stale can this data be?** Data whose answer is \"not at all\" — an account balance, a stock count — should be thought through twice before it's cached.\n\n## Questions to ask first\n\nBefore you turn on Redis:\n\n- **Is this really a read bottleneck?** \"The site is slow\" is a hypothesis, not a measurement. Adding a cache without seeing the source of the slowness is covering up something you don't understand.\n- **Is the slowness actually a missing index?** A cache papers over a bad query but doesn't fix it. Hiding behind a cache what a `CREATE INDEX` would solve is moving the problem, not solving it — this is exactly the wrong breaking point in [data-intensive systems](/engineering/systems/data-intensive-systems-breaking-points/).\n- **How stale can this data stay?** The answer determines which strategy you pick.\n\n## Why is invalidation hard?\n\nCache invalidation gets cited as one of the two hard problems in software — and that's no exaggeration. Putting the cache in is easy; **clearing** it at the right moment is hard.\n\nBecause a cached value must be cleared on every write path that affects it. A product price changes not only from the \"edit product\" screen; it also changes from a bulk price update, a discount job, an admin script. Finding **all** of those paths and clearing the cache — that's the hard part. Miss one, and you get the support ticket from the top of this post.\n\n## Two strategies\n\nIn practice there are two roads:\n\n- **TTL-based.** You assign the value a lifetime — 60 seconds, 5 minutes. When it expires, the cache refreshes itself. Simple, sturdy, and it doesn't require knowing every write path. In return: you accept staleness up to the TTL.\n- **Explicit invalidation.** You delete the cache by hand when the data changes. Exact and fresh — but only if you catch every write path completely.\n\nMy default is TTL. It's boring, predictable, and it doesn't break silently because of a forgotten write path. I move to explicit invalidation only when staleness is genuinely unacceptable and the number of write paths is limited and known. Most of the time the two are used together: a short TTL as a safety net, explicit invalidation for speed.\n\n## A cache doesn't reduce writes\n\nA common mistake: trying to rescue a system under write load by adding a cache. A cache reduces **read** load; it has no effect on write load — if anything, invalidation itself is extra write work. If your problem is on the write side, a cache is the wrong tool.\n\n\n> **Tan:** The staleness window should not be an engineer’s solo pick. How long a customer may see the old price after an update is a question for the product side to answer. Get that answer in writing: a sentence like “an update shows up within this long” is the contract you can point to before the support queue fills up.\n\n\n## Practical patterns\n\n- **Cache-aside.** Read: check the cache first, and on a miss fetch the value from the database and write it to the cache. The most common and most understandable pattern.\n- **Stampede protection.** The instant a popular key's TTL expires, hundreds of requests hit the database at once. On a cache miss, use a short lock so only one request goes to the database.\n- **Key discipline.** Namespace your cache keys — the pattern in the [shared Redis namespace isolation](/engineering/notes/shared-redis-namespace-isolation/) note applies to cache keys too.\n\n---\n\nA cache is not the cure for a slow system; it's the relief of a measured read bottleneck, bought with a deliberate consistency concession. A cache added without an invalidation plan takes back more as debt than it speeds up.\n\nBefore you add a cache, ask: how old can this data be — and who's going to clear it?\n\n## Sources\n\n- [RFC 9111: HTTP Caching — freshness lifetime, stale responses and invalidation](https://www.rfc-editor.org/rfc/rfc9111.html) — IETF\n- [Distributed Locks with Redis](https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/) — Redis\n\n## Frequently asked\n\n### Should you use a TTL or explicit invalidation?\n\nTTL is the safer default: it needs no knowledge of every write path and it does not break silently when one is forgotten, at the price of accepting staleness up to the TTL. Move to explicit invalidation only when staleness is genuinely unacceptable and the write paths are limited and known. Most of the time both run together.\n\n### Does a cache reduce write load?\n\nNo. A cache reduces read load and has no effect on write load; invalidation itself is extra write work. If the problem is on the write side, a cache is the wrong tool.", "date_published": "2026-08-15T00:00:00.000Z", "date_modified": "2026-09-30T00:00:00.000Z", "tags": [ "caching", "redis", "performance", "production" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/essays/the-nosql-trap/", "url": "https://muhammetsafak.com/engineering/essays/the-nosql-trap/", "title": "The NoSQL Trap: Starting Because \"It Has No Schema\"", "summary": "There is no such thing as schemaless data, only the question of where the shape is defined and who enforces it. Choosing a document database for the comfort of not wanting a schema does not delete the schema; it moves it into every read, every write and the team's heads, where nothing enforces it. NoSQL earns its place on horizontal write scale or genuinely heterogeneous documents, not on schema avoidance.", "content_html": "I once saw a project that had picked MongoDB on the logic of “no schema, we’ll move fast” — two years later. Every read was defensive: data?.user?.address?.city ?? null. The reason was simple — no two documents had exactly the same shape. The schema hadn’t disappeared; it had just left the database and scattered across every bit of code that reads a document.
There is no such thing as schemaless data. Data always has a shape: a user’s email, the line items of an order, the date on an invoice. The question is not whether there is a shape; it is where that shape is defined.
\nThere are two options. Either the schema lives in the database and is enforced by the database, or it lives in the application code and everyone is assumed to comply with it. Saying “schemaless” means choosing the second — not deleting the schema, but moving it somewhere it isn’t enforced.
\nWhen the database stops enforcing the schema, that work doesn’t vanish; it scatters to these places:
\nNOT NULL, type check, and foreign key that the database does in a single line all turn into hand-written checks.A schema is not bureaucracy; it is validation the database does for free on your behalf. Refusing it doesn’t erase the bill; it just charges it to someone else.
\nProjects that start with the comfort of “schemaless” sooner or later hit these:
\nALTER TABLE does in the relational world becomes, here, a script that walks millions of documents one by one and has to be batched.These are invisible in a small project; they blow up at exactly the moment the project grows, the team changes, and no one can answer “why is that field sometimes missing?”
\nThis is not an anti-NoSQL piece. NoSQL has real and justified uses:
\nThe trap isn’t NoSQL; it’s choosing NoSQL not for one of these real strengths but just because “I don’t want a schema.” Right tool, wrong reason.
\nIf the need for “let some fields be flexible” is real, you don’t have to sacrifice all your integrity for it. PostgreSQL’s jsonb column gives you a flexible document field within the integrity of a relational table — as I touched on in is PostgreSQL enough for everything. The fields that are fixed stay as schema, and the part that is genuinely variable lives in jsonb. The best of both, and in a single system.
A schema is not a burden; it is a shield. When you remove it from the database it doesn’t disappear — it just leaves the place where it protected you and moves to a place where it can’t.
\nThere is no such thing as “schemaless”; there is only the question of “who enforces the schema.”
", "content_text": "> **TL;DR:** There is no such thing as schemaless data, only the question of where the shape is defined and who enforces it. Choosing a document database for the comfort of not wanting a schema does not delete the schema; it moves it into every read, every write and the team's heads, where nothing enforces it. NoSQL earns its place on horizontal write scale or genuinely heterogeneous documents, not on schema avoidance.\n\nI once saw a project that had picked MongoDB on the logic of \"no schema, we'll move fast\" — two years later. Every read was defensive: `data?.user?.address?.city ?? null`. The reason was simple — no two documents had exactly the same shape. The schema hadn't disappeared; it had just left the database and scattered across every bit of code that reads a document.\n\n## \"Schemaless\" is a misnomer\n\nThere is no such thing as schemaless data. Data always has a shape: a user's email, the line items of an order, the date on an invoice. The question is not whether there is a shape; it is **where that shape is defined**.\n\nThere are two options. Either the schema lives in the database and is enforced by the database, or it lives in the application code and everyone is assumed to comply with it. Saying \"schemaless\" means choosing the second — not deleting the schema, but moving it somewhere it isn't enforced.\n\n## Where does the schema move?\n\nWhen the database stops enforcing the schema, that work doesn't vanish; it scatters to these places:\n\n- **To every read.** The code can no longer assume the incoming document has the expected fields; it checks every field defensively.\n- **To every write.** There is no guarantee you wrote the right shape; there is only hope.\n- **To scattered validation code.** The `NOT NULL`, type check, and foreign key that the database does in a single line all turn into hand-written checks.\n- **To the team's heads.** \"That field is sometimes a string, sometimes an array\" becomes undocumented tribal knowledge.\n\nA schema is not bureaucracy; it is validation the database does for free on your behalf. Refusing it doesn't erase the bill; it just charges it to someone else.\n\n## The crises you hit\n\nProjects that start with the comfort of \"schemaless\" sooner or later hit these:\n\n- **Data drift.** Over time, ten different versions of a document live in the same collection. A record written in 2023 doesn't have the same shape as one written in 2026, and no migration ever enforced it.\n- **No referential integrity.** An order pointing to a user that doesn't exist isn't blocked by the database. Orphan records pile up silently.\n- **Migration becomes an application job.** What a single `ALTER TABLE` does in the relational world becomes, here, a script that walks millions of documents one by one and has to be batched.\n\nThese are invisible in a small project; they blow up at exactly the moment the project grows, the team changes, and no one can answer \"why is that field sometimes missing?\"\n\n## Where NoSQL really is the right answer\n\nThis is not an anti-NoSQL piece. NoSQL has real and justified uses:\n\n- Workloads that exceed the write capacity of a single primary and genuinely need horizontal scale — the hardest breaking point in [data-intensive systems](/engineering/systems/data-intensive-systems-breaking-points/).\n- Documents that are genuinely heterogeneous by nature.\n- Specific access patterns — pure key-value, wide-column, graph.\n\nThe trap isn't NoSQL; it's choosing NoSQL not for one of these real strengths but just because \"I don't want a schema.\" Right tool, wrong reason.\n\n## If you want flexibility, PostgreSQL already gives it\n\nIf the need for \"let some fields be flexible\" is real, you don't have to sacrifice all your integrity for it. PostgreSQL's `jsonb` column gives you a flexible document field within the integrity of a relational table — as I touched on in [is PostgreSQL enough for everything](/engineering/essays/is-postgresql-enough-for-everything/). The fields that are fixed stay as schema, and the part that is genuinely variable lives in `jsonb`. The best of both, and in a single system.\n\n---\n\nA schema is not a burden; it is a shield. When you remove it from the database it doesn't disappear — it just leaves the place where it protected you and moves to a place where it can't.\n\nThere is no such thing as \"schemaless\"; there is only the question of \"who enforces the schema.\"\n\n## Sources\n\n- [Designing Data-Intensive Applications, Chapter 2: schema-on-read vs. schema-on-write](https://dataintensive.net/) — Martin Kleppmann, O'Reilly Media, 2017\n- [MongoDB Manual: Schema Validation](https://www.mongodb.com/docs/manual/core/schema-validation/) — MongoDB\n- [JSON Types: jsonb](https://www.postgresql.org/docs/current/datatype-json.html) — PostgreSQL\n\n## Frequently asked\n\n### Is any data really schemaless?\n\nNo. Data always has a shape; the only question is where that shape is defined. Either the database holds the schema and enforces it, or the application code holds it and everyone is assumed to comply. Schemaless means choosing the second.\n\n### When is NoSQL genuinely the right choice?\n\nWhen the workload exceeds the write capacity of a single primary and needs real horizontal scale, when documents are heterogeneous by nature, or when the access pattern is specific: pure key-value, wide-column, graph. Not wanting a schema is not one of those reasons.", "date_published": "2026-08-08T00:00:00.000Z", "date_modified": "2026-09-06T00:00:00.000Z", "tags": [ "nosql", "database", "architecture", "opinion" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/essays/is-postgresql-enough-for-everything/", "url": "https://muhammetsafak.com/engineering/essays/is-postgresql-enough-for-everything/", "title": "Is PostgreSQL Enough for Everything?", "summary": "Most of the list that puts Elasticsearch, MongoDB and RabbitMQ next to PostgreSQL is carried by a single PostgreSQL for a long time: tsvector with a GIN index for search, jsonb for documents, SELECT FOR UPDATE SKIP LOCKED for a moderate-volume queue. The limits are honest and real, but a second data system should arrive on a measured limit, not on a hunch or a blog post.", "content_html": "An architecture plan, drawn up before there was a single user, carried this list: PostgreSQL for durable data, Elasticsearch for search, Redis for cache, MongoDB for flexible documents, RabbitMQ for queueing. Five separate data systems — each with its own backups, monitoring, version upgrades, and distinct failure mode.
\nMost of that list, for a long time, a single PostgreSQL carries on its own. The question isn’t “can PostgreSQL do this”; it’s “did you really buy the operational burden of these five systems”.
\nTreating PostgreSQL as just a table-row store uses a small fraction of what it can do. Most of the needs that prompt a separate system are already inside it:
\ntsvector, tsquery, and a GIN index. It covers most apps’ “search products” and “search posts” needs, including stemming, weighting, and ranking.jsonb column, paired with a GIN index, is a queryable document store. You can keep the fields that want schema flexibility right inside the relational table.SELECT ... FOR UPDATE SKIP LOCKED and LISTEN/NOTIFY run a moderate-volume job queue without a separate broker. Laravel’s database queue driver uses the SKIP LOCKED half of it — its workers poll the jobs table at intervals rather than waiting on a LISTEN/NOTIFY signal.These features aren’t “present but unused”; they’re mature capabilities that run reliably in production.
\nEvery new data system isn’t just a box; it’s a maintenance commitment. What you gain by staying on one system:
\nThis is the data-layer version of the innovation-token logic from the boring architecture piece: every new system is a token, and tokens are limited.
\nPostgreSQL isn’t enough for everything — it has honest limits, and ignoring them is also a mistake:
\nWhen you hit one of these limits, bringing in an extra system is the right call. Bringing it in before you hit it is just a guess.
\nThe rule is plain: in a new project, make PostgreSQL do everything PostgreSQL can do. Let it be the first stop for search, queue, and cache-like needs. A separate system should arrive only when a measured limit is crossed — not from a hunch, a blog post, or résumé anxiety.
\nMost teams use maybe 20% of PostgreSQL, then say “it’s not enough”. What’s not enough is usually not PostgreSQL, but the effort to get to know it.
\nPostgreSQL isn’t enough for everything — but it’s enough for far more than you think. Before you bring in a second data system, ask whether you’ve truly reached the end of the first.
\nPostgreSQL first; let the rest follow from measurement.
", "content_text": "> **TL;DR:** Most of the list that puts Elasticsearch, MongoDB and RabbitMQ next to PostgreSQL is carried by a single PostgreSQL for a long time: tsvector with a GIN index for search, jsonb for documents, SELECT FOR UPDATE SKIP LOCKED for a moderate-volume queue. The limits are honest and real, but a second data system should arrive on a measured limit, not on a hunch or a blog post.\n\nAn architecture plan, drawn up before there was a single user, carried this list: PostgreSQL for durable data, Elasticsearch for search, Redis for cache, MongoDB for flexible documents, RabbitMQ for queueing. Five separate data systems — each with its own backups, monitoring, version upgrades, and distinct failure mode.\n\nMost of that list, for a long time, a single PostgreSQL carries on its own. The question isn't \"can PostgreSQL do this\"; it's \"did you really buy the operational burden of these five systems\".\n\n## PostgreSQL's little-known breadth\n\nTreating PostgreSQL as just a table-row store uses a small fraction of what it can do. Most of the needs that prompt a separate system are already inside it:\n\n- **Full-text search.** A real search built on `tsvector`, `tsquery`, and a GIN index. It covers most apps' \"search products\" and \"search posts\" needs, including stemming, weighting, and ranking.\n- **Document storage.** A `jsonb` column, paired with a GIN index, is a queryable document store. You can keep the fields that want schema flexibility right inside the relational table.\n- **Queues.** `SELECT ... FOR UPDATE SKIP LOCKED` and `LISTEN/NOTIFY` run a moderate-volume job queue without a separate broker. Laravel's `database` queue driver uses the `SKIP LOCKED` half of it — its workers poll [the jobs table](/engineering/notes/sizing-a-postgres-queue-table-by-its-live-set/) at intervals rather than waiting on a `LISTEN/NOTIFY` signal.\n- **Analytical queries.** Window functions, CTEs, materialized views — serious reporting without reaching for a separate analytical database.\n- **Geospatial data.** Location queries via the PostGIS extension.\n\nThese features aren't \"present but unused\"; they're mature capabilities that run reliably in production.\n\n## The quiet payoff of a single system\n\nEvery new data system isn't just a box; it's a maintenance commitment. What you gain by staying on one system:\n\n- **One backup line.** One backup/restore procedure, one recovery drill.\n- **One monitoring target.** One set of metrics to learn, one alerting setup to build.\n- **Cross-consistency.** This is the most important one. If your search index lives in the same database as your data, there **can be no drift** between them — both update in the same transaction. A separate Elasticsearch will eventually fall out of sync with the data, and fixing that becomes its own line item.\n- **One mental model.** The team learns the quirks of one system, not five.\n\nThis is the data-layer version of the innovation-token logic from the [boring architecture](/engineering/essays/why-boring-architecture/) piece: every new system is a token, and tokens are limited.\n\n## Where does the limit begin?\n\nPostgreSQL isn't enough for everything — it has honest limits, and ignoring them is also a mistake:\n\n- **Search.** When typo tolerance, advanced relevance tuning, faceted search, and multilingual analysis are needed at serious scale, a dedicated search engine genuinely pays off.\n- **Queues.** When you need very high throughput, complex routing, or multi-consumer fan-out, a real message broker is the right tool. A PostgreSQL queue is comfortable at moderate volume, not at the extremes.\n- **Write throughput.** Sustained writes beyond a single primary's fsync capacity — this is the hardest breaking point in [data-intensive systems](/engineering/systems/data-intensive-systems-breaking-points/), and a real limit.\n\nWhen you hit one of these limits, bringing in an extra system is the right call. Bringing it in before you hit it is just a guess.\n\n## The decision: PostgreSQL first, then measure\n\nThe rule is plain: in a new project, make PostgreSQL do everything PostgreSQL can do. Let it be the first stop for search, queue, and cache-like needs. A separate system should arrive only when a **measured** limit is crossed — not from a hunch, a blog post, or résumé anxiety.\n\nMost teams use maybe 20% of PostgreSQL, then say \"it's not enough\". What's not enough is usually not PostgreSQL, but the effort to get to know it.\n\n---\n\nPostgreSQL isn't enough for everything — but it's enough for far more than you think. Before you bring in a second data system, ask whether you've truly reached the end of the first.\n\nPostgreSQL first; let the rest follow from measurement.\n\n## Sources\n\n- [Full Text Search: tsvector, tsquery, stemming and ranking](https://www.postgresql.org/docs/current/textsearch-intro.html) — PostgreSQL\n- [JSON Types: jsonb and GIN indexing](https://www.postgresql.org/docs/current/datatype-json.html) — PostgreSQL\n- [SELECT: the locking clause and SKIP LOCKED](https://www.postgresql.org/docs/current/sql-select.html) — PostgreSQL\n- [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) — PostgreSQL\n\n## Frequently asked\n\n### Can PostgreSQL replace a separate search engine?\n\nFor most applications, yes. tsvector, tsquery and a GIN index give a real full-text search with stemming, weighting and ranking. A dedicated engine starts paying off when typo tolerance, advanced relevance tuning, faceted search and multilingual analysis are needed at serious scale.\n\n### Can PostgreSQL be used as a job queue?\n\nAt moderate volume, yes. SELECT ... FOR UPDATE SKIP LOCKED lets several consumers pull work from a queue-like table without lock contention, and LISTEN/NOTIFY signals that the table changed. Very high throughput, complex routing or multi-consumer fan-out is where a real message broker is the right tool.", "date_published": "2026-08-01T00:00:00.000Z", "date_modified": "2026-09-06T00:00:00.000Z", "tags": [ "postgresql", "database", "architecture", "opinion" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/essays/signals-of-an-over-complex-system/", "url": "https://muhammetsafak.com/engineering/essays/signals-of-an-over-complex-system/", "title": "Signals That a System Has Grown Too Complex", "summary": "Complexity does not arrive in one decision; it accumulates in defensible steps, which is why simplification starts from named signals rather than a feeling. Eight concrete signals, then a four-step subtraction: dead code, single-implementation interfaces, premature indirection, then clear boundaries. The target is not the least code but the least unnecessary code — essential complexity cannot be deleted.", "content_html": "At the end of her first week, an engineer who had just joined the team asked: “Why does adding a single field take two days?” The question was innocent; the diagnosis was clear. Nobody had sat down one day and decided to make that system complex. The complexity had simply accumulated.
\nThat is the heart of it: complexity doesn’t arrive in a single decision, it builds up unnoticed. To undo it, you first have to be able to see it.
\nNo complex system decides to be complex. Each step looks reasonable on its own: “let’s open an interface for now”, “let’s put a flag on this case”, “one more layer in between”. Each is small, each is defensible.
\nTheir sum is not defensible. Complexity is not an event but an accumulation — and an accumulation stays invisible until it is named. That is why simplification cannot start from a feeling; it starts from concrete signals.
\nThe red flags, not open to debate, that tell you a system is more complex than it needs to be:
\nIf two or three of these signals are present in a system, the debate isn’t “is it complex” but “where do we start”.
\nSimplification is not a rewrite; most of the time it’s a subtraction. In order:
\nAfter each step, stop and measure. Simplification can also go too far; the goal is not “the least code” but “the least unnecessary code”.
\n\nTan: Keep the newcomers’ questions in one place. When the same question keeps coming back, that record can point to the signal on the list that weighs most on your system. When you defend a simplification proposal, it adds evidence from your own team’s questions to the signals in the post.
An important distinction: there are two kinds of complexity. In Fred Brooks’s terms — essential and accidental.
\nEssential complexity comes from the domain itself. A tax calculation is complex because tax law is complex; you can’t delete that, you can only model it honestly. Accidental complexity is what you add: the unnecessary layer, the premature abstraction, the forgotten flag.
\nThe aim of simplification is to clear away the accidental — not the essential. Making a system “too simple” and ignoring the domain’s real complexity is also a mistake; that complexity doesn’t get deleted, it just moves to the wrong place — usually into the calling code.
\nGood architecture is exactly as complex as the domain is; no more.
\nComplexity accumulates unnoticed; that is why knowing by heart the signals that make it noticeable is an engineering skill. If you can’t name the signal, you can’t start simplifying either.
\nKeeping a system simple takes more effort than building it simple — and that effort is worth it.
", "content_text": "> **TL;DR:** Complexity does not arrive in one decision; it accumulates in defensible steps, which is why simplification starts from named signals rather than a feeling. Eight concrete signals, then a four-step subtraction: dead code, single-implementation interfaces, premature indirection, then clear boundaries. The target is not the least code but the least unnecessary code — essential complexity cannot be deleted.\n\nAt the end of her first week, an engineer who had just joined the team asked: \"Why does adding a single field take two days?\" The question was innocent; the diagnosis was clear. Nobody had sat down one day and decided to make that system complex. The complexity had simply accumulated.\n\nThat is the heart of it: complexity doesn't arrive in a single decision, it builds up unnoticed. To undo it, you first have to be able to **see** it.\n\n## How does complexity accumulate?\n\nNo complex system decides to be complex. Each step looks reasonable on its own: \"let's open an interface for now\", \"let's put a flag on this case\", \"one more layer in between\". Each is small, each is defensible.\n\nTheir sum is not defensible. Complexity is not an event but an accumulation — and an accumulation stays invisible until it is named. That is why simplification cannot start from a feeling; it starts from concrete signals.\n\n## The concrete signals\n\nThe red flags, not open to debate, that tell you a system is more complex than it needs to be:\n\n- **You open a lot of files to understand a single feature.** If seeing where a request goes takes six files and four levels of indirection, the problem isn't you, it's the structure.\n- **A one-line change touches a dozen files.** To add one field: migration, model, DTO, mapper, interface, factory, test fixture... The cost of a change is not proportional to the size of the change.\n- **Interfaces with a single implementation.** Abstractions opened \"just in case\" whose second implementation never arrived. This is the residue [speculative generality](/engineering/essays/the-cost-of-just-in-case-code/) leaves in a codebase.\n- **Onboarding takes weeks.** In a mature, lean system a new engineer is productive in the first days. If it takes weeks, the code isn't explaining itself.\n- **Nobody can draw the system on a whiteboard.** An architecture that can't be described as \"this box connects to that one\" is an architecture that isn't understood.\n- **The test setup is longer than the test itself.** The ten mocks you need to stand up to test one behavior are a confession that the behavior depends on far too much.\n- **\"Don't touch that, it'll break.\"** If a region of code has been declared untouchable, that region is already broken — it just hasn't blown up yet.\n- **Config, flags, and options nobody uses.** Every option that was opened and forgotten is a branch that everyone reading the code has to account for.\n\nIf two or three of these signals are present in a system, the debate isn't \"is it complex\" but \"where do we start\".\n\n## Where do you start simplifying?\n\nSimplification is not a rewrite; most of the time it's a **subtraction**. In order:\n\n1. **Delete dead code.** Unused flags, methods never called, branches never reached. The lowest-risk, highest-return step — deleted code has no bugs.\n2. **Collapse single-implementation interfaces.** Use the concrete class directly until the abstraction earns a real second use. When a seam is genuinely needed — [once the need is proven](/engineering/essays/the-cost-of-just-in-case-code/) — it comes back in a few hours of refactoring.\n3. **Inline premature indirection.** Remove the layers that are called from a single place and only pass an argument on to another function.\n4. **Clarify boundaries.** Gather the remaining complexity behind clear boundaries, like in a [modular monolith](/engineering/essays/why-i-start-with-a-modular-monolith/) — so the next accumulation shows up early.\n\nAfter each step, stop and measure. Simplification can also go too far; the goal is not \"the least code\" but \"the least unnecessary code\".\n\n\n> **Tan:** Keep the newcomers’ questions in one place. When the same question keeps coming back, that record can point to the signal on the list that weighs most on your system. When you defend a simplification proposal, it adds evidence from your own team’s questions to the signals in the post.\n\n\n## Complexity isn't always bad\n\nAn important distinction: there are two kinds of complexity. In Fred Brooks's terms — **essential** and **accidental**.\n\nEssential complexity comes from the domain itself. A tax calculation is complex because tax law is complex; you can't delete that, you can only model it honestly. Accidental complexity is what you add: the unnecessary layer, the premature abstraction, the forgotten flag.\n\nThe aim of simplification is to clear away the accidental — not the essential. Making a system \"too simple\" and ignoring the domain's real complexity is also a mistake; that complexity doesn't get deleted, it just moves to the wrong place — usually into the calling code.\n\nGood architecture is exactly as complex as the domain is; no more.\n\n---\n\nComplexity accumulates unnoticed; that is why knowing by heart the signals that make it noticeable is an engineering skill. If you can't name the signal, you can't start simplifying either.\n\nKeeping a system simple takes more effort than building it simple — and that effort is worth it.\n\n## Sources\n\n- [No Silver Bullet — Essence and Accidents of Software Engineering (TR86-020): essence, the difficulties inherent in the nature of software, versus accidents, those that attend its production but are not inherent](https://www.cs.unc.edu/techreports/86-020.pdf) — Frederick P. Brooks Jr., University of North Carolina\n- [Yagni: the cost of build, delay, carry and repair for a presumptive feature](https://martinfowler.com/bliki/Yagni.html) — Martin Fowler\n\n## Frequently asked\n\n### How do you tell an over-complex system from a genuinely hard domain?\n\nBy which kind of complexity you are looking at. Essential complexity comes from the domain — a tax calculation is complex because tax law is — and can only be modelled honestly. Accidental complexity is what you added: the extra layer, the premature abstraction, the forgotten flag. Only the second kind is worth clearing away.\n\n### Where do you start simplifying?\n\nIn order: delete dead code, collapse interfaces with a single implementation, inline indirection that is called from one place and only forwards an argument, then gather what is left behind clear boundaries. Stop and measure after each step — simplification can also go too far.\n\n### Is an interface with a single implementation always a mistake?\n\nIt is the residue that an abstraction opened just in case leaves behind when the second implementation never arrives. Use the concrete class directly until the abstraction earns a real second use; when a seam is genuinely needed, it comes back in a few hours of refactoring. That is cheaper than carrying it for years.", "date_published": "2026-07-25T00:00:00.000Z", "date_modified": "2026-09-30T00:00:00.000Z", "tags": [ "architecture", "simplicity", "decisions", "opinion" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/circuit-breaker-external-service-integration/", "url": "https://muhammetsafak.com/engineering/notes/circuit-breaker-external-service-integration/", "title": "Circuit Breakers for External Service Integrations", "summary": "A slow external service takes an application down faster than a crashed one: the call holds the process, the pool fills, and unrelated endpoints start timing out. Three defenses, in order — a short explicit timeout on every call, retry only with exponential backoff and jitter, and a circuit breaker that stops calling at all. Its value is decided by the fallback you serve while the circuit is open.", "content_html": "An SMS provider slowed down — it didn’t even crash, its response time just climbed to 30 seconds. Within half an hour the entire application became unresponsive; even endpoints with nothing to do with SMS were timing out. One slow external service had dragged the whole system down with it.
\nAn external service will fail sooner or later. That’s not the question. The question is: will your system go down with it?
\nThe logic is simple and merciless. A slow external call locks up the worker that makes it — or the PHP-FPM process. If the call takes 30 seconds, that process can’t look at any other request for 30 seconds.
\nRequests pile up, the process pool fills. Once the pool is full, new requests — including ones with nothing to do with SMS — wait too. Your healthy endpoints die because of one sick dependency. This is called cascading failure, and it almost always starts with a slowdown.
\nThe first and cheapest defense is to give every external call an explicit and short timeout. Default timeouts — 30 seconds or more in most HTTP clients — are unacceptable for production.
\n$response = Http::timeout(5) // total time\n ->connectTimeout(2) // connection establishment time\n ->get('https://sms-provider.example/send');The rule from the Laravel queue post applies here too: not the default 30 seconds, but 5. Don’t leave to chance how long a process waits on a dependency — that number should be a decision.
\nRetry makes sense for transient errors, but blind retry does harm. Immediately retrying a failing service piles more load onto it — this is called a retry storm; it knocks the service back down before it can fully recover.
\nIf you retry, two things are mandatory: exponential backoff (increasing wait on each attempt) and jitter (random offset — so all clients don’t retry at the same moment).
\nHttp::retry(3, function (int $attempt) {\n // exponential backoff (200, 400, 800 ms) plus jitter\n return 200 * (2 ** ($attempt - 1)) + random_int(0, 100);\n}, function ($exception) {\n return $exception instanceof ConnectionException;\n})->timeout(5)->get($url);Keep the retry count low. If there’s still an error after three attempts, the problem isn’t transient; continuing to retry only ties up your own process for longer.
\nA timeout caps a single call; a circuit breaker decides to stop making calls at all. Its logic comes from an electrical circuit breaker, and it has three states:
\nclosed; if not, back to open.The gain: when the external service is sick, you don’t wait on it. Even a 5-second timeout is 5 seconds per process; in the open state that drops to 0. Fail-fast beats slow-fail.
// Conceptual flow — leave the threshold, counter, and timer to a mature package.\nif ($breaker->isOpen('sms')) {\n return $this->queueForLater($message); // circuit open: no attempt at all\n}\n\ntry {\n $response = Http::timeout(5)->post($smsUrl, $payload);\n $breaker->recordSuccess('sms');\n} catch (\\Throwable $e) {\n $breaker->recordFailure('sms'); // circuit opens if the threshold is exceeded\n return $this->queueForLater($message);\n}Instead of writing this logic from scratch, use a mature package — getting the state counting, the threshold, and the timer right is subtler than it looks.
\nA circuit breaker’s value shows up in what you do in the open state. “Return an error” is the weakest option. Better ones:
Which fallback is right depends on the business — but “no fallback” is not an answer.
\nNot every external call wants a circuit breaker. A timeout is always mandatory. Retry, only if the operation is idempotent. A circuit breaker mainly adds value when the call is frequent and the service is on the user’s path — for an integration called a few times a day, a timeout and a reasonable retry are usually enough.
\nAn external service failing isn’t a question of “if,” it’s a question of “when.” Going down with it, though, is a design choice — and a choice you can change.
\nDon’t leave your system as fragile as your weakest dependency.
", "content_text": "> **TL;DR:** A slow external service takes an application down faster than a crashed one: the call holds the process, the pool fills, and unrelated endpoints start timing out. Three defenses, in order — a short explicit timeout on every call, retry only with exponential backoff and jitter, and a circuit breaker that stops calling at all. Its value is decided by the fallback you serve while the circuit is open.\n\nAn SMS provider slowed down — it didn't even crash, its response time just climbed to 30 seconds. Within half an hour the entire application became unresponsive; even endpoints with nothing to do with SMS were timing out. One slow external service had dragged the whole system down with it.\n\nAn external service will fail sooner or later. That's not the question. The question is: will your system go down with it?\n\n## How does cascading failure happen?\n\nThe logic is simple and merciless. A slow external call locks up the worker that makes it — or the PHP-FPM process. If the call takes 30 seconds, that process can't look at any other request for 30 seconds.\n\nRequests pile up, the process pool fills. Once the pool is full, new requests — including ones with nothing to do with SMS — wait too. Your healthy endpoints die because of one sick dependency. This is called **cascading failure**, and it almost always starts with a slowdown.\n\n## First defense: timeout\n\nThe first and cheapest defense is to give every external call an **explicit and short** timeout. Default timeouts — 30 seconds or more in most HTTP clients — are unacceptable for production.\n\n```php\n$response = Http::timeout(5) // total time\n ->connectTimeout(2) // connection establishment time\n ->get('https://sms-provider.example/send');\n```\n\nThe rule from the [Laravel queue post](/engineering/notes/laravel-queue-production-slowdown/) applies here too: not the default 30 seconds, but 5. Don't leave to chance how long a process waits on a dependency — that number should be a decision.\n\n## Second defense: retry — but carefully\n\nRetry makes sense for transient errors, but blind retry does harm. Immediately retrying a failing service piles more load onto it — this is called a **retry storm**; it knocks the service back down before it can fully recover.\n\nIf you retry, two things are mandatory: exponential backoff (increasing wait on each attempt) and jitter (random offset — so all clients don't retry at the same moment).\n\n```php\nHttp::retry(3, function (int $attempt) {\n // exponential backoff (200, 400, 800 ms) plus jitter\n return 200 * (2 ** ($attempt - 1)) + random_int(0, 100);\n}, function ($exception) {\n return $exception instanceof ConnectionException;\n})->timeout(5)->get($url);\n```\n\nKeep the retry count low. If there's still an error after three attempts, the problem isn't transient; continuing to retry only ties up your own process for longer.\n\n## Third defense: circuit breaker\n\nA timeout caps a single call; a circuit breaker decides to **stop making calls** at all. Its logic comes from an electrical circuit breaker, and it has three states:\n\n- **Closed** — everything's normal, calls pass through. Failures are counted.\n- **Open** — most of the last N calls failed; the circuit opens. The external service is no longer hit at all, and the call returns an error **instantly**.\n- **Half-open** — after a while the circuit allows a single trial call. If it succeeds, it returns to `closed`; if not, back to `open`.\n\nThe gain: when the external service is sick, you don't wait on it. Even a 5-second timeout is 5 seconds per process; in the `open` state that drops to 0. Fail-fast beats slow-fail.\n\n```php\n// Conceptual flow — leave the threshold, counter, and timer to a mature package.\nif ($breaker->isOpen('sms')) {\n return $this->queueForLater($message); // circuit open: no attempt at all\n}\n\ntry {\n $response = Http::timeout(5)->post($smsUrl, $payload);\n $breaker->recordSuccess('sms');\n} catch (\\Throwable $e) {\n $breaker->recordFailure('sms'); // circuit opens if the threshold is exceeded\n return $this->queueForLater($message);\n}\n```\n\nInstead of writing this logic from scratch, use a mature package — getting the state counting, the threshold, and the timer right is subtler than it looks.\n\n## Fallback: what do you do when the circuit is open?\n\nA circuit breaker's value shows up in what you do in the `open` state. \"Return an error\" is the weakest option. Better ones:\n\n- **Queue it, send it later.** If the SMS doesn't have to go out immediately, drop it on a [queue](/engineering/notes/sync-vs-async/) to be processed once the circuit closes.\n- **Serve stale but valid data.** If an exchange-rate service is down, showing the last known rate beats showing none.\n- **Degrade the feature gracefully.** A \"Recommendations can't load right now\" message beats crashing the whole page.\n\nWhich fallback is right depends on the business — but \"no fallback\" is not an answer.\n\n## When don't you need all this?\n\nNot every external call wants a circuit breaker. A timeout is **always** mandatory. Retry, only if the operation is idempotent. A circuit breaker mainly adds value when the call is frequent and the service is on the user's path — for an integration called a few times a day, a timeout and a reasonable retry are usually enough.\n\n---\n\nAn external service failing isn't a question of \"if,\" it's a question of \"when.\" Going down with it, though, is a design choice — and a choice you can change.\n\nDon't leave your system as fragile as your weakest dependency.\n\n## Sources\n\n- [Release It! Second Edition: Timeouts, Circuit Breaker, Bulkheads, Cascading Failures](https://pragprog.com/titles/mnee2/release-it-second-edition/) — Michael T. Nygard, Pragmatic Bookshelf\n- [Exponential Backoff And Jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) — AWS Architecture Blog\n- [HTTP Client: timeout, connectTimeout and retry](https://laravel.com/docs/12.x/http-client) — Laravel\n\n## Frequently asked\n\n### How does one slow external service take a whole application down?\n\nThe slow call locks the worker or the PHP-FPM process making it — a 30-second call means 30 seconds in which that process looks at nothing else. Requests pile up, the process pool fills, and once it is full even requests with no relation to that service wait too. Cascading failure almost always starts with a slowdown, not a crash.\n\n### Does every external call need a circuit breaker?\n\nNo. A timeout is always mandatory. Retry only if the operation is idempotent, and only with exponential backoff and jitter. A circuit breaker mainly earns its place when the call is frequent and the service sits on the user path; for an integration called a few times a day, a timeout and a reasonable retry are usually enough.", "date_published": "2026-07-18T00:00:00.000Z", "date_modified": "2026-09-06T00:00:00.000Z", "tags": [ "resilience", "architecture", "api", "production" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/serverless-migration-decision/", "url": "https://muhammetsafak.com/engineering/notes/serverless-migration-decision/", "title": "The Serverless Decision: Cold-Start and Vendor Lock-in", "summary": "Serverless is not an architectural fashion, it is a tool with a workload profile: variable, sparse, bursty. Steady traffic turns pay-per-use into pay-per-second, and on a low-traffic path a cold start is the rule rather than the exception, so the user pays it on nearly every request. The bond is not the runtime but the glue around it — event sources, IAM, gateway — which makes leaving a rewrite, not a migration.", "content_html": "A team built its new service on AWS Lambda with a “serverless first” decision. Yet the traffic was steady and predictable — roughly the same all day. At the end of the month the bill came in at several times what a single VPS doing the same work would cost; on top of that, p95 latency was worse.
\nServerless isn’t the wrong technology. It was the right technology applied to the wrong workload.
\nFaaS (Function as a Service) has real upsides:
\nThese three genuinely pay off on the right workload. The trick is diagnosing that workload correctly.
\nIf a function hasn’t been called for a while, the provider brings it up from scratch: the runtime loads, dependencies initialize, the code runs. This cold-start adds anywhere from under 100 ms to seconds of latency compared to an already-running process.
\nThe key distinction: for a low-traffic endpoint, cold-start is the rule, not the exception. On a function that gets a few calls an hour, nearly every request is cold. “Pay by scale” sounds appealing, but at near-zero traffic what you pay is the latency the user sees every single time. On a latency-sensitive user path, that’s a direct UX cost.
\nLock-in isn’t just the runtime; the real bond is in the glue around it. Over time a serverless application also couples to the provider’s event sources, identity/authorization model (IAM), managed queue, API gateway, and deployment tooling.
\nMoving a single function is easy. But in a real system, “getting off serverless” is usually not a migration, it’s a rewrite. This ties directly into the innovation-token logic from the boring architecture piece: what looks like a single decision is actually a bond that’s expensive to undo.
\nServerless shines where the load is variable and sparse:
\nOn these workloads, scale-to-zero is real money saved, and cold-start is usually irrelevant — nobody cares that a nightly report starts 800 ms late.
\nThe reverse is just as clear:
\nFor a workload with this profile, a boring VPS — like the setup in the multiple projects on a single VPS piece — is both cheaper and more predictable. The cost of a steady load matches a steady server.
\nOn the Laravel side, Laravel Vapor makes it possible to run the application on Lambda, and it works well. But that it works doesn’t mean it’s the right decision: with Vapor you also inherit cold-start and the AWS bond.
\nThe decision is, again, the workload. For a Laravel app with bursty, sparse, or unpredictable traffic, Vapor makes sense. For a continuous, latency-sensitive app, a classic server is still the plainer, cheaper answer.
\n\nTan: The real mistake in the story was not Lambda but the word first: the decision was made before anyone looked at the workload. Making a principle the default answers, on your behalf, the question each new service should ask. Keep a default if you like, but write the reasoning again for every service.
This isn’t a binary choice. A boring server carrying the continuous load + handing only the sparse/bursty work (webhooks, the nightly report) to serverless is often the most honest split. Put every workload where its own cost profile belongs.
\nServerless isn’t an architectural fashion, it’s a tool for a specific workload profile. If you have that profile, it pays off; if you don’t, it’s paid back as cold-start and lock-in.
\nMeasure your load first; the tool comes after.
", "content_text": "> **TL;DR:** Serverless is not an architectural fashion, it is a tool with a workload profile: variable, sparse, bursty. Steady traffic turns pay-per-use into pay-per-second, and on a low-traffic path a cold start is the rule rather than the exception, so the user pays it on nearly every request. The bond is not the runtime but the glue around it — event sources, IAM, gateway — which makes leaving a rewrite, not a migration.\n\nA team built its new service on AWS Lambda with a \"serverless first\" decision. Yet the traffic was steady and predictable — roughly the same all day. At the end of the month the bill came in at several times what a single VPS doing the same work would cost; on top of that, p95 latency was worse.\n\nServerless isn't the wrong technology. It was the right technology applied to the wrong workload.\n\n## What serverless buys you\n\nFaaS (Function as a Service) has real upsides:\n\n- **No server management.** Patching, scaling, capacity planning all sit with the provider.\n- **Scale to zero.** No requests means nothing running and nothing paid for.\n- **Pay per use.** You don't pay for idle capacity.\n\nThese three genuinely pay off on the right workload. The trick is diagnosing that workload correctly.\n\n## The real cost of cold-start\n\nIf a function hasn't been called for a while, the provider brings it up from scratch: the runtime loads, dependencies initialize, the code runs. This **cold-start** adds anywhere from under 100 ms to seconds of latency compared to an already-running process.\n\nThe key distinction: for a low-traffic endpoint, cold-start is **the rule, not the exception**. On a function that gets a few calls an hour, nearly every request is cold. \"Pay by scale\" sounds appealing, but at near-zero traffic what you pay is the latency the user sees every single time. On a latency-sensitive user path, that's a direct UX cost.\n\n## Vendor lock-in\n\nLock-in isn't just the runtime; the real bond is in the glue around it. Over time a serverless application also couples to the provider's event sources, identity/authorization model (IAM), managed queue, API gateway, and deployment tooling.\n\nMoving a single function is easy. But in a real system, \"getting off serverless\" is usually not a migration, it's a rewrite. This ties directly into the innovation-token logic from the [boring architecture](/engineering/essays/why-boring-architecture/) piece: what looks like a single decision is actually a bond that's expensive to undo.\n\n## Which workloads win?\n\nServerless shines where the load is **variable and sparse**:\n\n- Scheduled jobs — nightly reports, cron-like tasks.\n- Webhook receivers — uncertain when and how often the calls will arrive.\n- Low-volume internal tools — panels used a few times a day.\n- Bursty batch jobs — once an hour, briefly high parallelism.\n\nOn these workloads, scale-to-zero is real money saved, and cold-start is usually irrelevant — nobody cares that a nightly report starts 800 ms late.\n\n## Which workloads lose?\n\nThe reverse is just as clear:\n\n- **Steady, continuous traffic.** On an always-running API, \"pay per use\" turns into \"pay every second\" — and that's usually more expensive than a continuously running server.\n- **Latency-sensitive paths.** Cold-start is unacceptable on a request the user is waiting on.\n- **Long-running jobs.** FaaS has execution-time limits; long jobs either get split up or don't fit.\n\nFor a workload with this profile, a boring VPS — like the setup in the [multiple projects on a single VPS](/engineering/systems/single-vps-multi-project-architecture/) piece — is both cheaper and more predictable. The cost of a steady load matches a steady server.\n\n## Laravel and serverless\n\nOn the Laravel side, Laravel Vapor makes it possible to run the application on Lambda, and it works well. But that it works doesn't mean it's the right decision: with Vapor you also inherit cold-start and the AWS bond.\n\nThe decision is, again, the workload. For a Laravel app with bursty, sparse, or unpredictable traffic, Vapor makes sense. For a continuous, latency-sensitive app, a classic server is still the plainer, cheaper answer.\n\n\n> **Tan:** The real mistake in the story was not Lambda but the word first: the decision was made before anyone looked at the workload. Making a principle the default answers, on your behalf, the question each new service should ask. Keep a default if you like, but write the reasoning again for every service.\n\n\n## Mix them\n\nThis isn't a binary choice. A boring server carrying the continuous load + handing only the sparse/bursty work (webhooks, the nightly report) to serverless is often the most honest split. Put every workload where its own cost profile belongs.\n\n---\n\nServerless isn't an architectural fashion, it's a tool for a specific workload profile. If you have that profile, it pays off; if you don't, it's paid back as cold-start and lock-in.\n\nMeasure your load first; the tool comes after.\n\n## Sources\n\n- [Understanding the Lambda execution environment lifecycle: cold starts and latency](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html) — AWS\n- [Lambda quotas: function timeout is 900 seconds](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html) — AWS\n\n## Frequently asked\n\n### Which workloads does serverless actually pay off on?\n\nThe ones where the load is variable and sparse: scheduled jobs, webhook receivers whose timing is unknown, low-volume internal tools, and bursty batch work. There, scale-to-zero is real money saved and cold start is usually irrelevant — nobody minds a nightly report starting 800 ms late.\n\n### Why is cold start worse on a low-traffic endpoint?\n\nBecause there it is the rule, not the exception. On a function that gets a few calls an hour nearly every request is cold, so what you pay at near-zero traffic is the latency the user sees every single time. On a latency-sensitive user path, that is a direct UX cost.", "date_published": "2026-07-11T00:00:00.000Z", "date_modified": "2026-09-30T00:00:00.000Z", "tags": [ "serverless", "architecture", "cloud" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/bff-backend-for-frontend/", "url": "https://muhammetsafak.com/engineering/notes/bff-backend-for-frontend/", "title": "BFF Pattern: A Separate API Layer for Mobile and Web", "summary": "Bending one general-purpose API to fit every client is paid for in over-fetching, chatty flows, conditional fields and a versioning knot that locks web and mobile into one contract. A BFF cancels that bill with a thin adaptation layer per client — aggregation and response shaping, never business rules. What justifies the layer is the distance between clients, not their number.", "content_html": "I once watched a mobile app fire nine separate requests just to render a single home screen. The API had actually been designed for web; mobile was collecting the data the way web sliced it, piece by piece, and stitching it back together on screen. Nine round-trips per launch — under mobile network conditions, no less.
\nThat was the bill for the “one API fits everyone” assumption. The BFF — Backend for Frontend — exists precisely to cancel that bill.
\nA BFF is a thin API layer tailored to one specific client type. Web gets its own BFF, mobile gets its own. Each one calls the same underlying services and domain, but shapes the response to fit its own client’s screen.
\nThe real business logic doesn’t live in the BFF. The BFF is only an adaptation layer: it aggregates, trims unneeded fields, and gathers everything a client needs for one screen into a single response.
\nA single general-purpose API can’t fully satisfy two clients over time:
\nif mobile then ... else .... A single endpoint tries to carry the requirements of two clients at once and does a mediocre job for both.Each of these costs looks small on its own; their sum is an architectural problem.
\nA BFF is justified when client needs genuinely diverge:
\nWhen these differences exist, giving each client its own BFF is cheaper than constantly bending one API to fit both.
\nIf you have a single client, a BFF is more layer than solution — just an extra hop.
\nIf you have two clients but both want almost the same data in almost the same shape, it’s still unnecessary. What justifies a BFF isn’t the number of clients but the difference between them. No difference, no separate layer — otherwise it’s speculative generality in the shape of an API layer.
\nA common confusion: adding a BFF doesn’t drop you into distributed architecture. A BFF is a presentation/aggregation layer, not a domain service. Putting business rules inside it means copying the rule once per client.
\nThe BFF’s only kinship with microservices is this: both ask “is this worth a separate deployable?” The answer is usually “no” — the measured-signal threshold from the decision to move to microservices applies here too.
\n“BFF” doesn’t have to mean standing up a separate server. Within the same monolith:
\nroutes/web-api.php, routes/mobile-api.php — each with its own controllers and its own response shape.The real idea isn’t a box, it’s a boundary: collect the client-specific adaptation in one place and keep the domain clean of it. Move to a separate deployable only when a measured signal forces it.
\nThe BFF is the antidote to the “every client makes do with the same API” stubbornness — but only when clients genuinely diverge. No difference, no layer.
\nWhat adds the layer isn’t the number of clients, it’s the distance between them.
", "content_text": "> **TL;DR:** Bending one general-purpose API to fit every client is paid for in over-fetching, chatty flows, conditional fields and a versioning knot that locks web and mobile into one contract. A BFF cancels that bill with a thin adaptation layer per client — aggregation and response shaping, never business rules. What justifies the layer is the distance between clients, not their number.\n\nI once watched a mobile app fire nine separate requests just to render a single home screen. The API had actually been designed for web; mobile was collecting the data the way web sliced it, piece by piece, and stitching it back together on screen. Nine round-trips per launch — under mobile network conditions, no less.\n\nThat was the bill for the \"one API fits everyone\" assumption. The BFF — Backend for Frontend — exists precisely to cancel that bill.\n\n## What a BFF is\n\nA BFF is a thin API layer tailored to one specific client type. Web gets its own BFF, mobile gets its own. Each one calls the same underlying services and domain, but shapes the response to fit its own client's screen.\n\nThe real business logic doesn't live in the BFF. The BFF is only an **adaptation layer**: it aggregates, trims unneeded fields, and gathers everything a client needs for one screen into a single response.\n\n## The hidden cost of bending one API to fit everyone\n\nA single general-purpose API can't fully satisfy two clients over time:\n\n- **Over-fetching.** Mobile downloads a 40-field user object and uses 4 of them. The rest is wasted bandwidth and battery.\n- **Chatty flow.** The \"fetch the list first, then fetch details for each row\" flow that web finds perfectly normal turns into nine round-trips on a mobile network.\n- **Conditional fields.** The response starts branching with `if mobile then ... else ...`. A single endpoint tries to carry the requirements of two clients at once and does a mediocre job for both.\n- **Versioning knot.** A change needed for web risks breaking mobile's published version. Two clients get locked into one contract.\n\nEach of these costs looks small on its own; their sum is an architectural problem.\n\n## When is a BFF warranted?\n\nA BFF is justified when client needs **genuinely** diverge:\n\n- Mobile and web need distinctly different screens and different data shapes.\n- Mobile network and battery constraints make round-trip count and payload size a first-class concern.\n- The clients ship at different cadences — the mobile release is stuck in store review while web deploys every day; a single contract locks the two together.\n\nWhen these differences exist, giving each client its own BFF is cheaper than constantly bending one API to fit both.\n\n## When is a BFF unnecessary?\n\nIf you have a single client, a BFF is more layer than solution — just an extra hop.\n\nIf you have two clients but both want almost the same data in almost the same shape, it's still unnecessary. What justifies a BFF isn't the number of clients but the **difference between** them. No difference, no separate layer — otherwise it's [speculative generality](/engineering/essays/the-cost-of-just-in-case-code/) in the shape of an API layer.\n\n## A BFF is not a microservice\n\nA common confusion: adding a BFF doesn't drop you into distributed architecture. A BFF is a presentation/aggregation layer, not a domain service. Putting business rules inside it means copying the rule once per client.\n\nThe BFF's only kinship with microservices is this: both ask \"is this worth a separate deployable?\" The answer is usually \"no\" — the measured-signal threshold from [the decision to move to microservices](/engineering/essays/when-i-move-to-microservices/) applies here too.\n\n## Start light\n\n\"BFF\" doesn't have to mean standing up a separate server. Within the same monolith:\n\n- One route namespace per client — `routes/web-api.php`, `routes/mobile-api.php` — each with its own controllers and its own response shape.\n- Or a single GraphQL layer: each client queries exactly what it needs, and over-fetching resolves itself.\n\nThe real idea isn't a box, it's a boundary: collect the client-specific adaptation in one place and keep the domain clean of it. Move to a separate deployable only when [a measured signal](/engineering/essays/when-i-move-to-microservices/) forces it.\n\n---\n\nThe BFF is the antidote to the \"every client makes do with the same API\" stubbornness — but only when clients genuinely diverge. No difference, no layer.\n\nWhat adds the layer isn't the number of clients, it's the distance between them.\n\n## Sources\n\n- [Pattern: Backends For Frontends](https://samnewman.io/patterns/architectural/bff/) — Sam Newman\n- [Backends for Frontends pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/backends-for-frontends) — Microsoft Azure Architecture Center\n\n## Frequently asked\n\n### When is a BFF unnecessary?\n\nWhen there is only one client, a BFF is more layer than solution — just an extra hop. And with two clients that want almost the same data in almost the same shape it is still unnecessary: what justifies a BFF is the difference between clients, not their number.\n\n### Is a BFF a microservice?\n\nNo. A BFF is a presentation and aggregation layer, not a domain service, and adding one does not drop you into a distributed architecture. Putting business rules inside it means copying the rule once per client. It shares only one question with microservices: is this worth a separate deployable?", "date_published": "2026-07-04T00:00:00.000Z", "date_modified": "2026-09-06T00:00:00.000Z", "tags": [ "architecture", "api", "bff" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/sync-vs-async/", "url": "https://muhammetsafak.com/engineering/notes/sync-vs-async/", "title": "Synchronous or Asynchronous? HTTP or the Queue", "summary": "Synchronous or asynchronous is not a performance question, it is a question of who is waiting. If the user has to see the result, the job stays inside the request; if not, it goes to a queue and the request answers 202 Accepted. Failure sharpens the same line: a synchronous failure is an error screen, an asynchronous one is a retry — which is why an unstable third party never belongs on the synchronous path.", "content_html": "A signup form took eight seconds. Inside the request, in order: the user got saved, a welcome email went out, a PDF was generated, a record was pushed to a CRM. For all eight seconds the user stared at a spinner — when the only thing they cared about was whether their account had been created.
\nShould a job run inside the HTTP request, or go to a queue? The line is clearer than it looks.
\nOne question decides most of it: does the user have to see the result of this job?
\nIn the form above, the user is only waiting for the answer to “was my account created?” The email, the PDF, the CRM — none of their results need to show up on that screen. All three were on the wrong side.
\nJobs whose result the user sees directly stay inside the request:
\nPutting these on a queue means telling the user “check back later” — and that is usually a bad experience.
\nEvery job whose result the user doesn’t need right then is asynchronous:
\nFor these, the right response is 202 Accepted: “I got your request, I’m working on it.” The user doesn’t wait, and the system does the work at its own pace.
Every endpoint should have a p95 budget — say 300 ms. If a single step eats that budget on its own and the user doesn’t have to see that step’s result, the step goes to a queue.
\nA synchronous chain is only as fast as its slowest link. If PDF generation takes 4 seconds, user creation takes 4 seconds too — even though the PDF has nothing to do with the response.
\nThe real issue isn’t speed, it’s failure. Synchronous and asynchronous handle failure in completely different ways:
\nFrom this comes a single rule: an unstable third party should never sit on the synchronous path. If that CRM call is inside the request, your signup form goes down every time the CRM does. If the call is queued, the retry completes on its own once the CRM comes back. The third party’s uptime stops being your uptime.
\nOnce a job goes to a queue, leave the user a way to follow up:
\nPOST /exports → 202 Accepted { "id": "exp_1a2b", "status": "processing" }\nGET /exports/exp_1a2b → 200 OK { "status": "done", "url": "..." }The user isn’t waiting on the result, but sometimes they get curious. A status endpoint, or a notification sent once it’s ready, closes that gap.
\n\nTan: Moving a job to a queue can also move its failure from the user’s screen to a place nobody looks at. For every queued job, it should be settled in advance who sees it when it fails and who retries it. Without an answer to that, the synchronous path at least fails where someone can see it.
Some jobs sit between the two: the user wants to see the result, but the job is also slow. There’s a third path here — start the job synchronously, return the first meaningful result immediately, hand the rest off to a queue. Create the order record instantly (synchronous), generate the invoice in the background (asynchronous).
\nDraw the line per job; “everything to a queue” is as wrong as “everything synchronous.”
\nSynchronous or asynchronous isn’t a performance question, it’s a “who’s waiting” question. No job the user isn’t waiting on should make their request wait.
\nTie up the request only with the work whose answer they actually want.
", "content_text": "> **TL;DR:** Synchronous or asynchronous is not a performance question, it is a question of who is waiting. If the user has to see the result, the job stays inside the request; if not, it goes to a queue and the request answers 202 Accepted. Failure sharpens the same line: a synchronous failure is an error screen, an asynchronous one is a retry — which is why an unstable third party never belongs on the synchronous path.\n\nA signup form took eight seconds. Inside the request, in order: the user got saved, a welcome email went out, a PDF was generated, a record was pushed to a CRM. For all eight seconds the user stared at a spinner — when the only thing they cared about was whether their account had been created.\n\nShould a job run inside the HTTP request, or go to a queue? The line is clearer than it looks.\n\n## The decision line: is the user waiting on the result?\n\nOne question decides most of it: **does the user have to see the result of this job?**\n\n- If the answer is \"yes,\" the job stays synchronous — it finishes inside the request, before the response returns.\n- If the answer is \"no,\" the job goes to a queue — the request says \"accepted\" and returns.\n\nIn the form above, the user is only waiting for the answer to \"was my account created?\" The email, the PDF, the CRM — none of their results need to show up on that screen. All three were on the wrong side.\n\n## Which jobs have to stay synchronous?\n\nJobs whose result the user sees directly stay inside the request:\n\n- Input validation — the error must come back immediately.\n- Creating the actual resource — user creation, order creation.\n- The **result of a payment authorization** — did the card go through or not? The user has to know right then.\n- Data the user will see in the next step.\n\nPutting these on a queue means telling the user \"check back later\" — and that is usually a bad experience.\n\n## Which jobs should go to a queue?\n\nEvery job whose result the user doesn't need right then is asynchronous:\n\n- Email and notifications.\n- PDF and report generation.\n- Webhook delivery.\n- Third-party synchronization — CRM, analytics, search index.\n- Image processing, exports.\n\nFor these, the right response is `202 Accepted`: \"I got your request, I'm working on it.\" The user doesn't wait, and the system does the work at its own pace.\n\n## Response time sharpens the line\n\nEvery endpoint should have a p95 budget — say 300 ms. If a single step eats that budget on its own and the user doesn't have to see that step's result, the step goes to a queue.\n\nA synchronous chain is only as fast as its slowest link. If PDF generation takes 4 seconds, user creation takes 4 seconds too — even though the PDF has nothing to do with the response.\n\n## Fault tolerance: the real distinction\n\nThe real issue isn't speed, it's failure. Synchronous and asynchronous handle failure in completely different ways:\n\n- **Synchronous failure** = the user sees an error screen. The work is lost.\n- **Asynchronous failure** = the job is retried, drops to a DLQ if it has to, and the [queue](/engineering/notes/laravel-queue-production-slowdown/) processes it later.\n\nFrom this comes a single rule: **an unstable third party should never sit on the synchronous path.** If that CRM call is inside the request, your signup form goes down every time the CRM does. If the call is queued, the retry completes on its own once the CRM comes back. The third party's uptime stops being your uptime.\n\n## The 202-and-status-poll pattern\n\nOnce a job goes to a queue, leave the user a way to follow up:\n\n```\nPOST /exports → 202 Accepted { \"id\": \"exp_1a2b\", \"status\": \"processing\" }\nGET /exports/exp_1a2b → 200 OK { \"status\": \"done\", \"url\": \"...\" }\n```\n\nThe user isn't waiting on the result, but sometimes they get curious. A status endpoint, or a notification sent once it's ready, closes that gap.\n\n\n> **Tan:** Moving a job to a queue can also move its failure from the user’s screen to a place nobody looks at. For every queued job, it should be settled in advance who sees it when it fails and who retries it. Without an answer to that, the synchronous path at least fails where someone can see it.\n\n\n## Where does the line dissolve?\n\nSome jobs sit between the two: the user wants to see the result, but the job is also slow. There's a third path here — start the job synchronously, return the first meaningful result immediately, hand the rest off to a queue. Create the order record instantly (synchronous), generate the invoice in the background (asynchronous).\n\nDraw the line per job; \"everything to a queue\" is as wrong as \"everything synchronous.\"\n\n---\n\nSynchronous or asynchronous isn't a performance question, it's a \"who's waiting\" question. No job the user isn't waiting on should make their request wait.\n\nTie up the request only with the work whose answer they actually want.\n\n## Sources\n\n- [RFC 9110, 15.3.3: 202 Accepted](https://www.rfc-editor.org/rfc/rfc9110.html#name-202-accepted) — IETF\n- [Release It! Second Edition: stability patterns and antipatterns](https://pragprog.com/titles/mnee2/release-it-second-edition/) — Michael T. Nygard, Pragmatic Bookshelf\n- [Using dead-letter queues in Amazon SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html) — AWS\n\n## Frequently asked\n\n### Which jobs have to stay inside the HTTP request?\n\nThe ones whose result the user sees directly: input validation, creating the actual resource, the result of a payment authorization, and data the user needs in the next step. Putting these on a queue means telling the user to check back later, which is usually a bad experience.\n\n### Why should an unstable third party never sit on the synchronous path?\n\nBecause its downtime becomes yours. A CRM call inside the request takes the signup form down every time the CRM goes down. Queued, the same call is retried and completes on its own once the CRM is back, so the third party uptime stops being your uptime.", "date_published": "2026-06-27T00:00:00.000Z", "date_modified": "2026-09-30T00:00:00.000Z", "tags": [ "architecture", "queue", "async", "api" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/systems/transactional-outbox-dual-write-and-idempotent-consumption/", "url": "https://muhammetsafak.com/engineering/systems/transactional-outbox-dual-write-and-idempotent-consumption/", "title": "Transactional Outbox: Dual-Write, Idempotent Consumers", "summary": "An outbox collapses dual-write into a single write but gives no free guarantee — it gives at-least-once. Polling vs CDC, multi-relay with SKIP LOCKED, ordering, pruning processed_messages, the exactly-once illusion, and field-level encryption.", "content_html": "The symptom was small: one or two invoices a day sat in the database but the consumer never saw them. The user said “I issued it,” and there was no trace on the other side. Always at night, always during a deploy or a network blip. Once I laid the picture out, the cause was clear — the INSERT into the database and the publish to the queue were two separate systems, with no guarantee between them.
This post is the systems side of the transactional outbox pattern that closed that gap, the at-least-once repeats it opened in return, and finally encrypting the personal data that flows through the event at the field level. I wrote the decision-chain side as a log on muhammetsafak.com; here I collect the decisions you have to make for the pattern to be durable in production.
\nThe crux in one sentence: your local database transaction doesn’t cover RabbitMQ. The classic code is dangerous because it works most of the time:
\nDB::transaction(function () use ($invoice) {\n $invoice->save(); // 1) write to MySQL\n $this->publishToRabbit($invoice); // 2) publish to the queue\n});There are two distinct failure modes:
\nsave() succeeds, publish drops on a network error → the invoice is in the DB, the event isn’t. Lost event.publish succeeds, then the transaction rollbacks for another reason → the event went out, no counterpart in the DB. Phantom event.DB::transaction doesn’t save you, because commit/rollback only wraps the MySQL side; the broker isn’t part of that transaction. The practical way to hold two systems in one atomic step is to reduce the write to a single system.
The idea is plain: instead of pushing the message straight to the queue, write it as a row into an outbox table inside the same transaction. The invoice and the event either both exist or both don’t, in the same commit — dual-write collapses into a single write.
CREATE TABLE outbox (\n id BINARY(16) NOT NULL, -- event id = idempotency key\n aggregate_id BIGINT NOT NULL, -- ordering and partition key\n topic VARCHAR(120) NOT NULL,\n payload JSON NOT NULL,\n status ENUM('pending','published') NOT NULL DEFAULT 'pending',\n created_at DATETIME(6) NOT NULL,\n published_at DATETIME(6) NULL,\n PRIMARY KEY (id),\n KEY idx_dispatch (status, created_at) -- the relay scan goes through this index\n);The write is no longer the business code’s concern:
\nDB::transaction(function () use ($invoice) {\n $invoice->save();\n Outbox::write('invoice.issued', $invoice->id, $this->payload($invoice));\n});A separate relay does the queue push: it reads the pending rows, publishes them to RabbitMQ, and stamps them published on success. The critical fact here: the relay can fall into the “I published but crashed before stamping it published” state. So while the outbox solves dual-write, it gives you no free guarantee — it gives you at-least-once. The message isn’t lost, but it can repeat. Everything else arranges itself around this fact.
The relay can see pending rows two ways.
Polling — the relay scans the table periodically:
\nSELECT id, topic, payload\nFROM outbox\nWHERE status = 'pending'\nORDER BY created_at\nLIMIT 100;Simple, works on every database, low operational overhead. The cost is two things: latency up to the scan interval (a 1s poll = ~1s of queue latency) and an idling query. Without the idx_dispatch index this scan gets expensive as the table grows; even with it, you trade poll frequency against DB load. At small-to-medium volume, polling is the right answer — tune it frequent enough to keep latency acceptable, sparse enough not to hammer the DB (in practice 200ms–1s).
CDC (change data capture) — the relay listens to the database’s WAL/binlog rather than the table (Debezium is the typical tool). Every INSERT into outbox turns into an event almost instantly; polling latency and idle scanning disappear. The cost is operational: binlog access, a connector process, one more moving part. I turn CDC on when latency genuinely matters (sub-second) or when volume strains polling; otherwise I count polling’s simplicity as an advantage.
\n\nRule: your latency budget and your operational budget conflict. CDC buys you latency and charges you an infrastructure piece in return. Start with polling; switch to CDC when there’s a measured reason.
\n
SKIP LOCKEDA single relay is a bottleneck and a single point of failure. Running multiple relays over the same table creates a new risk: two of them grab the same row and publish the same message twice. The naive fix is to lock the table — which kills the parallelism.
\nThe right tool is SELECT ... FOR UPDATE SKIP LOCKED. Each relay claims a batch of rows by locking them; instead of queuing behind rows another relay has locked, it skips them:
BEGIN;\n SELECT id, topic, payload\n FROM outbox\n WHERE status = 'pending'\n ORDER BY created_at\n LIMIT 100\n FOR UPDATE SKIP LOCKED; -- skip rows another relay holds, don't wait\n\n -- publish this batch, then:\n UPDATE outbox SET status = 'published', published_at = NOW(6)\n WHERE id IN (...);\nCOMMIT;Without SKIP LOCKED, the relays queue for the same rows and the parallelism effectively runs serially. With it, each relay pulls a disjoint set and all of them move at once. PostgreSQL and MySQL 8+ support it.
A subtlety: if the relay crashes between publish and UPDATE ... published, the row stays pending and gets republished. That’s acceptable — we’re already at at-least-once; the fix is on the consumer side, below. The dangerous order is the reverse: stamping published first and then publishing. A crash there produces a lost event — exactly what we ran from. So the order is fixed: publish first, stamp second.
“Let events arrive in the order they were written” is an intuitive expectation, but global ordering is expensive and usually unnecessary. What you actually need is per-aggregate order: the same invoice’s created event should arrive before its updated; the order of two different invoices relative to each other is nobody’s concern.
You get this by partitioning on aggregate_id: events with the same aggregate_id go to the same queue partition (or, in RabbitMQ, the same queue via a consistent-hash exchange), and a single consumer processes that partition in order. Different aggregates flow in parallel.
outbox (in created_at order)\n │ hash(aggregate_id) % N\n ├── partition 0 ──► consumer-0 (events for aggregates A,D — ordered)\n ├── partition 1 ──► consumer-1 (events for aggregate B — ordered)\n └── partition 2 ──► consumer-2 (events for aggregate C — ordered)Two things still break ordering, and the design has to be ready for them: multiple relays under SKIP LOCKED can publish rows in a different order, and at-least-once repeats can cut in. So rather than leaning on a global order, you write the consumer to be resilient to out-of-order and duplicated events. The practical shield: put a monotonic version/sequence on the event and drop the stale one (lower version) at the consumer. If you genuinely need strict global ordering you drop to a single partition + single consumer — which then ties throughput to that one consumer; most systems don’t want to pay that.
Because the relay can republish, you have to assume every event arrives at least once, sometimes more. The fix isn’t to make the event unique — it’s to set up the consumer so that processing the same event twice has the same effect as once. Write the id of every processed event into a dedupe table, and do the work inside the same transaction:
DB::transaction(function () use ($event) {\n $inserted = DB::table('processed_messages')->insertOrIgnore([\n 'message_id' => $event['id'],\n 'processed_at' => now(),\n ]);\n\n if ($inserted === 0) {\n return; // it's a repeat; produce no side effects\n }\n\n $this->applyBusinessEffect($event); // the real work — exactly once\n});The unique constraint on processed_messages.message_id is the heart of it: a second arrival of the same id is silently dropped by insertOrIgnore. Putting the business effect in the same transaction is mandatory — otherwise the “I processed it but crashed before stamping” gap opens. Where the dedupe key comes from, and designing a dedupe that survives a crash, is a topic of its own; I covered it in a separate note.
If the side effect is an external system (an e-invoice integrator, a payment API), dedupe alone isn’t enough — because the side effect is outside the transaction. There you have to carry the key to the external service: pass the same id as an idempotency key to the integrator, or ask “does this already exist?” before sending. I worked through that boundary in detail in the JIT reservation post, where sealing the number before the external call turns recovery into a lookup.
processed_messages can’t grow foreverThe dedupe table accumulates a row per event; if you don’t prune it, it becomes a performance problem of its own. The key observation: you don’t need to keep an event id forever — only as long as the window in which a repeat could arrive. Relay retries and broker redeliveries are on the order of hours, not days.
In practice I pick a fixed retention window (say 7 days — comfortably above the longest possible redelivery) and prune older rows:
\nDELETE FROM processed_messages\nWHERE processed_at < NOW() - INTERVAL 7 DAY\nLIMIT 10000; -- small batches at a time, keeping replica lag and locks in checkIf message_id is a UUID the table and its index grow; pruning keeps that bounded. When choosing the window, frame the rule backwards: how late could the latest redelivery arrive? Make a safe upper bound of that your retention. Set the window too short and a late repeat misses the dedupe, so the side effect runs a second time.
In a distributed system “exactly-once delivery” is tempting but can’t be guaranteed end to end. The reason is the same dual-write, at the broker boundary: if the consumer did the work but crashed before getting its ack to the broker, the broker redelivers the message — because it never saw the ack. “I processed it” and “I reported that I processed it” are two separate steps, and a crash can fall between them. This is the practical face of the Two Generals problem.
So the goal isn’t exactly-once delivery, it’s exactly-once effect. And you get that by combining two cheap guarantees:
\nat-least-once delivery + idempotent consumer = effectively-onceThat is, you stop trying to deliver exactly once and instead make the repeat harmless. Under systems that market “exactly-once,” this is usually exactly what’s there: at-least-once plus a dedupe layer. Look for the guarantee in the effect, not in the delivery.
\nOnce the outbox settles in, a new surface appears: the events’ payload carries personal data — customer name, tax ID, address — and that payload now sits in a durable table (outbox) and passes through the broker. If I leave the data as-is, I’m copying GDPR/KVKK-scoped fields, unencrypted, into more than one place.
I didn’t want to encrypt the whole payload — I needed fields like topic, aggregate_id, and invoice_id in the clear for relay routing and observability. The need is field-level encryption: only the sensitive fields encrypted, the rest in the clear. I marked the sensitive fields in the schema with x-gdpr-sensitive and encrypted only those at serialization time:
$schema = [\n 'invoice_id' => ['x-gdpr-sensitive' => false],\n 'customer_name' => ['x-gdpr-sensitive' => true],\n 'tax_id' => ['x-gdpr-sensitive' => true],\n 'total' => ['x-gdpr-sensitive' => false],\n];\n\n$payload = collect($raw)->map(fn ($value, $field) =>\n ($schema[$field]['x-gdpr-sensitive'] ?? false)\n ? Crypt::encryptString((string) $value) // only the sensitive field is encrypted\n : $value\n)->all();That the schema declares the sensitive field is no accident — combine it with schema validation at the edge of the queue and both questions (“which field is personal data, and is it the right type?”) get answered at the edge. The real engineering decision isn’t the encryption itself, it’s the three questions around it:
\nKey management and rotation. Laravel’s Crypt uses AES-256 keyed by APP_KEY. Bind yourself to a single key and rotation becomes a nightmare: the new key can’t open outbox rows written with the old one. The fix is to store a key version (key id) alongside the encrypted value and keep a keyring that recognizes several keys at once — new writes use the current key, old values decrypt with their own version. In Laravel, key + previous_keys in config/app.php is exactly this; decryption tries the older keys in turn. That’s why you can rotate without migrating data.
Searching an encrypted field. Crypt::encryptString produces different ciphertext on every call (a random IV), which is correct for security but makes WHERE tax_id = ? impossible. If you need equality search you add a blind index: keep a deterministic HMAC of the searchable field in a separate column and query that. The encrypted column carries confidentiality, the blind index carries searchability; they’re separate columns because they do two separate jobs.
Losing the key is losing the data. Field encryption turns APP_KEY into an availability dependency: lose the key and the encrypted fields are permanent garbage. Key backup and access control are as much a part of the design as the encryption itself.
Outbox + idempotent consumption + field encryption isn’t cheap: it brings a table, a relay process (or a CDC connector), a dedupe table and its pruning, plus key management. If you don’t know what you’re buying in return, don’t pay for it. I don’t build it when:
\npublish is enough. You build the outbox because loss matters.The need for the pattern starts when three conditions arise together: event loss is unacceptable, production/consumption is parallel, and the data goes to a separate system (a broker). If all three aren’t present, there’s a plainer solution.
\nThe three pieces come down to one discipline: write the data to a single system, suppress repeats at the consumer, seal the sensitive field before it leaves. The outbox makes trust in the queue unnecessary, idempotent consumption makes trust in delivery unnecessary, and field encryption makes trust in the payload unnecessary — each moves a guarantee out of the code and into the design.
", "content_text": "> **TL;DR:** An outbox collapses dual-write into a single write but gives no free guarantee — it gives at-least-once. Polling vs CDC, multi-relay with SKIP LOCKED, ordering, pruning processed_messages, the exactly-once illusion, and field-level encryption.\n\nThe symptom was small: one or two invoices a day sat in the database but the consumer never saw them. The user said \"I issued it,\" and there was no trace on the other side. Always at night, always during a deploy or a network blip. Once I laid the picture out, the cause was clear — the `INSERT` into the database and the `publish` to the queue were two separate systems, with no guarantee between them.\n\nThis post is the systems side of the **transactional outbox** pattern that closed that gap, the **at-least-once** repeats it opened in return, and finally encrypting the personal data that flows through the event at the **field level**. I wrote the decision-chain side as [a log on muhammetsafak.com](/blog/from-dual-write-to-outbox-idempotent-consumption-and-field-encryption/); here I collect the decisions you have to make for the pattern to be durable in production.\n\n## Dual-write: you can't stretch a transaction across two systems\n\nThe crux in one sentence: your local database transaction doesn't cover RabbitMQ. The classic code is dangerous *because it works most of the time*:\n\n```php\nDB::transaction(function () use ($invoice) {\n $invoice->save(); // 1) write to MySQL\n $this->publishToRabbit($invoice); // 2) publish to the queue\n});\n```\n\nThere are two distinct failure modes:\n\n- `save()` succeeds, `publish` drops on a network error → the invoice is in the DB, the event isn't. **Lost event.**\n- `publish` succeeds, then the transaction `rollback`s for another reason → the event went out, no counterpart in the DB. **Phantom event.**\n\n`DB::transaction` doesn't save you, because commit/rollback only wraps the MySQL side; the broker isn't part of that transaction. The practical way to hold two systems in one atomic step is to reduce the write to **a single system**.\n\n## Outbox: write to one system first, let a separate process publish\n\nThe idea is plain: instead of pushing the message straight to the queue, write it as a row into an `outbox` table **inside the same transaction**. The invoice and the event either both exist or both don't, in the same commit — dual-write collapses into a single write.\n\n```sql\nCREATE TABLE outbox (\n id BINARY(16) NOT NULL, -- event id = idempotency key\n aggregate_id BIGINT NOT NULL, -- ordering and partition key\n topic VARCHAR(120) NOT NULL,\n payload JSON NOT NULL,\n status ENUM('pending','published') NOT NULL DEFAULT 'pending',\n created_at DATETIME(6) NOT NULL,\n published_at DATETIME(6) NULL,\n PRIMARY KEY (id),\n KEY idx_dispatch (status, created_at) -- the relay scan goes through this index\n);\n```\n\nThe write is no longer the business code's concern:\n\n```php\nDB::transaction(function () use ($invoice) {\n $invoice->save();\n Outbox::write('invoice.issued', $invoice->id, $this->payload($invoice));\n});\n```\n\nA separate **relay** does the queue push: it reads the `pending` rows, `publish`es them to RabbitMQ, and stamps them `published` on success. The critical fact here: the relay can fall into the \"I published but crashed before stamping it `published`\" state. So while the outbox solves dual-write, it gives you **no free guarantee** — it gives you **at-least-once**. The message isn't lost, but it can repeat. Everything else arranges itself around this fact.\n\n## How do you feed the relay: polling or CDC?\n\nThe relay can see `pending` rows two ways.\n\n**Polling** — the relay scans the table periodically:\n\n```sql\nSELECT id, topic, payload\nFROM outbox\nWHERE status = 'pending'\nORDER BY created_at\nLIMIT 100;\n```\n\nSimple, works on every database, low operational overhead. The cost is two things: **latency** up to the scan interval (a 1s poll = ~1s of queue latency) and an idling query. Without the `idx_dispatch` index this scan gets expensive as the table grows; even with it, you trade poll frequency against DB load. At small-to-medium volume, polling is the **right** answer — tune it frequent enough to keep latency acceptable, sparse enough not to hammer the DB (in practice 200ms–1s).\n\n> **Update · 2026-10-05** (about the paragraph above) — I later measured an index of the same shape on PostgreSQL 17. On a queue table with 10 million rows (5,000 of them `pending`), a `(status, created_at)` composite index grew to 310.4 MB while a partial index on the `pending` rows stayed at 7.6 MB — one container ([measurements](/engineering/notes/sizing-a-postgres-queue-table-by-its-live-set/)). This schema is MySQL, whose `CREATE INDEX` has no `WHERE` clause, and I have not measured it there.\n\n**CDC** (change data capture) — the relay listens to the database's **WAL/binlog** rather than the table (Debezium is the typical tool). Every `INSERT` into `outbox` turns into an event almost instantly; polling latency and idle scanning disappear. The cost is operational: binlog access, a connector process, one more moving part. I turn CDC on when latency genuinely matters (sub-second) or when volume strains polling; otherwise I count polling's simplicity as an advantage.\n\n> Rule: your latency budget and your operational budget conflict. CDC buys you latency and charges you an infrastructure piece in return. Start with polling; switch to CDC when there's a measured reason.\n\n## Multiple relays: collision-free dispatch with `SKIP LOCKED`\n\nA single relay is a bottleneck and a single point of failure. Running multiple relays over the same table creates a new risk: two of them grab the same row and publish the same message twice. The naive fix is to lock the table — which kills the parallelism.\n\nThe right tool is `SELECT ... FOR UPDATE SKIP LOCKED`. Each relay claims a batch of rows **by locking them**; instead of queuing behind rows another relay has locked, it **skips** them:\n\n```sql\nBEGIN;\n SELECT id, topic, payload\n FROM outbox\n WHERE status = 'pending'\n ORDER BY created_at\n LIMIT 100\n FOR UPDATE SKIP LOCKED; -- skip rows another relay holds, don't wait\n\n -- publish this batch, then:\n UPDATE outbox SET status = 'published', published_at = NOW(6)\n WHERE id IN (...);\nCOMMIT;\n```\n\nWithout `SKIP LOCKED`, the relays queue for the same rows and the parallelism effectively runs serially. With it, each relay pulls a disjoint set and all of them move at once. PostgreSQL and MySQL 8+ support it.\n\nA subtlety: if the relay crashes between `publish` and `UPDATE ... published`, the row stays `pending` and gets **republished**. That's acceptable — we're already at at-least-once; the fix is on the consumer side, below. The dangerous order is the reverse: stamping `published` first and then `publish`ing. A crash there produces a **lost event** — exactly what we ran from. So the order is **fixed**: `publish` first, stamp second.\n\n## Ordering: what you can guarantee, what you can't\n\n\"Let events arrive in the order they were written\" is an intuitive expectation, but global ordering is expensive and usually unnecessary. What you actually need is **per-aggregate** order: the same invoice's `created` event should arrive before its `updated`; the order of two different invoices relative to each other is nobody's concern.\n\nYou get this by partitioning on `aggregate_id`: events with the same `aggregate_id` go to the same queue partition (or, in RabbitMQ, the same queue via a consistent-hash exchange), and a **single consumer** processes that partition in order. Different aggregates flow in parallel.\n\n```\noutbox (in created_at order)\n │ hash(aggregate_id) % N\n ├── partition 0 ──► consumer-0 (events for aggregates A,D — ordered)\n ├── partition 1 ──► consumer-1 (events for aggregate B — ordered)\n └── partition 2 ──► consumer-2 (events for aggregate C — ordered)\n```\n\nTwo things still break ordering, and the design has to be ready for them: multiple relays under `SKIP LOCKED` can publish rows in a different order, and at-least-once repeats can cut in. So rather than leaning on a global order, you write the consumer to be resilient to **out-of-order and duplicated** events. The practical shield: put a monotonic `version`/`sequence` on the event and drop the stale one (lower version) at the consumer. If you genuinely need strict global ordering you drop to a single partition + single consumer — which then ties throughput to that one consumer; most systems don't want to pay that.\n\n## Once you have at-least-once: the consumer must be idempotent\n\nBecause the relay can republish, you have to assume every event arrives at least once, sometimes more. The fix isn't to make the event unique — it's to set up the **consumer** so that processing the same event twice has the same effect as once. Write the `id` of every processed event into a dedupe table, and do the work **inside the same transaction**:\n\n```php\nDB::transaction(function () use ($event) {\n $inserted = DB::table('processed_messages')->insertOrIgnore([\n 'message_id' => $event['id'],\n 'processed_at' => now(),\n ]);\n\n if ($inserted === 0) {\n return; // it's a repeat; produce no side effects\n }\n\n $this->applyBusinessEffect($event); // the real work — exactly once\n});\n```\n\nThe **unique** constraint on `processed_messages.message_id` is the heart of it: a second arrival of the same `id` is silently dropped by `insertOrIgnore`. Putting the business effect in the same transaction is mandatory — otherwise the \"I processed it but crashed before stamping\" gap opens. Where the dedupe key comes from, and designing a dedupe that survives a crash, is a topic of its own; I covered it in [a separate note](/engineering/notes/idempotency-duplicate-delivery/).\n\nIf the side effect is an external system (an e-invoice integrator, a payment API), dedupe alone isn't enough — because the side effect is outside the transaction. There you have to **carry the key to the external service**: pass the same `id` as an idempotency key to the integrator, or ask \"does this already exist?\" before sending. I worked through that boundary in detail in [the JIT reservation post](/engineering/systems/race-conditions-and-gaps-in-sequential-numbering/), where sealing the number before the external call turns recovery into a lookup.\n\n## `processed_messages` can't grow forever\n\nThe dedupe table accumulates a row per event; if you don't prune it, it becomes a performance problem of its own. The key observation: you don't need to keep an event `id` forever — only as long as the **window in which a repeat could arrive**. Relay retries and broker redeliveries are on the order of hours, not days.\n\nIn practice I pick a fixed **retention window** (say 7 days — comfortably above the longest possible redelivery) and prune older rows:\n\n```sql\nDELETE FROM processed_messages\nWHERE processed_at < NOW() - INTERVAL 7 DAY\nLIMIT 10000; -- small batches at a time, keeping replica lag and locks in check\n```\n\nIf `message_id` is a UUID the table and its index grow; pruning keeps that bounded. When choosing the window, frame the rule backwards: *how late could the latest redelivery arrive?* Make a safe upper bound of that your retention. Set the window too short and a late repeat misses the dedupe, so the side effect runs a second time.\n\n## Why exactly-once is usually an illusion\n\nIn a distributed system \"exactly-once delivery\" is tempting but can't be guaranteed end to end. The reason is the same dual-write, at the broker boundary: if the consumer did the work but crashed before getting its `ack` to the broker, the broker redelivers the message — because it never saw the `ack`. \"I processed it\" and \"I reported that I processed it\" are two separate steps, and a crash can fall between them. This is the practical face of the Two Generals problem.\n\nSo the goal isn't exactly-once **delivery**, it's exactly-once **effect**. And you get that by combining two cheap guarantees:\n\n```\nat-least-once delivery + idempotent consumer = effectively-once\n```\n\nThat is, you stop trying to deliver exactly once and instead make the **repeat harmless**. Under systems that market \"exactly-once,\" this is usually exactly what's there: at-least-once plus a dedupe layer. Look for the guarantee in the effect, not in the delivery.\n\n## Personal data flowing through the event: field-level encryption\n\nOnce the outbox settles in, a new surface appears: the events' payload carries personal data — customer name, tax ID, address — and that payload now sits in a **durable** table (`outbox`) and passes through the broker. If I leave the data as-is, I'm copying GDPR/KVKK-scoped fields, unencrypted, into more than one place.\n\nI didn't want to encrypt the whole payload — I needed fields like `topic`, `aggregate_id`, and `invoice_id` in the clear for relay routing and observability. The need is **field-level** encryption: only the sensitive fields encrypted, the rest in the clear. I marked the sensitive fields in the schema with `x-gdpr-sensitive` and encrypted only those at serialization time:\n\n```php\n$schema = [\n 'invoice_id' => ['x-gdpr-sensitive' => false],\n 'customer_name' => ['x-gdpr-sensitive' => true],\n 'tax_id' => ['x-gdpr-sensitive' => true],\n 'total' => ['x-gdpr-sensitive' => false],\n];\n\n$payload = collect($raw)->map(fn ($value, $field) =>\n ($schema[$field]['x-gdpr-sensitive'] ?? false)\n ? Crypt::encryptString((string) $value) // only the sensitive field is encrypted\n : $value\n)->all();\n```\n\nThat the schema declares the sensitive field is no accident — combine it with schema validation at the edge of the queue and both questions (\"which field is personal data, and is it the right type?\") get answered [at the edge](/engineering/notes/validate-schema-at-the-edge/). The real engineering decision isn't the encryption itself, it's the three questions around it:\n\n**Key management and rotation.** Laravel's `Crypt` uses AES-256 keyed by `APP_KEY`. Bind yourself to a single key and rotation becomes a nightmare: the new key can't open outbox rows written with the old one. The fix is to store a **key version** (key id) alongside the encrypted value and keep a keyring that recognizes several keys at once — new writes use the current key, old values decrypt with their own version. In Laravel, `key` + `previous_keys` in `config/app.php` is exactly this; decryption tries the older keys in turn. That's why you can rotate without migrating data.\n\n**Searching an encrypted field.** `Crypt::encryptString` produces different ciphertext on every call (a random IV), which is correct for security but makes `WHERE tax_id = ?` impossible. If you need equality search you add a **blind index**: keep a deterministic HMAC of the searchable field in a separate column and query that. The encrypted column carries confidentiality, the blind index carries searchability; they're separate columns because they do two separate jobs.\n\n**Losing the key is losing the data.** Field encryption turns `APP_KEY` into an **availability** dependency: lose the key and the encrypted fields are permanent garbage. Key backup and access control are as much a part of the design as the encryption itself.\n\n## When do you not build this pattern?\n\nOutbox + idempotent consumption + field encryption isn't cheap: it brings a table, a relay process (or a CDC connector), a dedupe table and its pruning, plus key management. If you don't know what you're buying in return, don't pay for it. I don't build it when:\n\n- **Event loss is tolerable.** If what you publish is a best-effort notification (cache invalidation, a \"new content available\" signal) and losing one or two doesn't matter, a direct `publish` is enough. You build the outbox because loss matters.\n- **The write and the publish are in the same system.** If your target is the same database (there's no separate broker), there's no dual-write; you don't need an outbox.\n- **Volume fits a single consumer and ordering is critical.** At low volume a single-consumer queue gives you both order and uniqueness simply; building the multi-relay-and-partition machinery is solving a problem you don't have.\n\nThe need for the pattern starts when three conditions arise **together**: event loss is unacceptable, production/consumption is parallel, and the data goes to a separate system (a broker). If all three aren't present, there's a plainer solution.\n\n---\n\nThe three pieces come down to one discipline: **write the data to a single system, suppress repeats at the consumer, seal the sensitive field before it leaves.** The outbox makes trust in the queue unnecessary, idempotent consumption makes trust in delivery unnecessary, and field encryption makes trust in the payload unnecessary — each moves a guarantee out of the code and into the design.\n\n## Sources\n\n- [Pattern: Transactional outbox](https://microservices.io/patterns/data/transactional-outbox.html) — microservices.io\n- [SELECT: the locking clause, FOR UPDATE and SKIP LOCKED](https://www.postgresql.org/docs/current/sql-select.html) — PostgreSQL\n- [InnoDB locking reads: SKIP LOCKED](https://dev.mysql.com/doc/refman/8.4/en/innodb-locking-reads.html) — MySQL\n- [Encryption: gracefully rotating encryption keys](https://laravel.com/docs/12.x/encryption) — Laravel\n\n## Frequently asked\n\n### Should the relay be fed by polling or by CDC?\n\nPolling works on every database and costs almost nothing operationally, but it adds latency up to the scan interval — in practice a 200 ms to 1 s poll — plus an idling query. CDC reads the WAL or binlog instead and that latency nearly disappears, at the price of binlog access and one more moving part. Start with polling and switch when there is a measured reason.\n\n### How long do rows have to stay in processed_messages?\n\nOnly as long as a repeat could still arrive. Relay retries and broker redeliveries are on the order of hours, not days, so a fixed retention window comfortably above the longest possible redelivery — seven days, for example — is enough, pruned in small batches. Set the window too short and a late repeat misses the dedupe, so the side effect runs a second time.", "date_published": "2026-06-22T00:00:00.000Z", "date_modified": "2026-10-05T00:00:00.000Z", "tags": [ "architecture", "messaging", "idempotency", "reliability", "security" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/systems/gpu-finops-with-ebpf/", "url": "https://muhammetsafak.com/engineering/systems/gpu-finops-with-ebpf/", "title": "GPU FinOps with eBPF", "summary": "You are billed for a whole GPU and you use 3% of it — and your monitoring cannot tell you whose 3% it was. eBPF attributes GPU work to a pod by watching the syscall boundary; DCGM measures how well the silicon ran. They answer different questions, and a real chargeback needs both.", "content_html": "A GPU node costs more per hour than a rack of CPU machines, and the question finance asks about it is the same question they ask about everything: who used this, and how much? For CPU, memory, and disk you can answer that — cgroups account it per process, cAdvisor reports it per pod, Prometheus rolls it up per team. For the GPU, the honest answer in most clusters is that nobody knows. You are paying for the most expensive resource in the fleet and you cannot attribute a dollar of it.
\nThis is not a tooling gap you forgot to fill. It is structural: the GPU is outside the accounting plane the rest of your stack relies on. This piece is about why, and about using eBPF to get attribution back — along with a hard line about what eBPF can and cannot see, because the most expensive mistake here is believing one tool answers a question it physically cannot.
\nThree separate blindnesses stack up.
\ncgroups don’t account the GPU. A cgroup is a kernel construct that accounts and limits CPU, memory, block I/O, and PIDs. The GPU is none of those. Compute and framebuffer (VRAM) are managed by the NVIDIA kernel driver and the userspace CUDA runtime, and are opaque to cgroup accounting. There is a device cgroup controller, but it only gates access to the device nodes (/dev/nvidia0 and friends) — allow or deny. It does not account a single second of GPU time or a megabyte of VRAM. So the entire cAdvisor → Prometheus pipeline that gives you per-pod CPU and memory has, by construction, nothing to say about the GPU.
nvidia.com/gpu is an allocation count, not a utilization signal. Kubernetes learns about GPUs through the NVIDIA device plugin, which registers them to the kubelet as the extended resource nvidia.com/gpu. Pods request it as an integer — whole GPUs. The scheduler matches requested count to available count. That is the only thing the allocation layer knows. Requesting a GPU is not using a GPU. A pod can hold a whole device — and be billed for the whole device — while driving it at 3%. The allocation number you could charge back against is precisely the number that tells you nothing about consumption.
Even nvidia-smi’s utilization number lies to you. Reach for nvidia-smi and the utilization.gpu field looks like salvation. It is not what you think. NVML defines it as the percent of time over the sample period during which one or more kernels was executing. It measures temporal presence — was the GPU busy at all — not how much of it was busy. A single-thread kernel occupying one SM out of dozens can report close to 100%. Microsoft has reported under 10% compute utilization during the memory-bound decode phase of serving an 8B-parameter model on A100s — while a naive reading of “utilization” would call those GPUs full. If you charge back on utilization.gpu, you are billing on a number that says “busy” when the silicon is nearly idle.
So: the allocation layer knows count-not-use, cgroups know nothing, and the one easy percentage is measuring the wrong thing. That is the gap.
\nBefore any tooling, separate the two questions, because conflating them is the core error — the same shape as confusing the problem a tool solves with the problem at hand.
\nThese map onto two different measurement planes, and the rule that organizes everything below is: eBPF answers attribution; the GPU’s own counters answer efficiency. Neither substitutes for the other.
\neBPF runs in the kernel and attaches to kprobes, uprobes, tracepoints, and syscalls. There is no GPU-utilization tracepoint to read — but there are two surfaces where GPU work is requested, and both are visible from the kernel:
\nioctl() to /dev/nvidiactl and /dev/nvidia0…N. A kprobe on the driver’s entry points (nvidia_unlocked_ioctl, nvidia_open) sees that traffic. (The closed driver historically exposes only one tracepoint, nvidia:nvidia_dev_xid, for hardware error events; everything else is kprobed.)libcuda.so / libcudart.so traces the API itself: cuLaunchKernel, cuMemAlloc, cuMemcpyHtoD, cuStreamSynchronize, and friends. Pairing an entry uprobe with a return uretprobe measures each call’s duration.The reason this is attribution is that at every hook point eBPF reads the calling PID/TGID and cgroup natively — bpf_get_current_pid_tgid and the cgroup id are right there. That gives you the chain PID → cgroup → pod → namespace → team with no application changes and no cooperation from NVIDIA. Per pod you get: kernel-launch counts, launch dimensions, VRAM allocation sizes, memcpy volume and direction, and call timing. That is a real, defensible chargeback signal derived entirely from the kernel side of the boundary.
The ecosystem here is young but real. The primitives work today — there are working write-ups and tutorials uprobing the CUDA libraries and kprobing the nvidia ioctl path, and bpftime explores tying eBPF logic to GPU events. Tetragon (Cilium) ships generic process_uprobe, process_kprobe, and ioctl tracing with Kubernetes pod identity, so it can be pointed at the CUDA symbols or the ioctl path — but understand that this is a TracingPolicy you author, not a shipped “GPU FinOps” feature. There is no dominant turnkey eBPF chargeback product yet. Anyone selling you “drop-in eBPF GPU FinOps” is selling further than the ecosystem currently reaches.
This is the honest limit, and it is not a maturity problem that will be fixed in a release — it is physics of where the data lives.
\neBPF sees the control plane: API calls, ioctls, allocation sizes, launch counts, and submit-to-complete timing where you can trace it. It does not see inside the GPU. It cannot read SM occupancy, tensor-core utilization, or achieved memory bandwidth, because those are hardware performance counters that live on the device and are exposed only through NVML / DCGM / CUPTI. eBPF has no path to them. It can tell you a pod launched ten thousand kernels and allocated 40 GB of VRAM; it cannot tell you whether those kernels saturated the SMs or left them 90% idle.
\nThere is a second, subtler limit. Even at the ioctl boundary eBPF can hook, the payloads are largely opaque. The driver’s command structures (the NV_ESC_* Resource Manager API) are complex and effectively proprietary. You can see that an ioctl happened — its command number, the calling PID, the timing — but decoding the semantic content of an arbitrary RM payload is impractical and fragile. You get the fact of the work and its attribution, not a free reading of its meaning.
A note that closes a tempting door: NVIDIA’s open kernel modules (open-gpu-kernel-modules, Turing and newer) open the kernel interface layer — the module init, the ioctl entry points, the NV_ESC_* command surface. That genuinely helps you understand what to hook. But the GPU’s brain stays closed: on Turing+ much of the management runs on on-GPU GSP firmware, and that firmware — along with the user-mode driver components the modules require — is still shipped closed. Opening the kernel modules does not expose hardware performance counters to eBPF. The on-device limit is unchanged. If someone claims the open modules let eBPF read SM occupancy, they are wrong on current evidence.
Because eBPF cannot see the silicon, the efficiency half of the answer comes from userspace, and the standard tool is NVIDIA DCGM (Data Center GPU Manager) with dcgm-exporter for Prometheus. DCGM reads the GPU’s hardware counters through the driver and exposes the fields eBPF can’t reach:
\nDCGM_FI_PROF_SM_ACTIVE — fraction of time at least one warp was active on a multiprocessor, averaged across all of them.DCGM_FI_PROF_SM_OCCUPANCY — fraction of resident warps relative to the maximum supported: true occupancy.DCGM_FI_PROF_PIPE_TENSOR_ACTIVE — fraction of cycles the tensor pipe was active.DCGM_FI_PROF_DRAM_ACTIVE — a memory-bandwidth proxy.DCGM_FI_DEV_FB_USED — framebuffer (VRAM) actually in use.These are the numbers that tell you whether a team’s expensive allocation was earning its keep. DCGM attributes them to pods through the kubelet Pod Resources API (a gRPC service at /var/lib/kubelet/pod-resources), which maps each GPU UUID to the pod that holds it.
But DCGM’s attribution has a hard edge that is exactly where eBPF earns its place: DCGM produces device-level metrics, so it cannot disambiguate consumers sharing one physical GPU. Under time-slicing or MPS (Multi-Process Service), all pods sharing a GPU receive identical, duplicated device-level values — including DCGM_FI_DEV_FB_USED. NVIDIA’s own exporter documents that it does not associate metrics to containers when time-slicing is enabled; with --kubernetes-virtual-gpus=true, every sharing pod mirrors the whole physical GPU’s state. Under MIG, attribution shifts to the GPU-instance level, not arbitrary pods. So precisely in the shared-GPU case — the case that exists because whole-GPU allocation wastes money — DCGM cannot tell you who consumed what. eBPF’s PID-level tracing can. The two tools are complementary at exactly the seam where each is weakest.
(For the deepest efficiency profiling there is CUPTI, the CUDA Profiling Tools Interface, which can read counters DCGM doesn’t surface — but metric-replay profiling like Nsight Compute re-runs kernels multiple times and carries heavy overhead. It is a profiling-session tool, not always-on per-tenant telemetry. DCGM is the lower-overhead, sampling-based continuous path; CUPTI is the heavy, deep one.)
\n| Plane | Tool | Sees | Cannot see |
|---|---|---|---|
| Kernel (control) | eBPF | PID→pod attribution, launch counts, alloc sizes, memcpy volume, call timing — including per-pod under time-slicing | on-device SM/tensor occupancy, memory bandwidth, ioctl payload internals |
| Userspace (device) | DCGM | true SM occupancy, tensor activity, DRAM activity, VRAM used — from hardware counters | per-pod attribution under shared GPU (time-slicing/MPS): all sharers get identical values |
eBPF is low-overhead, needs no app changes, needs no vendor cooperation, and attributes natively by PID/cgroup — it is the right tool for who and how much. DCGM reads the silicon — it is the right tool for how well. The costs are honest too: uprobes carry real overhead and the CUDA symbols are versioned (cuMemAlloc@CUDA_11.0 vs @CUDA_12.0), so probes need dynamic symbol resolution and break across driver upgrades; CUDA API calls can fire ten-thousand-plus times a second, so handlers must stay cheap or they slow the very workload they measure. DCGM is userspace polling with its own sampling overhead and a hardware constraint that only certain counter groups can be read together.
The wrong turn is picking one tool and asking it the other tool’s question — billing teams on DCGM’s device-level numbers and silently overcharging everyone who shares a GPU, or trusting eBPF’s launch counts as a proxy for efficiency and concluding a busy-looking workload was well-utilized. The order that actually works:
\n1. eBPF: attribute GPU work to pod/team (who, how much — the bill)\n2. DCGM: measure on-device efficiency (how well — was it justified)\n3. Join them on pod identity (the team's spend AND its efficiency)Attribution first, because that is the question finance actually asked and the one your existing stack cannot answer at all. Efficiency second, because it turns the bill into a decision — a team holding an expensive allocation at 4% occupancy is paying a just-in-case capacity tax you can now see, where before the GPU was a flat line item nobody could open. Joined on pod identity, you finally have what you have for every other resource: spend, attributed, with the efficiency context to act on it. The reason it took two planes is the same reason it was invisible — the silicon never lived in the accounting plane, and no single probe spans the boundary.
\nSee also: the DCGM exporter docs cover the pod-attribution path and the DCGM feature overview defines the profiling fields, and the eBPF primitives this leans on are documented at ebpf.io and in Tetragon.
", "content_text": "> **TL;DR:** You are billed for a whole GPU and you use 3% of it — and your monitoring cannot tell you whose 3% it was. eBPF attributes GPU work to a pod by watching the syscall boundary; DCGM measures how well the silicon ran. They answer different questions, and a real chargeback needs both.\n\nA GPU node costs more per hour than a rack of CPU machines, and the question finance asks about it is the same question they ask about everything: **who used this, and how much?** For CPU, memory, and disk you can answer that — cgroups account it per process, cAdvisor reports it per pod, Prometheus rolls it up per team. For the GPU, the honest answer in most clusters is that nobody knows. You are paying for the most expensive resource in the fleet and you cannot attribute a dollar of it.\n\nThis is not a tooling gap you forgot to fill. It is structural: the GPU is **outside the accounting plane the rest of your stack relies on.** This piece is about why, and about using eBPF to get attribution back — along with a hard line about what eBPF can and cannot see, because the most expensive mistake here is believing one tool answers a question it physically cannot.\n\n## Why is the GPU invisible to your existing metrics?\n\nThree separate blindnesses stack up.\n\n**cgroups don't account the GPU.** A cgroup is a kernel construct that accounts and limits CPU, memory, block I/O, and PIDs. The GPU is none of those. Compute and framebuffer (VRAM) are managed by the NVIDIA kernel driver and the userspace CUDA runtime, and are opaque to cgroup accounting. There is a `device` cgroup controller, but it only *gates access* to the device nodes (`/dev/nvidia0` and friends) — allow or deny. It does not account a single second of GPU time or a megabyte of VRAM. So the entire cAdvisor → Prometheus pipeline that gives you per-pod CPU and memory has, by construction, **nothing to say about the GPU.**\n\n**`nvidia.com/gpu` is an allocation count, not a utilization signal.** Kubernetes learns about GPUs through the NVIDIA device plugin, which registers them to the kubelet as the extended resource `nvidia.com/gpu`. Pods request it as an integer — whole GPUs. The scheduler matches requested *count* to available *count*. That is the only thing the allocation layer knows. **Requesting a GPU is not using a GPU.** A pod can hold a whole device — and be billed for the whole device — while driving it at 3%. The allocation number you could charge back against is precisely the number that tells you nothing about consumption.\n\n**Even `nvidia-smi`'s utilization number lies to you.** Reach for `nvidia-smi` and the `utilization.gpu` field looks like salvation. It is not what you think. NVML defines it as *the percent of time over the sample period during which one or more kernels was executing.* It measures **temporal presence — was the GPU busy at all — not how much of it was busy.** A single-thread kernel occupying one SM out of dozens can report close to 100%. Microsoft has reported under 10% compute utilization during the memory-bound decode phase of serving an 8B-parameter model on A100s — while a naive reading of \"utilization\" would call those GPUs full. If you charge back on `utilization.gpu`, you are billing on a number that says \"busy\" when the silicon is nearly idle.\n\nSo: the allocation layer knows count-not-use, cgroups know nothing, and the one easy percentage is measuring the wrong thing. That is the gap.\n\n## Two questions, and why they need different tools\n\nBefore any tooling, separate the two questions, because conflating them is the core error — the same shape as [confusing the problem a tool solves with the problem at hand](/engineering/systems/data-intensive-systems-breaking-points/).\n\n- **Attribution — *who* did GPU work, and *how much*?** This is the FinOps question. It needs per-pod, per-team accounting of GPU activity: which workload launched kernels, allocated VRAM, moved data, and for how long. This is what chargeback runs on.\n- **Efficiency — *how well* was the silicon used?** Were the SMs actually occupied, were the tensor cores active, was memory bandwidth the bottleneck? This is the performance question. It tells you whether a team's spend was *justified*, but it cannot tell you *whose* spend it was.\n\nThese map onto two different measurement planes, and the rule that organizes everything below is: **eBPF answers attribution; the GPU's own counters answer efficiency. Neither substitutes for the other.**\n\n## What eBPF can see: the control plane\n\neBPF runs in the kernel and attaches to kprobes, uprobes, tracepoints, and syscalls. There is no GPU-utilization tracepoint to read — but there are two surfaces where GPU *work is requested*, and both are visible from the kernel:\n\n- **The driver ioctl boundary.** Every piece of GPU work — kernel submission, memory allocation, synchronization — ultimately becomes an `ioctl()` to `/dev/nvidiactl` and `/dev/nvidia0…N`. A kprobe on the driver's entry points (`nvidia_unlocked_ioctl`, `nvidia_open`) sees that traffic. (The closed driver historically exposes only one tracepoint, `nvidia:nvidia_dev_xid`, for hardware error events; everything else is kprobed.)\n- **The CUDA library boundary.** A uprobe on `libcuda.so` / `libcudart.so` traces the API itself: `cuLaunchKernel`, `cuMemAlloc`, `cuMemcpyHtoD`, `cuStreamSynchronize`, and friends. Pairing an entry uprobe with a return uretprobe measures each call's duration.\n\nThe reason this is *attribution* is that at every hook point eBPF reads the calling PID/TGID and cgroup natively — `bpf_get_current_pid_tgid` and the cgroup id are right there. That gives you the chain **PID → cgroup → pod → namespace → team** with no application changes and no cooperation from NVIDIA. Per pod you get: kernel-launch counts, launch dimensions, VRAM allocation sizes, memcpy volume and direction, and call timing. That is a real, defensible chargeback signal derived entirely from the kernel side of the boundary.\n\nThe ecosystem here is young but real. The primitives work today — there are working write-ups and tutorials uprobing the CUDA libraries and kprobing the nvidia ioctl path, and `bpftime` explores tying eBPF logic to GPU events. Tetragon (Cilium) ships generic `process_uprobe`, `process_kprobe`, and ioctl tracing with Kubernetes pod identity, so it *can* be pointed at the CUDA symbols or the ioctl path — but understand that this is a `TracingPolicy` you author, not a shipped \"GPU FinOps\" feature. There is no dominant turnkey eBPF chargeback product yet. Anyone selling you \"drop-in eBPF GPU FinOps\" is selling further than the ecosystem currently reaches.\n\n## What eBPF cannot see: the silicon\n\nThis is the honest limit, and it is not a maturity problem that will be fixed in a release — it is physics of where the data lives.\n\neBPF sees the **control plane**: API calls, ioctls, allocation sizes, launch counts, and submit-to-complete timing where you can trace it. It does **not** see inside the GPU. It cannot read SM occupancy, tensor-core utilization, or achieved memory bandwidth, because those are hardware performance counters that live *on the device* and are exposed only through NVML / DCGM / CUPTI. eBPF has no path to them. It can tell you a pod launched ten thousand kernels and allocated 40 GB of VRAM; it cannot tell you whether those kernels saturated the SMs or left them 90% idle.\n\nThere is a second, subtler limit. Even at the ioctl boundary eBPF can hook, the *payloads* are largely opaque. The driver's command structures (the `NV_ESC_*` Resource Manager API) are complex and effectively proprietary. You can see *that* an ioctl happened — its command number, the calling PID, the timing — but decoding the semantic content of an arbitrary RM payload is impractical and fragile. You get the fact of the work and its attribution, not a free reading of its meaning.\n\nA note that closes a tempting door: NVIDIA's open kernel modules (`open-gpu-kernel-modules`, Turing and newer) open the **kernel interface layer** — the module init, the ioctl entry points, the `NV_ESC_*` command surface. That genuinely helps you understand *what* to hook. But the GPU's brain stays closed: on Turing+ much of the management runs on on-GPU GSP firmware, and that firmware — along with the user-mode driver components the modules require — is still shipped closed. Opening the kernel modules **does not** expose hardware performance counters to eBPF. The on-device limit is unchanged. If someone claims the open modules let eBPF read SM occupancy, they are wrong on current evidence.\n\n## The userspace path you still need: DCGM\n\nBecause eBPF cannot see the silicon, the efficiency half of the answer comes from userspace, and the standard tool is NVIDIA **DCGM** (Data Center GPU Manager) with **dcgm-exporter** for Prometheus. DCGM reads the GPU's hardware counters through the driver and exposes the fields eBPF can't reach:\n\n- `DCGM_FI_PROF_SM_ACTIVE` — fraction of time at least one warp was active on a multiprocessor, averaged across all of them.\n- `DCGM_FI_PROF_SM_OCCUPANCY` — fraction of resident warps relative to the maximum supported: *true* occupancy.\n- `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` — fraction of cycles the tensor pipe was active.\n- `DCGM_FI_PROF_DRAM_ACTIVE` — a memory-bandwidth proxy.\n- `DCGM_FI_DEV_FB_USED` — framebuffer (VRAM) actually in use.\n\nThese are the numbers that tell you whether a team's expensive allocation was earning its keep. DCGM attributes them to pods through the kubelet **Pod Resources API** (a gRPC service at `/var/lib/kubelet/pod-resources`), which maps each GPU UUID to the pod that holds it.\n\nBut DCGM's attribution has a hard edge that is exactly where eBPF earns its place: **DCGM produces device-level metrics, so it cannot disambiguate consumers sharing one physical GPU.** Under time-slicing or MPS (Multi-Process Service), all pods sharing a GPU receive *identical, duplicated* device-level values — including `DCGM_FI_DEV_FB_USED`. NVIDIA's own exporter documents that it does not associate metrics to containers when time-slicing is enabled; with `--kubernetes-virtual-gpus=true`, every sharing pod mirrors the whole physical GPU's state. Under MIG, attribution shifts to the GPU-instance level, not arbitrary pods. So precisely in the shared-GPU case — the case that exists *because* whole-GPU allocation wastes money — DCGM cannot tell you who consumed what. eBPF's PID-level tracing can. The two tools are complementary at exactly the seam where each is weakest.\n\n(For the deepest efficiency profiling there is **CUPTI**, the CUDA Profiling Tools Interface, which can read counters DCGM doesn't surface — but metric-replay profiling like Nsight Compute re-runs kernels multiple times and carries heavy overhead. It is a profiling-session tool, not always-on per-tenant telemetry. DCGM is the lower-overhead, sampling-based continuous path; CUPTI is the heavy, deep one.)\n\n## The kernel-vs-userspace tradeoff, stated plainly\n\n| Plane | Tool | Sees | Cannot see |\n| --- | --- | --- | --- |\n| Kernel (control) | eBPF | PID→pod attribution, launch counts, alloc sizes, memcpy volume, call timing — *including* per-pod under time-slicing | on-device SM/tensor occupancy, memory bandwidth, ioctl payload internals |\n| Userspace (device) | DCGM | true SM occupancy, tensor activity, DRAM activity, VRAM used — from hardware counters | per-pod attribution under shared GPU (time-slicing/MPS): all sharers get identical values |\n\neBPF is low-overhead, needs no app changes, needs no vendor cooperation, and attributes natively by PID/cgroup — it is the right tool for *who and how much*. DCGM reads the silicon — it is the right tool for *how well*. The costs are honest too: uprobes carry real overhead and the CUDA symbols are **versioned** (`cuMemAlloc@CUDA_11.0` vs `@CUDA_12.0`), so probes need dynamic symbol resolution and break across driver upgrades; CUDA API calls can fire ten-thousand-plus times a second, so handlers must stay cheap or they slow the very workload they measure. DCGM is userspace polling with its own sampling overhead and a hardware constraint that only certain counter groups can be read together.\n\n## Why you need both, in order\n\nThe wrong turn is picking one tool and asking it the other tool's question — billing teams on DCGM's device-level numbers and silently overcharging everyone who shares a GPU, or trusting eBPF's launch counts as a proxy for efficiency and concluding a busy-looking workload was well-utilized. The order that actually works:\n\n```\n1. eBPF: attribute GPU work to pod/team (who, how much — the bill)\n2. DCGM: measure on-device efficiency (how well — was it justified)\n3. Join them on pod identity (the team's spend AND its efficiency)\n```\n\nAttribution first, because that is the question finance actually asked and the one your existing stack cannot answer at all. Efficiency second, because it turns the bill into a decision — a team holding an expensive allocation at 4% occupancy is paying a [just-in-case capacity tax](/engineering/essays/the-cost-of-just-in-case-code/) you can now *see*, where before the GPU was a flat line item nobody could open. Joined on pod identity, you finally have what you have for every other resource: spend, attributed, with the efficiency context to act on it. The reason it took two planes is the same reason it was invisible — the silicon never lived in the accounting plane, and no single probe spans the boundary.\n\n---\n\n*See also:* the [DCGM exporter docs](https://docs.nvidia.com/datacenter/dcgm/latest/installation/install-dcgm-exporter.html) cover the pod-attribution path and the [DCGM feature overview](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html) defines the profiling fields, and the eBPF primitives this leans on are documented at [ebpf.io](https://ebpf.io/what-is-ebpf/) and in [Tetragon](https://tetragon.io/).\n\n## Steps\n\n**Setting up GPU chargeback** — Attributing GPU spend to teams and measuring whether that spend was justified.\n\n1. **Attribute with eBPF** — Watch GPU work at the syscall boundary and attribute it to a pod and a team. This answers \"who, how much\" — the bill.\n2. **Measure efficiency with DCGM** — Measure on-device efficiency from userspace: SM utilisation, memory, power. This answers \"how well did it run\".\n3. **Join on pod identity** — Join the two measurements on pod identity, so a team’s spend and its efficiency sit in one place.\n\n## Sources\n\n- [NVML: nvmlUtilization_t.gpu is the percent of time one or more kernels was executing](https://docs.nvidia.com/deploy/nvml-api/structnvmlUtilization__t.html) — NVIDIA\n- [DCGM feature overview: SM active, SM occupancy, tensor and DRAM activity, and the counter grouping limit](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html) — NVIDIA\n- [GPU Operator time-slicing: DCGM-Exporter does not associate metrics to containers when time-slicing is enabled](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html) — NVIDIA\n- [Kubernetes device plugins: extended resources are integer-only and devices cannot be shared between containers](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) — Kubernetes\n\n## Frequently asked\n\n### Why can cgroups and cAdvisor not attribute GPU usage?\n\nA cgroup accounts CPU, memory, block I/O and PIDs. GPU compute and framebuffer are managed by the NVIDIA kernel driver and the userspace CUDA runtime and are opaque to it. The device cgroup controller only gates access to the device nodes; it accounts no GPU time at all, so the cAdvisor to Prometheus pipeline has nothing to report.\n\n### Is the utilization.gpu number from nvidia-smi usable for chargeback?\n\nNo. NVML defines it as the percent of time over the sample period during which one or more kernels was executing. That is temporal presence, not how much of the device was busy — a single-thread kernel occupying one multiprocessor can report close to 100 percent. Billing on it means billing for silicon that is nearly idle.\n\n### If DCGM reads the real hardware counters, why is eBPF still needed?\n\nDCGM produces device-level metrics, so it cannot disambiguate consumers sharing one physical GPU. Under time-slicing or MPS every sharing pod receives identical duplicated values, and NVIDIA documents that its own exporter does not associate metrics to containers when time-slicing is enabled. eBPF attributes per PID, which covers exactly the case DCGM cannot.", "date_published": "2026-06-20T00:00:00.000Z", "date_modified": "2026-09-06T00:00:00.000Z", "tags": [ "ebpf", "gpu", "finops", "observability", "kubernetes" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/notes/when-event-driven-architecture/", "url": "https://muhammetsafak.com/engineering/notes/when-event-driven-architecture/", "title": "Event-Driven Architecture: What It Solves and Hides", "summary": "Events buy loose coupling by letting the producer stay unaware of its consumers, and the same move makes the flow invisible: no compiler verifies an event contract the way it verifies a method call. Two costs are paid up front or not at all — versioned, schema-validated payloads, and a correlation id carried into every log and every downstream event. A single-listener event is a method call in disguise.", "content_html": "I once watched a team where no one could give a straight answer to “what happens when an order is created?” In the codebase, eight separate listeners were subscribed to the OrderCreated event; you couldn’t read from any single place which one ran in which order, under which condition. The flow didn’t live in the code — it lived in people’s heads, and incompletely at that.
Event-driven architecture loosens coupling. This post is a reminder that something else loosens along with it: causality.
\nCode that publishes an event doesn’t know who listens to it. OrderCreated is published; billing, notifications, and analytics each listen to it separately. The order code is unaware of all three.
The benefit is real: adding a new consumer happens without touching the producer. Independent development, independent deployment, fan-out. Parts that don’t know each other.
\nThe flip side of the same coin: as coupling loosens, the flow becomes invisible.
\nIn a direct method call, the flow is the code — you go to the definition and read on. Not so with an event. To see what happens when OrderCreated is published, you have to find every listener by hand. The compiler won’t help you: a compiler verifies the contract of a method call, but no one verifies the contract of an event.
The line from the modular monolith post applies here too — like a network contract, an event contract sits in the compiler’s blind spot.
\nBefore moving to an event-driven architecture, you pay two costs up front.
\nSchema discipline. An event is a contract. When the payload of OrderCreated changes, the eight places listening to it won’t know — not until it breaks in production. Events must be versioned, fields must be added in a backward-compatible way, payloads must be validated against a schema. Without this discipline, loose coupling means “silent breakage.”
Traceability. To be able to see a request travel through the system, every event must carry a correlation ID:
\nevent(new OrderCreated(\n orderId: $order->id,\n correlationId: request()->header('X-Correlation-Id') ?? (string) Str::uuid(),\n));If this ID isn’t carried into the logs and into subsequent events, then in a multi-listener flow there is no answer to “why did this job run?” Tracing isn’t an optional add-on to event-driven; it’s a prerequisite.
\nLaravel’s event()/listener mechanism, when it runs synchronously, is really an organized method call — same process, same transaction, same stack trace. This is safe and traceable; use it freely.
The difficulties event-driven architecture brings start when an event crosses a boundary: into a queue, into a message broker, into another process. The moment you add ShouldQueue to a listener and send the event to RabbitMQ, the schema and tracing debt kicks in. Don’t conflate the two: using in-process listeners does not move you into “event-driven architecture.”
Actually carrying an event across a boundary is justified in these cases:
\nWithout these, a direct service call is both more readable and safer. An event with a single listener is just a method call in disguise — and a harder one to trace, at that.
\nEvent-driven architecture loosens coupling; in doing so, it loosens causality too. If you want the first, pay the bill for the second — schema and tracing — up front.
\nLoose coupling isn’t free; you pay for it with invisible flow.
", "content_text": "> **TL;DR:** Events buy loose coupling by letting the producer stay unaware of its consumers, and the same move makes the flow invisible: no compiler verifies an event contract the way it verifies a method call. Two costs are paid up front or not at all — versioned, schema-validated payloads, and a correlation id carried into every log and every downstream event. A single-listener event is a method call in disguise.\n\nI once watched a team where no one could give a straight answer to \"what happens when an order is created?\" In the codebase, eight separate listeners were subscribed to the `OrderCreated` event; you couldn't read from any single place which one ran in which order, under which condition. The flow didn't live in the code — it lived in people's heads, and incompletely at that.\n\nEvent-driven architecture loosens coupling. This post is a reminder that something else loosens along with it: causality.\n\n## What do events solve?\n\nCode that publishes an event doesn't know who listens to it. `OrderCreated` is published; billing, notifications, and analytics each listen to it separately. The order code is unaware of all three.\n\nThe benefit is real: adding a new consumer happens without touching the producer. Independent development, independent deployment, fan-out. Parts that don't know each other.\n\n## What do events hide?\n\nThe flip side of the same coin: as coupling loosens, **the flow becomes invisible**.\n\nIn a direct method call, the flow *is* the code — you go to the definition and read on. Not so with an event. To see what happens when `OrderCreated` is published, you have to find every listener by hand. The compiler won't help you: a compiler verifies the contract of a method call, but no one verifies the contract of an event.\n\nThe line from [the modular monolith post](/engineering/essays/why-i-start-with-a-modular-monolith/) applies here too — like a network contract, an event contract sits in the compiler's blind spot.\n\n## Don't choose it before paying: two prerequisites\n\nBefore moving to an event-driven architecture, you pay two costs up front.\n\n**Schema discipline.** An event is a contract. When the payload of `OrderCreated` changes, the eight places listening to it won't know — not until it breaks in production. Events must be versioned, fields must be added in a backward-compatible way, [payloads must be validated against a schema](/engineering/notes/validate-schema-at-the-edge/). Without this discipline, loose coupling means \"silent breakage.\"\n\n**Traceability.** To be able to see a request travel through the system, [every event must carry a correlation ID](/engineering/systems/polyglot-queue-distributed-tracing/):\n\n```php\nevent(new OrderCreated(\n orderId: $order->id,\n correlationId: request()->header('X-Correlation-Id') ?? (string) Str::uuid(),\n));\n```\n\nIf this ID isn't carried into the logs and into subsequent events, then in a multi-listener flow there is no answer to \"why did this job run?\" Tracing isn't an optional add-on to event-driven; it's a prerequisite.\n\n## In Laravel: in-process events ≠ event-driven architecture\n\nLaravel's `event()`/listener mechanism, when it runs synchronously, is really an organized method call — same process, same transaction, same stack trace. This is safe and traceable; use it freely.\n\nThe difficulties event-driven architecture brings start when an event crosses a **boundary**: into a queue, into a message broker, into another process. The moment you add `ShouldQueue` to a listener and send the event to RabbitMQ, the schema and tracing debt kicks in. Don't conflate the two: using in-process listeners does not move you into \"event-driven architecture.\"\n\n## When should you move to event-driven?\n\nActually carrying an event across a boundary is justified in these cases:\n\n- **There's a real fan-out.** A single event is listened to by many independent consumers that shouldn't know about each other.\n- **The consumer must be asynchronous.** The work shouldn't keep the producer waiting for a response; \"accepted\" is enough.\n- **The consumers must scale or be distributed independently.** This also brings us to [the microservices decision](/engineering/essays/when-i-move-to-microservices/) — the same measured-signal threshold.\n\nWithout these, a direct service call is both more readable and safer. An event with a single listener is just a method call in disguise — and a harder one to trace, at that.\n\n---\n\nEvent-driven architecture loosens coupling; in doing so, it loosens causality too. If you want the first, pay the bill for the second — schema and tracing — up front.\n\nLoose coupling isn't free; you pay for it with invisible flow.\n\n## Sources\n\n- [Enterprise Integration Patterns: Publish-Subscribe Channel](https://www.enterpriseintegrationpatterns.com/patterns/messaging/PublishSubscribeChannel.html) — Hohpe and Woolf\n- [Enterprise Integration Patterns: Correlation Identifier](https://www.enterpriseintegrationpatterns.com/patterns/messaging/CorrelationIdentifier.html) — Hohpe and Woolf\n- [Events: queued event listeners and ShouldQueue](https://laravel.com/docs/12.x/events) — Laravel\n\n## Frequently asked\n\n### Does using in-process Laravel events mean you have an event-driven architecture?\n\nNo. A synchronous listener is an organized method call — same process, same transaction, same stack trace — and it is safe and traceable. The difficulties start when the event crosses a boundary into a queue, a broker or another process; that is where the schema and tracing debt kicks in.\n\n### What has to be paid for before moving to event-driven?\n\nTwo things, up front. Schema discipline: an event is a contract, so it must be versioned, extended in a backward-compatible way and validated against a schema, or loose coupling just means silent breakage. And traceability: every event carries a correlation id into the logs and into the events it triggers, otherwise there is no answer to why a job ran.", "date_published": "2026-06-20T00:00:00.000Z", "date_modified": "2026-09-06T00:00:00.000Z", "tags": [ "architecture", "event-driven", "messaging" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://muhammetsafak.com" } ] }, { "id": "https://muhammetsafak.com/engineering/systems/zero-static-authority-multi-cluster-gitops/", "url": "https://muhammetsafak.com/engineering/systems/zero-static-authority-multi-cluster-gitops/", "title": "Zero Static Authority in Multi-Cluster GitOps", "summary": "A stored kubeconfig is a long-lived credential pointed at your whole fleet — the exact thing an attacker wants. Replace it with a workload identity that expires in an hour and is never written to disk. Here is the rule, the order, and the new control plane you take on in return.", "content_html": "A GitOps controller — Argo CD, Flux — sits in one cluster and reconciles many. To reach the others it needs to authenticate to each remote kube-apiserver. The default way to do that is to store a credential: a bearer token, a client certificate, a kubeconfig, kept as a Kubernetes Secret in the control cluster. In Argo CD these are the objects labelled argocd.argoproj.io/secret-type: cluster.
Stop and look at what that Secret is. It is a long-lived credential, usually with broad rights, pointing at your entire fleet, sitting at rest in one place. It does not rotate on its own. It is in your backups. It is the single most valuable thing an attacker can find in your control cluster, because owning it is owning every cluster it can reach. The whole discipline of boring architecture is about not keeping liabilities you don’t have to — and a stored fleet-wide token is a liability you keep purely because issuing a fresh, short-lived one at the moment of use seemed like more work.
This piece is about doing that more work, in order. The target is zero static authority: no long-lived credential that authenticates the controller to a remote cluster is ever written down. The identity the controller presents is minted on demand, expires in about an hour, and lives only in memory. The mechanism is SPIFFE for the identity model and SPIRE for issuing it.
\nBefore any of it, this is not a tool for three clusters you could count on one hand. SPIRE is a control plane you operate, not a library you import — it is exactly the kind of capability you should be slow to take on, the way modular monolith beats microservices until you’ve earned the split. It earns its weight only when you genuinely run a fleet — many clusters, many teams, an audit requirement that a stolen Secret would fail. If a single shared cluster covers you, the cheapest secure credential is the one you never stand up the machinery to issue. The rest of this assumes you have actually arrived at the fleet.
State it before the mechanics: the credential that crosses a cluster boundary must be short-lived and never stored. Everything below is in service of that one sentence. If at the end you still have a long-lived token written to disk somewhere on the authentication path, you have built complexity without buying the property you came for — the same wrong turn as adding a read replica to dodge a missing index.
\n“Short-lived and never stored” has a precise meaning here. The identity is an X.509 certificate whose default lifetime is one hour — or a JWT, whose default is five minutes — fetched by the workload from a local socket with no token of its own to present, and rotated automatically at half its life. There is no secret to steal that is worth stealing tomorrow.
\nSPIFFE is a spec; SPIRE is the reference implementation that issues the things the spec describes. Four nouns carry the whole design.
\nA SPIFFE ID is a URI: spiffe://<trust-domain>/<path>, e.g. spiffe://prod.example.org/ns/argocd/sa/application-controller. The authority part is the trust domain — one logical root of trust, one CA. The path identifies the workload. That is the name a workload proves it holds.
An SVID (SPIFFE Verifiable Identity Document) is the proof. Two forms:
\nsub is the SPIFFE ID and whose aud you scope to the intended verifier. You present it as a bearer token.The lifetimes are the point. In SPIRE’s server config the defaults are default_x509_svid_ttl: 1h and default_jwt_svid_ttl: 5m, and the SVIDs are signed by a CA whose own ca_ttl defaults to 24h. These are short on purpose: a leaked SVID is worthless within the hour, which is exactly why it is safe to use it where you used to store a token.
The thing that makes “never stored” possible is the Workload API, served by a SPIRE agent over a local Unix domain socket (the documented default is /tmp/spire-agent/public/api.sock; many deployments mount it at /run/spire/sockets/...). The workload connects to that socket and asks for its SVID. It presents no credential to do so. The agent identifies the caller by inspecting the calling process through the kernel — its PID, and from that its container, namespace, service account — and performs workload attestation against a set of selectors. Identity is established by what the process verifiably is, not by what secret it holds. That inversion is the entire reason there is nothing to store.
Underneath, SPIRE is two components and two attestations:
\nUpstreamAuthority plugin — a decision you make once and live with.k8s_psat, which validates a projected service-account token through the Kubernetes TokenReview API. No shared secret is planted on the node. Then it does workload attestation for each local process before handing it an SVID.A registration entry is the server-side record that ties a SPIFFE ID to a parent (the node/agent) and a set of selectors (namespace, service account, image). It is the policy: this workload, attested this way, gets this identity. Managing the lifecycle of these entries is real work, and it is where most of the ongoing operational cost lives.
\nOne trust domain is one root of trust. A multi-cluster fleet usually means multiple trust domains — one per cluster, or per region — and now a workload in domain A must be able to validate an SVID minted in domain B. That is federation, and its unit is the trust bundle: the public CA certs and JWT signing keys (a JWKS) that let you verify a domain’s SVIDs.
\nEach domain stands up a bundle endpoint — a URL serving its current bundle, the SPIFFE analogue of OIDC’s jwks_uri — and each peer polls it to stay current. Two profiles, and the difference is exactly the bootstrap question:
https_web — the endpoint is fronted by a Web-PKI certificate from a public CA. The peer validates it with the ordinary public trust store, so no initial bundle has to be exchanged out of band. Trust bootstraps off the existing Web PKI.https_spiffe — the endpoint authenticates with its own X.509-SVID. To talk to it the first time, the peer must already hold that domain’s initial bundle. So the very first bundle has to arrive out of band, after which the latest fetched bundle is used going forward.On a registration entry, federatesWith lists the foreign trust domains a workload is allowed to authenticate against; the agent then delivers those foreign bundles to the workload over the Workload API. Bundles refresh on the spiffe_refresh_hint (commonly around five minutes), and you publish a new signing key several refresh cycles before you start using it, so federated peers have already learned it when the rotation lands.
Note the honest shape of “zero static authority”: federation relocates the bootstrap trust decision, it does not erase it. With https_spiffe you ship one initial bundle by hand; with https_web you lean on the public CA system. Either way the root of trust still comes from somewhere — what you’ve eliminated is the long-lived, broadly-scoped, per-cluster credential, not the one-time trust anchor. That is the right trade, but call it what it is.
Here is where intent meets the Kubernetes API, and where it is easy to be wrong. There are two mechanisms, and only one is clean today.
\nThe JWT-SVID-as-OIDC path (the one that works). Kubernetes can be told to trust an external OIDC issuer. SPIRE ships an OIDC Discovery Provider that serves /.well-known/openid-configuration and a JWKS backed by SPIRE’s JWT-SVID signing keys. You point the remote (spoke) cluster’s apiserver at that provider as an OIDC issuer. The controller fetches a fresh JWT-SVID from its local Workload API and presents it as a bearer token; the apiserver validates it against SPIRE’s JWKS and maps it to a subject. No Secret, no stored token — a new JWT per request, expiring in minutes.
This is not a thought experiment. Red Hat ships SPIFFE/SPIRE support as the Zero Trust Workload Identity Manager operator starting in OpenShift 4.20, and documents exactly this wiring as a how-to guide rather than a shipped OpenShift GitOps feature: Argo CD uses a client-go ExecCredential plugin (apiVersion: client.authentication.k8s.io/v1beta1) via execProviderConfig, which on each call reads the SPIFFE socket (SPIFFE_ENDPOINT_SOCKET), requests a JWT-SVID with the audience the spoke apiserver expects (SPIFFE_JWT_AUDIENCE), and hands it over. The credential is manufactured at the moment of use and discarded.
The mTLS-with-X.509-SVID path (the trap). The instinct is to do straight mTLS: the controller presents its X.509-SVID, the apiserver trusts the SPIFFE bundle as a client CA, done. It is not done. Kubernetes client-cert auth derives the username from the certificate’s Subject CN and groups from the Subject O — and the SPIFFE ID lives in the URI SAN, which apiserver client-cert auth does not read. A raw SPIFFE X.509-SVID therefore does not map to a Kubernetes user. To use mTLS you insert a proxy — Ghostunnel or Envoy — in front of the apiserver: it terminates SPIFFE mTLS, validates the URI-SAN SPIFFE ID, and forwards with an identity the apiserver does understand. Ghostunnel consumes the Workload API directly for its own rotating SVIDs. This works, but it is a moving part in front of every apiserver, and the OIDC path avoids it.
\nOn maturity, be honest. Upstream Argo CD has no native SPIFFE auth; it works through the generic ExecCredential plugin pattern, and the most mature productization is OpenShift’s. Flux can be GitOps-managed to deploy SPIRE, and its OCI registry auth can consume a JWT-SVID, but a first-class “authenticate Flux to a remote cluster via SPIFFE instead of a kubeconfig” feature is not documented — it rides the same OIDC/exec-credential/proxy plumbing. Treat native remote-cluster SPIFFE auth as a pattern you assemble, not a checkbox you enable.
This is the part the architecture diagrams omit. You have removed a stored credential; in return you have made an identity service a hard dependency on the critical path, and short-lived things fail in ways long-lived things didn’t.
\navailability_target knob (must be ≥ 24h if set) makes it rotate early to bank headroom for graceful downtime. But the failure mode is real and new: identity-plane down means fleet auth down.The steps are not a menu; they are a sequence, and skipping it is how teams get the cost without the benefit.
\n1. Stand up SPIRE: server + agents, node attestation (foundation)\n2. Issue workload SVIDs to the controller via Workload API\n3. Make the spoke apiserver trust SPIRE (OIDC path first)\n4. Federate trust domains across clusters\n5. Delete the stored kubeconfig Secrets (the payoff)Step 5 is the whole point, and it is only safe once 1–4 actually work. The common failure is to keep the old Secret “as a fallback” — which means the long-lived fleet-wide credential is still at rest, still in your backups, still the thing an attacker takes, and you are now also running SPIRE. You have paid for the control plane and kept the liability. Either the stored credential is gone or you have not done this; there is no half-credit.
The reverse error is reaching for SPIRE before the order can pay off — standing up a federated identity plane for a couple of clusters whose Secret would be fine. That is just-in-case complexity wearing a security badge: machinery built for a blast radius you don’t yet have. The machinery is justified by the fleet, the audit requirement, and the blast radius of a stolen token. Below that line, the cheapest secure credential really is the boring one you never have to issue.
Where this approach genuinely ends is clear: a fleet large enough that a static per-cluster credential is an unacceptable blast radius, with the operational maturity to run a stateful, HA, on-the-critical-path identity service and keep its clocks in sync. A team that has arrived there already pays for that maturity elsewhere — and at that point zero static authority is not gold-plating, it is the credential model the blast radius was always demanding. Not before.
\nSee also: the SPIFFE and SPIRE docs define the identity model and the server/agent components, and the SPIFFE Federation spec specifies the bundle-endpoint profiles this design relies on.
", "content_text": "> **TL;DR:** A stored kubeconfig is a long-lived credential pointed at your whole fleet — the exact thing an attacker wants. Replace it with a workload identity that expires in an hour and is never written to disk. Here is the rule, the order, and the new control plane you take on in return.\n\nA GitOps controller — Argo CD, Flux — sits in one cluster and reconciles many. To reach the others it needs to authenticate to each remote `kube-apiserver`. The default way to do that is to store a credential: a bearer token, a client certificate, a kubeconfig, kept as a Kubernetes `Secret` in the control cluster. In Argo CD these are the objects labelled `argocd.argoproj.io/secret-type: cluster`.\n\nStop and look at what that `Secret` is. It is a **long-lived credential, usually with broad rights, pointing at your entire fleet, sitting at rest in one place.** It does not rotate on its own. It is in your backups. It is the single most valuable thing an attacker can find in your control cluster, because owning it is owning every cluster it can reach. The whole discipline of [boring architecture](/engineering/essays/why-boring-architecture/) is about not keeping liabilities you don't have to — and a stored fleet-wide token is a liability you keep purely because issuing a fresh, short-lived one at the moment of use seemed like more work.\n\nThis piece is about doing that more work, in order. The target is **zero static authority**: no long-lived credential that authenticates the controller to a remote cluster is ever written down. The identity the controller presents is minted on demand, expires in about an hour, and lives only in memory. The mechanism is SPIFFE for the identity model and SPIRE for issuing it.\n\nBefore any of it, this is not a tool for three clusters you could count on one hand. SPIRE is a control plane you operate, not a library you import — it is exactly the kind of capability you should be slow to take on, the way [modular monolith beats microservices until you've earned the split](/engineering/essays/why-i-start-with-a-modular-monolith/). It earns its weight only when you genuinely run a fleet — many clusters, many teams, an audit requirement that a stolen `Secret` would fail. If a single shared cluster covers you, the cheapest secure credential is the one you never stand up the machinery to issue. The rest of this assumes you have actually arrived at the fleet.\n\n## The one rule up front\n\nState it before the mechanics: **the credential that crosses a cluster boundary must be short-lived and never stored.** Everything below is in service of that one sentence. If at the end you still have a long-lived token written to disk somewhere on the authentication path, you have built complexity without buying the property you came for — the same wrong turn as adding a read replica to dodge a missing index.\n\n\"Short-lived and never stored\" has a precise meaning here. The identity is an X.509 certificate whose default lifetime is **one hour** — or a JWT, whose default is five minutes — fetched by the workload from a local socket with **no token of its own to present**, and rotated automatically at half its life. There is no secret to steal that is worth stealing tomorrow.\n\n## What SPIFFE and SPIRE actually are\n\nSPIFFE is a spec; SPIRE is the reference implementation that issues the things the spec describes. Four nouns carry the whole design.\n\nA **SPIFFE ID** is a URI: `spiffe://