# Engineering: Backend and Systems

> Production lessons, architecture decisions, database, server and application patterns, and AI-assisted engineering: notes, system write-ups, essays and developer tools that run in your browser.

- Articles: 51
- Tools: 14
- Source: https://muhammetsafak.com/engineering/
- Language: en-US
- Author: Muhammet Şafak

---
A notebook on backend and systems engineering.

## How every piece is built

1. Context
2. Constraint
3. Approach
4. Pitfalls
5. Expiry

## Notes

- [Upgrade Triggers: Mandatory, Locking, Opportunity](https://muhammetsafak.com/engineering/notes/upgrade-triggers/): A Dependabot PR, an EOL notice and a CVE bulletin share one queue but not one urgency. Sort signals into three types and give each a response time. (Markdown: https://muhammetsafak.com/engineering/notes/upgrade-triggers.md)
- [CQRS: When You Need It, When It's Dead Weight](https://muhammetsafak.com/engineering/notes/when-is-cqrs-necessary/): Separating read and write models: the clarity CQRS buys in a complex domain, and the dead weight it adds to plain CRUD (Markdown: https://muhammetsafak.com/engineering/notes/when-is-cqrs-necessary.md)
- [An Archiving Strategy: Old Data Without the Slowdown](https://muhammetsafak.com/engineering/notes/archiving-strategy/): Separating old records that bloat the live table: partitioning, cold storage, and keeping query access while you archive (Markdown: https://muhammetsafak.com/engineering/notes/archiving-strategy.md)
- [Sizing a PostgreSQL Queue Table by Its Live Set](https://muhammetsafak.com/engineering/notes/sizing-a-postgres-queue-table-by-its-live-set/): On a queue table, an index and an autovacuum threshold that grow with the table scale wrong; the measured case for sizing both to the pending jobs (Markdown: https://muhammetsafak.com/engineering/notes/sizing-a-postgres-queue-table-by-its-live-set.md)
- [Indexes: Too Few Slow Reads, Too Many Kill Writes](https://muhammetsafak.com/engineering/notes/index-management/): A missing index slows the query; too many indexes make every write expensive. Striking the right balance by measurement, not by guesswork (Markdown: https://muhammetsafak.com/engineering/notes/index-management.md)
- [Changing Schema in Production With Zero Downtime](https://muhammetsafak.com/engineering/notes/zero-downtime-database-migrations/): Running migrations without causing downtime in production: backward-compatible steps, multi-phase column changes, and avoiding table locks (Markdown: https://muhammetsafak.com/engineering/notes/zero-downtime-database-migrations.md)
- [Read/Write Splitting: Separating Read and Write Load](https://muhammetsafak.com/engineering/notes/read-write-splitting/): Before scaling up the database: moving read traffic to replicas, the traps replication lag creates, and when you actually need it (Markdown: https://muhammetsafak.com/engineering/notes/read-write-splitting.md)
- [Before You Reach for Redis: The Right Cache Strategy](https://muhammetsafak.com/engineering/notes/the-right-cache-strategy/): The questions to ask before adding a cache, why cache invalidation is hard, and the choice between TTL and consistency (Markdown: https://muhammetsafak.com/engineering/notes/the-right-cache-strategy.md)
- [Circuit Breakers for External Service Integrations](https://muhammetsafak.com/engineering/notes/circuit-breaker-external-service-integration/): Connecting a critical system to external APIs: using timeouts, retries, and circuit breakers to stop one service's collapse from taking the whole system down. (Markdown: https://muhammetsafak.com/engineering/notes/circuit-breaker-external-service-integration.md)
- [The Serverless Decision: Cold-Start and Vendor Lock-in](https://muhammetsafak.com/engineering/notes/serverless-migration-decision/): Before you move to FaaS: the real cost of cold-start latency, vendor lock-in risk, and the workloads where serverless actually pays off (Markdown: https://muhammetsafak.com/engineering/notes/serverless-migration-decision.md)
- [BFF Pattern: A Separate API Layer for Mobile and Web](https://muhammetsafak.com/engineering/notes/bff-backend-for-frontend/): Separate API layers for mobile and web: the hidden cost of bending one API to fit every client, and when a BFF is actually warranted (Markdown: https://muhammetsafak.com/engineering/notes/bff-backend-for-frontend.md)
- [Synchronous or Asynchronous? HTTP or the Queue](https://muhammetsafak.com/engineering/notes/sync-vs-async/): Should a job run inside the HTTP request or go to a queue? The decision line, drawn through response time and fault tolerance (Markdown: https://muhammetsafak.com/engineering/notes/sync-vs-async.md)
- [Event-Driven Architecture: What It Solves and Hides](https://muhammetsafak.com/engineering/notes/when-event-driven-architecture/): The price of loose coupling through events: balancing the flexibility it buys against the risk of making your system's flow impossible to trace (Markdown: https://muhammetsafak.com/engineering/notes/when-event-driven-architecture.md)
- [Replaying a Dead-Letter Queue Without Making It Worse](https://muhammetsafak.com/engineering/notes/replaying-a-dead-letter-queue/): Naive replay re-poisons the queue or double-fires side effects. The safe shape: reset, dry-run, sandbox-first, select. (Markdown: https://muhammetsafak.com/engineering/notes/replaying-a-dead-letter-queue.md)
- [Validating the Schema at the Edge of the Queue](https://muhammetsafak.com/engineering/notes/validate-schema-at-the-edge/): A message's data is an untyped contract the queue won't check. Validate it at the edge: producer before publish, consumer as a safety net. (Markdown: https://muhammetsafak.com/engineering/notes/validate-schema-at-the-edge.md)
- [Idempotency: When the Same Message Arrives Twice](https://muhammetsafak.com/engineering/notes/idempotency-duplicate-delivery/): At-least-once delivery means a handler will see the same message twice. The idempotency key, where it comes from, and the dedupe that survives a crash (Markdown: https://muhammetsafak.com/engineering/notes/idempotency-duplicate-delivery.md)
- [Namespace Isolation for a Shared Redis](https://muhammetsafak.com/engineering/notes/shared-redis-namespace-isolation/): The pattern I use to avoid key collisions and accidental deletes when a single Redis instance is shared across several projects (Markdown: https://muhammetsafak.com/engineering/notes/shared-redis-namespace-isolation.md)
- [A PostgreSQL Backup Strategy with pgBackRest](https://muhammetsafak.com/engineering/notes/pgbackrest-postgresql-backup-strategy/): pg_dump alone is not a backup. A PITR-capable, compressed, verifiable backup architecture with pgBackRest. (Markdown: https://muhammetsafak.com/engineering/notes/pgbackrest-postgresql-backup-strategy.md)
- [Why Do Laravel Queues Slow Down in Production?](https://muhammetsafak.com/engineering/notes/laravel-queue-production-slowdown/): Laravel queue workers are flawless locally and slow in production. The five most common reasons developers overlook. (Markdown: https://muhammetsafak.com/engineering/notes/laravel-queue-production-slowdown.md)
- [Nginx + PHP-FPM Pool Separation](https://muhammetsafak.com/engineering/notes/nginx-php-fpm-pool-separation/): The concrete benefits of defining a separate pool per application instead of a single PHP-FPM pool when running multiple PHP apps on one VPS (Markdown: https://muhammetsafak.com/engineering/notes/nginx-php-fpm-pool-separation.md)
- [Multiple DB Users with pgBouncer auth_query](https://muhammetsafak.com/engineering/notes/pgbouncer-auth-query/): A practical setup for dynamic user authentication via `auth_query` instead of `userlist.txt` when running pgBouncer (Markdown: https://muhammetsafak.com/engineering/notes/pgbouncer-auth-query.md)

## Systems

- [Transactional Outbox: Dual-Write, Idempotent Consumers](https://muhammetsafak.com/engineering/systems/transactional-outbox-dual-write-and-idempotent-consumption/): Closing dual-write with an outbox, suppressing the at-least-once repeats it creates, and encrypting the personal data inside the event. (Markdown: https://muhammetsafak.com/engineering/systems/transactional-outbox-dual-write-and-idempotent-consumption.md)
- [GPU FinOps with eBPF](https://muhammetsafak.com/engineering/systems/gpu-finops-with-ebpf/): Why GPU spend is invisible to cgroup and cAdvisor metrics, how eBPF attributes it to teams — and the CUDA boundary only DCGM can cover. (Markdown: https://muhammetsafak.com/engineering/systems/gpu-finops-with-ebpf.md)
- [Zero Static Authority in Multi-Cluster GitOps](https://muhammetsafak.com/engineering/systems/zero-static-authority-multi-cluster-gitops/): Multi-cluster GitOps with no long-lived kubeconfig: SPIFFE IDs, SPIRE-issued short-lived SVIDs — the order to build it in, and what each step costs. (Markdown: https://muhammetsafak.com/engineering/systems/zero-static-authority-multi-cluster-gitops.md)
- [Distributed Tracing Across a Polyglot Queue](https://muhammetsafak.com/engineering/systems/polyglot-queue-distributed-tracing/): Turning a message that crosses PHP, Go and Python over a frozen envelope into one OpenTelemetry trace — no new field, no new dependency. (Markdown: https://muhammetsafak.com/engineering/systems/polyglot-queue-distributed-tracing.md)
- [Race Conditions and Gaps in Sequential Numbering](https://muhammetsafak.com/engineering/systems/race-conditions-and-gaps-in-sequential-numbering/): Generating legally sequential, gap-free numbers with parallel workers: the JIT reservation pattern that solves race conditions and gaps together (Markdown: https://muhammetsafak.com/engineering/systems/race-conditions-and-gaps-in-sequential-numbering.md)
- [Scale Breaking Points in Data-Intensive Systems](https://muhammetsafak.com/engineering/systems/data-intensive-systems-breaking-points/): As data load grows, where does boring architecture break? Working set, read replicas, partitioning, write load, a separate data layer — in order. (Markdown: https://muhammetsafak.com/engineering/systems/data-intensive-systems-breaking-points.md)
- [Multiple Projects on a Single VPS](https://muhammetsafak.com/engineering/systems/single-vps-multi-project-architecture/): One VPS, several independent apps. Deliberate minimalism over Kubernetes: user isolation, separate PHP-FPM pools, shared PostgreSQL/Redis, a plain deploy. (Markdown: https://muhammetsafak.com/engineering/systems/single-vps-multi-project-architecture.md)
- [Laravel Production Stack: Why Each Piece Is There](https://muhammetsafak.com/engineering/systems/laravel-production-stack/): A production-grade Laravel setup with Nginx + PHP-FPM + Redis + PostgreSQL + Supervisor + Horizon. Each piece's job and what you lose if you drop it (Markdown: https://muhammetsafak.com/engineering/systems/laravel-production-stack.md)

## Essays

- [Which Test Catches the Production Bug?](https://muhammetsafak.com/engineering/essays/integration-vs-unit-tests/): The real payoff of unit and integration tests: which one actually prevents production bugs, and where to spend your testing budget. (Markdown: https://muhammetsafak.com/engineering/essays/integration-vs-unit-tests.md)
- [The Cost of Deferring an Upgrade](https://muhammetsafak.com/engineering/essays/the-cost-of-deferring-an-upgrade/): A PHP 7.1 → 8.2 upgrade was put off because "it works." The bill came later: a dependency lock, batched breaking changes, and bugs spread over months. (Markdown: https://muhammetsafak.com/engineering/essays/the-cost-of-deferring-an-upgrade.md)
- [Chasing 100% Coverage With Tests That Do Nothing](https://muhammetsafak.com/engineering/essays/a-lean-testing-culture/): Test coverage is not a target, it's a byproduct. The cost of tests written to push the number up that catch no bugs at all (Markdown: https://muhammetsafak.com/engineering/essays/a-lean-testing-culture.md)
- [The Cost of Installing a Package for Everything](https://muhammetsafak.com/engineering/essays/third-party-package-dependency/): The security, maintenance, and supply-chain cost of adding a dependency for a small function — and when writing it yourself is cheaper (Markdown: https://muhammetsafak.com/engineering/essays/third-party-package-dependency.md)
- [Is Your Data Really Big, or Just Badly Modeled?](https://muhammetsafak.com/engineering/essays/big-data-syndrome/): Most big-data problems are really bad modeling or a missing index; before moving to a distributed system, ask whether the data is actually big (Markdown: https://muhammetsafak.com/engineering/essays/big-data-syndrome.md)
- [ORM or Native SQL?](https://muhammetsafak.com/engineering/essays/orm-vs-native-sql/): Is the ORM making us lazy, or actually protecting us? The balance of safety, readability, and performance between the two approaches (Markdown: https://muhammetsafak.com/engineering/essays/orm-vs-native-sql.md)
- [The NoSQL Trap: Starting Because "It Has No Schema"](https://muhammetsafak.com/engineering/essays/the-nosql-trap/): The consistency and data-integrity crises that projects which moved to NoSQL for the comfort of going schemaless eventually hit (Markdown: https://muhammetsafak.com/engineering/essays/the-nosql-trap.md)
- [Is PostgreSQL Enough for Everything?](https://muhammetsafak.com/engineering/essays/is-postgresql-enough-for-everything/): Before adding a separate search engine, queue, or document DB: how far PostgreSQL gets you on its own, and where its limit begins. (Markdown: https://muhammetsafak.com/engineering/essays/is-postgresql-enough-for-everything.md)
- [Signals That a System Has Grown Too Complex](https://muhammetsafak.com/engineering/essays/signals-of-an-over-complex-system/): The concrete red flags that reveal code or architecture has turned needlessly complex — and where to start simplifying (Markdown: https://muhammetsafak.com/engineering/essays/signals-of-an-over-complex-system.md)
- [Architecture Decisions Rot Because Nothing Runs Them](https://muhammetsafak.com/engineering/essays/architecture-decisions-rot/): An architecture decision written in a wiki is a wish. The boundaries you can't enforce quietly erode until the diagram stops matching the code. (Markdown: https://muhammetsafak.com/engineering/essays/architecture-decisions-rot.md)
- [AI Didn't Remove the Bottleneck — It Moved It](https://muhammetsafak.com/engineering/essays/ai-didnt-remove-the-bottleneck/): Cheap code production doesn't raise throughput if review and integration can't keep up. The bottleneck moves downstream — that's where the work is (Markdown: https://muhammetsafak.com/engineering/essays/ai-didnt-remove-the-bottleneck.md)
- [Flaky Tests: Retrying Isn't a Fix](https://muhammetsafak.com/engineering/essays/flaky-tests-and-determinism/): AI made writing tests nearly free, and flaky suites exploded. Why retry and quarantine are a treadmill, and what catches flakiness deterministically (Markdown: https://muhammetsafak.com/engineering/essays/flaky-tests-and-determinism.md)
- [Schema Evolution Without Breaking the Contract](https://muhammetsafak.com/engineering/essays/schema-evolution-without-breaking/): You can't change a cross-service contract atomically. The rule: additive is safe, a breaking change needs a new identity — consumers upgrade first (Markdown: https://muhammetsafak.com/engineering/essays/schema-evolution-without-breaking.md)
- [Can You Ship Vibe-Coded Software to Production?](https://muhammetsafak.com/engineering/essays/can-you-ship-vibe-coded-to-production/): Vibe coding lowers the cost of a first version, not the cost of being wrong. That bill arrives in production, not at build time. (Markdown: https://muhammetsafak.com/engineering/essays/can-you-ship-vibe-coded-to-production.md)
- [Layered Architecture or Clean Architecture?](https://muhammetsafak.com/engineering/essays/layered-vs-clean-architecture/): Why the classic layered structure is enough for most projects while we drown in abstraction layers, and when Clean Architecture actually pays off. (Markdown: https://muhammetsafak.com/engineering/essays/layered-vs-clean-architecture.md)
- [The Production Cost of "Just-in-Case" Code](https://muhammetsafak.com/engineering/essays/the-cost-of-just-in-case-code/): Applying YAGNI in practice: the price of adding flexibility you don't need today, repaid daily in maintenance, readability, and bug surface (Markdown: https://muhammetsafak.com/engineering/essays/the-cost-of-just-in-case-code.md)
- [When I Move to Microservices](https://muhammetsafak.com/engineering/essays/when-i-move-to-microservices/): The measured signals that justify moving from a monolith to a distributed architecture — and the common false reasons that say it's still too early (Markdown: https://muhammetsafak.com/engineering/essays/when-i-move-to-microservices.md)
- [Gitflow or GitHub Flow? How I Choose](https://muhammetsafak.com/engineering/essays/gitflow-vs-github-flow/): Practical criteria for picking a branching strategy by product model, version count, and deploy rhythm — and how each model breaks in the wrong place. (Markdown: https://muhammetsafak.com/engineering/essays/gitflow-vs-github-flow.md)
- [When I Switch to Trunk-Based Development](https://muhammetsafak.com/engineering/essays/when-i-switch-to-trunk-based/): Why release branches break for high-frequency deploy teams and how much of trunk-based is really discipline — the thresholds I cross and the ones I don't (Markdown: https://muhammetsafak.com/engineering/essays/when-i-switch-to-trunk-based.md)
- [Why I Start Projects With a Modular Monolith](https://muhammetsafak.com/engineering/essays/why-i-start-with-a-modular-monolith/): The case for drawing clean module boundaries inside a single deployable app before jumping to microservices — and when I stop doing it. (Markdown: https://muhammetsafak.com/engineering/essays/why-i-start-with-a-modular-monolith.md)
- [Why I Prefer Boring Architecture](https://muhammetsafak.com/engineering/essays/why-boring-architecture/): The pragmatic case for choosing proven tools over chasing new technology. This is not cowardice, it is a budget decision. (Markdown: https://muhammetsafak.com/engineering/essays/why-boring-architecture.md)
- [My AI-Assisted Engineering Workflow](https://muhammetsafak.com/engineering/essays/ai-assisted-engineering-workflow/): The seven rules I follow to use Claude Code and similar agents on production code with confidence — not a skeptical take, a discipline manifesto (Markdown: https://muhammetsafak.com/engineering/essays/ai-assisted-engineering-workflow.md)

## Series

- [PostgreSQL in Production](https://muhammetsafak.com/engineering/series/postgresql-in-production/): Keeping a single PostgreSQL cluster healthy in production: connection pooling, backups, read/write splitting, zero-downtime migrations and restore testing. (5-part series)
- [Upgrading in Production](https://muhammetsafak.com/engineering/series/production-upgrade/): Treating runtime, framework and dependency upgrades as a risk decision: triggers, inventory, decision matrix, rollout and the decision not to upgrade. (2-part series)

## Tools

- [.env Parser](https://muhammetsafak.com/engineering/tools/env-parser/): Splits dotenv content into a key/value table, flags invalid lines, and masks values of sensitive keys such as secret, token and password. Runs in your browser. (Markdown: https://muhammetsafak.com/engineering/tools/env-parser.md)
- [Backoff & Jitter Calculator](https://muhammetsafak.com/engineering/tools/backoff-calculator/): Compute an exponential backoff schedule and see how full, equal, and decorrelated jitter spread the retries. Client-side, nothing is retried. (Markdown: https://muhammetsafak.com/engineering/tools/backoff-calculator.md)
- [Base64 Encoder / Decoder](https://muhammetsafak.com/engineering/tools/base64/): Encode and decode text as Base64, with a URL-safe variant and full UTF-8 support. Runs in your browser; your text is never sent anywhere. (Markdown: https://muhammetsafak.com/engineering/tools/base64.md)
- [Code Image Generator](https://muhammetsafak.com/engineering/tools/code-image/): Turn a code block into a shareable PNG. The code itself never leaves your browser; pick a theme, export scale, and window chrome. (Markdown: https://muhammetsafak.com/engineering/tools/code-image.md)
- [gitignore Generator](https://muhammetsafak.com/engineering/tools/gitignore-generator/): Generate a .gitignore from your project folder. Unlike the competition, nothing is uploaded — not even the file names ever leave the page. (Markdown: https://muhammetsafak.com/engineering/tools/gitignore-generator.md)
- [HTTP Status Codes Reference](https://muhammetsafak.com/engineering/tools/http-status/): What common HTTP status codes mean and where each shows up. Search by code, name or keyword. Runs in your browser; nothing you type is sent anywhere. (Markdown: https://muhammetsafak.com/engineering/tools/http-status.md)
- [JSON Formatter](https://muhammetsafak.com/engineering/tools/json-formatter/): Format JSON with 2-space, 4-space or tab indentation, or minify it; invalid JSON gets an error message. Runs in your browser; your data never leaves it. (Markdown: https://muhammetsafak.com/engineering/tools/json-formatter.md)
- [JSON Schema Backward-Compatibility Diff](https://muhammetsafak.com/engineering/tools/schema-compat/): Diff two JSON Schemas: is the change backward-compatible, or a breaking one that needs a new message version? Runs in your browser. (Markdown: https://muhammetsafak.com/engineering/tools/schema-compat.md)
- [JWT Decoder](https://muhammetsafak.com/engineering/tools/jwt-decoder/): Decodes a JWT header and payload and shows the exp and iat claims as ISO timestamps. It does not verify the signature; the token stays in your browser. (Markdown: https://muhammetsafak.com/engineering/tools/jwt-decoder.md)
- [Password Generator](https://muhammetsafak.com/engineering/tools/password-generator/): Cryptographically secure password generator: set the length and character classes, and optionally exclude look-alike and shell characters. Runs in your browser. (Markdown: https://muhammetsafak.com/engineering/tools/password-generator.md)
- [Typing Speed Test](https://muhammetsafak.com/engineering/tools/typing-test/): Measures Turkish and English typing speed honestly: CPM is the primary metric, and a code mode exercises real brackets and template syntax. (Markdown: https://muhammetsafak.com/engineering/tools/typing-test.md)
- [Unix Timestamp Converter](https://muhammetsafak.com/engineering/tools/timestamp/): Convert both ways between Unix timestamp and ISO 8601. Seconds and milliseconds are auto-detected; shows UTC, local and relative time. Runs in your browser. (Markdown: https://muhammetsafak.com/engineering/tools/timestamp.md)
- [URL Parser](https://muhammetsafak.com/engineering/tools/url-parser/): Breaks a URL into protocol, host, port, pathname, hash and credentials, and lists its query parameters in a table. Runs in your browser; the URL stays there. (Markdown: https://muhammetsafak.com/engineering/tools/url-parser.md)
- [UUID Generator (v4 & v7)](https://muhammetsafak.com/engineering/tools/uuid/): Generates UUID v4 (random) and v7 (time-ordered, RFC 9562), one at a time or in bulk. Runs in your browser with the Web Crypto API; nothing is sent anywhere. (Markdown: https://muhammetsafak.com/engineering/tools/uuid.md)

## Glossary

- [Glossary](https://muhammetsafak.com/engineering/glossary/)
- [Stack](https://muhammetsafak.com/engineering/stack/) (Markdown: https://muhammetsafak.com/engineering/stack.md)

## About this section

## Vision

What Turkish technical writing is missing is not another introduction; it is a reference
an engineer can return to when the same decision lands on their desk. That is where
the Engineering section is going: an archive of decisions that still show which constraint produced
them, years after they were made.

I measure that three ways. A piece cited as a source in Turkish engineering
conversations. An engineer who arrived for a tool staying for the writing. A site that
opens like documentation rather than like a marketing page.

The notebook will get larger. It will not get faster.