Expertise
Apache Kafka developer — since 2022
Adding partitions is easy; the real work is accounting up front for the same key landing in another partition and breaking ordering — I have made Kafka decisions with that cost in view since 2022.
- years
- 4
Me in the Apache Kafka ecosystem
Kafka is not a queue for me but a replayable event log. That difference decides where I pick it: if a message does not need to stay reachable after it is consumed, a simpler queue often does the job. The general frame for that choice is on the Software Architecture page.
Most of my Kafka work goes into small decisions on the producer and consumer side: what the partition key is tied to, what happens when the consumer count exceeds the partition count, what a redelivery would break. I account for the fact that the partition count can be raised later, but doing so changes the key-to-partition mapping and breaks the earlier ordering guarantee.
The same backbone holds on the AI side: for event-driven workflows and data pipelines in the AI Systems role, Kafka is the message layer I use. The deep write-ups and long essays live in the Engineering section.
What is the Apache Kafka Developer role?
Apache Kafka is a distributed event streaming platform that stores events in topics for a configured period, so multiple consumers can read them as often as needed. Topics are split into partitions, and the ordering guarantee holds only within a partition. A Kafka developer is the person who builds the producer and consumer side, the key choice and the scaling decisions on top of that model.
What does the Apache Kafka Developer do?
I use Kafka as the message backbone of event-driven workflows and data pipelines. My work turns on four decisions: making the narrowest entity that truly needs ordering the partition key, treating consumer parallelism as a ceiling bounded by partition count, assuming at-least-once delivery and building idempotent consumers, and choosing Kafka only where a replayable event log is actually needed.
Technologies I pair it with
-
Apache Kafka
The event streaming platform I use as the message backbone of event-driven workflows and data pipelines.
-
RabbitMQ
Asynchronous delivery between services; the exit door of the outbox relay.
-
PostgreSQL
The database I measured queue-table and index behaviour on.
-
Redis
The fast layer that works alongside the relational store in a cache layer.
-
Docker
Development environment and deployment unit.
-
Kubernetes
The orchestration layer where containers are run, scaled and restarted across machines.
Questions in this area
Frequently asked
8 questions
-
How long have you worked with Kafka?
Since 2022 — 4 years.
-
What does a Kafka developer do?
A Kafka developer designs the producers that write events to Kafka and the consumers that read them, and decides the partition key, consumer scaling and delivery guarantee. For me that means setting Kafka up as the message backbone of event-driven workflows and data pipelines.
-
How do you choose between Kafka and RabbitMQ?
I look at the need before the technology. I use RabbitMQ for asynchronous delivery between services, as the exit door of the outbox relay. I choose Kafka where a message must stay readable after it is consumed, that is, where a real event log is wanted.
-
How do you keep per-entity ordering in Kafka?
Kafka guarantees order only within a partition, not across a topic. So I make the narrowest entity that truly needs ordering the partition key.
-
Do you chase exactly-once delivery?
No, I build on at-least-once. A stable dedup key per message, a processed-messages table and an outbox on the producer side make the consumer idempotent, so a redelivery becomes harmless.
-
How do you reduce rebalancing disruptions when scaling Kafka consumers?
First I set partition.assignment.strategy to CooperativeStickyAssignor so only the moved partitions pause. Then I cut membership churn with static membership and an autoscaler cooldown.
-
Do you relay the outbox table by polling or with Debezium CDC?
A polling relay tuned with SKIP LOCKED, batching and a partial index is enough for most teams. I move to Debezium where measured latency or database load pinches.
-
Where do you use Kafka in AI systems?
As the message backbone of event-driven AI workflows and data pipelines. The [AI Systems page](/expertise/ai-systems/) covers that context.
Let us work together
If you have work in this ecosystem, tell me what you are building and we will talk about how to build it.