Should I put a Kafka/Redis buffer between Fluent Bit and OpenSearch when shipping logs?

Short answer: yes, a buffer is the right move — but the choice between a Redis list and Kafka comes down to the guarantee level you’re willing to accept.

Short answer

What you’re hitting is classic backpressure: once OpenSearch’s indexing rate drops below your production rate, the chain clogs backwards until the app container’s stdout blocks. The fix is a durable layer that decouples the producer (your app) from the consumer (OpenSearch).

Why

1. A Redis list is cheap but risky. In-memory; RPUSH/LPOP is simple, but with AOF off you can lose the latest writes when Redis stops uncleanly and a growing list eats Redis’s RAM. Fine for logs you don’t mind losing, not for financial/audit logs.

2. 40k/s is Kafka’s home turf. Disk-based, with retention, replayable, horizontal consumption via consumer groups. It gives you at-least-once naturally. The cost is operational — you have to stand up and feed a cluster. If the volume is permanent, it’s the right tool.

What to do

1. Turn on Fluent Bit’s own disk buffer first. With storage.type filesystem on the input and storage.total_limit_size on the output, Fluent Bit spools to disk instead of RAM when OpenSearch slows down, so no pressure reaches the app. Once the output hits that limit, the oldest chunk is discarded, so size it for the longest slowdown you expect. Measure this before adding an external queue — it’s often all you need.

2. Move to Kafka only if the volume is permanent. Decide by asking whether 40k/s is a temporary peak or the new floor; disk buffering covers temporary peaks.

3. Make at-least-once idempotent at the consumer. When writing to OpenSearch, give each document a deterministic _id (e.g. a hash of source+offset). A message delivered twice overwrites the same _id instead of creating a duplicate. For the worker and retry mechanics on the queue side, see the Laravel queue and Supervisor post.

Bottom line: start with Fluent Bit filesystem buffering plus bulk tuning and ISM (rollover) on the OpenSearch side. If that’s not enough, move to Kafka — keep Redis for logs you can afford to lose. And the golden rule: never let the app container block while logging. Logging should be fire-and-forget; don’t carry critical data (payments, audit) over the log pipeline — run that on a separate, guaranteed path.