Time-series data: TimescaleDB or InfluxDB?
100M rows a day breaks a plain table, so chunk it by time with hypertables, materialize the averages as continuous aggregates, and compress the old chunks.
Just Ask
Ask me anything about software architecture, careers, PHP, Go and the craft of building software; I answer here, in the open, for everyone. (Page 5/8)
Whatever’s on your mind, don’t hold back. Questions reach me directly; I answer the good ones and publish them on this page. Your email is never published.
100M rows a day breaks a plain table, so chunk it by time with hypertables, materialize the averages as continuous aggregates, and compress the old chunks.
Instead of a blocking `ALTER TABLE` on 20M rows run expand/contract: add nullable columns, dual-write, backfill in throttled batches, then drop the old.
Use a Sorted Set instead of plain string keys sorted in the app, since `ZADD` updates in O(log N) and `ZRANGE ... REV` returns the top 100 already ordered.
Move reporting to a read replica and adopt CQRS only when query shapes outgrow one schema, feeding the read model from outbox events, not dual-writes.
Keep products, prices and orders in PostgreSQL and put the variable attributes in one GIN-indexed `JSONB` column; reporting alone rules out splitting.
Keep the invariant in the DB with an atomic `UPDATE ... WHERE balance >= 40` or a `SELECT ... FOR UPDATE`, and save Redlock for non-DB resources.
Put `tries` and a `backoff` on the job so it lands in `failed_jobs` at the cap, alert from a `JobFailed` listener and replay it with `queue:retry`.
Keep liveness dependency-free and check dependencies in readiness with a cached probe, since liveness restarts the pod while readiness only sheds traffic.
Stand up monthly RANGE declarative partitioning on `created_at`, backfill history in batches while the app writes, then swap names in a single transaction.
Acquire locks in one global order such as ascending PK, keep transactions short, target `FOR UPDATE` at the fewest rows, and retry the rest with backoff.