Is Your Data Really Big, or Just Badly Modeled?
Most big-data problems are really bad modeling or a missing index; before moving to a distributed system, ask whether the data is actually big
TL;DR
A team that says our data is big usually rests on a feeling of slowness, not a measurement; a 12 GB table is small for a modern PostgreSQL. Moving to a distributed system before the data is genuinely big is the most expensive wrong turn; the tax is paid up front and nothing gets solved in return. If there is no number, what you have is not big data but unmeasured data.
A sentence came up in a meeting: “Our data has grown so much, we need to move to a distributed database now.” Then we looked at the actual size: 12 GB. For PostgreSQL on a modern server, 12 GB isn’t big — it’s small.
“Big data” is said most often about data that isn’t big at all. And that sentence isn’t a diagnosis; more often it’s an excuse.
”Big” is a feeling, not a measurement
The sentence “our data is big” almost never rests on a number. It rests on a sense of slowness, a fear, a conference talk. Yet a modern PostgreSQL comfortably carries hundreds of GB on an ordinary server, and low TB if it’s set up carefully.
Whether data is big isn’t felt; it’s measured. A “big” verdict reached without measuring is the first step toward the wrong architecture.
What most “big data” problems actually are
The real cause of the slowness mistaken for big data is almost always one of three:
- Missing index. A query doing a sequential scan makes a 5 GB table feel like 500 GB. The problem isn’t the size, it’s the scanned size.
- Bad modeling. Unnormalized tables, giant JSON blobs embedded in every row, an append-heavy table that was never partitioned. The data isn’t “big,” it’s badly placed.
- Wrong access pattern. N+1 queries, pulling the whole row when one column would do, queries inside a loop. The application uses small data in a big way.
All three are solved by discipline, not by size — and none of the three requires a distributed database.
Measuring the real size
To answer the “is it big” question, ask concrete questions:
- How much space does the table actually take? Look with
pg_total_relation_size— not by guessing. - Does the frequently accessed data (working set) fit in RAM? Has the cache hit ratio from
pg_stat_databasedropped below 99%? - What’s the growth rate? 1 GB per month and 100 GB per month are entirely different decisions.
- Which query is slow, and why? What does the
EXPLAIN ANALYZEoutput say?
If the answers to these questions reveal a real breaking point, what to do next — index, read replica, partitioning — is in the breaking points of data-intensive systems. This piece is about the measurement that comes before those steps.
The distributed-system tax
Moving to a distributed system before the data is genuinely big is the most expensive wrong turn. Because you pay the costs up front — with manual sharding or distributed NoSQL, losing the JOIN across tables and eventual consistency; even with distributed SQL, twice the operational load and distributed debugging — and nothing gets solved in return. A real tax for a problem that doesn’t exist.
The rule from is PostgreSQL enough for everything applies here too: most teams use a small fraction of PostgreSQL’s jsonb, partitioning, BRIN, and indexing capabilities and then conclude it “isn’t enough.”
If it really is big
Let’s be honest: real big data exists. Throughput that sustainably exceeds the write capacity of a single large machine, or a genuinely multi-TB hot working set — these are real limits. A system that gets there has also reached a scale that can finance that complexity; at that point a distributed architecture isn’t a debt, it’s an earned cost.
The point isn’t to be against distributed systems; it’s to avoid moving to them before they’re truly needed — that is, before measuring.
When you hear “our data is big,” the only thing to ask for is a number. If there’s no number, what you have isn’t big data, it’s unmeasured data.
Real size is measured; felt size is, most of the time, a missing index.