Skip to content
Muhammet Şafak
tr

Postgres never turned the partial index into a generic plan: forty executions, forty custom plans

Does Postgres switch a partial-index query to a generic plan on its own inside a prepared statement — or is the 1,635-fold cliff something you have to opt into?

Finding

Postgres declines. On the partial index all forty executions used a custom plan — the counter reads 40/0. The reason it declines is the disaster itself: a generic plan cannot use the partial index, so its estimated cost comes out high and the planner does not choose it. The composite index switches at the sixth execution exactly as documented (5/35) and loses nothing by it. So the 1,635-fold cliff is real but fenced: reaching it takes writing `plan_cache_mode = force_generic_plan`.

Partial index, generic plans
0 of 40 executions
Composite index, switch
execution 6
Partial, first 5 → last 5
0.36 → 0.20 ms
What reaching the cliff takes
force_generic_plan

Method

The same harness and the same 10-million-dead-plus-5,000-live table, across three index strategies. For each, one prepared statement was executed forty times in a single psql session and `pg_prepared_statements` was queried after every execution — the counters are session-local, which is why this measurement cannot be made from pgbench. The decision is not inferred from timings but read from the counters: `custom_plans` and `generic_plans` say directly what the planner chose on that execution. Timings arrive in pairs (the statement, then the counter query) and the parser keeps only the first. psql's aligned output pads the counter value with spaces, so output was switched to unaligned mode — the first run produced an empty series because of exactly that.

High confidence Repeated runs, controlled environment, raw data published.
Measured on

measured 43 days ago

Published

Environment

Postgres
17-alpine · plan_cache_mode at its default (auto)
Table
10M dead + 5,000 live rows
Measurement
one session · one prepared statement · 40 executions
Source of the decision
pg_prepared_statements.custom_plans / generic_plans
Hardware
Apple M4 Pro · 12 cores · 24 GB · Docker Desktop 29.7.2

Technologies

PostgreSQL SQL Docker

To reproduce

EXECUTIONS=40 ./bench/plan-switch.sh

The partial index measurement pinned plan_cache_mode to each extreme and found a factor of 1,635 between them: 11,752 tps on a custom plan, 7 on a generic one. What it did not measure is the setting everyone actually runs — auto.

That record wrote a sentence there: “the profile degrades as it warms up… it does not show up in staging.” It was a reasonable inference and it turned out to be wrong.

The decision was read, not guessed

Postgres runs a prepared statement on a custom plan for its first five executions, then compares the generic plan’s estimated cost against the average of the custom ones, and keeps the custom plan only when the generic one looks so much more expensive that re-planning each time seems preferable. pg_prepared_statements keeps that decision as a counter, so nothing has to be inferred from timings.

Strategy First generic plan Final counter (custom/generic) First 5 (ms) Last 5 (ms)
partial the switch was declined never 40 / 0 0.36 0.20
composite execution 6 5 / 35 0.40 0.21
(status) never 40 / 0 0.98 0.78
One session, one prepared statement, forty executions. The counters were read after every one.

The composite index switches where the documentation says it will, right after the fifth:

#5  0.266 ms   custom=5  generic=0
#6  0.275 ms   custom=5  generic=1   ← switched
#7  0.185 ms   custom=5  generic=2

On the partial index it never happens. Forty executions, forty custom plans.

It declines because of the disaster itself

The planner does not choose the generic plan because it estimates its cost correctly. A generic plan does not know what $1 is; to use the partial index it would have to prove $1 = 'pending', it cannot, so that plan falls back to a sequential scan and its estimate goes through the roof. The average custom plan sits far below it, and the comparison comes out in favour of custom every time.

The real trade is smaller, and elsewhere

The protection has a price: a query on the partial index is re-planned on every execution. Forty executions, forty plans. The composite one reuses its plan from the sixth onwards.

At this scale the difference is unmeasurable — 0.20 ms against 0.21 ms. But planning is cheap on a cheap query, not on a many-table join or a long IN list. The partial index’s hidden costs are bloat (see the endurance measurement) and constant re-planning; both are small in this workload, and neither is guaranteed to stay small in another.

It corrects the earlier record

The “degrades as it warms up” paragraph in the partial index record went too far, and a note pointing at this finding has been added there. A measurement correcting its own publication is what this section’s contract asks for: if a number is published, so is the number that refutes it.

Other records

All records

The partial index grew three hundred and five times in fifteen minutes — and autovacuum never ran

Under sustained churn, does a partial index stay small on a queue table, and do the default autovacuum settings keep up with it?

Finding

With the live set holding at five thousand rows until about 845 seconds, the partial index went from 0.125 MB to 38.2 MB — three hundred and five times. Its smallness comes from the live set, its bloat rate comes from throughput, and nothing connects the two. The composite index bloated less in proportion (42%) and more in absolute terms (+126 MB), and while bloating it fell over — most likely because it no longer fit in memory, a cause this run did not measure: its latency went from 0.52 ms to 61 seconds and its backlog climbed to 126,000. Fifteen minutes produced 1.75 million dead rows and autovacuum **did not run once** — the default threshold scales with the whole table (50 + 0.2 × 10 million ≈ 2 million) while the churn happens in a tiny subset.

measured 43 days ago

High confidence

A partial index makes a queue table forty-one times smaller — for as long as the planner picks it

On a Postgres queue table with millions of dead rows, what does a partial index buy, and when does the planner refuse to use it?

Finding

At 10 million dead rows a partial index sustains 11,537 claims per second where the same table without one manages 7. Against a composite index the throughput difference is small (6.9%) but the size difference is not: 7.6 MB against 310.4 MB, and the partial one does not grow with the table because it indexes only the 5,000 live rows. None of that is the real finding. The moment the planner switches a prepared statement to a generic plan the partial index stops being used at all — 11,752 tps becomes 7, and 0.68 ms becomes 1.1 seconds. A factor of 1,635. The composite index is untouched under the same conditions.

measured 43 days ago

High confidence
Service & load Measurement

One core carries 14,330 OAuth2 requests in Go and 5,152 in PHP-FPM

The same API verifies an RS256 bearer token on every request and then reads or writes one row in PostgreSQL — on one, two and four cores, how much mixed traffic does it carry in Go, PHP-FPM and FrankenPHP worker mode?

Finding

On four cores Go carried 57,321 mixed requests a second, FrankenPHP 25,659 and php-fpm 20,606 — 14,330, 6,415 and 5,152 per core. The number that goes into a capacity plan is not that one but application CPU per request: 66.8, 110.7 and 187.7 microseconds. At saturation FrankenPHP uses only 2.84 of its four cores, against Go's 3.83 and php-fpm's 3.87. The database is not the constraint: on the same four cores PostgreSQL alone writes 68,212 rows a second, above the mixed ceiling of the fastest candidate.

measured 17 days ago

Medium confidence

Search the site

Start typing to search posts, projects and pages.

Esc to close Powered by Pagefind