# 18,750 req/s: precise, repeated, and three times wrong

> Two phases of the same run measured the same service three times apart. What caught the error was a second method that measured the same thing.

- Kind: Presentation
- Published: 2026-10-03
- Slides: 14
- Based on these articles: https://muhammetsafak.com/blog/precise-repeated-and-three-times-wrong/ · https://muhammetsafak.com/research/capacity-per-core-go-php-oauth2/
- Source: https://muhammetsafak.com/visuals/d/precise-repeated-and-three-times-wrong/
- Language: en-US
- Author: Muhammet Şafak

---
## Slide 1 — Precise, repeated, wrong

Measurement · Load test

18,750 req/s: three repetitions agreed closely, and the number was three times wrong.

Muhammet Şafak October 3, 2026

## Slide 2 — Two phases, a threefold gap

Same run, same service

|  | Ladder (open loop) | Tuning phase (closed loop) |
| --- | --- | --- |
| Go, 4 cores | 18,750 req/s | 55,715 req/s |
| Within itself | Consistent | Consistent |
| In hindsight | Turned out wrong | Turned out right |

## Slide 3

Which one is wrong?

Consistency is not accuracy.

Both measurements were consistent within themselves, and nothing said which of them was wrong.

## Slide 4

Looking at the data

The repetitions ranged from 1,250 to 48,000 .

The median was 18,750, because that is where the middle repetition happened to land.

## Slide 5

First diagnosis

It was correct and not enough.

The writeback of the CHECKPOINT issued before every step spilled into the measurement window: at 3,000 req/s, the write p99 was 10.6 ms, the read 3.3 ms.

## Slide 6 — A fivefold divergence, zero spoiled windows

Second attempt

Req/s measured in three repetitions of the same cell after the fix

| Cell | Repetition 1 | Repetition 2 | Repetition 3 |
| --- | --- | --- | --- |
| Go, 1 core | 1,250 | 6,500 | 3,500 |
| PHP-FPM, 1 core | 1,750 | 2,750 | 750 |

## Slide 7 — oha reports two latencies

PHP-FPM, 1 core, 3,000 req/s

p50, p75, p99 and slowest values of time to first byte and latency-corrected latency in the same measurement

| Latency | p50 | p75 | p99 | Slowest |
| --- | --- | --- | --- | --- |
| Time to first byte | 1.42 ms | 47 ms | 191 ms | 267 ms |
| Latency-corrected | 1.44 ms | 1,539 ms | 3,718 ms | 3,819 ms |

## Slide 8 — The correction turns the limit into a cliff

Mechanism

1.   Step 1: Counts from the schedule `the moment it should have been sent`
2.   Step 2: Responses slow down `connections fill up`
3.   Step 3: The schedule slips `every request inherits it`
4.   Step 4: Cliff `either passes or misses`

## Slide 9

What the search could not measure

The cliff moves between repetitions.

Exponential search and bisection assume that pass/fail is monotonic in the rate. Neither was true.

## Slide 10 — Drop the search, set fixed rates

The fix

-   7 Fixed rates Per cell, the same in every repetition
-   Vote Majority rule If most repetitions carried it, it is carried
-   115% Probe threshold Before every repetition

## Slide 11 — Chromium on the host machine

One more lie

The effect on the measurement of headless Chromium processes on the host machine during one run

| Metric | Value |
| --- | --- |
| Headless Chromium processes | About thirty |
| Total CPU | 1,019% |
| Left for the Docker VM | 94% |
| Go, 4 cores | 17,490 req/s |

## Slide 12 — Two methods, within a few percent

Result · Go, 4 cores

Go, four cores: the tuning phase in the closed loop versus the result of five repetitions on a quiet machine

| Measurement | Req/s | Note |
| --- | --- | --- |
| Tuning phase (closed loop) | 55,715 | Twelve hours earlier |
| Five repetitions (quiet machine) | 55,836–57,321 | Median 56,571; 45/45 repetitions passed the probe |

## Slide 13 — Three concrete additions

For your own setup

-   A second phase that measures in a different loop, and a rule that voids the run when the two diverge (pending)
-   A probe before every repetition that runs no application code (pending)
-   A record of the environment alongside the repetition (pending)

## Slide 14 — The second road

The measurement itself

[muhammetsafak.com/research/capacity-per-core-go-php-oauth2/](https://muhammetsafak.com/research/capacity-per-core-go-php-oauth2/)

The raw data of the numbers and the discarded attempts is in the research notebook.
