Which model reviews better? I finally measured it

My OpenAI default raised a false alarm on 29 percent of clean diffs. I didn’t know that, because none of CommitBrief’s default models had been picked by measuring. Gemini cleared the threshold; the OpenAI one didn’t, but it still stands as the default.

If you’ve followed the series, you know I’ve covered why I wrote the tool, how I moved it to the team and how I added a filter that needs no model. In all three there was a question I never answered: is the model that produces the finding actually better than another model?

Defaults I picked without measuring

v1.18.0 (September 24, 2026) changed the default model for two providers. For Gemini, gemini-3.8-flash was chosen, but not by looking at quality: by a stopgap rule that holds until there is a measurement. The rule takes the cheapest model in Google’s mid tier; the estimated cost of a typical diff has to stay under the default warning threshold of $0.50. So the rule looks at list price.

For Anthropic, claude-opus-5-5 was chosen, and that was my preference, not the rule’s. The rule would have picked the mid-tier claude-sonnet-5; I wanted the current flagship. On the OpenAI side, gpt-5.4-mini stayed the same.

I attached no quality claim to any of them. The commitbrief setup wizard writes the list price and “not measured” next to every model. The label was honest but uncomfortable: a review tool answers the question “is this code good?”, while I couldn’t answer “which model answers that question better?” The reasoning behind the defaults is in more detail in the v1.18.0 post, and the change record is in the CHANGELOG.

Threshold first, measurement second

Before measuring, I wrote down what passing means (decision record ADR-0043). To become a default, a model has to meet three conditions at once:

• Recall at least 0.90. It must find at least 90 percent of the planted bugs.
• FPR at most 0.10 on clean fixtures. It may raise an alarm on at most 10 percent of the diffs that contain no bug.
• Recall at least 0.70 in every language. It can’t look good on average and collapse in a single language.

The thresholds are hard-coded. They can’t be tuned with an environment variable, and the decision record forbids loosening them even for a single run. The reason is simple: if I move the threshold after seeing the result, I haven’t measured anything; I’ve found a justification for the model I wanted. When the corpus grew, I had to re-derive the thresholds; I did that too before the live measurement.

In v1.18.0 the corpus was 40 fixtures, exactly fitting the qualification floor. Then it grew to 129: 99 planted bugs across 96 buggy fixtures, 33 clean controls; 26 of the 129 fixtures are held-out. It covers Go, TypeScript, JavaScript, Python, Java and PHP; cross-file, large refactor and framework-specific cases are in there too. The qualification floor rose along with the corpus.

A provider doesn’t give the same answer to the same diff every time. So I run the corpus k times with COMMITBRIEF_EVAL_RUNS=k and write the min/max spread across runs next to the aggregate result. I can’t hide behind one lucky or unlucky pass. I took the measurements with k=3.

First bug: I was scoring a prompt the CLI never sent

I found a flaw in the measurement setup, in the live eval harness. The harness sent the model the raw diff instead of the line-numbered diff that production sends. The measurement was scoring a prompt the CLI never sends.

The fix is in the Fixed section of v1.18.0. I don’t count any measurement from before that fix as valid. The lesson is small but expensive: the measuring tool is code too, and what it measures has to be what the product actually sends.

OpenAI failed the threshold

I took the live measurement with OpenAI on October 26; the raw values are in the repository’s results file. For gpt-5.4-mini, recall is 0.9259; the threshold is 0.90, so it passes. FPR is 0.2929; the threshold is 0.10. The FPR of the three runs ranges from 0.2727 to 0.3030, so even its best run is far above the threshold.

The false alarms pile up mostly in JavaScript, TypeScript and PHP: 8 of 15 clean JavaScript fixture runs, and 6 of 15 each in TypeScript and PHP. None of the per-language recalls falls below 0.70. So the model finds the bug, but it also accuses clean code. In a review tool, that means the user stops reading the warnings.

Second bug: Gemini cut the JSON while thinking

I measured Gemini too. On the first attempt some reviews came back with an unexpected end of JSON input or empty content error. The harness counted these as missed findings, so the numbers were wrong in the bad direction. What actually bothered me was something else: the cost and the output size looked too small for a thinking model.

The cause: in Gemini’s thinking models, thinking tokens share the same budget as MaxOutputTokens. The default limit of 4096 was often spent on thinking, and the visible JSON was either cut halfway or missing entirely. On top of that, because thinking tokens weren’t included in the output count in the report, the cost looked incomplete too.

In v1.21.2 (October 31) three changes matter for this post. The output ceiling for thinking models became 16384. Thinking tokens were included in the reported output, the cost and the eval numbers. If the model spends the whole budget on thinking and returns no visible text, the request is retried once with double the ceiling.

I renewed the measurement after this fix, on October 31; the results file now records a single error. The cost per review is $0.006407, thinking tokens included. As long as thinking tokens weren’t in the report, that number came out too low: the model didn’t get more expensive, the first measurement had under-measured.

Result

The permanent home of these numbers is the FAQ page on commitbrief.com and the repository’s results file; I summarize them here only to tell how I set up the measurement. Corpus of 129 fixtures, three runs, measurement dates October 26 for OpenAI and October 31 for Gemini:

Model	Recall	FPR	Cost per review	Passed?
gemini-3.8-flash	0.9731	0.0303	$0.006407	Yes
gpt-5.4-mini	0.9259	0.2929	$0.003674	No (FPR)

Cost is list price, no cache discount; for Gemini it includes thinking tokens. Gemini’s lowest per-language recall is in Go, at 0.9306. The recall of the three runs stayed between 0.9697 and 0.9798.

This is an uncomfortable table. gpt-5.4-mini is still OpenAI’s default. The decision record says that if no candidate passes the threshold, the current default stays and the result must be recorded; I recorded it. But that doesn’t change the fact that I’ve left a model measured to fail the threshold as the default. I haven’t measured any other OpenAI model.

The Gemini side isn’t a victory story either. A rule that looks at price had chosen the default, and the measurement later confirmed the model the rule chose. A single example doesn’t show that the rule predicts quality.

Limits — an honest list again

• Only two models were measured. Anthropic and the other providers, claude-opus-5-5 included, weren’t measured and aren’t ranked. Being unmeasured doesn’t mean being bad.
• setup still shows “not measured”. The wizard doesn’t show the results; it only writes the list price. It has nothing but price to compare on.
• The corpus is synthetic. The fixtures contain deliberately planted bugs, not real PRs. On real diffs the result may differ.
• Few clean controls. 33 clean fixtures don’t resolve FPR as finely as 99 bugs resolve recall. I count this as a known weakness; the remedy is more clean fixtures, not a tighter threshold.
• A snapshot. Providers change their models over time; these numbers are numbers from two dates.

To try it

The thresholds, the corpus and the harness are in the CommitBrief repository. If you want to reproduce the measurement, you can run the harness with COMMITBRIEF_EVAL_PROVIDER=<name> make eval-live; the command appears on the Providers wiki page and in the repository’s Makefile, and it costs API money because it calls a live provider. The corpus makeup and thresholds are on the FAQ page; documentation and installation are at commitbrief.com.

If you’re choosing a default model in your own tool, write the threshold down before measuring: what rate must a model clear to be good enough? Then measure, and even if you don’t like the number, leave the threshold where it is.