Same bug, different title: giving an LLM finding an identity

A finding I had put in the baseline kept coming back, even though the bug was the same. The only thing that had changed was the title the model gave it.

For a long time I never asked what makes a finding “the same” in CommitBrief. This post is the story of writing the answer down as the finding’s identity, and of the features that followed from that one decision. I had already told you why I wrote the tool and how I carried it into the PR flow in earlier posts.

v1: the identity was the title

The baseline arrived in v1.8.0 (2026-06-20). In an existing repo you accept today’s findings once, and in later runs you see only the new ones. To recognize a finding, I used this fingerprint:

sha256(File + Severity + normalize(Title))

It survived line shifts: even if a finding moved up or down in the file, it stayed in the baseline. But it depended on the title, and the one writing the title was not me but the model. When the model wrote the same bug up under a different title on the next run, the finding I had baselined counted as “new” and came back. The model re-grading the severity gave the same result.

I had hit the same ceiling in the policy gate in v1.10.0. That day I wrote this in the changelog: “Rule-id-scoped allow/deny lists are deferred (findings carry no stable rule id).” I couldn’t build per-rule allow and deny lists, because a finding had no stable rule ID.

Defining “the same”

In v1.19.0 (2026-10-18) I pulled the identity away from the title and split it into two parts.

The first is the rule ID (rule_id). It is a closed taxonomy of 30 IDs: sec.injection, correctness.error-handling, test.flaky and general.other, among others. The model picks it itself; in the OpenAI, Anthropic and Gemini schemas rule_id is required and its values are restricted to the taxonomy. If an empty or unknown value comes back, there is no retry; it is simply pulled to general.other. I didn’t want a model that is sloppy about labels to spoil a valid review. Deterministic producers supply the ID themselves: the flaky filter writes test.flaky, and leaks writes sec.hardcoded-secret.

The second is the symbol. It doesn’t come from the model; it is derived locally from the diff: in the hunk containing the finding’s line, up to that line, the last line that matches git’s default funcname rule, meaning a line whose first character is a letter, _ or $. Because anything inside parentheses is reduced to (), the symbol doesn’t change when the parameter list changes. So the identity became this:

fingerprint = sha256(file, rule_id, symbol)

Title, severity and line number are not in the formula. The identity is computed locally on every run and is not written to the cache; for a finding that comes from the cache it is recomputed from the current diff. I did it this way so that if the formula ever changes, old identities don’t linger in the cache and match wrongly.

What came with the same identity

Once the identity was stable, the features that had been waiting fell into place one by one.

• Baseline v2 now stores identities. A finding whose title changes doesn’t drop out of the baseline.
• Policy gained rules.deny and rules.allow. A single finding from a rule on the deny list closes the gate; the ones on the allow list are exempt from the thresholds and the total. An entry is either an ID from the taxonomy or <category>.*.
• Inline markers gained rule scope: commitbrief-ignore[sec.injection]: reason silences only that rule’s finding.
• SARIF output carries the identity under partialFingerprints["commitbrief/v1"]. Why SARIF writes no file when it fails is a separate topic; I only mention it here.

The most visible link in the chain is the remote pr change in v1.20.0. It used to post the comments afresh on every run. Now there is a hidden marker at the end of each inline comment:

<!-- commitbrief:fp=<fingerprint> v=1 -->

The next run recognizes its own threads from this marker. It leaves alone a finding that is still present (if the text changed, it edits the comment in place), writes a new finding once, closes the thread of a finding that disappeared, and reopens the thread of a finding that came back. It never touches threads a human has resolved or reopened. Repeats of the same identity are collected in a single thread under Also at:. If it lacks permission to close a thread, it replies No longer present as of <sha>.

--since-last-review arrived in the same release: a <!-- commitbrief:reviewed=<head sha> --> marker is appended to the end of every full-PR review, and with the flag only the new commit range is reviewed. If there is a force-push, a merge commit, or the range can’t be determined safely, it falls back to a full review. Since v1.21.0, remote pr also honors inline silencing; the baseline is not applied to remote pr.

The cost

• The move from v1 to v2 is lossy. The rule ID and symbol can’t be recovered from title hashes; you have to retake the baseline with --update-baseline.
• Comments from before v1.20 have no marker. On a PR that was reviewed earlier, the first v1.20 run writes the findings once more.
• The taxonomy block adds roughly 690 tokens to the prompt, by the changelog’s estimate. Because the system prompt is part of the cache key, existing cache entries miss once after the upgrade.

Limits — an honest list again

• The identity still depends partly on the model. The model picks rule_id and supplies the finding’s line; the line in turn determines which symbol it falls under. If the same bug comes back with a different rule in two runs, the identity changes. I have no measurement to give here of how well this stability holds; the earlier post where I measured models on review quality did not measure rule_id stability.
• The closed taxonomy is coarse. Findings that land in general.other can’t be told apart; their scope in the baseline is also broad.
• Two findings of the same rule in the same symbol collide. A finding taken into the baseline also hides future findings from the same rule in the same symbol.
• An indented line is not a symbol. In Python class methods the symbol can fall to the class line or to an empty value; sibling methods share the same symbol.
• The symbol depends on the kind of diff. The scan looks at the new side’s lines first and falls back to the hunk header, which git writes from the old code. For a function renamed within the range, --staged and main..HEAD can give a different symbol, and therefore a different identity. Compare with the same kind of diff you took the baseline from. The identity also changes when the file name changes.
• CLI-based providers produce plain text, so there is no rule_id and no fingerprint.

Try it

• github.com/CommitBrief/commitbrief
• commitbrief.com

Put a finding into the baseline, in your own tool or in CommitBrief, and see whether it comes back on the next run. If it does, today is a good day to write down what your tool ties “the same finding” to.