CcrBench: telling reasoning from memorization by looking at the path
A CLI that solves a problem repeatedly in isolated sessions and measures how diverse the solution paths are, with JSON and HTML reports. Not on PyPI.
What it does today
Has a model solve the problems of a suite YAML repeatedly in isolated sessions, measures how diverse the solution paths are and how much accuracy drops when a problem is changed on the surface, writes a JSON and HTML report per problem and fails CI on suspected memorization. Engineers choosing a model or checkpoint, or producing fine-tuning data, can clone the repository and install it with uv; it is not on PyPI.
- Started
- November 2026 — Ongoing
Form
Technologies
A correct answer does not show that the model solved the problem; it may have seen the answer before. I cannot tell the two apart by looking at the output, so I wrote a tool that looks at the path instead: CcrBench.
Looking at the path, not the output
The method is Cross-Context Review: I have the model solve each problem in N completely independent sessions (default N=10), with temperature and prompt variations, and nothing carried between sessions. Then I look at how much the solution paths differ: edit distance between step sequences, the distribution of transcript embeddings, AST difference on code tasks and branching entropy at decision points. Embeddings are computed locally with an ONNX model, so transcripts never leave the machine. API access is enough; no weights are needed.
One signal is not enough
Low diversity alone raises false alarms: on an easy problem with one natural solution path, everyone takes the same path. So I split the score into two independent axes and do not reduce it to one number. The first is path diversity. The second is variant robustness: I change the problem on the surface (identifier names, numbers) and see how much accuracy drops. I do not have a model generate the variants; they are rule-based and deterministic, so their output is reproducible. The “suspected memorization” label comes out only when low diversity, high original accuracy and a high variant drop occur together; when an axis cannot be computed, the label is “undetermined”, not a guess. The default thresholds (0.3 / 0.8 / 0.3) are provisional; I have not calibrated them yet.
Run, cost and report
ccrbench run suite.yaml --model anthropic:<model> generates the variants, shows a cost preview, runs the sessions, scores them and writes a JSON and an HTML report. Provider adapters exist for Claude, OpenAI, Gemini and OpenAI-compatible local endpoints. Each run gets a session nonce in its system message; transcripts are stored under the hash of their inputs, so repeating a run makes no model call. If the suite sets budget_usd and the estimate exceeds it, the run does not start in a non-interactive environment. A model missing from the bundled price table counts as having an unknown price. Exit code 1 means only “memorization suspected”; configuration and budget errors are 2, an unexpected internal error is 3.
On code problems the correctness check runs the code the model wrote. By default I do that in a Docker container with no network and a read-only filesystem; running it directly on the machine is a separate, explicit flag (--unsafe-local-exec).
Alpha: ten phases closed, installed from a clone
Alpha. The first commit is from 14 November 2026, there are twelve commits as of today and all ten phases of the MVP plan are closed. It is not on PyPI; to try it you clone the repository, run uv sync --locked --extra all and use uv run ccrbench (Python 3.11–3.14). The embedding library (tunedness-embed) is not on PyPI either; it comes as a git dependency. Anthropic’s 2026-10-21 API version rejects temperature; the adapter sends neither temperature nor seed, so on Claude models those two variation axes are not applied, and the report says so as a warning on every affected problem.
I have no published model measurement yet. The next job is testing the discriminating power: deliberately fine-tune an open-weight model on a known subset of the problem set and see whether I can tell the contaminated model from the clean one; the target is AUC ≥ 0.8, and the thresholds will be calibrated against that result.