Skip to content
Muhammet Şafak
tr
Cover — Tan the beaver smiles while holding a laptop, a branching flow diagram glowing above it

CcrBench: telling reasoning from memorization by looking at the path

A CLI that solves a problem repeatedly in isolated sessions and measures how diverse the solution paths are, with JSON and HTML reports. Not on PyPI.

What it does today

Has a model solve the problems of a suite YAML repeatedly in isolated sessions, measures how diverse the solution paths are and how much accuracy drops when a problem is changed on the surface, writes a JSON and HTML report per problem and fails CI on suspected memorization. Engineers choosing a model or checkpoint, or producing fine-tuning data, can clone the repository and install it with uv; it is not on PyPI.

Started
November 2026 — Ongoing

Form

Open-Source

Technologies

Python Typer Pydantic YAML Docker ONNX Runtime

A correct answer does not show that the model solved the problem; it may have seen the answer before. I cannot tell the two apart by looking at the output, so I wrote a tool that looks at the path instead: CcrBench.

Looking at the path, not the output

The method is Cross-Context Review: I have the model solve each problem in N completely independent sessions (default N=10), with temperature and prompt variations, and nothing carried between sessions. Then I look at how much the solution paths differ: edit distance between step sequences, the distribution of transcript embeddings, AST difference on code tasks and branching entropy at decision points. Embeddings are computed locally with an ONNX model, so transcripts never leave the machine. API access is enough; no weights are needed.

One signal is not enough

Low diversity alone raises false alarms: on an easy problem with one natural solution path, everyone takes the same path. So I split the score into two independent axes and do not reduce it to one number. The first is path diversity. The second is variant robustness: I change the problem on the surface (identifier names, numbers) and see how much accuracy drops. I do not have a model generate the variants; they are rule-based and deterministic, so their output is reproducible. The “suspected memorization” label comes out only when low diversity, high original accuracy and a high variant drop occur together; when an axis cannot be computed, the label is “undetermined”, not a guess. The default thresholds (0.3 / 0.8 / 0.3) are provisional; I have not calibrated them yet.

Run, cost and report

ccrbench run suite.yaml --model anthropic:<model> generates the variants, shows a cost preview, runs the sessions, scores them and writes a JSON and an HTML report. Provider adapters exist for Claude, OpenAI, Gemini and OpenAI-compatible local endpoints. Each run gets a session nonce in its system message; transcripts are stored under the hash of their inputs, so repeating a run makes no model call. If the suite sets budget_usd and the estimate exceeds it, the run does not start in a non-interactive environment. A model missing from the bundled price table counts as having an unknown price. Exit code 1 means only “memorization suspected”; configuration and budget errors are 2, an unexpected internal error is 3.

On code problems the correctness check runs the code the model wrote. By default I do that in a Docker container with no network and a read-only filesystem; running it directly on the machine is a separate, explicit flag (--unsafe-local-exec).

Alpha: ten phases closed, installed from a clone

Alpha. The first commit is from 14 November 2026, there are twelve commits as of today and all ten phases of the MVP plan are closed. It is not on PyPI; to try it you clone the repository, run uv sync --locked --extra all and use uv run ccrbench (Python 3.11–3.14). The embedding library (tunedness-embed) is not on PyPI either; it comes as a git dependency. Anthropic’s 2026-10-21 API version rejects temperature; the adapter sends neither temperature nor seed, so on Claude models those two variation axes are not applied, and the report says so as a warning on every affected problem.

I have no published model measurement yet. The next job is testing the discriminating power: deliberately fine-tune an open-weight model on a known subset of the problem set and see whether I can tell the contaminated model from the clean one; the target is AUC ≥ 0.8, and the thresholds will be calibrated against that result.

Tags: #ai#benchmark#evaluation#llm
Share:

More Records

All records

A Python library I wrote to get schema-valid output from small on-device language models; its first reference benchmark missed some targets.

What it does today

Takes a Pydantic model or a JSON Schema and gets schema-valid output from a local model: grammar (llama.cpp only), deterministic repair, re-ask and a cloud fallback that is off by default (Claude, OpenAI, Gemini) work; there is a metrics collector, cost estimation and an OTLP observer. The MVP is complete but no release is published; in the first reference benchmark two models missed the targets on the ONNX path. Python developers embedding a local model in an application can try it by cloning the repository; it is not on PyPI.

Open-Source Python Pydantic llama.cpp +2 more
November 2026 — Ongoing

A proxy between an agent and its MCP servers that cuts semantic loops, budget overruns and risky tool calls; its default mode only observes.

What it does today

Sits between an agent and its MCP server, sees every tools/call, and stops repeating loops, sessions that exceed a duration or call budget, and tools the policy forbids. Anyone running agents that call tools over MCP can try it with `npm i -g @tunedness/agentfuse` (0.1.0).

Open-Source TypeScript Node.js MCP +2 more
September 2026 — Ongoing

A proxy between an agent and its MCP servers: scans results for injection, masks PII, enforces access and logs decisions to a hash-chained audit log.

What it does today

Sits between an agent and its MCP servers over stdio (`wrap`) or HTTP (`serve`), scans tool results and resource contents for prompt injection, masks personal data, enforces access by tool, role and dangerous capability combination, pins a server's tool definitions and writes every decision to a hash-chained audit log. Anyone wiring MCP servers into agents can try it by cloning and building the repository; it is not published on npm.

Open-Source TypeScript Node.js MCP +2 more
September 2026 — Ongoing

Search the site

Start typing to search posts, projects and pages.

Esc to close Powered by Pagefind