# SlmStrict: getting structured output from a small model through a graded ladder

> A Python library I wrote to get schema-valid output from small on-device language models; its first reference benchmark missed some targets.

- What it does today: Takes a Pydantic model or a JSON Schema and gets schema-valid output from a local model: grammar (llama.cpp only), deterministic repair, re-ask and a cloud fallback that is off by default (Claude, OpenAI, Gemini) work; there is a metrics collector, cost estimation and an OTLP observer. The MVP is complete but no release is published; in the first reference benchmark two models missed the targets on the ONNX path. Python developers embedding a local model in an application can try it by cloning the repository; it is not on PyPI.
- Form: Open-Source
- Maturity: Alpha
- State: In development
- Focus: AI and AI-assisted development
- Started: 2026-11-14
- Technologies: Python, Pydantic, llama.cpp, ONNX Runtime, OpenTelemetry
- Tags: #ai, #slm, #structured-output, #local-models
- Source code: https://github.com/Tunedness/SlmStrict
- Source: https://muhammetsafak.com/labs/slmstrict-lab/
- Language: en-US
- Author: Muhammet Şafak

---
In every project that expects JSON from a small language model, the same defensive code gets written: strip the markdown fence, close the missing bracket, retry if that fails. SlmStrict is the Python library I started to write that layer once, with tests.

## The four rungs of the ladder

From models under 3B I expect format errors such as drifting from the schema, adding code fences or misreading a field. SlmStrict handles these with a four-rung ladder. First, grammar: if the backend supports it, the JSON Schema is compiled to a grammar and decoding is constrained. Then repair: fence stripping, bracket completion, type coercion; it fixes only form and produces no meaning, because I do not want a model doing work a deterministic script can do. Then re-ask: the validation errors go back to the model, twice by default. The last rung is the cloud fallback. `result.path` says which rungs actually took effect.

## What works in the MVP

`generate()` and `agenerate()` follow the same ladder and return the same result; the schema can be a Pydantic model or a JSON Schema. There are two backends: llama.cpp is the only carrier of the grammar rung, and ONNX Runtime (`onnxruntime-genai`) has no grammar, so its ladder starts at repair. A backend failure does not silently fall through to the cloud. A metrics collector counts runs without storing content, cost estimation uses a price table you supply and says "unknown" without one, and an optional OTLP observer exists.

## When data leaves the device

The cloud fallback is off by default: it never runs unless a provider (Claude, OpenAI or Gemini) is configured, and `local_only` shuts it off again. One default to watch: unless you pass a masking hook, the prompt goes to the cloud as it is; content logging is also refused without a hook.

## The first benchmark: targets missed in places

The reference benchmark measures Qwen3-4B, Llama-3.2-3B and Gemma-3-1B on both backends over 36 schema tasks in six categories. The first run was on 16 November 2026, in arm64 Docker (Linux aarch64, 12 CPUs), CPU only, temperature 0, one run per task and mode, with Q4_K_M GGUF files and third-party int4 ONNX conversions; the numbers hold for that setup only. Targets: strict pass rate ≥ 95%, grammar-path overhead ≤ 15%, fallback rate < 10%. The fallback was replaced by a stub that only counts.

On llama.cpp all three models passed strict at 100% with 0% fallback, from raw rates of 94.4% (Qwen3-4B), 22.2% (Llama-3.2-3B) and 2.8% (Gemma-3-1B). The grammar-path overhead was +25.1% for Qwen3-4B, a missed target; for the other two the figure is negative only because the unconstrained model writes longer outputs, so those rows do not show the target is met. On ONNX, Qwen3-4B passed at 100%. Llama-3.2-3B ended at 38.9% strict with 61.1% fallback and Gemma-3-1B at 0% strict with 100% fallback, both far from the targets. A possible cause is how the ONNX path feeds the prompt, but I have not verified it: a hypothesis, not a diagnosis. Only schema validity was measured, not whether the values are correct.

## Alpha: MVP complete, no release

The first commit is from 14 November 2026. The MVP's feature scope is complete and the plan-end security review closed with no findings in its first round. No release is published and it is not on PyPI; you clone the repository and install it with an extra such as `uv sync --extra llamacpp`. MLX and Android adapters, and routing that splits a task into local subtasks, are outside the MVP.
