Skip to content
Muhammet Şafak
tr
Cover — Tan the beaver smiles at a laptop while a column behind it turns from scattered pixels into orderly indigo blocks

SlmStrict: getting structured output from a small model through a graded ladder

A Python library I wrote to get schema-valid output from small on-device language models; its first reference benchmark missed some targets.

What it does today

Takes a Pydantic model or a JSON Schema and gets schema-valid output from a local model: grammar (llama.cpp only), deterministic repair, re-ask and a cloud fallback that is off by default (Claude, OpenAI, Gemini) work; there is a metrics collector, cost estimation and an OTLP observer. The MVP is complete but no release is published; in the first reference benchmark two models missed the targets on the ONNX path. Python developers embedding a local model in an application can try it by cloning the repository; it is not on PyPI.

Started
November 2026 — Ongoing

Form

Open-Source

Technologies

Python Pydantic llama.cpp ONNX Runtime OpenTelemetry

In every project that expects JSON from a small language model, the same defensive code gets written: strip the markdown fence, close the missing bracket, retry if that fails. SlmStrict is the Python library I started to write that layer once, with tests.

The four rungs of the ladder

From models under 3B I expect format errors such as drifting from the schema, adding code fences or misreading a field. SlmStrict handles these with a four-rung ladder. First, grammar: if the backend supports it, the JSON Schema is compiled to a grammar and decoding is constrained. Then repair: fence stripping, bracket completion, type coercion; it fixes only form and produces no meaning, because I do not want a model doing work a deterministic script can do. Then re-ask: the validation errors go back to the model, twice by default. The last rung is the cloud fallback. result.path says which rungs actually took effect.

What works in the MVP

generate() and agenerate() follow the same ladder and return the same result; the schema can be a Pydantic model or a JSON Schema. There are two backends: llama.cpp is the only carrier of the grammar rung, and ONNX Runtime (onnxruntime-genai) has no grammar, so its ladder starts at repair. A backend failure does not silently fall through to the cloud. A metrics collector counts runs without storing content, cost estimation uses a price table you supply and says “unknown” without one, and an optional OTLP observer exists.

When data leaves the device

The cloud fallback is off by default: it never runs unless a provider (Claude, OpenAI or Gemini) is configured, and local_only shuts it off again. One default to watch: unless you pass a masking hook, the prompt goes to the cloud as it is; content logging is also refused without a hook.

The first benchmark: targets missed in places

The reference benchmark measures Qwen3-4B, Llama-3.2-3B and Gemma-3-1B on both backends over 36 schema tasks in six categories. The first run was on 16 November 2026, in arm64 Docker (Linux aarch64, 12 CPUs), CPU only, temperature 0, one run per task and mode, with Q4_K_M GGUF files and third-party int4 ONNX conversions; the numbers hold for that setup only. Targets: strict pass rate ≥ 95%, grammar-path overhead ≤ 15%, fallback rate < 10%. The fallback was replaced by a stub that only counts.

On llama.cpp all three models passed strict at 100% with 0% fallback, from raw rates of 94.4% (Qwen3-4B), 22.2% (Llama-3.2-3B) and 2.8% (Gemma-3-1B). The grammar-path overhead was +25.1% for Qwen3-4B, a missed target; for the other two the figure is negative only because the unconstrained model writes longer outputs, so those rows do not show the target is met. On ONNX, Qwen3-4B passed at 100%. Llama-3.2-3B ended at 38.9% strict with 61.1% fallback and Gemma-3-1B at 0% strict with 100% fallback, both far from the targets. A possible cause is how the ONNX path feeds the prompt, but I have not verified it: a hypothesis, not a diagnosis. Only schema validity was measured, not whether the values are correct.

Alpha: MVP complete, no release

The first commit is from 14 November 2026. The MVP’s feature scope is complete and the plan-end security review closed with no findings in its first round. No release is published and it is not on PyPI; you clone the repository and install it with an extra such as uv sync --extra llamacpp. MLX and Android adapters, and routing that splits a task into local subtasks, are outside the MVP.

Tags: #ai#slm#structured-output#local-models
Share:

More Records

All records

A CLI that solves a problem repeatedly in isolated sessions and measures how diverse the solution paths are, with JSON and HTML reports. Not on PyPI.

What it does today

Has a model solve the problems of a suite YAML repeatedly in isolated sessions, measures how diverse the solution paths are and how much accuracy drops when a problem is changed on the surface, writes a JSON and HTML report per problem and fails CI on suspected memorization. Engineers choosing a model or checkpoint, or producing fine-tuning data, can clone the repository and install it with uv; it is not on PyPI.

Open-Source Python Typer Pydantic +3 more
November 2026 — Ongoing

A proxy between an agent and its MCP servers that cuts semantic loops, budget overruns and risky tool calls; its default mode only observes.

What it does today

Sits between an agent and its MCP server, sees every tools/call, and stops repeating loops, sessions that exceed a duration or call budget, and tools the policy forbids. Anyone running agents that call tools over MCP can try it with `npm i -g @tunedness/agentfuse` (0.1.0).

Open-Source TypeScript Node.js MCP +2 more
September 2026 — Ongoing

How a Claude Code skill reads a Sentry issue against the working tree and drafts the fix as a diff and a regression test, touching no file before approval.

What it does today

Reads a Sentry issue against the working tree, produces a root-cause analysis and a fix plan, and touches no file without explicit approval. Anyone using Claude Code who needs to triage a production error can install the skill.

Open-Source Python Bash Sentry API +3 more
July 2026 — September 2026

Search the site

Start typing to search posts, projects and pages.

Esc to close Powered by Pagefind