Skip to content

Benchmark run — 2026-06-12, Ollama (llama3.2)

Published results of the TypeCast benchmark harness (typecast-bench; see docs/benchmarks.md for the harness guide). Raw per-case records are committed alongside this file: 2026-06-12-ollama/results.jsonl, with the rendered matrix in 2026-06-12-ollama/table.md.

Results

raw +repair +repair+targeted retry
ollama (llama3.2) 76% (1.0 att) 76% (1.0 att) 76% (1.0 att)

Per-suite breakdown (from results.jsonl):

suite cases raw +repair +repair+targeted retry
extraction 20 20/20 20/20 20/20
union 15 15/15 15/15 15/15
constraint 15 3/15 3/15 3/15

Findings

  1. The tiers are flat on this target because Ollama enforces the schema server-side. The transport sends the lowered wire schema as a json_schema response_format, and Ollama applies grammar-constrained decoding: every response is already syntactically valid JSON in the right shape, so the raw tier's strict parse never sees malformed output and repair has nothing to fix. The repair tiers' value proposition targets providers/modes without server-side enforcement; this run shows the end-to-end pipeline holds, not repair/retry lift.
  2. Every constraint-suite failure is a mismatch, not a validation failure — the model emits values that satisfy the constraints but differ from the expected answer (e.g. an in-range but wrong score where the prompt demands clamping an out-of-range raw reading to the nearest bound, or transposed digits in the unit code). Because the emitted values validate, targeted retry is never triggered (attempts = 1.0 everywhere) — grammar-constrained decoding pre-empts exactly the validation failures the bottom-right cell exists to recover. Measuring that cell needs a target without server-side constraint enforcement.
  3. A first sweep (same day, same invocation) found a real lowering bug. Ollama 0.30.7's json_schema/GBNF converter rejects unanchored regex patterns with HTTP 400 (Pattern must start with '^' and end with '$'), which killed all 45 constraint case-runs of that sweep at the transport stage. Fixed in commit 9c40073 (Ollama's schema subset now declares pattern unsupported, so it lowers off the wire and the validate-and-retry backstop enforces it post-hoc); the published numbers above are from a full rerun after that fix.

Targets

Targets are discovered from the environment. In this run:

  • ollama — ran. Server reachable at localhost:11434, Ollama version 0.30.7. Model: llama3.2:latest (digest a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72, 3.2B parameters, Q4_K_M quantization, GGUF). Transport: the OpenAI-compatible endpoint (http://localhost:11434/v1) in JSON mode.
  • anthropic — did not run (ANTHROPIC_API_KEY not set in the environment).
  • openai — did not run (OPENAI_API_KEY not set in the environment).

Methodology

Tiers

The sweep axis is how much of the TypeCast pipeline is switched on (definitions as implemented in typecast.bench.Tier / TierRunner):

  • raw — direct transport call with the wire schema, then a STRICT parse (any tolerant repair counts as failure) and strict decode; no alignment, no validation-driven retry.
  • repair (+repair) — the full TypeCast pipeline with RetryPolicy.none (repair + alignment + validation, fail fast).
  • repair+retry (+repair+targeted retry) — the full pipeline with RetryPolicy.targeted(3): validation failures become JSONPath-targeted reprompts.

Scoring is typed equality of the extracted value against the expected value (BigDecimal fields compare numerically, so wire-scale differences like 13.5 vs 13.50 never fail). Requests are strictly sequential — one in flight at any time — at temperature 0.

Datasets

Committed, deterministic synthetic datasets, generated by typecast.bench.DatasetGenerator with fixed master seed 20260611 (each suite derives its own offset) and pinned by a regenerate-on-mismatch golden test — never fetched or regenerated at run time:

  • extraction — 20 invoice-style nested documents (BigDecimal money, nested line-item lists); every prompt states each expected field value verbatim.
  • union — 15 discriminated-union triage cases over a 3-variant sealed trait (refund / escalate / reply), discriminated on the derived "type" tag.
  • constraint — 15 refinement-constrained readings (a score bounded 0–100, a [A-Z]{3}-[0-9]{2} unit code) where the literally-stated answer violates a constraint, so only constraint-aware behaviour scores.

Full sweep: 50 cases × 3 tiers × 1 target = 150 case-runs.

Hardware and timing

  • Apple M1 Max (arm64), 64 GB RAM, macOS (Darwin 25.0.0); Ollama running locally on the same machine.
  • Sweep wall time: ~197 s of model time (~220 s end to end including sbt). Mean per-case latency ~1.3 s (max ~3.1 s). Token totals across the run: 23,223 in / 8,415 out.

Reproduction

./bench.sh

which probes Ollama, refuses with a clear message if it is unreachable, and runs the exact published invocation (writing to bench-out by default; pass an argument to change the output directory):

sbt "bench/runMain typecast.bench.BenchMain --suites extraction,union,constraint --tiers raw,repair,repair+retry --targets ollama --out bench-out"

The published results.jsonl / table.md above are the verbatim outputs of one such ./bench.sh run at commit 9c40073.

Caveats

  • Single machine, single run, no statistical treatment. N = 20/15/15 per suite, one pass, no confidence intervals (v1). Numbers vary run-to-run even at temperature 0 (local inference is not bit-reproducible across Ollama/llama.cpp versions); expect cell-level movement of a few percentage points on a reproduction — the table SHAPE (rows × columns) and the per-suite pass/fail pattern are what should match.
  • One small local model. llama3.2 3.2B at Q4_K_M says nothing about other models or quantizations; the hosted rows (anthropic/openai) are absent, so the headline local-vs-frontier comparison in docs/benchmarks.md remains open until a keyed run is published.
  • This target cannot show repair/retry lift (findings 1–2 above): server-side grammar-constrained decoding makes the raw baseline as strong as the full pipeline on these suites. The tier deltas the harness exists to measure need providers/modes without that enforcement.
  • A transient Ollama timeout would have been counted as a failure for that case; none occurred in this run (every failure is a scored mismatch, none transport/parse/validate).