Benchmarks

The benchmark harness (typecast-bench, not published) measures the thing the library claims: how much of the extraction failure rate repair and targeted retry remove, per provider, on the same code. No numbers are quoted here — the published table comes from real harness runs at launch (task D3 in the project plan); anything else would be fabrication.

Running it

sbt "bench/runMain typecast.bench.BenchMain --suites extraction,union --tiers raw,repair+retry --limit 5 --out bench-out"

usage: BenchMain [--suites a,b] [--tiers a,b] [--targets a,b] [--limit N] [--out DIR]
  --suites   comma-separated subset of: extraction, union, constraint (default: all)
  --tiers    comma-separated subset of: raw, repair, repair+retry (default: all)
  --targets  comma-separated target names (default: every available target)
  --limit    run only the first N cases of each suite
  --out      output directory (default: bench-out)

Targets are discovered from the environment — only configured providers run:

Target	Enabled by	Default model (override)
anthropic	ANTHROPIC_API_KEY	claude-haiku-4-5 (ANTHROPIC_MODEL)
openai	OPENAI_API_KEY	gpt-4o-mini (OPENAI_MODEL)
ollama	a reachable server at localhost:11434	llama3.2 (OLLAMA_MODEL)

Requests are strictly sequential (one in flight at any time — rate-kindness) at temperature 0. Outputs: results.jsonl (one record per case: target, suite, tier, outcome, tokens, latency) and table.md (the success-rate matrix).

The tiers

The sweep axis is how much of the TypeCast pipeline is switched on — each tier is a column:

• raw — direct transport call with the wire schema, then a STRICT parse and strict decode. Any tolerant repair counts as failure; no alignment, no retry. This is the "just ask for JSON and hope" baseline.
• repair (+repair in the table) — the full pipeline with RetryPolicy.none: tolerant parse, schema alignment, validation — fail fast, zero extra round-trips. Everything this tier gains over raw is free (deterministic, no tokens spent).
• repair+retry (+repair+targeted retry) — the full pipeline with RetryPolicy.targeted(3): validation failures become JSONPath-targeted reprompts.

The suites

• extraction — invoice-style nested documents (the rich-types story: BigDecimal, dates, nested lists).
• union — discriminated-union triage (the sealed-trait story).
• constraint — refinement-constrained extraction (the validate-and-retry story).

Cases are committed, deterministic datasets — never fetched at run time.

The claim under test

The hypothesis the published table will confirm or refute: small local models become production-reliable with repair+retry — i.e. the ollama row's journey from the raw column to the repair+retry column closes most of the gap to hosted frontier models, which changes the cost conversation for high-volume extraction. Published results: 2026-06-12 Ollama run.