A10 SPIKE — `Partial[A]` derivation (SPEC §5.6 streaming partial decode)

Status: complete — GO, with a deliberate de-scope of the representation (see §5). Evidence: scratch tests in modules/derivation/src/test/scala/typecast/spike/ (PartialSketchSuite, 5 tests; ReparseBenchSuite, 1 test + printed numbers), package typecast.spike, test scope only. All green under sbt derivation/test.

The target DX is EXAMPLES.md example 5: tc.stream[MarketReport](prompt) emits Partial[MarketReport] where "summary fills in, competitors appear one by one", every already-emitted field is already validated, and .compile.lastOrError is the fully-validated MarketReport. SPEC §10 flags this as the hardest piece; this spike de-risks it.

1. REPRESENTATION — what is `Partial[A]`?

(a) Derived nominal mirror — every field `Option`-wrapped recursively — REJECTED

A per-A generated case class (PartialMarketReport(summary: Option[String], competitors: List[PartialCompetitor], ...)) is the "obvious" reading of SPEC §5.6, but Scala 3 stable cannot synthesize new nominal classes from a derives clause:

Macro annotations (the only mechanism that can introduce new classes) are @experimental on 3.7.1 and cannot add public members visible to downstream code anyway. Out for a Maven-Central library.
sbt-side codegen would work but couples user builds to a plugin — disproportionate for one feature, and hostile to the Java facade (C5).
Structural mirror via recursive match types / named tuples (type DeepPartial[A] = NamedTuple.Map[NamedTuple.From[A], DeepField]) is derivable, but: recursive ADTs (which A5 supports via SRef) make DeepPartial[A] an infinite type; compile-time cost is a second full type-level traversal per schema type; and hover/error types explode into raw NamedTuple[("summary", ...), (Option[String], List[NamedTuple[...]])] spaghetti — the "type-name explosion" risk on the task card, confirmed.
A fully recursive Option-mirror is also semantically worse for the demo: it emits half-built Competitors, which breaks the "each already-emitted field is already validated" wow-moment. "Collections grow by complete, validated elements" is the stronger guarantee.

(b) Generic structure-typed wrapper — viable core, insufficient DX alone

Partial[A](snapshot: Js, complete: Set[JsonPath], schema: LlmSchema[A]) with a typed getter get[B](path)(using LlmSchema[B]): Option[B]. Derivation cost: zero (it rides on A5's LlmSchema[A]). But path-based getters alone make example 5 stringly-typed — not acceptable as the only surface.

(c) Staged hybrid — CHOSEN

Generic core from (b) plus a derived typed view that costs almost nothing:

incomplete: Set[JsonPath] — the rightmost spine of the snapshot (see §3); everything off it is final. (Equivalent to the card's complete: Vector[JsonPath] stated negatively — the incomplete set is finite and tiny, the complete set is not.)
fields: Fields[A] — a named tuple (stable since Scala 3.7) mirroring A's fields one level deep: scalars/nested objects as Option[F] (Some once complete and strict-decoded), collections as-built (List[E] of complete, validated elements so far). Statically typed: p.fields.summary: Option[String], p.fields.competitors: List[Competitor] — example 5's DX exactly. One inline expansion; no new nominal types; no recursion limit issues.
narrow[B](path): Option[Partial[B]] — a typed cursor that rebases the spine onto a subtree. This recovers recursive inspection (look inside the in-flight Competitor) without recursive types: p.narrow[Competitor](root / "competitors" at 1).get.fields.name.
value: Option[A] — whole-document strict decode, Some only once the document closed.

Coexistence with A5: total. The sketch summons ordinary LlmSchema givens per field (the derived Competitor instance, the String primitive given, ...) and never touches typecast.derive.Derive internals. C1 should introduce a small derived PartialSchema[A] typeclass that captures per-field decoders once (instead of re-expanding fields per call site), mirroring how Derive.Part works — but the spike proves the inline route compiles and behaves on 3.7.1 with zero new dependencies.

Verified in PartialSketchSuite for the 2-level EXAMPLES-5 shape (SpMarketReport / SpCompetitor), including static field types, mid-stream as-built lists, narrow, and final-equals-strict.

2. INCREMENTAL PARSE — re-parse per chunk vs. true streaming parser

Decision: re-parse the accumulated buffer with TolerantParser on every chunk. No streaming parser for v1.

A partial token stream is a truncated document, and A1's truncation completion already turns any prefix into a well-formed snapshot ("inserted missing } at end of input", unterminated strings closed, dangling keys dropped) and tells us it did so via RepairKind.Truncated notes — which is exactly the completeness signal §3 needs. The engine is ~10 lines (PartialSketch.fromBuffer).

Micro-benchmark (ReparseBenchSuite, System.nanoTime, Apple-silicon dev box, JDK 17, after warmup; numbers printed by the test). Two recorded runs — a quiet box and one under heavy shared load (parallel sbt builds) — bound the realistic range:

metric	chunk = 100 B (~25-token SSE deltas)	chunk = 400 B
re-parses over a 102,376 B doc	1,024	256
cumulative bytes parsed (the O(n²))	52 MB	13 MB
total parse time (quiet → loaded)	618 ms → 2,091 ms	122 ms → 546 ms
mean / max per chunk (quiet)	0.60 ms / 14.7 ms (JIT/GC outlier)	0.48 ms / 2.1 ms
mean / max per chunk (loaded)	2.04 ms / 49.3 ms	2.13 ms / 5.4 ms
single full 100KB parse	0.87 ms (quiet) / 4.25 ms (loaded)	—
unparseable prefixes (boundary inside `true`/`false`/`null`)	37 (3.6%) — chunk skipped	11 (4.3%)

Feasibility bar: an LLM emitting 100KB takes tens of seconds of wall clock; even the worst measured case (~2.1 s of total CPU spread across ~1,000 chunks, ≤ ~5 ms per chunk at the end-of-stream point on a contended machine) is noise. O(n²) is real but n is bounded by response size, and even 4× the size (400KB → ~16× work ≈ 10 s total) stays under typical generation wall-clock. Mitigations if ever needed, in order: parse every k-th chunk or on a time budget (trivially correct — emissions are snapshots), then a resumable parser (C1 follow-up, not v1).

Two engine rules fall out of the failure rows: (1) a prefix that fails to parse ⇒ skip emission for that chunk (never an error mid-stream); (2) the final buffer goes through the full strict pipeline regardless (§4), so a skipped last chunk cannot lose data.

3. PROGRESSIVE VALIDATION — when is a field "complete and validated" mid-stream?

Rule (rightmost spine). Because the buffer grows strictly by appending, the only values that can still change are those on the path from the root to the last parsed token: at each object take the last field, at each array the last element — the rightmost spine. Therefore:

A path p is complete in snapshot S(buffer) iff a value exists at p and p is not on the rightmost spine (computed only when the parse carried RepairKind.Truncated notes; a clean parse of a closed document has an empty spine).

A complete path is validated when its subtree strict-decodes under the field's LlmSchema (structural) — C1 additionally runs Validator.validate (A3) on the same subtree for ConstraintIR. Whole-A semantic validators (withValidation) run only on the final element.

This matches the card's intuition (closed scalar with all bytes in; closed container) and is conservative by construction: a just-closed value stays "incomplete" for at most one token (until its successor token arrives) but a value is never marked complete while it can still change. Specifically it subsumes the awkward cases for free: a number at EOF (2 of an eventual 25), an unterminated string the parser closed, a closed-but-last array element.

Property verified exhaustively in PartialSketchSuite ("prefix stability"): for every prefix of the test document, every leaf reported complete already equals its value in the final strict document (1,400+ path checks). C1 must keep this as a ScalaCheck property over generated documents and chunkings. One caveat to carry into C1: the property leans on the parser being prefix-monotone; TolerantParser's prose-stripping retry could in principle reinterpret earlier text, so the streaming engine should anchor parsing at the first structural opener of the buffer (one indexOf + offset) — fence streaming already works because an unterminated opening fence extends to EOF.

4. EMISSION SEMANTICS — when to emit, and the final-element guarantee

Compute per chunk, emit on change. Each chunk: append → re-parse → build Partial → emit iff the observable state changed (incomplete spine + the set of complete paths / decoded view). Emitting every raw chunk would spam consumers with identical mid-string states; field granularity is what the UI use case wants. PartialSketchSuite ("chunked emission") demonstrates the dedup (fewer emissions than chunks) and monotone growth (as-built list sizes are non-decreasing, ending at the full list).
Final-element guarantee, by construction: the stream's terminal step does NOT reuse the last per-chunk snapshot. It runs the accumulated buffer through the same strict pipeline as non-streaming extraction (parse → align → validate → decode, i.e. the ExtractionSession code path) and emits Partial.done(a) / fails typed. Last element == strict decode is then an identity, not a theorem; the C1 acceptance property (final element == strict-mode result on fixtures) verifies the wiring. Note: per-chunk partials deliberately skip alignment (key fuzzing etc. is not stable on prefixes); only the terminal element gets stages 2–5, so retries remain a non-streaming concern in v1 (a retry restarts the partial stream).

5. GO / NO-GO + fallback

GO — with the representation de-scoped from "recursive nominal mirror" to the staged hybrid (§1c). The C1 card as literally written ("derived Partial[A]" read as a generated field-mirror class) is NOT achievable on stable Scala 3.7 and shouldn't be attempted; the hybrid delivers example 5's DX verbatim, is strictly stronger on the validated-elements guarantee, and collapses the riskiest derivation work to near zero. With it, C1 at size XL is realistic: the engine (§2) and completeness rule (§3) are ~150 lines of pure core code already sketched here; the bulk of XL is the fs2 surface, ExtractionSession integration, the PartialSchema[A] typeclass, and the property suites. No experimental flag needed for the core design; if anything ships flagged, it is only the fields named-tuple view (the one 3.7-specific surface), with get/narrow/value as the unflagged baseline.

Fallback if even the hybrid view slips: ship representation (b) alone (get/narrow/value, no named-tuple view) — everything in §2–§4 is unchanged, only the typed-view sugar drops out.

RECOMMENDATION (concrete)

Chosen representation — staged hybrid (§1c). API sketch (~30 lines, proven compiling in typecast.spike.PartialSketch):

/** Progressive, validated view of an A being streamed (SPEC §5.6). */
final case class Partial[A] private[stream] (
    snapshot: Js,                  // tolerant parse of the accumulated buffer
    incomplete: Set[JsonPath],     // rightmost spine; empty once the document closed
    schema: LlmSchema[A]
):
  def isDone: Boolean = incomplete.isEmpty
  def isComplete(path: JsonPath): Boolean            // present and off the spine
  def get[B: LlmSchema](path: JsonPath): Option[B]   // Some iff complete AND strict-decodes
  def narrow[B: LlmSchema](path: JsonPath): Option[Partial[B]]  // typed descent, spine rebased
  def value: Option[A]                               // whole-doc strict decode once closed

  /** Typed one-level view: scalars/objects Option-wrapped, collections as-built. */
  inline def fields(using PartialSchema[A]): Fields[A]

/** Named-tuple projection of A's fields — no generated nominal types. */
type FieldView[F] = F match
  case List[e]   => List[e]      // complete, validated elements so far ("lists grow")
  case Vector[e] => Vector[e]
  case Option[t] => Option[t]
  case _         => Option[F]    // Some once complete + validated ("options fill")
type Fields[A] = NamedTuple.Map[NamedTuple.From[A], FieldView]

/** Derived once per A (Mirror + compiletime, summons A5's LlmSchema per field). */
trait PartialSchema[A]:
  def view(p: Partial[A]): Fields[A]

object Stream:  // engine, pure core
  def step[A: LlmSchema](buffer: String): Option[Partial[A]]          // re-parse + spine
  def finish[A: LlmSchema](buffer: String): Either[ExtractionError, A] // strict pipeline

Incremental-parse decision: re-parse accumulated buffer per chunk via TolerantParser (truncation completion = snapshot, Truncated repair notes = completeness signal). Measured on a 102 KB document: 1,024 re-parses at 100 B chunks = 618 ms total (2.1 s on a contended box), 0.6–2.0 ms mean, ≤49 ms max (52 MB cumulative, ~25–85 MB/s); 400 B chunks = 122–546 ms total; single full parse 0.9–4.3 ms. O(n²) is immaterial against LLM generation wall-clock at realistic response sizes. No streaming parser in v1; "parse every k-th chunk" is the escape hatch before ever writing one.

Proposed amended C1 task card

[ ] C1 Streaming Partial[A] — deps: A10 B1 — owns: modules/core/.../stream/, modules/runner-ce/.../stream/, modules/derivation/.../partial/ Per A10 (docs/spikes/partial-derivation.md): generic Partial[A] core (snapshot Js + incomplete rightmost-spine Set[JsonPath] + get/narrow/value) — NOT a generated field-mirror class — plus derived PartialSchema[A] providing the named-tuple fields view (one level deep; collections as-built with complete validated elements; FieldView match type). Incremental engine = TolerantParser re-parse of the accumulated buffer per chunk (A10-benchmarked feasible to ≥100KB; unparseable prefix ⇒ skip emission), parsing anchored at the first structural opener; emit on observable-state change only; terminal element produced by the SAME strict pipeline as extract (parse→align→validate→decode), per-chunk partials are structural-decode + subtree Validator.validate only (no alignment, no semantic validators mid-stream). fs2 surface first; ZStream parity if cheap, else follow-up. Accept: EXAMPLES 5 runs on fixtures (scripted token stream); property: final element == strict-mode result; property: prefix stability (a path reported complete never changes value in any longer prefix, over generated docs × chunkings); fields view compiles with the exact static types of EXAMPLES 5 (Option[String], List[Competitor]). Size: XL. Critical path.

Scratch-test inventory (acceptance evidence)

typecast.spike.PartialSketch — compiling sketch of the chosen representation (core + spine + named-tuple view + narrow), test scope.
typecast.spike.PartialSketchSuite — 5 tests: 2-level typed view mid-stream, typed descent, final == strict, exhaustive prefix-stability property, chunked emission monotonicity/dedup.
typecast.spike.ReparseBenchSuite — 1 test: the §2 micro-benchmark (numbers above printed on each run).

A10 SPIKE — Partial[A] derivation (SPEC §5.6 streaming partial decode)

1. REPRESENTATION — what is Partial[A]?

(a) Derived nominal mirror — every field Option-wrapped recursively — REJECTED