Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates "prolepsis" in small transformers (Gemma 2B, Llama 3.2 1B), defined as early commitment to a decision that is sustained by task-specific attention heads without later correction. Through a series of five targeted experiments, the authors find that commitment requires more than 16 layers, planning sites are invisible to residual stream methods without causal language tracing, and specific attention heads route decisions to the output. They also show that factual recall exhibits the same architectural motif as planning, but at a different network depth and with different heads.
Even in small transformers, early commitment to a decision ("prolepsis") is an architectural feature, with task-specific attention heads acting as irrevocable routing mechanisms.
When do transformers commit to a decision, and what prevents them from correcting it? We introduce \textbf{prolepsis}: a transformer commits early, task-specific attention heads sustain the commitment, and no layer corrects it. Replicating \citeauthor{lindsey2025biology}'s (\citeyear{lindsey2025biology}) planning-site finding on open models (Gemma~2 2B, Llama~3.2 1B), we ask five questions. (Q1)~Planning is invisible to six residual-stream methods; CLTs are necessary. (Q2)~The planning-site spike replicates with identical geometry. (Q3)~Specific attention heads route the decision to the output, filling a gap flagged as invisible to attribution graphs. (Q4)~Search requires ${\leq}16$ layers; commitment requires more. (Q5)~Factual recall shows the same motif at a different network depth, with zero overlap between recurring planning heads and the factual top-10. Prolepsis is architectural: the template is shared, the routing substrates differ. All experiments run on a single consumer GPU (16\,GB VRAM).