Search papers, labs, and topics across Lattice.
Pretrained transformers exhibit limited context following capabilities, with base models reliably attending to only a few lines of context, a limitation not significantly improved by additional pretraining. A rank-8 LoRA fine-tuned on a specific task, applied to an early layer with frozen base model weights, dramatically extends this computational depth, achieving near-perfect accuracy on 24-line chains and significantly longer chains with further training. This LoRA initiates a "relay" mechanism where program lines pass chain identity through middle layers, enabling
Key contribution not extracted.
Pretrained transformers use little of their depth to follow references in context. Thirteen base models reliably follow only 1.4-3.6 lines, and extra pretrained loops add little. A task-trained rank-8 LoRA at one early layer extends this computation with all model weights frozen. Qwen3-8B improves from 15.5% to 99% exact accuracy on 24-line chains; a longer-trained LoRA reaches 50 lines. Ouro-1.4B reaches 60 lines after four loops and at least 160 after eight. The LoRA starts a relay: program lines pass on their chain identity through a short range of middle layers. Frozen heads read progressively further up the chain, and removing parent-line attention stops the relay. A frozen-model measurement locates the last useful intervention layer within tolerance in three of four held-out models. Task-specific LoRAs also improve MuSiQue. Default answers therefore understate the computation accessible through a tiny edit. Code and an interactive demo are available at https://lunamos.github.io/stop-thinking-too-early/
