Search papers, labs, and topics across Lattice.
This study investigates how different input pathways affect the compositional binding of information in small transformers, revealing that no informative route achieves converged zero-shot composition despite a theoretical ceiling of 1.0. The authors identify a two-factor model explaining few-shot binding efficiency through input-pathway parameter sharing and code readability, while also detailing the transient behaviors of distributed codes during training. Key findings include the diagnostic constraints of symbolic and index routes, which struggle with readout accuracy and binding, respectively, highlighting the nuanced interplay between input format and model performance.
Few-shot binding in transformers hinges on the interplay of input pathways and code readability, revealing that traditional symbolic routes may fail at critical readout stages.
How does the way information reaches a transformer -- as symbolic tokens, a clean per-factor "oracle" code, or an entangled perceptual vector -- shape whether it binds that information compositionally? We study ~6-10K-parameter transformers on finite factored worlds enumerated exhaustively, so every measurement covers the whole input space (zero sampling variance) and the informative routes are information-matched (exact Bayes ceiling 1.0). We report four findings. (1) Endpoint invariance: on held-out binding queries no informative route reaches converged zero-shot composition -- each ends at or below chance despite a ceiling of 1.0, so within a bounded sweep the failure reflects inductive bias under a lookup-sufficient objective, not missing information. (2) A two-factor account of few-shot binding: sample efficiency is best explained by input-pathway parameter sharing and code readability; a dimension-matched control and a graded readability sweep isolate readability from input dimension, and the clean oracle is not the most sample-efficient readable route. (3) A double dissociation: early in training, distributed -- but not index-like -- codes pass through a transient above-chance phase (tracking code format), while few-shot efficiency tracks pathway sharing. (4) Failure anatomy: symbolic routes lose the answer at the readout; index routes mis-bind (the answer stays decodable, yet an input intervention shows the output tracks the wrong slot); entangled routes inherit their input's readability. The central claim is the two-factor account; the endpoint and anatomy results are diagnostic constraints. All code, manifests, and per-seed logs are released for exact reproduction.
