Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates how the site of adaptation in Transformer models influences learning outcomes, generalization, and selectivity, using a benchmark that encompasses five distinct objectives. The authors define "adaptation geometry" for each objective, revealing that different learning tasks benefit from specific adaptation layers—early layers favor lexical binding, while later layers are more effective for factual associations and behavioral learning. The results indicate that the adaptation site is a critical factor in model design, as it significantly impacts the learning dynamics and transfer capabilities across various tasks.
Adapting Transformers at different layers can drastically alter their learning outcomes, with distinct tasks requiring tailored adaptation strategies for optimal performance.
Transformer adaptation is typically distributed across model depth, even when the intended change is narrow. We investigate how adaptation site shapes what a model learns, how well that learning generalizes, and how selectively it is applied. We introduce a controlled benchmark spanning five objectives (lexical binding, factual association, behavioral policy learning, causal mapping, and procedural reasoning) and define each objective's"adaptation geometry"as its profile of acquisition, transfer, and boundedness under full-stack and early-, middle-, or late-layer LoRA. The objectives exhibit distinct geometries. Lexical binding favors early-layer adaptation for acquisition and boundedness but requires broader updates for transfer; factual association favors later layers among localized adapters; behavioral learning separates late-layer action acquisition from middle-layer policy gating; and causal and procedural transfer benefit most from middle- or full-stack adaptation. These patterns largely persist under parameter-matched controls, and most corresponding directional contrasts replicate across five model families. These findings establish adaptation site as a key design variable for controlling what models learn, generalize, and leave unchanged.