Search papers, labs, and topics across Lattice.
The paper introduces Selective Access Transformer (SATFormer), a novel architecture that selectively reuses early-layer representations in later Transformer layers using a context-dependent gate. This approach addresses the limitations of static value residuals, which uniformly expose the first-layer value projection, by allowing for more nuanced access to early lexical or semantic information. Experiments on models ranging from 130M to 1.3B parameters demonstrate that SATFormer consistently improves validation loss and zero-shot accuracy, particularly on retrieval-intensive benchmarks, while maintaining comparable throughput and memory usage to baseline Transformers.
SATFormer shows that selectively gating access to early-layer representations boosts Transformer performance, especially in retrieval tasks, without sacrificing efficiency.
Several recent Transformer architectures expose later layers to representations computed in the earliest layers, motivated by the observation that low-level features can become harder to recover as the residual stream is repeatedly transformed through depth. The cheapest among these methods add static value residuals: learned mixing coefficients that expose the first-layer value projection V_1 uniformly across tokens and heads. More expressive dense or dynamic alternatives recover finer-grained access, but at higher memory cost and lower throughput. The usefulness of V_1 is unlikely to be constant across tokens, heads, and contexts; different positions plausibly require different amounts of access to early lexical or semantic information. We therefore treat early-representation reuse as a retrieval problem rather than a connectivity problem, and introduce Selective Access Transformer (SATFormer), which preserves the first-layer value pathway while controlling access with a context-dependent gate. Across models from 130M to 1.3B parameters, SATFormer consistently improves validation loss and zero-shot accuracy over the static value-residual and Transformer baselines. Its strongest gains appear on retrieval-intensive benchmarks, where it improves over static value residuals by approximately 1.5 average points, while maintaining throughput and memory usage close to the baseline Transformer. Gate analyses suggest sparse, depth-dependent, head-specific, and category-sensitive access patterns, supporting the interpretation that SATFormer learns selective reuse of early representations rather than uniform residual copying. Our code is available at https://github.com/SkyeGunasekaran/SATFormer.
