Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Kan Extension Transformers (KETs) are introduced as a categorical framework that unifies various Transformer implementations, framing a Transformer layer as a weighted structured extension operator. This perspective connects attention mechanisms to diffusion-style completion and introduces a self-conditioning mechanism using detached predictive carriers to expose noncausal structure. Empirical validation across 12 Transformer implementations on Penn Treebank, WikiText-2, and WikiText-103 demonstrates that predict-detach self-conditioning yields the most significant performance gains compared to varying neighborhood families alone.
Predict-detach self-conditioning, not just fancy attention neighborhoods, unlocks the biggest performance gains in Transformers.
We propose Kan Extension Transformers (KETs) as a unifying categorical framework for a diverse group of Transformer implementations. The core claim is that a Transformer layer can be viewed as a weighted structured extension operator: standard attention is the singleton-neighborhood case, Geometric Transformer style incidence mixing is a sparse edge-restricted case, and KET is the higher-order simplicial case. This lens also clarifies a bridge to diffusion-style completion. When the extension operator acts on detached predictive carriers instead of teacher-forced hidden states, it becomes a valid self-conditioning mechanism that exposes noncausal structure without leaking gold future tokens. We include a comprehensive experimental validation of 12 different Transformer implementations varying across strict-causal and predict-detach regimes on Penn Treebank, WikiText-2, and WikiText-103. In the strict-causal setting, quadratic KET is the strongest model among the compared causal architectures on WikiText-2 and WikiText-103. Across all datasets, however, the largest gains come from the predict-detach regime rather than from changing the neighborhood family alone.