Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Continuum Attention (CA) replaces discrete key-value lookups in causal Transformers with integration over a continuous, piecewise linear memory curve, where queries induce an exponential relevance density. This continuous integration, computed via closed-form exponential moments, yields exact normalized attention including half-token boundary contributions and analytically derived gradients, enabling efficient GPU implementation. CA outperforms standard softmax and focal attention on perplexity benchmarks, demonstrating the efficacy of continuous integration for attention mechanisms.
Continuous integration over a piecewise linear memory curve unlocks exact normalized attention with analytical gradients, outperforming softmax and focal attention in causal Transformers.
We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Integrating the density and interpolated values yields exact normalized attention from closed form exponential moments, including half token boundary contributions. The gradients follow analytically from moments of order at most two, enabling parallel GPU implementations while retaining quadratic sequence complexity and the standard Transformer decoder. Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by 0.0200 and 0.0480 nats per byte, respectively, winning all paired comparisons. An additional disjoint validation slice yields a 0.0457 nats per byte advantage over the strongest selected softmax baseline, with a paired 95% confidence interval. These findings motivate continuous integration as a promising attention alternative while leaving larger scale generalization open.