Search papers, labs, and topics across Lattice.
This paper explores the integration of a slow coupling mechanism in hierarchical pretraining of neural networks, framed through the lens of singularly perturbed ordinary differential equations (ODEs). By implementing a fast path of standard causal attention alongside a slower path utilizing pooled tokens, the authors demonstrate that this dual approach can maintain performance while significantly reducing computational costs. The key finding reveals that, at a certain token threshold, the slow coupling mechanism does not enhance performance but provides a structured mapping that could inform future model architectures.
A dual-path attention mechanism reveals that slower coupling does not improve performance but offers a structured mapping that could redefine hierarchical pretraining strategies.
Causal self-attention is a coupling mechanism: each token's hidden state is updated by a learned mixture of preceding tokens at the same timescale. This paper asks whether a second, temporally slower coupling-a slow sub-system operating on a temporally-downsampled view of the sequence and fed back into the fast path through a zero-initialised gate-complements it. The question is framed in the language of singularly perturbed ordinary differential equations (ODEs), where the fast variable $x$ evolves at the token rate, the slow variable $y$ evolves at one update per $P$ tokens, and the timescale ratio $\varepsilon = 1/P$ is enforced structurally by causal block-mean pooling. The paper instantiates the fast-slow ODE formalism as a concrete neural network: a fast path of standard causal attention over $T$ tokens, a slow path of full attention over $T/P$ pooled tokens ($P^2 \times$ cheaper per layer), and a zero-initialised additive gate. In addition, under a linear-generator assumption on the fast dynamics, we prove that the equilibrium manifold $x = 蠁(y)$ is exactly the master-equation (ME) stationary distribution $p_{\mathrm{st}}(y)$; in that regime a learned MLP $蠁_胃(y)$ is a variational approximation of it (the trained block is not a generator, so this identity is the structured limit, not a claim about the network as trained). Empirically, at $500$k tokens the coupling is neutral -- the gate stays closed and the coupled and frozen ablations are within run-to-run noise -- at a wall-clock cost comparable to a dense baseline. The contribution is the precise, gap-marked mapping itself, not a performance gain.
