Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors design the Graph Machine (GM), an architecture that replaces dense self-attention with sparse, dynamic routing over an unconstrained $O(n)$ state using differentiable pointer-like edges. This design overcomes the expressive bottlenecks of $O(1)$ recurrent states and the inflexibility of static sparsity by enabling pointer-chasing referral across the full sequence context. When replacing 75% of dense attention layers in a Qwen3-0.6B model pretrained on 15.7B tokens, the architecture matches the dense baseline by retrieving only 2 of 4,096 tokens per KV head, and marginally outperforms it when retrieving 4.
Retrieving as few as 4 out of 4,096 tokens per head via dynamic pointer chasing is enough to match—and even marginally beat—dense Transformer pretraining loss.
We introduce the Graph Machine (GM), an architecture that maintains an $O(n)$-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves $O(n)$ complexity in its sparse layers without restricting the potentially accessible state size to $O(1)$. Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.