Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Continuous Acoustic Wave Networks (CAWN), a novel sequence-mixing architecture for autoregressive language modeling that replaces discrete attention with a continuous, complex-domain phase accumulation mechanism. CAWN incorporates innovations like Selective Phase Resonance with Frequency-Dependent Retention, Hard-Threshold Gating, and Depth-wise Harmonic Convolutions to prevent signal degradation and improve spatial frequency mixing. Evaluated on a 150M-parameter model trained on 100B tokens, CAWN demonstrates robust vocabulary acquisition and contextual denoising, achieving targeted information retrieval across 2M tokens with only 8.72 GB peak VRAM.
Forget quadratic scaling – CAWN's continuous wave approach lets you retrieve information from 2 million tokens while barely touching 8GB of VRAM.
Modern Large Language Models (LLMs) rely on Transformer self-attention, which scales quadratically with sequence length. Recent linear-time alternatives, like State Space Models (SSMs), often suffer from signal degradation over extended contexts. We introduce the Continuous Acoustic Wave Network (CAWN), a fully continuous sequence-mixing architecture. Instead of discrete matrix-based attention, CAWN projects hidden states into multi-headed complex-domain phasors, achieving sequence mixing through a causal, $O(L)$ Phase Accumulation mechanism. To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism incorporating Frequency-Dependent Retention, Hard-Threshold Gating via Straight-Through Estimation, and a Temporal Syntax Cache to capture short-term local dependencies. We also replace standard dense linear projections with Depth-wise Harmonic Convolutions for optimal spatial frequency mixing, augmented by Block Attention Residuals for depth-wise state routing. Scaled to a 150M-parameter model, CAWN utilizes custom Triton kernels for hardware-efficient, true-complex phase accumulation in float32. Trained via a continuous streaming loop on a 100-Billion-token corpus, the prototype is evaluated at a 5-Billion-token milestone. Empirical evaluations via a Targeted Semantic Retrieval protocol demonstrate robust vocabulary acquisition and extended explicitly learned contextual denoising. By leveraging $O(1)$ state-passing via chunked prefill, the model retrieves targeted information across 2,000,000 tokens while strictly plateauing at 8.72 GB of Peak VRAM, empirically overcoming the $O(L^2)$ context memory wall.