Search papers, labs, and topics across Lattice.
This paper introduces DeltaFlow, a noise-adaptive bidirectional Gated Delta Network (GDN) designed to enhance the efficiency of Embedded Language Flows (ELF) by addressing the high computational costs associated with full non-causal attention. By comparing two variants鈥擠eltaFlow-A and DeltaFlow-P鈥攖he authors demonstrate that DeltaFlow-P significantly reduces perplexity from 24.218 to 21.228 on OpenWebText while achieving a 2.72x throughput speedup in denoising tasks. These findings highlight DeltaFlow's potential as a more efficient alternative to traditional dense attention mechanisms in continuous language denoising applications.
DeltaFlow-P achieves a remarkable 2.72x speedup in denoising while reducing perplexity, challenging the dominance of full attention in language modeling.
Embedded Language Flows (ELF) rely primarily on full non-causal attention for iterative denoising, repeatedly incurring quadratic sequence-mixing cost at each sampling step. Gated Delta Networks (GDNs) provide an efficient recurrent alternative, but their standard causal formulation cannot directly capture the bidirectional context required by ELF. We introduce DeltaFlow, a noise-adaptive bidirectional GDN backbone for continuous language denoising. We study two variants: DeltaFlow-A, which alternates scan directions across layers, and DeltaFlow-P, which performs parallel forward and backward scans within each layer. We further introduce noise-adaptive memory control and scheduled Temporal State Consistency (TSC) to stabilize hidden representations across nearby noise levels. On OpenWebText, using a 32-step stochastic differential equation sampler, DeltaFlow-P reduces generated perplexity from 24.218 for the full-attention ELF baseline to 21.228 while maintaining comparable unigram entropy, with 36B training-token exposure compared with 45B for the baseline. In a denoiser-only benchmark, DeltaFlow-P achieves a 2.72x throughput speedup over the full-attention baseline at a sequence length of 16k. These results show that DeltaFlow is a promising alternative to dense attention for efficient continuous language denoising.
