Search papers, labs, and topics across Lattice.
This paper introduces the Parallel Causal Associative Field (PCAF), a novel architecture that leverages a parallel content-addressed memory to enhance long-context language modeling while mitigating the quadratic scaling of causal self-attention. By writing local records into hash buckets and retrieving a sparse cache of successor tokens, PCAF maintains efficient access to long contexts without the limitations of fixed-size recurrent states. The model achieves competitive perplexity scores on benchmark datasets while processing tokens at a higher rate than traditional dense Transformers, demonstrating a significant advancement in efficiency and performance for long-context tasks.
PCAF achieves a remarkable balance of speed and accuracy, processing over 0.61M tokens/s while outperforming dense Transformers in perplexity on long-context language modeling tasks.
Transformers achieve strong language modeling performance by providing direct token-to-token communication paths, but causal self-attention scales quadratically with context length. Recurrent and state-space models reduce this cost, yet compress history into sequentially updated fixed-size states. This paper studies a third primitive: a parallel content-addressed memory over causal successor records. The proposed Parallel Causal Associative Field (PCAF) writes local records from a context window into hash buckets, retrieves a bounded candidate set for the current query, forms a sparse cache distribution over successor tokens, and mixes that cache with a parametric local language model through a learned gate. The resulting model maintains sparse long-context access while avoiding a single fixed recurrent state bottleneck. We evaluate PCAF under full autoregressive pretraining on WikiText-103 and PG-19 using a distributed Google Cloud TPU v4-32 pod. At 303M parameters and context length T = 2048, PCAF-semantic reaches 36.31 perplexity on WikiText-103 and 52.45 perplexity on PG-19, compared with 47.49 and 53.84 for a matched dense Transformer. PCAF-semantic simultaneously processes 0.61-0.62M tokens/s across the TPU pod, versus 0.43M tokens/s for dense and local attention baselines. Supporting 41M-parameter multi-seed sweeps and single-GPU component ablations show that the associative cache, retrieval capacity, and learned gate materially affect the speed-quality trade-off.
