Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
We introduce Copy-Paste Language Models (CPLMs) by augmenting standard Transformer LMs with a lightweight Copy-Paste head, demonstrating that explicit modeling significantly enhances in-context copying capabilities. CPLMs improve training efficiency and downstream performance across various scales (135M to 1.7B parameters) with minimal overhead. This success is attributed to the Copy-Paste head providing an alternative gradient path that alleviates the LM head's gradient bottleneck.
Explicitly modeling copy operations unlocks substantial improvements in LLM efficiency and performance, even for models already adept at in-context learning.
Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.