Search papers, labs, and topics across Lattice.
This paper introduces Randomized YaRN, a novel training method that enhances length generalization in large language models (LLMs) by integrating YaRN-based positional extrapolation with randomized positional encoding and a structured length curriculum. The approach allows models to learn from out-of-distribution positional representations during training on short-context data, significantly improving their reasoning capabilities on longer sequences. Evaluated on the BABILong and Multi-Round Coreference Resolution benchmarks, Randomized YaRN demonstrates substantial performance gains, particularly at extreme context lengths, outperforming traditional fine-tuning methods.
Randomized YaRN boosts long-context reasoning performance by exposing models to out-of-distribution positional encodings, yielding impressive gains at extreme lengths.
Large language models (LLMs) are typically pretrained on short sequences and then extended to work on longer sequences with additional training. However, such LLMs still struggle to further generalize to very long sequences. We propose Randomized YaRN, a training method that improves length generalization by combining YaRN-based positional extrapolation with randomized positional encoding and a length curriculum. During training on short context data, tokens are assigned YaRN positional encodings sampled from a larger position range, exposing the model to out-of-distribution positional representations even on short-context inputs. We evaluate Randomized YaRN on two challenging long-context reasoning benchmarks, BABILong and Multi-Round Coreference Resolution (MRCR). When training on data with <8K context, Randomized YaRN consistently improves reasoning performance on context lengths from 16K to 128K and outperforms standard fine-tuning, with the largest gains appearing at far out-of-distribution lengths. Our results suggest that progressively exposing models to OOD positional distributions provides an effective recipe for generalizable long-context reasoning.
