Search papers, labs, and topics across Lattice.
This paper introduces the Generalization Spectrum, a novel evaluation framework that assesses the extent of generalization in learning algorithms by measuring performance across varying transfer distances from training examples. By employing a controlled suite of test variants, the authors reveal how learning from specific instances translates to broader contexts, providing insights into the generalization capabilities of different learning paradigms. The findings indicate that while reinforcement learning (RL) efficiently converts memorization into near-transfer, other methods like self-distillation can inadvertently reduce far transfer despite improvements in local transfer metrics.
Learning algorithms might excel in memorization but can falter in broader generalization, with RL outperforming SFT in transferring knowledge across contexts.
Traditional evaluations measure a learning algorithm's final performance on an i.i.d. test set, reducing learning to a single aggregate score. This approach obscures a fundamental question: to what extent does learning from a specific example generalize to others? Such per-sample generalization, akin to learning by analogy in human cognition, captures how far the knowledge extracted from one example can transfer, yet remains invisible to standard benchmarks. We introduce the Generalization Spectrum, an evaluation framework designed to expose this hidden dimension. For each training example, we construct a controlled suite of test variants arranged by increasing transfer distance, from exact recall to implementation transfer across languages, context transfer under complete narrative re-framing, category-matched in-domain problems, and an unpaired baseline. By tracking performance across these distances, we reveal not just whether an algorithm learns, but how far that learning extends. We instantiate this framework on competitive programming, using a selection-and-synthesis pipeline seeded with recent problems to mitigate contamination. We first compare three canonical learning paradigms under matched memorization. RL converts memorization into near-transfer more efficiently than SFT-family baselines, while ICL exhibits strong but correspondence-dependent transfer. We then use the Spectrum to diagnose within-family variants. The resulting profiles show that local gains need not expand the generalization radius: abstractions and hints mainly lift local transfer, RFT preserves a stronger far-transfer tail than reference SFT, and self-distillation or hint-assisted RL can reduce far transfer even when local transfer or optimization improves.
