Search papers, labs, and topics across Lattice.
This paper establishes the first non-vacuous generalization bounds for reinforcement learning with verifiable rewards (RLVR) at the billion-parameter scale, addressing the challenge of generalizability in fine-tuned large language models. By adapting PAC-Bayes compression bounds and employing the Gumbel-max reparameterization trick, the authors introduce the Progressive RLVR framework, which combines RLVR with on-policy distillation, TinyLoRA, and model quantization. Empirical results demonstrate that this framework retains 84-97% of standard LoRA fine-tuning performance while achieving a remarkable 14,796x increase in model compressibility and improved accuracy across multiple domains.
Non-vacuous generalization bounds reveal that RLVR fine-tuning can significantly enhance model performance while achieving unprecedented levels of compressibility.
While reinforcement learning with verifiable rewards (RLVR) is widely used to improve the reasoning capabilities of large language models (LLMs), the generalizability of the resulting models remains poorly understood. In this work, we establish the first non-vacuous generalization bounds for parameter-efficient RLVR fine-tuning at the billion-parameter scale. Our approach adapts PAC-Bayes compression bounds to this setting, and addresses the inherent stochasticity of token generation by applying the Gumbel-max reparameterization trick. To operationalize these bounds, we propose the Progressive RLVR framework, which integrates RLVR with on-policy distillation, TinyLoRA, and model quantization. Progressive RLVR empirically retains 84-97% performance of standard LoRA fine-tuning while producing models that are 14,796x more compressible. We show that this framework yields non-vacuous generalization bounds in four domains: mathematical problem-solving, programming, general-knowledge reasoning, and Text-to-SQL. Our bounds exceed the accuracy of the base model by 9-51% and lie within 6-11% of the accuracy of the fine-tuned models.
