Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the use of offline reinforcement learning (RL) to improve the performance of large language models (LLMs) for code generation, aiming to reduce the computational cost associated with online RL. They train LLMs using offline RL on existing code datasets and find that this approach enhances performance, particularly for smaller models and complex coding tasks. The results suggest that offline RL offers a viable and efficient alternative to online RL for post-training LLMs in code generation.
Offline RL can give smaller LLMs a surprising edge on challenging coding problems, rivaling the performance of larger models without the costly online interaction.
Post-training using online reinforcement learning (RL) is an important training step for LLMs, including code-generating models. However, online RL for code generation involves LLM inference and verification of the generated output, which can take considerable time and resources. In this paper, we explore the application of offline RL to code-generating models by leveraging existing code datasets. Our experiments demonstrate that offline RL is an effective training strategy for improving LLM performance. We show that offline RL can be especially beneficial for small LLMs and challenging coding problems.