Search papers, labs, and topics across Lattice.
This paper introduces Instruction-Conditioned Exploration (ICE), a novel approach that enhances the exploration capabilities of Large Language Models (LLMs) during training by supplementing task prompts with diverse instructions. By employing an Asymmetric Reinforcement Learning and Self-Distillation (Asymmetric-RL/SD) framework, the authors effectively transfer explored behaviors to the model's unconditioned test-time policy. The proposed method yields a significant 5.0% relative improvement in the Qwen3-1.7B model's performance on mathematical reasoning tasks, demonstrating its efficacy in increasing the model's behavioral diversity and adaptability.
Instruction-Conditioned Exploration boosts LLM performance by 5% on complex reasoning tasks by strategically diversifying training instructions.
Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration. New methods are required that leverage the broad knowledge and flexibility of pre-trained LLMs to deliberately generate diverse experience at training time. We propose Instruction-Conditioned Exploration (ICE), which supplements task prompts during training with one of several distinct instructions, increasing the coverage of behaviours attempted. To facilitate ICE, we propose Asymmetric-RL/SD, a combined Reinforcement Learning and Self-Distillation training objective, to transfer explored behaviours to the unconditioned test-time policy. ICE with the Asymmetric-RL/SD objective improves Qwen3-1.7B held-out pass@1 performance at $4$K response length on mathematical reasoning tasks by $5.0\%$ relative to training with DAPO, with improvement persisting at a longer 8K context.
