Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of long-context reasoning in large language models by proposing a data-centric approach that emphasizes the importance of diverse training data over traditional reward engineering methods. By curating eight datasets totaling approximately 14,000 examples across retrieval, multi-evidence synthesis, and reasoning tasks, the authors demonstrate significant improvements in long-context reasoning capabilities. Their experiments show that models trained with this data recipe achieve average gains of +7.2, +3.2, and +6.4 points across seven benchmarks, with notable performance enhancements in agentic tasks as well.
A simple data recipe can outperform complex reward engineering in enhancing long-context reasoning for large language models.
Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.
Tsinghua AI