Search papers, labs, and topics across Lattice.
This paper introduces LongCrafter, a structured synthesis framework designed to enhance the long-context understanding of large language models by addressing limitations in existing supervised fine-tuning methods. By employing a hierarchical task taxonomy and an evidence-grounded pipeline, LongCrafter generates diverse instruction-response pairs that are both task-aligned and grounded in explicit evidence graphs, ensuring controllable difficulty and faithful reasoning. The results show that models fine-tuned with LongCrafter data significantly outperform SFT baselines and post-trained models on various benchmarks, particularly excelling in high-difficulty tasks.
Models fine-tuned with LongCrafter data achieve unprecedented performance on long-context tasks, particularly in high-difficulty scenarios.
Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task coverage, insufficient instruction difficulty, and a lack of faithfulness supervision. We propose \textbf{LongCrafter}, a structured synthesis framework that couples a hierarchical task taxonomy with an evidence-grounded pipeline. The taxonomy organizes long-context understanding into local/shallow and global/deep levels and yields 32 fine-grained task types that serve as a global generative prior. Guided by this taxonomy, LongCrafter constructs task-aligned long contexts, decomposes them into explicit evidence graphs that model cross-paragraph dependencies, and generates instruction--response pairs strictly grounded in the located evidence spans, ensuring both controllable difficulty and faithful, traceable reasoning. Models fine-tuned on LongCrafter data outperform all SFT baselines and even the official post-trained models on LongBench, LongBench~v2, and LooGLE across both Qwen2.5-7B and LLaMA-3.1-8B, with the largest gains on high-difficulty tasks. Further analysis shows that LongCrafter data is more diverse and better spread across difficulty levels, and that the trained models locate evidence robustly regardless of position, effectively mitigating the ``lost in the middle''problem.
