Search papers, labs, and topics across Lattice.
The paper explores using rule-generated synthetic data for RL fine-tuning of LLMs to improve multi-hop reasoning, addressing the limitations of human-annotated, LLM-generated, and LLM-verified data. They find that LLMs fine-tuned on synthetic data significantly outperform those trained on real-world data on question-answering benchmarks, even with fictional knowledge. The improvement is attributed to the synthetic data's ability to teach knowledge composition, a generalizable reasoning skill.
Forget expensive human annotations or hallucination-prone LLM-generated data: rule-generated synthetic data can teach LLMs to compose knowledge and significantly boost multi-hop reasoning on real-world tasks.
Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule-generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge -- a fundamental and generalizable reasoning skill. Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.
NVIDIA
Stanford HAI