Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to financial question answering that combines tabular and textual data by employing execution-verified Python programs for programmatic distillation. By transferring reliable numerical reasoning from a larger teacher model to a smaller student model, the method mitigates the arithmetic errors commonly found in natural-language rationales. Experimental results on the TAT-QA dataset reveal that the 7B student model outperforms the 72B teacher model and various LLM-based baselines, achieving 87.00 EM and 87.18 F1 scores, highlighting the effectiveness of execution-verified supervision in enhancing numerical reasoning capabilities.
Execution-verified programmatic distillation allows smaller models to outperform larger ones in financial reasoning tasks, achieving remarkable accuracy through reliable numerical computation.
Financial question answering over hybrid tabular and textual data may require multi-source reasoning and precise numerical computation. While large language models (LLMs) can generate intermediate reasoning steps, natural-language rationales remain prone to arithmetic errors, making them an unreliable supervision source for distillation. Building on programmatic distillation, we develop an approach that transfers reliable numerical reasoning from a large teacher model to a compact student using execution-verified Python programs instead of free-form textual rationales. It leverages gold derivations to guide teacher-side program synthesis and retains only programs that execute correctly and produce the gold answer, ensuring high-quality supervision. We further introduce an iterative recovery stage that revisits teacher-failed examples, enabling the student to recover and incorporate newly verified programs into training. Experiments on TAT-QA show that our framework is highly effective for hybrid financial reasoning. Our best 7B student achieves 87.00 EM / 87.18 F1 on the test set, substantially outperforming the 72B teacher (78.46 EM) as well as traditional and strong LLM-based baselines, including TAGOP and TAT-LLM. These results demonstrate that execution-verified programmatic distillation provides an effective and extensible framework for training smaller models to perform reliable numerical reasoning.
