Search papers, labs, and topics across Lattice.
This paper introduces FormalRx, a diagnostic evaluation framework designed to enhance the transparency and interpretability of autoformalization assessments in formal mathematical reasoning. By employing the SCI Error Taxonomy, which categorizes errors into 28 distinct types, FormalRx provides actionable feedback through alignment verdicts, error categorization, localization, and correction capabilities. The framework's implementation, FormalRx-8B, demonstrates superior performance over existing models, achieving high F1-scores and accuracies, thereby facilitating systematic improvements in autoformalization systems.
FormalRx transforms opaque autoformalization evaluations into clear, actionable insights, enabling targeted improvements in formal reasoning systems.
The veracious semantic alignment in autoformalization is significant for formal mathematical reasoning. However, existing evaluations provide only opaque binary verdicts or scalar scores, offering no interpretable insight into where or why translations fail. This opacity severely limits both human understanding and automated system improvement. To bridge this gap, we introduce FormalRx, a comprehensive diagnostic evaluation framework that transforms autoformalization assessment from black-box judgments into actionable feedback. At its core is SCI Error Taxonomy, a hierarchical classification scheme decomposing autoformalization errors into 28 distinct categories with strict priority ordering. Building on this taxonomy, FormalRx provides four critical diagnostic capabilities: alignment verdicts, error categorization, error localization, and correction. We instantiate the framework with a diagnostic model FormalRx-8B, trained on 56,287 NL-FL pairs with fine-grained diagnostic annotations, and release FormalRx-Test as the first fine-grained diagnostic benchmark. FormalRx-8B achieves F1-scores of 0.88 (verdict) and 0.71 (categorization), along with accuracies of 0.75 (localization) and 0.73 (correction), substantially outperforming both general-purpose LLMs and specialized baselines. By connecting evaluation with actionable insights, FormalRx enables systematic diagnosis and improvement of autoformalization systems.
