Search papers, labs, and topics across Lattice.
This study investigates cross-format robustness in large language models, revealing that models can struggle to provide consistent answers across different formats of the same question. By comparing full-format training to a novel approach called FormatMix, which selectively expands training items into multiple equivalent formats, the authors demonstrate that multi-format supervision significantly enhances both task performance and cross-format robustness. Notably, expanding only 30% of the training set into multiple formats can yield most of the benefits of full-format training, highlighting the importance of format diversity in improving model robustness.
Multi-format training can drastically enhance language model consistency across different answer formats, with just 30% of training data needing augmentation to achieve significant gains.
Large language models often remain sensitive to answer format: a question solved correctly in one form may fail in another semantically equivalent form. To study this gap, we define cross-format robustness as the extent to which a model answers the same underlying question consistently across formats. We then compare full-format training with FormatMix, which expands only a subset of training items into multiple equivalent formats using either random or targeted selection. Across GLM4 and Llama-3.1, multi-format supervision consistently improves both task performance and cross-format robustness, whereas Multiple-choice question (MCQ)-only supervision alone brings little benefit and can even reduce robustness. We further find that expanding only about 30% of the training set into multiple formats often recovers most of the gain from full-format training, and this effect appears across the model families and sizes we study. These results suggest that format diversity, rather than additional supervision alone, is the key driver of robustness. That lightweight multi-format augmentation is a practical way to make LLMs less sensitive to answer format without changing the base model.
