Search papers, labs, and topics across Lattice.
This study investigates Test-Time Augmentation (TTA) as a method to enhance the accuracy of large language models (LLMs) by introducing input diversity alongside traditional output diversity methods, such as self-consistency. Through a systematic evaluation across six diverse datasets, the authors demonstrate that TTA, particularly through semantic rephrasing, achieves approximately 1.8 times more accuracy per dollar spent compared to self-consistency, significantly improving cost-effectiveness. The findings suggest that for mid-tier models, input perturbation is a more efficient strategy for converting compute into accuracy than merely varying reasoning paths.
Input diversity can yield 1.8X more accuracy per dollar for mid-tier LLMs compared to traditional output diversity methods.
Test-time scaling improves LLM accuracy but multiplies inference cost, making the accuracy gained per unit of compute the metric that matters in deployment. Self-consistency is one of the established approaches, which spends this budget entirely on the output side by sampling repeated reasoning paths. We study Test-Time Augmentation (TTA), which extends self-consistency by also perturbing the input, aggregating predictions across transformed versions of the input, and ask whether input-side diversity converts compute into accuracy more efficiently than output-side diversity. We perform a systematic, matched-compute comparison: we evaluate three simple input-side strategies (semantic rephrasing, lexical perturbations, and visual transformations) across six datasets covering general and multilingual knowledge, mathematical reasoning, multi-modal question answering, and sentiment classification, against chain-of-thought prompting and self-consistency. Semantic rephrasing delivers consistent and statistically significant accuracy gains while Pareto-dominating self-consistency on cost-effectiveness, delivering roughly 1.8X more accuracy per dollar and outperforming it on five of six tasks. We further analyze the number of augmentations, multi-modal strategies, and base model scaling, finding that TTA is most cost-effective for mid-tier models where a stronger model is unavailable or too expensive. Our findings indicate that for current mid-tier LLMs, varying the input converts inference compute into accuracy more efficiently than varying the reasoning path alone. The TTA implementation is available at https://github.com/aws-samples/sample-genai-reflection-for-bedrock.
Amazon Science