Search papers, labs, and topics across Lattice.
This study introduces BudgetDoc, a novel multimodal benchmark that provides explicit supervision for optimizing model-budget-performance trade-offs in document tasks. The researchers developed DRB, a lightweight pre-flight estimator with approximately 1 billion parameters, which predicts ordinal performance across varying reasoning budgets and achieves a weighted F1 score of 0.753. By dynamically allocating reasoning budgets, DRB outperforms or matches maximum-budget baselines in 9 out of 15 configurations, significantly reducing computational costs while maintaining performance.
DRB can optimize reasoning budgets to improve LLM performance while cutting costs, achieving better results than traditional maximum-budget approaches.
Uniformly allocating inference reasoning budgets to LLMs is expensive and prone to over-thinking penalties; especially in document tasks where visual layouts drive complexity. To address this, we introduce BudgetDoc, the first multimodal benchmark providing explicit supervision for model-budget-performance trade-offs across three document tasks. Using BudgetDoc, we train DRB (Document-Reasoning Balancer), an approx. 1B-parameter pre-flight estimator (SigLIP-2 + Qwen3-0.6B) that predicts ordinal model performance across budget levels, achieving a 0.753 weighted F1. When dynamically allocating reasoning budgets across five frontier models and three datasets, DRB matches or improves F1 scores compared to always-maximum-budget baselines in 9 of 15 configurations while drastically reducing cost. Finally, preliminary evaluations demonstrate DRB's potential to generalize to cross-model selection.
