Search papers, labs, and topics across Lattice.
This paper addresses the economic challenges of fine-tuning large language models (LLMs) by introducing a risk decomposition framework for pre-hoc performance prediction, which is formulated as a stochastic estimation problem. The authors decompose prediction risk into intrinsic limits related to data-model compatibility and reducible optimization variance, proving that the latter has a fundamental lower bound on its decay rate. Their findings lead to a budget-optimal probing principle and a predictability phase diagram that categorizes tasks into three regimes, validated through extensive experiments on both synthetic and real-world benchmarks.
Pre-hoc performance prediction can significantly reduce the costs of fine-tuning LLMs by revealing fundamental limits on uncertainty decay.
The high cost of fine-tuning LLMs poses a significant economic barrier; pre-hoc performance prediction offers a critical solution to substantially reduce this expense. However, the theoretical limits of pre-hoc performance prediction remain unexplored. We formulate it as a stochastic estimation problem under information constraints, decomposing prediction risk into two components: an intrinsic limit (static data-model compatibility) and a reducible optimization variance. We prove that optimization variance admits a necessary lower bound on its decay rate, implying fundamental constraints on how quickly uncertainty dissipates, regardless of the predictor used. Based on these dynamics, we derive a budget-optimal probing principle and introduce a predictability phase diagram that organizes tasks into three distinct regimes: Static-Sufficient, Dynamic-Critical, and Noise-Dominant. Extensive experiments on synthetic and real-world benchmarks validate these theoretical regimes and demonstrate the efficiency of our probing strategy.
