Search papers, labs, and topics across Lattice.
This paper introduces double preconditioning (DoPr), an innovative optimization approach designed to enhance test-time performance in deep learning models, particularly in settings affected by test-time feedback (TTF). By integrating gradient-wise preconditioning with activation-wise preconditioning, DoPr effectively mitigates error accumulation during model deployment, leading to improved downstream metrics such as task success rate and generation quality. Notably, the findings reveal that enhancements in test-time performance do not necessarily correlate with reductions in validation loss, challenging traditional evaluation methods for one-step supervised objectives.
Test-time performance can be significantly improved without a corresponding drop in validation loss, raising critical questions about model evaluation in deep learning.
Many modern applications of deep learning involve training a neural network via a one-step prediction loss (e.g., $L^2$ regression, cross-entropy), but deploy the network by rolling out along its own predictions. Key examples include autoregressive language modeling, flow-based generative modeling, and robot policy learning. It is well-documented that these settings induce a phenomenon we call test-time feedback (TTF): the mismatch between the training/validation loss and downstream metrics of interest, such as task success rate and generation quality, which grows with task length. While data curation, architecture, and objective design have been proposed to combat train-test shift in TTF settings, this paper proposes optimization as a new design axis to mitigate error accumulation. Specifically, we introduce a new optimization paradigm called double-preconditioning (DoPr) uniquely tailored to the challenges of TTF. DoPr combines gradient-wise preconditioning, as in Adam and Muon, with activation-wise preconditioning (AP), such as in KFAC. We show that the addition of AP yields a drop-in intervention for increasing downstream model performance across a range of TTF settings. Interestingly, these gains in test-time performance do not consistently accompany improvements in validation loss, opening new questions about how to properly evaluate models trained with one-step supervised objectives.
CMU ML