Search papers, labs, and topics across Lattice.
The paper introduces Forward-Only Zeroth-Order Optimization (FOZO), a backpropagation-free test-time adaptation (TTA) method that optimizes prompts using zeroth-order gradient estimation based on feature statistics and prediction entropy. FOZO employs a dynamically decaying perturbation scale to ensure efficient and stable adaptation to out-of-distribution data streams, with theoretical convergence guarantees. Experiments on ImageNet-C, ImageNet-R, and ImageNet-Sketch demonstrate that FOZO outperforms both gradient-based and existing forward-only TTA methods, even on quantized models.
Forget backpropagation for test-time adaptation: FOZO achieves state-of-the-art accuracy with a memory-efficient, forward-only zeroth-order prompt optimization, even on quantized models.
Test-Time Adaptation (TTA) is essential for enabling deep learning models to handle real-world data distribution shifts. However, current approaches face significant limitations: backpropagation-based methods are not suitable for low-end deployment devices, due to their high computation and memory requirements, as well as their tendency to modify model weights during adaptation; while traditional backpropagation-free techniques exhibit constrained adaptation capabilities. In this work, we propose Forward-Only Zeroth-Order Optimization (FOZO), a novel and practical backpropagation-free paradigm for TTA. FOZO leverages a memory-efficient zeroth-order prompt optimization, which is led by objectives optimizing both intermediate feature statistics and prediction entropy. To ensure efficient and stable adaptation over the out-of-distribution data stream, we introduce a dynamically decaying perturbation scale during zeroth-order gradient estimation and theoretically prove its convergence under the TTA data stream assumption. Extensive continual adaptation experiments on ImageNet-C, ImageNet-R, and ImageNet-Sketch demonstrate FOZO's superior performance, achieving 59.52% Top-1 accuracy on ImageNet-C (5K, level 5) and outperforming main gradient-based methods and SOTA forward-only FOA (58.13%). Furthermore, FOZO exhibits strong generalization on quantized (INT8) models. These findings demonstrate that FOZO is a highly competitive solution for TTA deployment in resource-limited scenarios.
