Search papers, labs, and topics across Lattice.
This paper introduces Dynamic Scaled Gradient Descent (DSGD), a novel optimization algorithm for fine-tuning pretrained models on sparse, imbalanced datasets. DSGD addresses the problem of gradient cancellation by dynamically scaling down gradients from correctly classified examples, thereby promoting more stable and effective training. Experiments across diverse datasets and large pretrained models show that DSGD reduces performance variance and improves accuracy compared to existing fine-tuning methods.
Gradient cancellation during fine-tuning can be tamed by simply scaling down the gradients of correctly classified examples, leading to more stable and accurate models.
Fine-tuning pretrained models has become a standard approach to adapting pretrained knowledge to improve the accuracy on new sparse, imbalance datasets. However, issues arise when optimization falls into a collapsed state, where the model gets stuck, leading to degraded performance and unstable training. One possible reason for this is the cancellation of gradients across training examples. To address this problem, we propose a novel algorithm, dynamic scaled gradient descent (\mName), that directly modifies the gradients returned by training examples, specifically, scaling down the gradients of correctly classified examples using a dynamic scaler. This strategy offers both theoretical and empirical advantages in improving training stability. Experiments on a variety of benchmark datasets, spanning multiple tasks and large pretrained models, demonstrate that our method consistently reduces performance variance and surpasses the accuracy of existing approaches.
