Search papers, labs, and topics across Lattice.
This paper introduces For-Value, a computationally efficient, forward-pass-only data valuation framework for LLMs and VLMs. It leverages the theoretical insight that data value can be approximated by the alignment between final hidden representations and prediction errors, enabling a closed-form value calculation without backpropagation. Experiments demonstrate that For-Value matches or exceeds the performance of gradient-based methods in identifying influential and mislabeled data, while significantly improving computational efficiency and enabling batch parallelization.
Data valuation for LLMs doesn't need backpropagation: a single forward pass is enough to match gradient-based methods, unlocking massive speedups.
Data valuation is essential for enhancing the transparency and accountability of large language models (LLMs) and vision-language models (VLMs). However, existing methods typically rely on gradient computations, making them computationally prohibitive for billion-parameter models and precluding batch parallelization. In this work, we introduce For-Value, a forward-only data valuation framework that enables efficient batch-scalable value estimation while maintaining effectiveness. Leveraging the expressive power of pretrained LLMs/VLMs, we theoretically demonstrate that data valuation can be captured by the alignment between the final hidden representations and prediction errors at the last layer. In light of this insight, For-Value computes data value using a simple closed-form expression with a single forward pass, eliminating the need for costly backpropagation and enabling efficient batch calculating at scale. Extensive experiments show that For-Value matches or outperforms gradient-based baselines in detecting influential data and mislabeled data, while achieving significant efficiency improvements.
