Search papers, labs, and topics across Lattice.
This paper introduces Influcoder, a novel method that distills the gradient influence rankings of decoders into a more efficient encoder for data attribution in large language models. The motivation behind this work is to enhance the speed and storage efficiency of influence function-based data attribution methods, which are crucial for identifying the impact of training samples on model outputs. Key results demonstrate that Influcoder significantly improves processing speed while maintaining accuracy, making it feasible to apply data attribution techniques to large datasets effectively.
Influcoder achieves rapid and compact data attribution for LLMs, transforming how we identify the influence of training samples on model behavior.
With the growth of LLMs'(Large Language Models) capabilities, there has been an increasing push to curate high quality datasets by filtering samples in the training data. In general, Data Attribution (DA) methods aim to estimate how individual samples in a training dataset can precondition a model to generate certain outputs. As an example, one might be interested in which samples in the data could be the source of toxic behavior after training the LLM. Many methods quantify this conditioning through the paradigm of influence functions. While methods of this family are effective in its function, they lack the necessary processing speed and storage compactness to be practically implemented on large datasets. We propose a method, Influcoder, as a quick and cost-effective approach to influence-based Data Attribution at scale.
