Search papers, labs, and topics across Lattice.
This paper introduces STRIDE, a novel framework for Training Data Attribution (TDA) that leverages sparse recovery techniques to model the influence of training data on Large Language Models (LLMs) in the activation space rather than the parameter space. By employing lightweight "steering operators" to simulate the behavioral shifts induced by data subsets, STRIDE significantly accelerates the attribution process, achieving state-of-the-art performance while being 13 times faster than previous methods. The practical applications of STRIDE include enhanced data selection and contamination analysis, demonstrating its utility in real-world scenarios.
STRIDE reveals that training data influences can be efficiently traced in LLMs using sparse recovery, achieving attribution 13 times faster than traditional methods.
Training Data Attribution (TDA) seeks to trace a model's predictions back to its training data. The gold standard for TDA relies on causal interventions, observing how a model changes when data is added or removed, but repeated retraining is computationally challenging for Large Language Models (LLMs). Consequently, most approaches approximate this effect in the parameter space using gradients. However, tracking gradients across billions of parameters is not only prohibitively expensive but relies on local approximations. In this work, we propose a shift: rather than estimating parameter changes, we model the functional effect of training data in the activation space. We introduce STRIDE (Steering-based Training Data Influence Decomposition), a framework that formulates TDA as a sparse recovery problem in the spirit of compressive sensing. STRIDE learns lightweight "steering operators" that mimic the behavioral shift caused by training on data subsets. By measuring how these operators perturb test predictions, we recover individual training example influences via sparse linear decomposition. STRIDE achieves state-of-the-art for LLM pre-training attribution while being an order of magnitude (13times) faster than previous art. We further validate its practical utility through downstream applications including data selection, data contamination, and qualitative analysis.
