Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To resolve the prohibitive storage bottleneck of caching training gradients for large language model influence functions, this work characterizes the optimal linear gradient representation and develops Eigenbasis-Corrected One-Bit Gradient Projection (EOGP). The framework combines EK-FAC subspace reduction with in-subspace PCA and one-bit coordinate quantization to maximize retained directional information under tight memory constraints. Across OLMo 2 models up to 32B parameters, EOGP matches the predictive fidelity of standard compression baselines while slashing per-example storage by more than two orders of magnitude.
Training data attribution on 32B-parameter models no longer requires intractable gradient storage: extreme 1-bit eigenbasis projection achieves competitive influence estimation accuracy at over 100× lower per-example footprint than standard baselines.
Influence functions estimate how individual training examples affect the behavior of large language models (LLMs). Analyzing how training data influence different behaviors of an LLM involves repeated influence computation. Reusing stored training gradients reduces the computational cost, but storing full gradients is prohibitively expensive at LLM scale. We study how to compress these gradients while preserving influence estimates for future queries that are unknown at storage time. Through a worst-case analysis, we characterize the optimal fixed-dimensional linear representation and propose eigenbasis-corrected one-bit gradient projection (EOGP) to approximate it at scale. Specifically, EOGP uses EK-FAC to reduce gradient dimensionality, then applies PCA within the retained subspace to learn compression directions from the training gradients. We then apply one-bit quantization to the resulting coordinates, allowing more coordinates to be retained within a fixed storage budget. On GPT-2, EOGP predicts retraining outcomes more accurately than the evaluated compression baselines while using one-sixteenth of their per-example storage. On OLMo 2 SFT models from 1B to 32B parameters, EOGP remains competitive with the baselines allocated over 100 times as much storage per example.