Search papers, labs, and topics across Lattice.
This paper introduces progressive cramming, a method that incrementally builds target prefixes for token compression, allowing for reliable reconstruction while revealing the limitations of fixed token budgets. The authors find that while progressive cramming achieves low-dimensional representation in embedding space, it leads to a significant accuracy drop in multiple-choice tasks and a near-total collapse in generative evaluations. Causal attention-knockout interventions indicate that this degradation stems from the embedding's interactions within the model's early layers, highlighting the distinction between perfect reconstruction and effective compression.
Progressive cramming reveals that achieving perfect reconstruction is not enough, as it can lead to substantial performance degradation in downstream tasks.
Token cramming compresses sequences into learned embeddings with near-perfect reconstruction, but fixed token budgets and 99\% accuracy thresholds leave it unclear whether residual errors reflect optimization failures or fundamental limits. We introduce progressive cramming, which grows the target prefix token-by-token, stopping only when reconstruction is no longer achievable within a fixed optimization budget. Progressive trajectories occupy low-dimensional structure in embedding space. Prepending a crammed embedding causes a moderate but consistent accuracy drop on multiple-choice benchmarks even with the original prefix in context, and collapses capability almost entirely under generative evaluation. Causal attention-knockout interventions trace this degradation to the embedding's interactions in the model's early layers. These results position progressive cramming as a tool for studying compression limits and show that perfect reconstruction - achievable through brittle steering rather than transferable semantics - is insufficient for meaningful compression.
