Search papers, labs, and topics across Lattice.
The paper introduces a method for compressing transformer weight matrices by parameterizing them in the 2D Discrete Cosine Transform (DCT) domain and retaining only the lowest-frequency coefficients. This allows for significant parameter reduction without sacrificing performance, as demonstrated by matching the perplexity of a standard transformer on character-level language modeling with only 52% of the parameters. The approach is simple to implement, requiring only a replacement of standard linear layers with spectral layers.
Compress transformers by 2x with no perplexity loss, or 3x with only a minor hit, simply by operating in the frequency domain.
We parameterize the weight matrices of a transformer in the two-dimensional discrete cosine transform (DCT) domain, retaining only the lowest-frequency coefficients. At each forward pass the full weight matrix is reconstructed via the inverse DCT; gradients propagate through the reconstruction to update the spectral coefficients directly. On character-level language modeling (Shakespeare, 1M characters), a 4-layer transformer trained from scratch in this representation matches the perplexity of the standard parameterization (6.1 vs.\ 6.1) while storing 52\% of the parameters. At 4$\times$ compression (29\% of parameters), the model reaches perplexity 6.9 -- outperforming a low-rank baseline (perplexity 8.8 at 21\% of parameters) at a comparable reduction. The method requires no architectural changes, no pre-trained checkpoint, and no auxiliary loss. It reduces to replacing each \texttt{nn.Linear} with a drop-in spectral layer that stores $K$ DCT coefficients instead of $n \times m$ weights.
