Search papers, labs, and topics across Lattice.
This study investigates the optimization of feed-forward network (FFN) widths in Transformers by allocating capacity based on layer behavior measured during forward passes. By employing geometric metrics such as Gromov-Wasserstein distortion and persistent homology, the authors develop a layerwise approximation that allows for effective width allocation, demonstrating that this method can significantly reduce validation loss compared to traditional uniform and heuristic width strategies. The results show that geometry-guided width allocation consistently outperforms both uniform and cosine taper approaches, particularly in larger model configurations.
Geometry-guided width allocation in Transformers can lead to substantial improvements in model performance, reducing validation loss more effectively than traditional methods.
Feed-forward networks (FFNs) account for a large fraction of Transformer parameters, yet their hidden width is usually constant across depth. We ask whether this capacity can instead be allocated from a forward-pass measurement of layer behavior. We view each FFN as transporting a cloud of token representations and quantify the induced geometric change using correspondence-preserving shift, Gromov-Wasserstein distortion, and degree-one persistent homology under raw and scale-normalized metrics. A layerwise approximation surrogate yields an exact fixed-budget optimizer. Across seven pretrained language models, raw Euclidean work largely tracks residual-norm growth, whereas normalized work is predominantly front-loaded. Gromov-Wasserstein work is more consistently associated with perturbation-based layer sensitivity than the finite-sample topological estimate. In paired 128M and 256M training runs, several normalized-work schedules reduce mean validation loss relative to both uniform width and a hand-designed cosine taper. With the amplified paired differences at 440M, the best geometry-based allocations improve over uniform substantially larger than the cosine taper, while the anti-topological raw control is worse than uniform.
