Search papers, labs, and topics across Lattice.
This paper explores the concept of "enlightenment" in large-scale models, hypothesizing that they can experience sudden boosts in capability akin to human "aha moments." The authors introduce a novel training-free post-tuning approach called Enlightenment, which modifies key module shortcuts without weight updates, contrasting with existing methods that primarily adjust attention weights. Their experiments demonstrate that this method significantly enhances performance across various benchmarks for both language and vision-language models, revealing the latent potential of pre-trained networks.
Large models can achieve sudden performance leaps without retraining, unlocking their hidden capabilities through innovative shortcut modifications.
The pursuit of autonomously self-improving models has attracted growing interest in the era of large-scale foundation models. Drawing inspiration from the concept of"enlightenment"or"aha moment"in human brain, we hypothesize that large models exhibit an analogous enlightenment phenomenon-a latent capacity for sudden capability boost. Then, we propose Enlightenment, a novel training-free post-tuning paradigm for large-scale models. Our approach modifies shortcuts for key modules/layers without weight updates, while existing training-free ones predominantly manipulate attention weights. We introduce two architecture-specific instantiations: i) For large language models, we propose attention head-mixing shortcuts that recalibrate attention weights by linking the initial attention head's output to all other target heads, modulated by an adaptive scaling factor initialization strategy. ii) For vision-language models, we apply a lightweight scalar-modulated factor to residual connections in the decoder layers, regulating information flow. Extensive experiments show that Enlightenment efficiently unlocks the latent potential of pre-trained networks, yielding remarkable performance improvements across diverse benchmarks and models.
