Search papers, labs, and topics across Lattice.
This paper explores Manifold-Constrained Hyper-Connections (mHC), a novel parameter-efficient finetuning (PEFT) method that modifies the residual connections in Transformer architectures. The authors find that while mHC can finetune frozen OLMo-2 backbones, fixing the residual mixing matrix to identity often enhances performance, revealing a fundamental difference in the role of residuals during finetuning compared to pre-training. Although mHC does not consistently outperform LoRA as a standalone method, its combination with LoRA yields improved language modeling and task-specific gains at both 1B and 7B model scales, highlighting the potential of residual routing in PEFT strategies.
Fixing the residual mixing matrix to identity during finetuning can lead to unexpected performance improvements in Transformer models.
Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections. This paper investigates Manifold-Constrained Hyper-Connections (mHC), a generalisation of residual connections, as a novel PEFT approach, wrapping frozen OLMo-2 backbones with learned residual routing modules. We find that mHC can finetune frozen Transformers, but that its role differs fundamentally from the original pre-training setting: in finetuning, fixing the residual mixing matrix to identity often improves performance. As a standalone PEFT method, mHC does not consistently outperform LoRA. However, at matched trainable parameter budgets, mHC+LoRA combinations improve language-modelling loss and show task-dependent benchmark gains at both 1B and 7B scale. Overall, our results identify residual routing as a distinct and promising novel PEFT axis.
