Search papers, labs, and topics across Lattice.
This paper introduces Mixture-of-Control (MoC), a novel fine-tuning framework for transformer models that leverages both local and global control signals to improve representation learning. By treating block-wise control states as experts in a sparse mixture-of-experts approach, MoC facilitates efficient inter-block communication without incurring significant computational overhead. Empirical evaluations show that MoC surpasses existing state-based fine-tuning methods while preserving memory and computational efficiency, making it a compelling alternative for transformer adaptation.
MoC achieves superior representation learning in transformers by enabling efficient cross-block communication without the computational burden of traditional methods.
State-based fine-tuning has emerged as a compelling alternative to weight-based adaptation for transformers, updating lightweight controls into states rather than model weights, offering substantial memory savings while retaining parameter efficiency. However, most existing state-based methods typically apply only per-block control updates, which limits inter-block information exchange and restricts representational adaptation. Meanwhile, prior mechanisms that enable cross-block communication often introduce considerable computational overhead, reducing their practicality for efficient fine-tuning. We introduce Mixture-of-Control (MoC), a lightweight fine-tuning framework that adaptively integrates local and global control signals to enhance representation learning. MoC treats block-wise control states as experts in a sparse mixture-of-experts process, enabling efficient communication across transformer blocks. Empirical results across diverse transformer-based benchmarks demonstrate that MoC outperforms state-based methods while maintaining a comparable memory and computational efficiency.
