Search papers, labs, and topics across Lattice.
This study investigates the effects of low-rank adaptation on vision-language alignment, contrasting it with traditional full-parameter updates. The authors find that low-rank alignment not only reduces computational costs but also enhances performance across various benchmarks, shifting model behavior from hallucinatory to conservative. They provide a theoretical framework explaining how low-rank adaptation preserves linear separability of visual features and induces stability in visual representations, leading to improved alignment outcomes.
Low-rank adaptation in vision-language alignment not only cuts costs but also boosts performance, revealing a surprising shift from hallucination to conservatism in model behavior.
Vision-language alignment, the stage that bridges pretrained vision encoders and large language models, is widely treated as a form of pretraining requiring full-parameter updates. We challenge this view and investigate what happens when low-rank adaptation is applied to the LLM during this stage instead. We find that low-rank alignment not only reduces computational costs but also outperforms full-parameter alignment on most benchmarks. To understand this phenomenon, we systematically characterize the implicit biases introduced by low-rank adaptation during alignment. Empirically, we find that low-rank alignment shifts model behavior from hallucinatory to conservative and preserves per-token linear separability of visual features that full-parameter alignment disrupts, a phenomenon we term LS-curse. Geometrically, low rank aligned models exhibit more homogeneous and structurally stable visual representations, maintaining modality-specific knowledge rather than prematurely fusing entity-level semantics. Theoretically, we establish two theorems showing that low-rank alignment induces preferences for parameter subspaces with flat gradients and feature subspaces robust to perturbations, providing a principled explanation for the observed structure-preserving behavior. Extensive experiments cover ablation over 100 alignment configurations, three families of low-rank operators, and various rank, encoder, and other settings.
