Search papers, labs, and topics across Lattice.
This paper introduces ScAle, a lightweight adaptation method that enhances spatial reasoning in vision language models (VLMs) by learning a small set of scalar coefficients to modulate attention and MLP activations without altering pretrained weights. The method was evaluated on both synthetic and real-world datasets, achieving up to 134.1% relative accuracy gains with only 1,000 trainable parameters, significantly outperforming traditional parameter-efficient fine-tuning methods like LoRA. The findings underscore the effectiveness of bounded activation reweighting as a compact and architecture-agnostic solution for improving VLM performance in spatial reasoning tasks.
Achieving over 134% accuracy gains with just 1,000 trainable parameters reveals a game-changing approach to enhancing spatial reasoning in vision language models.
Spatial reasoning remains a persistent challenge for many vision language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters. Our preliminary analysis reveals that rescaling activations in selected transformer layers-without modifying pretrained weights-can significantly influence downstream performance. Motivated by this observation, we propose ScAle, an ultra-lightweight adaptation method that learns a small set of scalar coefficients to modulate last-token attention and MLP activations in a fully frozen backbone. We evaluate our method on the synthetic spatial reasoning benchmark SpatialEval and on real-world VQA datasets (COCOQA and VGQA) across multiple model families. Our method, ScAle, achieves up to 134.1% relative accuracy gains using only 1K trainable parameters without requiring millions of trainable parameters as in standard PEFT methods such as LoRA. Despite its extreme compactness, our approach recovers a substantial fraction of standard PEFT performance while preserving strong non-spatial VQA accuracy. These results demonstrate that bounded activation reweighting provides a simple, architecture-agnostic, and highly parameter-efficient alternative for adapting pretrained VLMs.
