Search papers, labs, and topics across Lattice.
This study investigates how visual tokens are integrated into Large Language Models (LLMs) through two distinct architectural approaches: in-context prompting and layer-wise injection. By conducting a controlled comparison across various benchmarks, the authors reveal that the integration method significantly influences the transformation of visual information into meaningful representations, with each approach capturing different frequency characteristics of the visual signal. The findings highlight that the evolution of visual tokens within the LLM not only affects the alignment of visual features with language but also determines overall task performance, emphasizing that attention allocation alone is not sufficient for optimal results.
Visual tokens undergo a hidden evolution in LLMs, transforming from raw signals into contextually rich representations that dictate task performance.
Visual tokens enter Large Language Models (LLMs) as raw, foreign signals. How they are transformed into meaningful representations and interact with the language space depends entirely on the integration architecture. Whether by treating visual tokens as in-context prompts within the input sequence or injecting them directly into the LLM's intermediate layers. A controlled comparison and understanding of how these architectural choices affect visual information and its internal transformation to integrate with the LLM remains underexplored. We provide a fair comparison by evaluating in-context and layer-wise injection VLM integration paradigms under identical training conditions across single image, multi-image, and video benchmarks. In doing so, we uncover a hidden evolution where visual tokens enter the LLM as disguised visual context, raw representations lacking linguistic structure, but are progressively reshaped depending on the integration paradigm, each capturing fundamentally different frequency characteristics of the visual signal. We show that this evolution inside the LLM determines what visual features the VLM can utilize effectively, how visual representations align with the language space, and ultimately how each paradigm performs across different tasks. We further demonstrate that attention allocation alone is insufficient, and that performance is driven by the quality of visual representations at each layer.
