Search papers, labs, and topics across Lattice.
Hydra is a novel vision-language model architecture that unifies document retrieval and autoregressive generation into a single model using a dual-head approach. By toggling a LoRA adapter trained only for retrieval, Hydra can switch between ColBERT-style late-interaction retrieval and high-quality generation, achieving byte-identical outputs to the base model when the adapter is disabled. The single-model design reduces peak GPU memory by 41% compared to separate retrieval and generation models, while maintaining competitive performance on visual document understanding tasks.
Why juggle separate retrieval and generation models when a single vision-language model can do both, cutting memory footprint by 41% without sacrificing generation quality?
Visual document understanding typically requires separate retrieval and generation models, doubling memory and system complexity. We present Hydra, a dual-head approach that provides both ColBERT-style late-interaction retrieval and autoregressive generation from a single vision-language model (VLM). A single LoRA adapter, trained only for retrieval, is toggled at inference: enabling it produces multi-vector embeddings; disabling it recovers the base model's generation quality -- byte-identical outputs in 100% of 10,500 greedy and stochastic samples, with max delta-ANLS = 0.0044 across 15,301 samples on four VQA benchmarks (three informative; ChartQA is near-zero for both models under greedy decoding) when compared against an independent base-model pipeline. We identify three engineering requirements (attention-mode restoration, lm_head preservation, KV-cache-aware decoding) whose omission silently breaks generation despite correct weight recovery. On ViDoRe V1, Hydra (4B) is within 1 percentage point of a controlled single-head baseline in a single training run, with higher aggregate scores on V2 and V3 that are concentrated on a subset of tasks; multi-seed experiments are needed to confirm these trends. The single-model design reduces peak GPU memory by 41%, though adapter switching introduces throughput overhead under concurrent serving loads. An ablation shows that GritLM-style joint training provides no benefit within the LoRA-based (r=16) training regime. A proof-of-concept extension to Qwen2.5-Omni-3B demonstrates that the mechanism generalizes to audio retrieval and video embedding, with speech generation.
