Search papers, labs, and topics across Lattice.
This paper introduces hLLM, a novel decoding strategy that optimizes generative reranking by leveraging a permutation-structured output format, allowing for the decoding of all ordinal values in constant time. By employing the Hungarian algorithm on an $N \times K$ item-position score matrix derived from LLM prefill hidden states, hLLM achieves a remarkable 64-fold speed-up in end-to-end inference while maintaining ranking quality comparable to traditional methods. The study further explores the impact of training signals and backbone adaptation, providing a comprehensive analysis of architectural contributions to the method's efficiency.
Achieving 64x faster generative reranking without sacrificing quality, hLLM redefines efficiency in language model output.
Large language models (LLMs) achieve state-of-the-art generative ranking quality, but the ranking they produce must be decoded, and autoregressive decoding spends one sequential forward pass per emitted token. We observe that the only tokens a ranker must emit are the $N$ ordinal values naming the items in ranked order, and that this narrow, permutation-structured output format admits decoding strategies which are much more efficient than left-to-right generation. We introduce hLLM (Hungarian LLM), a format-specialized decoding strategy that decodes all $N$ ordinals in $O(1)$ forward passes. hLLM reads an $N \times K$ item-position score matrix off the LLM's prefill hidden states with a lightweight self-attention head, then decodes the ordinals as the optimal bipartite assignment of that matrix via the Hungarian algorithm, yielding a valid permutation by construction rather than by repair. Through a systematic study of training signals and backbone adaptation, we show that LoRA-based fine-tuning combined with teacher ranking distillation reaches 28 ms end-to-end inference, a speed-up of $64\times$ while maintaining ranking quality on par with the teacher. We provide a complete ablation decomposing the contributions of architecture, training signal, and backbone adaptation. Our framework connects generative ranking to combinatorial optimization, opening a path toward other $O(1)$-decode mechanisms for real-time ranking.
