Search papers, labs, and topics across Lattice.
This paper introduces Polaris, a system that fine-tunes a large language model (LLM) to generate table descriptions by leveraging retrieval feedback from existing table retrieval benchmarks. By utilizing Direct Preference Optimization (DPO) on candidate descriptions ranked by their BM25 effectiveness, Polaris significantly enhances the quality of table descriptions compared to the state-of-the-art AutoDDG solution. The findings highlight the potential of repurposing retrieval benchmarks as effective supervision for training LLMs in generating retrieval-oriented metadata.
Polaris shows that optimizing LLM-generated table descriptions for retrieval effectiveness can lead to significant performance improvements over existing methods.
Many table-centric NLP tasks such as NL2SQL first retrieve relevant tables from large collections using keyword search. Recent work uses LLMs to generate natural-language table descriptions to improve retrieval, but they are typically optimized for fluency rather than retrieval effectiveness. We present Polaris, a system that trains an LLM to generate table descriptions directly from retrieval feedback. Our key insight is that existing table retrieval benchmarks already contain the supervision needed for this task: given query-table relevance judgments, we generate multiple candidate descriptions for each table, rank them by their BM25 retrieval effectiveness, and use the resulting preference pairs to fine-tune the LLM with Direct Preference Optimization (DPO). Polaris further expands abbreviated table and column names before generation to reduce vocabulary mismatch. Extensive experiments show that Polaris outperforms the state-of-the-art AutoDDG solution, often by a significant margin. More broadly, our results demonstrate that retrieval benchmarks can be repurposed as supervision for training LLMs to generate retrieval-oriented metadata.
