Search papers, labs, and topics across Lattice.
To eliminate the need for brittle, hand-engineered prompts in conformal language retrieval, this work establishes the Conformal Relevance framework, which automatically generates score functions via curated in-context learning and ensembling. Applying conformal prediction to language tasks like summarization and extractive QA provides rigorous coverage guarantees, but practical utility depends entirely on scoring functions that minimize prediction set size (conciseness). Across seven NLP tasks, the approach maintains valid coverage while boosting conciseness, backed by theoretical proofs that formalize the complementarity conditions required for ensemble diversity to improve worst-case conformal scores alongside a fundamental saturation bound.
Brittle, hand-crafted prompts are no longer necessary for distribution-free NLP: ensembling diverse in-context demonstrations provably minimizes prediction set size while maintaining strict coverage guarantees.
Many NLP tasks, such as summarization and extractive question answering, reduce to retrieving relevant content from documents under two constraints: coverage, retaining enough pertinent information to achieve some goal, and conciseness, removing as much irrelevant information as possible. Conformal prediction methods have been used to guarantee coverage, and must be optimized for conciseness through design of a score function. State-of-the-art scoring functions use hand-engineered LLM prompts asking the model to rate the importance of content, but manual prompt engineering is labor-intensive and task-specific. We introduce the Conformal Relevance framework which uses in-context learning example curation and ensembling to create a score function which maintains coverage while improving conciseness with minimal manual input. We demonstrate this framework's application on seven NLP tasks, and also theoretically study the impact of diversity for ensembled conformal scores, giving a complementarity condition that characterizes when ensembling improves worst-case sentence scores, and a saturation bound on ensemble improvement.
