Search papers, labs, and topics across Lattice.
The paper introduces CLSGen, a fine-tuning framework for LLMs that enables joint probabilistic classification and explanation generation for binary tasks. CLSGen uses a dual-head architecture with a classification head and a generation head, trained with a combination of cross-entropy loss and a novel data construction strategy to prevent linguistic collapse. Experiments show CLSGen outperforms baselines in AUROC and F1-score while maintaining explanation quality.
LLMs can provide reliable probabilities for binary classification *without* sacrificing their ability to generate high-quality, aligned explanations, thanks to a novel dual-head fine-tuning approach.
With the recent progress of Large Language Models (LLMs), there is a growing interest in applying these models to solve complex and challenging problems. Modern LLMs, capable of processing long contexts and generating verbalized explanations, offer significant potential in addressing real-world applications. However, a critical hurdle in deploying LLMs for practical decision-making is their inability to provide reliable, quantitative probabilities. While task-specific fine-tuning of LLMs using traditional discriminative objectives (similar to encoder-only models) can yield probability estimates, this often leads to catastrophic forgetting and linguistic collapse. Consequently, the model loses its ability to generate explanations, severely undermining its interpretability and usability. To address this challenge, we propose CLSGen, a novel LLM fine-tuning framework designed for binary classification tasks. The CLSGen framework encompasses a new model architecture, training methodology, and data construction strategy to enable robust probability estimation without sacrificing the model's inherent explanation-generation capabilities. Experimental results across multiple benchmark datasets demonstrate that models fine-tuned with CLSGen outperform existing baselines in classification metrics (AUROC and F1-score). Regarding explanation, the results showed strong alignment between predicted labels and generated justifications, as well as high readability.
