Search papers, labs, and topics across Lattice.
This paper introduces spec learning, a novel framework that streamlines the alignment of large language models (LLMs) using brief user instructions and a small set of preference judgments to create natural-language prompts. By avoiding the need for parameter updates, spec learning offers a more efficient and interpretable alternative to traditional preference-based fine-tuning methods, which can be costly and cumbersome. The authors demonstrate that LLMs conditioned on these specifications outperform those optimized through direct preference optimization on specialized datasets, highlighting the effectiveness of this approach in enhancing model behavior while maintaining transparency.
Spec learning enables LLMs to align with user preferences using just a few instructions, outperforming traditional optimization methods without requiring costly parameter updates.
Steering a large language model (LLM) toward a desired behavior typically relies on an iterative process of hand-crafting a prompt based on a careful inspection of the model's responses. This is an involved, brittle, and error-prone process. Preference-based fine-tuning is a more rigorous but often prohibitively expensive solution. We propose spec learning, a framework that relies on a brief user instruction and a small set of preference judgments. These are compiled into specifications in the form of natural-language prompts for an LLM. Specifications condition LLMs at inference time, and no parameter updates to the underlying models are required. We show that the responses generated based on the compiled specifications often outperform direct preference optimization (DPO) on datasets from specialized domains whose preference signal is dense. Unlike opaque weight updates, the resulting specifications are human-readable and double as interpretable and transparent written embodiments of the preference signal that produced them.
CMU ML