Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
GrammarRL addresses the semantic quality degradation inherent in grammar-constrained text generation by employing a label-free reinforcement learning approach. The method optimizes language models using two self-supervised rewards—direct and reverse likelihoods—derived from the model's own predictions, mitigating the need for annotated data. Experiments on diverse tasks with Llama models demonstrate significant improvements in generation quality over existing methods, matching or exceeding beam search performance at greedy decoding costs.
LLMs can learn to generate syntactically valid text without sacrificing semantic quality, even with underspecified prompts, by leveraging self-supervised rewards derived from their own likelihoods.
Grammar-constrained generation guarantees syntactic validity, but can substantially degrade semantic quality when the model's preferred outputs are poorly aligned with the imposed grammar. This trade-off is particularly severe when the prompt is underspecified or the model has limited instruction-following ability. Beam search can partially mitigate these failures by exploring multiple valid sequences, but its computational cost grows with beam width, while sequence-level probability is only an imperfect proxy for semantic quality. We introduce GrammarRL, a label-free reinforcement learning method that adapts language models to grammar constraints without requiring annotated data. GrammarRL optimizes the model using two complementary self-supervised rewards derived from its own likelihoods: a direct reward, measuring how likely the constrained output is given the input, and a reverse reward, measuring how well the input can be reconstructed from the generated output. We optimize these rewards with a Reinforce Leave-One-Out (RLOO) objective over groups of grammar-constrained rollouts, augmented with the top-1 beam-search hypothesis and regularized towards a frozen base model. We evaluate GrammarRL on sign language gloss translation, hierarchical text classification, and named entity recognition using Llama models ranging from 1B to 8B parameters. GrammarRL consistently outperforms constrained greedy decoding, with an average improvement of 9.8 points and gains of up to 22.8 BLEU. It matches or outperforms beam search on two of the three tasks while preserving greedy-decoding inference cost. Ablations further show that the two rewards are complementary: either reward alone can underperform the untrained baseline, whereas their combination consistently improves upon it.