Search papers, labs, and topics across Lattice.
This paper introduces TUP, a novel policy for distillation that enhances generation by truncating low-ranked completions and upweighting only the higher-ranked candidates based on a reward model. By employing a sharper reweighting strategy, TUP eliminates low-ranked options from the support, thereby reducing reliance on potentially brittle top rankings while maintaining competitive performance against existing alignment methods. Theoretical analysis supports the effectiveness of this approach, demonstrating that lower-tail truncation can match the performance of traditional rank-reweighting methods under certain conditions.
Truncating low-ranked completions can outperform traditional rank-based policies by reducing reliance on brittle top rankings while still achieving strong alignment results.
Inference-time selection methods, such as Best-of-N, improve generation by sampling a pool of candidates and selecting the top-ranked completion according to a reward model. Distillation seeks to amortize this procedure into a single policy by replacing raw rewards with in-pool ranks and learning a policy that upweights higher-ranked completions. However, existing rank-based policies typically use smooth full-support reweighting, so low-ranked completions receive less mass but remain in the target support. Although a sharper reweighting reduces lower-tail mass, it also increases reliance on brittle ranking at the top made by a single reward model. We propose TUP: a Truncate-bad, Upweight-good Policy that removes low-ranked completions from the support and reweights only the retained upper tail with a tunable sharpness. TUP admits a closed-form, prompt-independent normalization and can be trained fully offline via binary cross-entropy, using shifted-truncated win-rates as soft labels and distilled-to-reference log-likelihood ratios as logits. Theoretically, under certain assumptions, we show that for any unknown oracle reward, the best monotone rank-reweighting can be matched by a lower-tail truncation rule, providing formal support for removing the lower tail rather than merely downweighting it. Empirically, we show that TUP is competitive with strong offline alignment baselines.
