Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of training large language models (LLMs) to manage multi-level instruction hierarchies, which is crucial for mitigating risks from conflicting prompts. By formalizing a five-level instruction hierarchy and introducing Gravity-Weighted Direct Preference Optimization (GW-DPO), the authors enhance the model's ability to prioritize instructions based on their structural significance. The results show that GW-DPO significantly improves adherence to priority relations while reducing over-refusal rates compared to standard methods, demonstrating a more nuanced approach to instruction conflict resolution.
Gravity-Weighted DPO enables LLMs to navigate complex instruction hierarchies, achieving better compliance with priority relations while halving over-refusal rates.
Production LLMs receive instructions from sources with very different levels of trust, yet attend to every token with uniform architectural privilege. This is the structural vulnerability that enables malicious prompt injections and, more broadly, leaves models without a principled way to resolve conflicts between legitimate but competing instructions. A common training-based response is to teach models an explicit instruction hierarchy; existing approaches, however, formalize hierarchies of only three or four levels, treat all violations as equally severe, and rarely evaluate the full set of pairwise level interactions. We formalize a k-level instruction hierarchy problem and instantiate it for k=5, yielding ten pairwise priority relations that a compliant model must enforce. We then introduce Gravity-Weighted DPO (GW-DPO), a preference-optimization objective whose per-sample offset scales with the structural distance between conflicting levels under a linear or bilateral schedule, the latter weighting severity by both the privilege gap and the privilege of the victim level. Combined with hierarchy-specific delimiter tokens (Chen et al., 2025) and Instructional Segment Embeddings (ISE; Wu et al., 2025), GW-DPO with the bilateral schedule Pareto-improves over standard DPO and the linear variant on Llama-3.1-8B-Instruct, raising macro pairwise priority adherence while keeping over-refusal at half the standard DPO rate. Ablations isolate ISE as a refusal-threshold calibrator and recast five- versus three-level training as a generality-specialization tradeoff.