Search papers, labs, and topics across Lattice.
This paper introduces IFHierBench, a hierarchical benchmark designed to evaluate the instruction-following capabilities of large language models (LLMs) across multiple constraint levels. The study reveals that even the best-performing models achieve only marginally above 50% accuracy when handling complex, nested constraints, with performance significantly declining as the depth of constraints increases. These findings highlight a critical gap in current LLMs' ability to adhere to structured instructions, suggesting a need for improved training methodologies focused on hierarchical constraint adherence.
Current LLMs struggle to follow nested constraints, achieving only 50% accuracy on complex tasks, revealing a significant gap in their instruction-following capabilities.
Instruction-following ability is critical for deploying large language models in real-world applications, where downstream components depend on the output satisfying specific constraints. Modern deployments increasingly handle the full task in a single LLM call, with one prompt specifying a layered output whose overall artifact, structural sections, and nested fields must each satisfy concrete constraints. Existing instruction-following benchmarks treat the constraint set as a flat list applied uniformly to the response, so they cannot scope a check to a particular section of the output. We introduce IFHierBench, a hierarchical instruction-following benchmark of 600 prompts stratified across four constraint-tree depths and 35 distinct constraints, each prompt paired with a deterministic checker that verifies satisfaction at every scope. Evaluating seven leading proprietary and open-weight models, we find that even the strongest model only marginally exceeds 50% prompt-level accuracy and that accuracy degrades sharply as constraint depth grows. Reliably following nested constraints remains a substantial gap for current LLMs, motivating future training methods that consider constraint adherence at finer granularity to achieve better instruction-following ability.
