Search papers, labs, and topics across Lattice.
This paper introduces CDR-Bench, a novel benchmark designed to evaluate the faithful execution of compositional, order-sensitive data refinement recipes across 3,462 tasks in four real-world domains. The study reveals that state-of-the-art LLMs exhibit significant performance degradation in both compositional settings and when processing order-sensitive recipes, indicating a critical gap in their procedural faithfulness. These findings highlight the limitations of current models in reliably executing complex data refinement tasks, raising concerns about their practical applicability in real-world scenarios.
LLMs struggle with compositional reasoning, showing sharp performance drops in executing order-sensitive data refinement tasks.
Data refinement involves executing multi-step recipes over evolving text states, where both composition and execution order of processing operators determine the outcome. While existing benchmarks either isolate text editing or entangle it with code and tool execution, it remains unclear whether LLMs can directly and faithfully execute these compositional, order-sensitive data refinement recipes. To fill this gap, we introduce CDR-Bench, a comprehensive benchmark featuring 3,462 high-quality tasks spanning four real-world data refinement domains and 29 distinct operators. Our benchmark evaluates models across atomic, order-agnostic, and order-sensitive settings, leveraging deterministic reference outputs to enable exact evaluation. Experiments on 10+ state-of-the-art LLMs reveal consistent failure patterns: performance degrades sharply in compositional settings, and order-sensitive recipe success collapses. These findings underline that current LLMs lack the procedural faithfulness required for reliable compositional data refinement.
DAMO