Search papers, labs, and topics across Lattice.
The paper introduces \$OneMillion-Bench, a new benchmark comprising 400 expert-curated tasks across five domains (Law, Finance, Industry, Healthcare, and Natural Science) designed to evaluate language agents in complex, real-world professional scenarios. The benchmark emphasizes retrieval of authoritative sources, conflict resolution, domain-specific rule application, and constraint-based decision-making. Evaluation uses a rubric-based protocol assessing factual accuracy, logical coherence, practical feasibility, and professional compliance, providing a rigorous testbed for agentic capabilities.
Current language agents are still far from matching human expert performance when faced with real-world professional tasks requiring complex reasoning, authoritative source retrieval, and domain-specific knowledge, as revealed by the new \$OneMillion-Bench benchmark.
As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce \OneMillion-Bench OneMillion-Bench, a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focused on expert-level problems to ensure meaningful differentiation across agents. Together, \$OneMillion-Bench provides a unified testbed for assessing agentic reliability, professional depth, and practical readiness in domain-intensive scenarios.
Tsinghua AI