Search papers, labs, and topics across Lattice.
The paper introduces HardMTBench, a new Chinese-English translation benchmark designed to stress-test models on knowledge-intensive domains like finance, healthcare, and law, where existing benchmarks have saturated. They created a dataset of 10,000 hand-curated sentences across 12 domains using a three-stage pipeline involving LLM-based difficulty assessment and hardness fusion. Evaluation of 22 systems on HardMTBench reveals a doubled performance range compared to FLORES-200, significant rank reorderings, and exposes domain-specific weaknesses masked by quality-only metrics.
LLMs might seem fluent in Chinese-English translation, but HardMTBench reveals their surprising struggles with domain-specific knowledge, exposing weaknesses hidden by standard benchmarks.
General-purpose machine translation benchmarks such as FLORES-200 have reached a saturation regime on Chinese-English pairs, where modern large language models cluster within a narrow band of high scores. Across 22 systems, FLORES-200 zh-en GEMBA scores fall in a 7.87-point range with a standard deviation of 2.29, which compresses the separation between systems on knowledge-intensive domains such as finance, healthcare, law, and science and technology. We introduce HardMTBench, a difficulty-aware diagnostic benchmark for bidirectional Chinese-English domain translation. HardMTBench covers 12 domains and contains 10,000 hand-curated source sentences with reference translations, packaged as 20,000 directional test items. A three-stage construction pipeline builds a domain-balanced candidate pool of 84{,}566 pairs, applies an LLM-based multi-signal judge over knowledge density, translation difficulty, terminology load and reference correctness, and assembles the final test set under a hardness fusion rule with per-domain quotas. Across 22 systems spanning general LLMs, commercial engines and specialised MT models, HardMTBench widens the cross-system GEMBA range by roughly a factor of two over FLORES-200, induces visible rank reorderings, and exposes domain-specific terminology and knowledge weaknesses that quality-only metrics tend to flatten. All data and code are open-sourced at https://github.com/jasonNLP/HardMTBench.
