Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the limitations of existing machine translation (MT) metrics by focusing on extrinsic discourse evaluation in both static and interactive contexts. The authors introduce an entity counting task to assess referential consistency and demonstrate that high intrinsic quality does not guarantee effective downstream discourse, revealing that even strong MT systems can produce significant referential inconsistencies. Additionally, through the Welfare Diplomacy game, they show that translation failures in interactive settings adversely affect coordination outcomes, underscoring the importance of evaluating MT in goal-oriented scenarios.
High intrinsic quality in machine translation doesn't ensure effective communication, as referential inconsistencies can still derail downstream discourse success.
Existing machine translation (MT) metrics and discourse-focused evaluations primarily assess translation quality intrinsically, without measuring the downstream consequences of translation errors. In this work, we focus on extrinsic discourse evaluation of machine translation under two distinct regimes: static and interactive. Under the static regime, we propose an entity counting task as a probe of referential consistency in discourse. We show that high intrinsic MT quality does not reliably predict downstream discourse success and strong MT systems still produce referential inconsistencies. For the interactive regime, we study the goal-oriented multi-agent Welfare Diplomacy game as a probe of long-horizon communication and coordination. We find that interaction-specific translation failures impact downstream coordination. Our results highlight goal-oriented environments as a viable framework for discourse-sensitive extrinsic MT evaluation.