Search papers, labs, and topics across Lattice.
The authors evaluate standard translation metrics and frontier LLM judges on social media text, uncovering a severe failure mode where metrics like COMET and XCOMET exhibit negative correlations with human judgments and actively reward worse translations. To remedy this lack of cultural grounding, they introduce CuRIL, a reinforcement learning framework that prepends loss-masked cultural hints to model reasoning and anneals their presence to zero over training. Applied to Qwen3-8B, CuRIL improves Cohen's kappa from 0.162 to 0.370鈥攎atching Gemini-3.1-Pro with 30x fewer parameters鈥攁nd cuts downstream low-quality translation generation by over 20 percentage points when serving as an RL reward signal.
Standard translation metrics and 235B-parameter LLM judges actively reward the worst social media translations due to cultural blindness, but progressively decaying masked cultural scaffolding enables an 8B model to rival Gemini-3.1-Pro.
Automatic translation quality metrics trained on general-domain corpora systematically fail on social media content, where communicative intent is encoded in culturally loaded expressions (internet slang, homophonic ciphers, and platform-specific idioms) rather than surface token patterns. We conduct a systematic empirical analysis demonstrating that standard metrics including COMET, XCOMET, and BERTScore exhibit near-zero or negative correlation with human cultural judgments, and even display a severity inversion in which scores increase as translation quality deteriorates. We further show that this failure extends to large language model judges: Qwen3-235B achieves Cohen's kappa of only 0.162, revealing that the bottleneck is not reasoning capacity but cultural grounding: models lack the domain-specific cultural knowledge needed to identify which aspects of a translation require scrutiny. To address this, we propose CuRIL, a reinforcement learning framework that internalizes cultural reasoning: cultural annotations are prepended inside the model's reasoning, excluded from policy gradients via a token-level loss mask, and injected with a probability that decays to zero over training, progressively forcing autonomous cultural judgment. On a 1,444-sample human-annotated social media translation benchmark, Qwen3-8B trained with CuRIL achieves Cohen's kappa 0.370 and Exact Match accuracy of 45.22%, approaching Gemini-3.1-Pro with 30x fewer parameters and surpassing models up to 235B in scale. We further demonstrate that our judge produces reliable reward signals for downstream translation optimization, reducing the low-quality translation rate by over 20 percentage points under independent human evaluation.
