Search papers, labs, and topics across Lattice.
This study evaluates explainability in media bias detection using encoder-based models, specifically BERT and RoBERTa, across predictive performance, explanation plausibility, and mechanistic faithfulness. By employing the Bias Annotations By Experts (BABE) dataset and introducing attention-supervised finetuning, the authors demonstrate that model architectures exhibit significant differences in how well they align with expert rationales and recover predictive signals. The findings highlight the necessity of assessing multiple dimensions of explainability, rather than relying solely on predictive accuracy, to better understand model behavior in bias detection tasks.
Attention-supervised finetuning reveals that model architecture significantly impacts the plausibility of explanations in media bias detection, challenging assumptions about scale and performance.
Detecting media bias automatically is difficult because biased framing is often subtle, yet in domains such as news analysis, accurate predictions alone are insufficient without explanations that reflect the model's underlying reasoning. We present a multi-dimensional evaluation of explainability in encoder-based media bias detection using the Bias Annotations By Experts (BABE) dataset. Specifically, we study BERT and RoBERTa as classifiers (base and large variants) along three complementary axes: predictive performance, explanation plausibility (token-level alignment with expert rationales), and mechanistic faithfulness (whether compact sets of attention heads recover predictive signal under counterfactual rationale masking). To induce variation in plausibility, we additionally investigate attention-supervised finetuning, which incorporates expert rationale annotations as an auxiliary training signal. Attention supervision serves as an intervention on attribution plausibility, while the effectiveness of attribution methods varies substantially across architectures. Circuit analysis further reveals substantial variation in mechanistic recoverability across architectures, suggesting that model scale alone does not determine circuit compressibility. Taken together, our findings suggest that predictive performance, attribution plausibility, and mechanistic faithfulness characterize different aspects of model behavior and should be evaluated separately when studying explainability in media bias detection.
CMU ML