Search papers, labs, and topics across Lattice.
This paper introduces the Embedded Attack method, which reveals how harmful supervision can be concealed within benign training samples, effectively bypassing existing defenses. The authors demonstrate that typical guardrails are insufficient for detecting these hidden threats at the example level. To counter this vulnerability, they propose Dual-Reference SFT (DR-SFT), a novel approach that employs token-level regularization to enhance the robustness of supervised fine-tuning against harmful influences.
Hidden harmful supervision can infiltrate training data without detection, undermining the effectiveness of current safety measures.
Existing defenses are effective when harmful content is explicitly mixed into downstream fine-tuning data, but crafted samples can instead hide harmful supervision inside benign tasks. We propose Embedded Attack, where harmful QA pairs are embedded within benign training samples, and show that representative guardrails often fail to detect them at the example level. To address this, we propose Dual-Reference SFT (DR-SFT), which adapts DPO-style contrastive objective design to SFT through token-level regularization, mitigating harmful fine-tuning beyond coarse data filtering.
