Search papers, labs, and topics across Lattice.
This paper introduces WaterMoE, a novel watermarking scheme designed for Mixture-of-Experts (MoE) large language models that addresses the dual challenges of performance degradation and inference overhead associated with existing watermarking techniques. By embedding watermarking signals directly into the expert selection process, WaterMoE achieves high fidelity in generated outputs while maintaining a minimal increase in inference latency. Experimental results show that WaterMoE not only preserves model performance comparable to unwatermarked outputs but also provides a significant speedup over state-of-the-art methods, making it suitable for real-world applications.
WaterMoE achieves watermarking with only 1% additional latency while maintaining fidelity nearly indistinguishable from unwatermarked outputs.
Large language models (LLMs) have achieved remarkable success but raise growing concerns about content provenance and misuse, motivating the need for reliable watermarking techniques. However, these techniques have rarely been adopted in practice mainly for two reasons: i) severely degraded model performance, and ii) additional inference overhead. To confirm the problem, we construct a comprehensive benchmark spanning different generation tasks to systematically evaluate 9 representative watermarking methods. We found almost all existing methods are designed for text fluency, but not for restricted and complicated tasks, and their overhead prevents them from deployment in latency-critical systems. To address i) and ii), we propose an LLM watermarking scheme \textit{WaterMoE} for the growingly popular Mixture-of-Experts (MoE) LLMs. WaterMoE embeds watermarking signals through controlled perturbation into the expert selection at each router, which accumulates to token selection shift at the final output. In contrast to watermarking as a post-processing token-sampling approach, WaterMoE embeds watermark within the inference loop incurring negligible quality degradation and computational overhead. Extensive experiments demonstrate that our method achieves a fidelity performance close to the unwatermarked and consistently outperforms state-of-the-art watermarking methods on the benchmark, with up to $4\times$ speedup, incurring merely 1\% additional inference latency compared to native generation. The results demonstrate the capability of WaterMoE to be deployed in real-world tasks.
