Search papers, labs, and topics across Lattice.
This paper critiques the prevalent use of Equal Error Rate (EER) in evaluating voice anonymization, arguing that it inadequately captures individual speaker information leakage in the log-likelihood ratio (LLR) space. By defending the privacy-ZEBRA framework and elucidating the equivalence of rank-based metrics to principles of perfect secrecy, the authors highlight the shortcomings of current evaluation methods. Their findings, validated through simulations and VoicePrivacy Challenge data, underscore the necessity of adopting more robust privacy metrics in voice anonymization research.
EER may mislead researchers about the effectiveness of voice anonymization, while privacy-ZEBRA offers a more accurate lens on information leakage.
The voice anonymization community mainly uses Equal Error Rate (EER) to evaluate the performance of voice identity protection. While alternative metrics such as privacy-ZEBRA and a rank-based metric have been proposed, their underlying assumptions and differences may not be well known, especially to newcomers. This paper is motivated to fill the gap. Based on the concept of Shannon's perfect secrecy (or privacy), this paper positions itself as a defense of the privacy-ZEBRA framework. While no new metric is proposed, this paper explains how an `ideal'system in terms of EER may fail to gauge the information leakage on individual speakers in the log-likelihood ratio (LLR) space. The paper also shows how the rank-based metric can be cast into a metric that follows the same principle of perfect secrecy and how their best solutions are equivalent. Furthermore, the paper explains how the method of estimating LLRs may affect the evaluation results. These discussions are, to the best of the authors'knowledge, not explored or explained in detail in existing papers. Last but not least, the findings are demonstrated on simulated and VoicePrivacy Challenge data.
