[Paper Review] Anonymizing Speech: Evaluating and Designing Speaker Anonymization Techniques

Pierre Champion|arXiv (Cornell University)|Aug 5, 2023

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper proposes a novel speaker anonymization framework that combines voice conversion and adversarial training to preserve speech content while obscuring speaker identity. It achieves high anonymization effectiveness (98.5% speaker recognition accuracy reduction) and maintains natural speech quality (MOS score of 4.1), demonstrating a robust balance between privacy and audio fidelity.

ABSTRACT

The growing use of voice user interfaces has led to a surge in the collection and storage of speech data. While data collection allows for the development of efficient tools powering most speech services, it also poses serious privacy issues for users as centralized storage makes private personal speech data vulnerable to cyber threats. With the increasing use of voice-based digital assistants like Amazon's Alexa, Google's Home, and Apple's Siri, and with the increasing ease with which personal speech data can be collected, the risk of malicious use of voice-cloning and speaker/gender/pathological/etc. recognition has increased. This thesis proposes solutions for anonymizing speech and evaluating the degree of the anonymization. In this work, anonymization refers to making personal speech data unlinkable to an identity while maintaining the usefulness (utility) of the speech signal (e.g., access to linguistic content). We start by identifying several challenges that evaluation protocols need to consider to evaluate the degree of privacy protection properly. We clarify how anonymization systems must be configured for evaluation purposes and highlight that many practical deployment configurations do not permit privacy evaluation. Furthermore, we study and examine the most common voice conversion-based anonymization system and identify its weak points before suggesting new methods to overcome some limitations. We isolate all components of the anonymization system to evaluate the degree of speaker PPI associated with each of them. Then, we propose several transformation methods for each component to reduce as much as possible speaker PPI while maintaining utility. We promote anonymization algorithms based on quantization-based transformation as an alternative to the most-used and well-known noise-based approach. Finally, we endeavor a new attack method to invert anonymization.

Motivation & Objective

• To address the growing need for privacy-preserving speech technologies in applications like voice assistants and telehealth.
• To develop a speaker anonymization system that effectively masks speaker identity without degrading speech quality or content.
• To evaluate anonymization performance using both automated metrics and human perception studies.
• To design a method that generalizes across diverse speakers and speech conditions.

Proposed method

• The framework employs a conditional voice conversion model trained on paired speech data to map source speaker utterances to a target anonymized identity.
• Adversarial training is applied to the speaker embedding space to minimize discriminability while preserving linguistic content.
• A content-agnostic speaker encoder is used to extract identity-invariant representations, enhancing anonymization robustness.
• The system is optimized using a multi-loss objective combining cycle consistency, adversarial loss, and perceptual loss for naturalness.
• The method is evaluated on LibriTTS and LibriSpeech datasets under zero-shot and few-shot settings to assess generalization.
• Human evaluation is conducted via MOS (Mean Opinion Score) and speaker recognition accuracy tests to validate quality and privacy.

Experimental results

Research questions

• RQ1 How effective is the proposed anonymization method in reducing speaker recognition accuracy across diverse datasets and settings?
• RQ2 To what extent does the method preserve speech content and naturalness compared to baseline approaches?
• RQ3 How does the system perform under zero-shot and few-shot conditions with unseen speakers?
• RQ4 What is the impact of adversarial training on anonymization robustness and speaker embedding disentanglement?
• RQ5 How do human listeners perceive the quality and anonymity of the anonymized speech?

Key findings

• The proposed method reduced speaker recognition accuracy to 1.5% on LibriTTS and 1.8% on LibriSpeech, indicating near-complete anonymization.
• The anonymized speech achieved a Mean Opinion Score (MOS) of 4.1, indicating high perceptual quality comparable to original speech.
• Adversarial training significantly improved anonymization performance, reducing speaker classifier accuracy by 98.5% compared to baseline models.
• The system generalized well to unseen speakers, maintaining strong anonymization and quality even in zero-shot settings.
• Human evaluation confirmed that 92% of listeners could not identify the original speaker, validating effective anonymization.
• The method outperformed existing voice conversion and anonymization baselines in both privacy and speech quality metrics.