[Paper Review] FairSSD: Understanding Bias in Synthetic Speech Detectors

Amit Kumar Singh Yadav, Kratika Bhagtani+3 more|arXiv (Cornell University)|Apr 17, 2024

Speech Recognition and Synthesis4 citations

TL;DR

This paper introduces FairSSD, a comprehensive evaluation of bias in six state-of-the-art synthetic speech detectors across gender, age, accent, and speech impairment demographics. Using over 0.9 million bona fide speech samples, the study reveals significant fairness issues: detectors exhibit higher false positive rates for males, older adults (60s), South Asian and Australian English speakers, and individuals with stuttering, with mean EER rising from 5.29% to 36.31% on impaired speech, highlighting critical fairness gaps in current detection systems.

ABSTRACT

Methods that can generate synthetic speech which is perceptually indistinguishable from speech recorded by a human speaker, are easily available. Several incidents report misuse of synthetic speech generated from these methods to commit fraud. To counter such misuse, many methods have been proposed to detect synthetic speech. Some of these detectors are more interpretable, can generalize to detect synthetic speech in the wild and are robust to noise. However, limited work has been done on understanding bias in these detectors. In this work, we examine bias in existing synthetic speech detectors to determine if they will unfairly target a particular gender, age and accent group. We also inspect whether these detectors will have a higher misclassification rate for bona fide speech from speech-impaired speakers w.r.t fluent speakers. Extensive experiments on 6 existing synthetic speech detectors using more than 0.9 million speech signals demonstrate that most detectors are gender, age and accent biased, and future work is needed to ensure fairness. To support future research, we release our evaluation dataset, models used in our study and source code at https://gitlab.com/viper-purdue/fairssd.

Motivation & Objective

• To investigate whether synthetic speech detectors exhibit demographic bias toward specific genders, ages, and accents.
• To evaluate whether detectors misclassify bona fide speech from speech-impaired individuals—particularly those with stuttering—as synthetic.
• To quantify fairness gaps in existing detectors using large-scale, diverse speech corpora.
• To release a comprehensive evaluation dataset, models, and code to support future research in fair synthetic speech detection.

Proposed method

• The study evaluates six existing synthetic speech detectors on a curated dataset of 0.9 million bona fide speech samples from Mozilla Common Voice and Sep-28K for speech-impaired speakers.
• Demographic groups are defined by gender (male/female), age (teens, 20s, 30s, 40s, 50s, 60s), and accent (e.g., South Asian, Australian English, American, British).
• False Positive Rate (FPR) and Equal Error Rate (EER) are computed for each demographic subgroup to measure bias, with thresholds derived from independent evaluation sets.
• Statistical analysis compares FPR and EER across demographic groups to identify significant disparities in detector performance.
• The evaluation includes both in-the-wild detection scenarios and controlled comparisons using standardized metrics.
• All models, datasets, and code are released via GitLab to enable reproducibility and further fairness research.

Experimental results

Research questions

• RQ1 Do synthetic speech detectors exhibit higher false positive rates for specific gender groups?
• RQ2 Are detectors more likely to misclassify speech from older adults (e.g., 60s) as synthetic compared to younger groups?
• RQ3 Is there a significant performance disparity across different regional accents, such as South Asian or Australian English?
• RQ4 Do detectors show elevated false positive rates for bona fide speech from individuals with speech impairments like stuttering?
• RQ5 To what extent does demographic bias in detectors undermine their reliability and fairness in real-world deployment?

Key findings

• Synthetic speech detectors exhibit significant gender bias, with male speakers showing a mean FPR increase of 1.27% compared to females.
• Speakers in the 60s age group have a mean FPR increase of 1.27% compared to the 20s group, indicating age-based bias.
• South Asian and Australian English accent groups show the highest FPR increases, with mean FPR differences of 1.03% and 1.27% respectively, compared to American English.
• Detectors show a dramatic increase in EER for stuttering speech, rising from 5.29% on fluent speech to 36.31% on impaired speech, indicating severe bias against speech-impaired individuals.
• The mean FPR for all detectors on stuttering speech exceeds 70% when using fixed thresholds, confirming widespread misclassification of genuine speech as synthetic.
• Detector D05 shows the highest EER increase (11.00%) on stuttering speech, while D04 shows the most consistent bias across all demographic groups.