[Paper Review] SASV 2022: The First Spoofing-Aware Speaker Verification Challenge

Jee-weon Jung, Hemlata Tak+7 more|arXiv (Cornell University)|Mar 28, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper introduces the SASV 2022 challenge, the first to promote jointly optimized spoofing-aware speaker verification systems that integrate anti-spoofing and speaker verification into a single, unified framework. The top-performing system achieved a 0.13% equal error rate (EER), reducing the conventional speaker verification EER of 23.83% by over 99%, demonstrating the effectiveness of integrated solutions over separate systems.

ABSTRACT

The first spoofing-aware speaker verification (SASV) challenge aims to integrate research efforts in speaker verification and anti-spoofing. We extend the speaker verification scenario by introducing spoofed trials to the usual set of target and impostor trials. In contrast to the established ASVspoof challenge where the focus is upon separate, independently optimised spoofing detection and speaker verification sub-systems, SASV targets the development of integrated and jointly optimised solutions. Pre-trained spoofing detection and speaker verification models are provided as open source and are used in two baseline SASV solutions. Both models and baselines are freely available to participants and can be used to develop back-end fusion approaches or end-to-end solutions. Using the provided common evaluation protocol, 23 teams submitted SASV solutions. When assessed with target, bona fide non-target and spoofed non-target trials, the top-performing system reduces the equal error rate of a conventional speaker verification system from 23.83% to 0.13%. SASV challenge results are a testament to the reliability of today's state-of-the-art approaches to spoofing detection and speaker verification.

Motivation & Objective

• Address the growing need for reliable speaker verification systems resilient to spoofing attacks, which can bypass traditional systems using synthesized or manipulated speech.
• Overcome the limitations of the ASVspoof challenge, which treats spoofing detection and speaker verification as separate, independently optimized systems.
• Promote the development of integrated, jointly optimized solutions that simultaneously assess speaker identity and utterance authenticity, improving both security and usability.
• Establish a common evaluation protocol and benchmark using the ASVspoof 2019 LA database to enable fair comparison and reproducibility across teams.
• Demonstrate that fusion of pre-trained ASV and spoofing countermeasure (CM) models, or end-to-end integration, can achieve significantly lower error rates than conventional systems.

Proposed method

• Introduce a new evaluation protocol that includes target trials, bona fide non-target trials, and spoofed non-target trials to assess system reliability under realistic spoofing conditions.
• Provide open-source pre-trained models for both speaker verification (ECAPA-TDNN) and spoofing detection (based on ASVspoof 2019), enabling participants to build upon existing state-of-the-art components.
• Support two solution strategies: (1) back-end fusion of independent ASV and CM systems at score, decision, or embedding levels; (2) end-to-end training of a single model for joint speaker and spoofing classification.
• Use the minimum tandem detection cost function (min t-DCF) as the primary metric, which evaluates the combined performance of ASV and CM systems in a unified framework.
• Enable ensemble methods by allowing teams to combine multiple systems, with some teams using score normalization and quality-aware features to improve robustness.
• Implement a common development and evaluation split using the ASVspoof 2019 Logical Access (LA) database, ensuring consistency and reproducibility across submissions.

Experimental results

Research questions

• RQ1 Can jointly optimized or fused speaker verification and spoofing countermeasure systems achieve significantly lower error rates than conventional, separate systems?
• RQ2 How does the performance of spoofing-aware speaker verification vary with different proportions of spoofed non-target trials, and is the system robust to varying spoofing priors?
• RQ3 To what extent can score-level, decision-level, or embedding-level fusion of pre-trained ASV and CM models improve overall system reliability?
• RQ4 Can end-to-end integrated models that learn a shared latent space for speaker identity and spoofing artifacts outperform modular fusion approaches?
• RQ5 Does the use of meta-features (e.g., speech duration) in score normalization improve system robustness across diverse trial conditions?

Key findings

• The top-performing system achieved a spoofing-aware equal error rate (SASV-EER) of 0.13%, representing a 99.45% reduction from the conventional speaker verification EER of 23.83%.
• All three highest-performing systems used ensemble fusion of independent ASV and CM sub-systems, with the best system employing score-level fusion and quality-aware normalization.
• The top system’s DET curve was below all others, indicating superior trade-off between false acceptances and false rejections across operating points.
• The system achieved similarly low SV-EER and SPF-EER values, indicating insensitivity to the proportion of spoofed non-target trials, suggesting robustness to varying spoofing priors.
• The results demonstrate that integrated solutions significantly outperform conventional speaker verification systems, even when the latter are not exposed to spoofed trials.
• Despite strong performance, all top systems remained ensembles of separate ASV and CM components, indicating that truly end-to-end integrated models remain an open challenge for future work.