[Paper Review] ASVspoof 2021: Automatic Speaker Verification Spoofing and Countermeasures Challenge Evaluation Plan

Héctor Delgado, Nicholas Evans+9 more|arXiv (Cornell University)|Sep 1, 2021

Speech Recognition and SynthesisComputer Science6 references130 citations

TL;DR

This paper outlines the ASVspoof 2021 challenge plan with three tasks (Logical Access, Physical Access, and Speech Deepfake), evaluation metrics (ASV-constrained t-DCF and EER), data partitions, baselines, and submission rules to promote robust countermeasures against spoofing.

ABSTRACT

The automatic speaker verification spoofing and countermeasures (ASVspoof) challenge series is a community-led initiative which aims to promote the consideration of spoofing and the development of countermeasures. ASVspoof 2021 is the 4th in a series of bi-annual, competitive challenges where the goal is to develop countermeasures capable of discriminating between bona fide and spoofed or deepfake speech. This document provides a technical description of the ASVspoof 2021 challenge, including details of training, development and evaluation data, metrics, baselines, evaluation rules, submission procedures and the schedule.

Motivation & Objective

• Promote spoofing countermeasures robust to channel variability and real-world conditions.
• Evaluate countermeasures on recordings captured in real physical spaces and compressed data.
• Assess data augmentation effects and broaden relevance to non-ASV deepfake detection.
• Provide a common evaluation framework with defined protocols, metrics, and baselines.

Proposed method

• Introduce three tasks: Logical Access (LA), Physical Access (PA), and Speech Deepfake (DF).
• Use ASV-constrained t-DCF as the primary metric for LA and PA, and EER for DF.
• Provide labeled training/development data and baseline codes; evaluation data is unlabelled.
• Common ASV system based on deep speaker embeddings with PLDA scoring.
• Baseline countermeasures include LFCC-GMM, CQCC-GMM, LFCC-LCNN, and RawNet2, with source code available in a public repository.

Experimental results

Research questions

• RQ1 How can countermeasures generalize across codec and transmission variability in LA PA scenarios?
• RQ2 Can countermeasures trained on ASVspoof 2019 data adapt to more realistic, real-space recordings?
• RQ3 What is the impact of data augmentation and domain mismatch on spoofing detection performance?
• RQ4 How effective are deepfake detection approaches in DF without an ASV system?
• RQ5 What are the defined evaluation protocols and metrics that enable fair, scalable comparison of countermeasures?

Key findings

• The plan introduces three tasks (LA, PA, DF) to broaden spoofing detection beyond traditional ASV constraints.
• Primary metric is ASV-constrained t-DCF for LA and PA, with EER as a secondary metric; DF uses EER.
• Training uses ASVspoof 2019 data for CA development, with new evaluation data reflecting real-world conditions.
• Baseline countermeasures and open-source code are provided to facilitate reproducible comparisons.
• Evaluation rules restrict external data usage and mandate a single submission per task during the evaluation phase.