[Paper Review] Do You Remember? Overcoming Catastrophic Forgetting for Fake Audio Detection

Xiaohui Zhang, Jiangyan Yi+3 more|arXiv (Cornell University)|Aug 7, 2023

Music and Audio Processing4 citations

TL;DR

This paper proposes Regularized Adaptive Weight Modification (RAWM), a continual learning method for fake audio detection that adaptively adjusts model weights based on the ratio of real to fake audio during fine-tuning, while applying regularization to preserve knowledge under distribution shifts. RAWM outperforms existing methods on cross-dataset detection benchmarks, achieving state-of-the-art performance without requiring access to old training data.

ABSTRACT

Current fake audio detection algorithms have achieved promising performances on most datasets. However, their performance may be significantly degraded when dealing with audio of a different dataset. The orthogonal weight modification to overcome catastrophic forgetting does not consider the similarity of genuine audio across different datasets. To overcome this limitation, we propose a continual learning algorithm for fake audio detection to overcome catastrophic forgetting, called Regularized Adaptive Weight Modification (RAWM). When fine-tuning a detection network, our approach adaptively computes the direction of weight modification according to the ratio of genuine utterances and fake utterances. The adaptive modification direction ensures the network can effectively detect fake audio on the new dataset while preserving its knowledge of old model, thus mitigating catastrophic forgetting. In addition, genuine audio collected from quite different acoustic conditions may skew their feature distribution, so we introduce a regularization constraint to force the network to remember the old distribution in this regard. Our method can easily be generalized to related fields, like speech emotion recognition. We also evaluate our approach across multiple datasets and obtain a significant performance improvement on cross-dataset experiments.

Motivation & Objective

• To address catastrophic forgetting in fake audio detection when fine-tuning on new datasets without access to previous data.
• To improve cross-dataset generalization by adapting weight modification direction based on real/fake audio ratios.
• To mitigate performance degradation when genuine audio distributions differ across datasets due to varying acoustic conditions.
• To develop a method applicable beyond fake audio detection, such as in speech emotion recognition.
• To achieve strong performance with minimal inductive bias, avoiding reliance on experience replay or data augmentation.

Proposed method

• RAWM adaptively computes the direction of weight modification based on the proportion of genuine and fake utterances in the new dataset.
• It uses a projector orthogonal to the subspace spanned by old inputs, but adjusts the direction toward the old input subspace when genuine audio dominates.
• When fake audio is more prevalent, the modification direction shifts closer to orthogonal to the old input subspace.
• A regularization constraint is introduced to preserve the feature distribution of old genuine audio, especially when new and old datasets have different acoustic conditions.
• The method does not require storing or replaying old data, making it suitable for real-world deployment with proprietary models.
• The approach is generalized to image recognition via experiments on the CLEAR-10 benchmark, confirming broad applicability.

(a)

Experimental results

Research questions

• RQ1 How can catastrophic forgetting be mitigated in fake audio detection when fine-tuning on new datasets without access to old training data?
• RQ2 What is the optimal weight update direction when the ratio of real to fake audio in the new dataset varies?
• RQ3 How can knowledge of old genuine audio distributions be preserved when new and old datasets are collected under significantly different acoustic conditions?
• RQ4 Can the proposed method generalize to other machine learning tasks beyond fake audio detection?
• RQ5 How does RAWM compare to existing continual learning baselines in terms of performance and robustness across diverse datasets?

Key findings

• RAWM achieved the highest average accuracy of 92.53% across 10 experiences on the CLEAR-10 image classification benchmark, outperforming all baselines including Replay, EWC, LwF, GDF, CWR, and OWM.
• On the ASVspoof and ADD fake audio detection benchmarks, RAWM achieved a 95.25% accuracy on the final experience (Exp10), surpassing OWM (95.05%) and other methods.
• In cross-dataset detection, RAWM significantly outperformed fine-tuning, EWC, LwF, GDF, CWR, and OWM, demonstrating robustness to distribution shifts.
• RAWM closely approached the upper bound performance of the Replay method in Exp3 and Exp8, indicating strong knowledge retention.
• The method maintained high performance even when training on only a few samples, showing strong sample efficiency.
• The regularization component effectively preserved old feature distributions under acoustic condition shifts, improving generalization in challenging cross-dataset scenarios.

(b)