[论文解读] Universal Speech Enhancement with Score-based Diffusion

Joan Serrà, Santiago Pascual+3 more|arXiv (Cornell University)|Jun 7, 2022

Speech and Audio Processing被引用 63

一句话总结

本文提出了一种基于分数扩散的通用语音增强方法，旨在通过扩散概率建模提升单条件和多条件语音质量。

ABSTRACT

Removing background noise from speech audio has been the subject of considerable effort, especially in recent years due to the rise of virtual communication and amateur recordings. Yet background noise is not the only unpleasant disturbance that can prevent intelligibility: reverb, clipping, codec artifacts, problematic equalization, limited bandwidth, or inconsistent loudness are equally disturbing and ubiquitous. In this work, we propose to consider the task of speech enhancement as a holistic endeavor, and present a universal speech enhancement system that tackles 55 different distortions at the same time. Our approach consists of a generative model that employs score-based diffusion, together with a multi-resolution conditioning network that performs enhancement with mixture density networks. We show that this approach significantly outperforms the state of the art in a subjective test performed by expert listeners. We also show that it achieves competitive objective scores with just 4-8 diffusion steps, despite not considering any particular strategy for fast sampling. We hope that both our methodology and technical contributions encourage researchers and practitioners to adopt a universal approach to speech enhancement, possibly framing it as a generative task.

研究动机与目标

• 提出一个在多样声学条件下均可工作的通用、普适的语音增强解决方案。
• 利用基于分数的扩散模型来建模以嘈杂输入为条件的干净语音分布。
• 开发基于分数匹配的训练与采样过程，以实现跨场景的有效降噪。

提出的方法

• 提出一种利用基于分数生成建模的扩散框架用于语音增强。
• 利用去噪分数匹配和随机微分方程（SDE）表述来建模数据分布的梯度。
• 采用退火采样并对嘈杂输入进行条件化以产生增强的波形估计。
• 将该方法置于先前的扩散与分数匹配文献之中，以实现波形域中的鲁棒降噪。

实验结果

研究问题

• RQ1 基于分数的扩散模型能否在不同噪声类型和录音条件下为语音增强提供普遍适用性？
• RQ2 如何对嘈杂语音进行条件化以引导扩散过程，产生高质量、无伪影的增强语音？
• RQ3 哪些训练与采样策略能在保持感知质量的同时实现有效降噪？
• RQ4 所提出的方法与现有的基于扩散或非扩散的语音增强方法相比如何？

主要发现

• 提出了一种基于分数扩散的通用语音增强方法。
• 描述了利用分数匹配进行波形降噪的训练与采样过程。
• 将该方法置于更广义的音频生成与推断扩散框架之中。
• 讨论在多样声学条件下泛化性和鲁棒性方面的潜在收益。