[论文解读] Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation

Rongzhi Gu, Yuexian Zou|arXiv (Cornell University)|Jan 2, 2020

Speech and Audio Processing参考文献 66被引用 10

一句话总结

本文提出一种时频-空间神经滤波器，用于在混响环境中实现端到端的多通道目标语音分离，通过联合建模时序、频谱、空间及方向特征，提升分离精度。该方法采用全卷积自编码器结构，实现低延迟、单次遍历推理，当说话人方向夹角超过15°时，对方向估计误差具有鲁棒性，且在参数量更少、速度更快的前提下，优于当前最先进方法。

ABSTRACT

Target speech separation refers to extracting the target speaker's speech from mixed signals. Despite the recent advances in deep learning based close-talk speech separation, the applications to real-world are still an open issue. Two main challenges are the complex acoustic environment and the real-time processing requirement. To address these challenges, we propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture in reverberant environments, assisted with directional information of the speaker(s). Firstly, against variations brought by complex environment, the key idea is to increase the acoustic representation completeness through the jointly modeling of temporal, spectral and spatial discriminability between the target and interference source. Specifically, temporal, spectral, spatial along with the designed directional features are integrated to create a joint acoustic representation. Secondly, to reduce the latency, we design a fully-convolutional autoencoder framework, which is purely end-to-end and single-pass. All the feature computation is implemented by the network layers and operations to speed up the separation procedure. Evaluation is conducted on simulated reverberant dataset WSJ0-2mix and WSJ0-3mix under speaker-independent scenario. Experimental results demonstrate that the proposed method outperforms state-of-the-art deep learning based multi-channel approaches with fewer parameters and faster processing speed. Furthermore, the proposed temporal-spatial neural filter can handle mixtures with varying and unknown number of speakers and exhibits persistent performance even when existing a direction estimation error. Codes and models will be released soon.

研究动机与目标

• 解决在复杂声学条件下混响、多说话人环境中真实世界语音分离的挑战。
• 通过直接估计时域波形，避免相位重建误差，克服时频掩码方法的局限性。
• 通过全卷积、单次遍历的端到端架构实现鲁棒的实时处理，具备低延迟特性。
• 通过联合建模时序、频谱、空间及方向可分性，提升分离精度。
• 确保对方向估计误差的鲁棒性，尤其在说话人方向不接近时表现更优。

提出的方法

• 将时序、频谱、空间及方向感知特征整合为统一的联合声学表征，以增强语音源的区分能力。
• 设计两种方向特征——cosIPD与DPR，基于通道间相位差与方向网格分辨率，表征每个时频点上各源的主导性。
• 实现全卷积自编码器网络，单次遍历处理输入混合信号，支持实时、低延迟推理。
• 端到端训练模型，直接从多通道混合信号预测目标语音波形，跳过中间的时频掩码估计步骤。
• 将说话人方向作为输入，引导网络聚焦于目标源，实现说话人特定的输出分配。
• 利用方向特征增强空间区分能力，尤其在角度差超过15°时效果显著。

实验结果

研究问题

• RQ1 联合建模时序、频谱、空间及方向特征是否能提升混响环境中目标语音分离的精度？
• RQ2 当说话人方向接近时，该方法在方向估计误差下的表现如何？
• RQ3 方向特征在多大程度上增强了对说话人定位角度不确定性的鲁棒性？
• RQ4 全卷积、单次遍历架构能否在模型参数更少、延迟更低的前提下实现高性能分离？
• RQ5 该方法如何处理未知或变化的说话人数量的混合信号？

主要发现

• 在WSJ0-2mix与WSJ0-3mix数据集上，该方法在说话人无关条件下优于当前最先进的基于深度学习的多通道语音分离方法。
• 当方向估计准确且说话人角度差超过15°时，与cosIPD+AF基线相比，SI-SDRi提升超过0.3 dB。
• 方向估计误差导致的性能下降极小——当角度差超过15°且误差在±10°以内时，2-mix的SI-SDRi下降不足0.1 dB，3-mix下降不足0.4 dB。
• 在所有测试的方向估计误差下，系统保持鲁棒性，WSJ0-2mix的性能下降小于1.2 dB，WSJ0-3mix的下降小于1.7 dB。
• 与现有端到端方法相比，该模型处理速度更快、参数量更少，支持实时部署。
• 该方法能成功处理最多三名说话人同时存在的混合信号，并根据方向输入正确分配输出至对应说话人。