[论文解读] An online sequence-to-sequence model for noisy speech recognition

Chung‐Cheng Chiu, Dieterich Lawson+5 more|arXiv (Cornell University)|Jun 16, 2017

Speech Recognition and Synthesis参考文献 9被引用 3

一句话总结

本文提出了一种改进的在线序列到序列模型，采用改进的策略梯度训练方法，用于端到端语音识别，支持使用因果注意力机制的实时推理。在噪声单通道混合语音（Multi-TIMIT）上实现优异性能，当混合比例为10%时，音素错误率降低至25.9%，在保持低延迟的同时展现出对干扰的强鲁棒性。

ABSTRACT

Generative models have long been the dominant approach for speech recognition. The success of these models however relies on the use of sophisticated recipes and complicated machinery that is not easily accessible to non-practitioners. Recent innovations in Deep Learning have given rise to an alternative - discriminative models called Sequence-to-Sequence models, that can almost match the accuracy of state of the art generative models. While these models are easy to train as they can be trained end-to-end in a single step, they have a practical limitation that they can only be used for offline recognition. This is because the models require that the entirety of the input sequence be available at the beginning of inference, an assumption that is not valid for instantaneous speech recognition. To address this problem, online sequence-to-sequence models were recently introduced. These models are able to start producing outputs as data arrives, and the model feels confident enough to output partial transcripts. These models, like sequence-to-sequence are causal - the output produced by the model until any time, $t$, affects the features that are computed subsequently. This makes the model inherently more powerful than generative models that are unable to change features that are computed from the data. This paper highlights two main contributions - an improvement to online sequence-to-sequence model training, and its application to noisy settings with mixed speech from two speakers.

研究动机与目标

• 解决标准序列到序列模型需等待完整输入后才生成输出的局限性，使其不适合实时应用。
• 开发一种因果在线推理机制，实现音频到达时逐段生成转录结果。
• 通过引入带二值随机变量的改进策略梯度方法，提升在线序列到序列模型的训练稳定性和性能。
• 在双说话人混合的噪声单通道语音上评估模型，模拟真实干扰场景。
• 展示模型在保持低延迟和高精度的同时，能够从混合音频中识别目标说话人。

提出的方法

• 采用带软注意力机制的序列到序列架构，解码器通过编码器状态生成的上下文向量自回归地生成输出标记。
• 引入二值随机变量以决定模型何时应输出标记，通过在输入处理过程中允许部分输出，实现在线推理。
• 使用策略梯度方法训练输出决策机制，采用改进的目标函数以提升训练稳定性和收敛性。
• 在编码器和解码器中均使用单向LSTM，采用两层结构，每层256个单元，以实现高效的序列建模。
• 将模型应用于新构建的数据集Multi-TIMIT，该数据集通过以不同音量比（10%、25%、50%）混合男性和女性TIMIT语音样本生成。
• 使用标准的对数梅尔滤波器组特征（123维），并在标记级别预测上端到端训练，采用交叉熵损失。

实验结果

研究问题

• RQ1 能否将改进训练方法的在线序列到序列模型有效应用于噪声条件下的实时语音识别？
• RQ2 当面对来自两名说话人的单通道混合语音时，模型性能如何，尤其在干扰水平增加时？
• RQ3 模型的因果性和自回归特性是否使其能够在不依赖完整输入序列的情况下适应退化输入？
• RQ4 与标准模型相比，模型能否在保持竞争性准确率的同时维持低延迟？
• RQ5 模型在清洁和噪声输入下的输出时机有何差异，这对其鲁棒性有何启示？

主要发现

• 在Multi-TIMIT数据集上，当混合比例为10%时，所提模型的音素错误率（PER）达到25.9%，优于CTC（27.3%）和RNN-Transducer（25.7%）的性能。
• 在25%混合比例下，模型PER为32.5%，相较于干净数据出现适度下降，但仍优于基线模型。
• 在50%混合比例下，模型PER为42.9%，仍优于RNN-Transducer（48.9%），表明其对高干扰具有强鲁棒性。
• 与干净的TIMIT数据相比，模型在噪声Multi-TIMIT输入中输出标记的时间更晚，表明其能自适应调整以应对退化输入。
• 当每个目标语音样本与多个干扰说话人配对时，过拟合现象加剧，表明模型存在记忆特定输入-输出对的能力。
• 模型的因果架构支持实时推理，并可生成多个转录结果，提升了其在实际部署中的灵活性。