[论文解读] Extracting the Locus of Attention at a Cocktail Party from Single-Trial EEG using a Joint CNN-LSTM Model.

Ivine Kuruvila, Jan Muncke+2 more|arXiv (Cornell University)|Feb 8, 2021

EEG and Brain-Computer Interfaces参考文献 28被引用 6

一句话总结

该论文提出了一种联合CNN-LSTM模型，通过分析单次试验的EEG信号和多说话人频谱图，实现对鸡尾酒会环境中听觉注意力的推断，三秒内的中位解码准确率达到77.2%。该模型对稀疏性具有鲁棒性，在参数剪枝达50%时仍能保持性能。

ABSTRACT

Human brain performs remarkably well in segregating a particular speaker from interfering speakers in a multi-speaker scenario. It has been recently shown that we can quantitatively evaluate the segregation capability by modelling the relationship between the speech signals present in an auditory scene and the cortical signals of the listener measured using electroencephalography (EEG). This has opened up avenues to integrate neuro-feedback into hearing aids whereby the device can infer user's attention and enhance the attended speaker. Commonly used algorithms to infer the auditory attention are based on linear systems theory where the speech cues such as envelopes are mapped on to the EEG signals. Here, we present a joint convolutional neural network (CNN) - long short-term memory (LSTM) model to infer the auditory attention. Our joint CNN-LSTM model takes the EEG signals and the spectrogram of the multiple speakers as inputs and classifies the attention to one of the speakers. We evaluated the reliability of our neural network using three different datasets comprising of 61 subjects where, each subject undertook a dual-speaker experiment. The three datasets analysed corresponded to speech stimuli presented in three different languages namely German, Danish and Dutch. Using the proposed joint CNN-LSTM model, we obtained a median decoding accuracy of 77.2% at a trial duration of three seconds. Furthermore, we evaluated the amount of sparsity that our model can tolerate by means of magnitude pruning and found that the model can tolerate up to 50% sparsity without substantial loss of decoding accuracy.

研究动机与目标

• 开发一种深度学习模型，用于在多说话人环境中从EEG信号推断听觉注意力。
• 通过端到端学习原始EEG和频谱图输入，超越线性模型。
• 利用多样化数据集评估模型在不同语言间的泛化能力。
• 通过幅度剪枝评估模型在结构稀疏性下的鲁棒性。

提出的方法

• 联合卷积神经网络（CNN）与长短期记忆网络（LSTM）架构处理多个说话人的EEG信号和频谱图。
• CNN从EEG中提取空间和时间特征，LSTM则建模神经反应中的序列依赖关系。
• 输入特征包括德语、丹麦语和荷兰语的双说话人语音刺激的EEG时间序列和频谱图。
• 基于联合输入表征，模型分类听者正在关注的说话人。
• 通过幅度剪枝评估模型鲁棒性，减少参数数量的同时监控准确率。

实验结果

研究问题

• RQ1 联合CNN-LSTM模型能否在多说话人场景中准确解码单次试验EEG的听觉注意力？
• RQ2 在双说话人实验中，模型性能在不同语言间如何变化？
• RQ3 模型在不造成显著准确率下降的前提下，能容忍多大程度的结构稀疏性？
• RQ4 联合架构在注意力解码中是否优于传统线性模型？

主要发现

• 联合CNN-LSTM模型在61名受试者中，三秒试验时长的中位解码准确率达到77.2%。
• 该模型在德语、丹麦语和荷兰语三种不同语言数据集上均保持高性能，表明具备跨语言泛化能力。
• 幅度剪枝结果表明，模型在稀疏度达50%时仍能保持准确率，无显著下降。
• 结果表明，深度学习模型可有效从单次试验EEG中解码听觉注意力，支持实时神经反馈应用。