[Paper Review] EEG based Continuous Speech Recognition using Transformers

Gautam Krishna, Co Tran+2 more|arXiv (Cornell University)|Dec 31, 2019

EEG and Brain-Computer Interfaces16 references4 citations

TL;DR

This paper investigates end-to-end EEG-based continuous speech recognition using a transformer architecture, demonstrating faster training and better performance than RNN-based models on small vocabularies (e.g., 5 unique sentences), but showing RNN-CTC models outperform transformers as vocabulary size increases (e.g., 30 sentences), with WER rising significantly for transformers beyond small-scale tests.

ABSTRACT

In this paper we investigate continuous speech recognition using electroencephalography (EEG) features using recently introduced end-to-end transformer based automatic speech recognition (ASR) model. Our results demonstrate that transformer based model demonstrate faster training compared to recurrent neural network (RNN) based sequence-to-sequence EEG models and better performance during inference time for smaller test set vocabulary but as we increase the vocabulary size, the performance of the RNN based models were better than transformer based model on a limited English vocabulary.

Motivation & Objective

• To explore the feasibility of using transformer-based models for end-to-end EEG-to-text continuous speech recognition.
• To compare transformer-based models with established RNN-CTC models in terms of training speed, inference performance, and robustness to increasing vocabulary size.
• To assess the interpretability of attention mechanisms in EEG-based sequence modeling, particularly in alignment between EEG inputs and predicted text.
• To evaluate the impact of data size and model capacity on performance, especially given the complexity and noise in EEG signals.

Proposed method

• The study employs a standard transformer encoder-decoder architecture with 8 stacked layers in both encoder and decoder, each containing multi-head self-attention and feed-forward sublayers with residual connections and layer normalization.
• Positional encoding using sine and cosine functions is applied to preserve sequence order, with d_model = 256, d_ff = 1024, d_k = d_v = 8, and h = 32 attention heads.
• The model uses shared embedding weights between encoder and decoder, and predicts one word per time step using a dense layer with softmax activation.
• Training is performed using the Adam optimizer for 120 epochs with a batch size of 100, using cross-entropy loss.
• Inference employs beam search with a 4-gram language model (shallow fusion), and generation stops at the end-of-sequence token.
• The model is evaluated on two EEG datasets (A and B) from prior work, using word-level WER as the primary metric across varying test set sizes.

Experimental results

Research questions

• RQ1 Does the transformer architecture outperform RNN-CTC models in EEG-based continuous speech recognition for small-vocabulary test sets?
• RQ2 How does model performance, particularly WER, change as the size of the test vocabulary increases?
• RQ3 Can the attention mechanism in transformers be meaningfully interpreted for EEG signals, given their complex and noisy nature?
• RQ4 Does the transformer model offer faster training and inference times than RNN-CTC models on EEG data?
• RQ5 Can improved data scaling mitigate the performance drop of transformers on larger vocabularies?

Key findings

• For test sets with only 5 unique sentences, the transformer model achieved a WER of 67.7% on Data set A and 62.5% on Data set B, outperforming RNN-CTC models.
• As vocabulary size increased to 30 unique sentences, the transformer WER rose to 93.95% on Data set A and 96.8% on Data set B, indicating a significant performance drop.
• The RNN-CTC model from [12] consistently outperformed the transformer model on larger test sets, achieving lower WER across all vocabulary sizes tested.
• The transformer model trained 2.4x faster per example than the RNN-CTC model (0.017 vs. 0.041 minutes per example on Data set A), though inference was slower due to larger model size.
• Despite faster training, the transformer model required longer inference time per example (0.016 min) than the RNN-CTC model (0.0015 min), primarily due to higher parameter count and longer weight loading times.
• Attention visualization for EEG inputs remained uninterpretable, with no clear alignment between attention weights and meaningful linguistic or neural features.