[Paper Review] Constrained Variational Autoencoder for improving EEG based Speech Recognition Systems

Gautam Krishna, Co Tran+2 more|arXiv (Cornell University)|Jun 1, 2020

EEG and Brain-Computer Interfaces4 citations

TL;DR

This paper proposes a constrained variational autoencoder (VAE) with a recurrent neural network architecture to learn meaningful, low-dimensional EEG representations from raw EEG signals, significantly improving EEG-based continuous and isolated speech recognition. The method outperforms prior approaches, especially as vocabulary size increases, demonstrating superior performance on limited-vocabulary datasets for both continuous and isolated speech tasks.

ABSTRACT

In this paper we introduce a recurrent neural network (RNN) based variational autoencoder (VAE) model with a new constrained loss function that can generate more meaningful electroencephalography (EEG) features from raw EEG features to improve the performance of EEG based speech recognition systems. We demonstrate that both continuous and isolated speech recognition systems trained and tested using EEG features generated from raw EEG features using our VAE model results in improved performance and we demonstrate our results for a limited English vocabulary consisting of 30 unique sentences for continuous speech recognition and for an English vocabulary consisting of 2 unique sentences for isolated speech recognition. We compare our method with another recently introduced method described by authors in [1] to improve the performance of EEG based continuous speech recognition systems and we demonstrate that our method outperforms their method as vocabulary size increases when trained and tested using the same data set. Even though we demonstrate results only for automatic speech recognition (ASR) experiments in this paper, the proposed VAE model with constrained loss function can be extended to a variety of other EEG based brain computer interface (BCI) applications.

Motivation & Objective

• To address the challenge of noisy and high-dimensional EEG signals in brain-computer interfaces for speech recognition.
• To improve the performance of EEG-based automatic speech recognition (ASR) systems using deep representation learning.
• To develop a constrained VAE that generates more semantically meaningful EEG features than standard VAEs.
• To demonstrate the effectiveness of the proposed method on both continuous and isolated speech recognition tasks.
• To enable broader applicability of the model to other EEG-based brain-computer interface (BCI) applications.

Proposed method

• A recurrent neural network-based variational autoencoder (RNN-VAE) is designed to model temporal dependencies in raw EEG sequences.
• A novel constrained loss function is introduced to regularize the latent space, encouraging disentangled and meaningful representations.
• The model learns a posterior distribution over latent variables conditioned on raw EEG inputs, with reparameterization for end-to-end training.
• The constrained loss incorporates structural inductive biases to improve disentanglement and generalization in the learned features.
• The VAE is fine-tuned to generate compact, discriminative EEG embeddings suitable for downstream ASR tasks.
• The method is evaluated on two ASR benchmarks: one with 30 continuous sentences and another with 2 isolated sentences.

Experimental results

Research questions

• RQ1 Can a constrained VAE with RNN architecture learn more meaningful and disentangled representations from raw EEG signals than standard VAEs?
• RQ2 Does the proposed method improve recognition accuracy in EEG-based continuous and isolated speech recognition systems?
• RQ3 How does the performance of the proposed method scale with increasing vocabulary size compared to prior approaches?
• RQ4 Can the learned EEG features generalize across different speech recognition tasks and be extended to other BCI applications?
• RQ5 What is the impact of the constrained loss function on the quality and discriminability of the learned latent representations?

Key findings

• The proposed constrained VAE significantly improves speech recognition performance on both continuous and isolated speech recognition tasks using EEG signals.
• The method outperforms a recently proposed baseline method described in [1], particularly as vocabulary size increases.
• For continuous speech recognition with a 30-sentence vocabulary, the model achieves improved recognition accuracy over the baseline.
• For isolated speech recognition with a 2-sentence vocabulary, the model demonstrates robust performance using the generated EEG features.
• The constrained loss function leads to more disentangled and meaningful latent representations, enhancing downstream ASR performance.
• The model's architecture and training scheme are generalizable to other EEG-based BCI applications beyond speech recognition.