[Paper Review] NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals

Wei-Bang Jiang, Yansen Wang+2 more|arXiv (Cornell University)|Aug 27, 2024

EEG and Brain-Computer Interfaces4 citations

TL;DR

NeuroLM proposes the first universal multi-task foundation model for EEG signal processing by treating EEG as a 'foreign language' via a text-aligned neural tokenizer, enabling end-to-end integration with Large Language Models (LLMs). It achieves state-of-the-art performance across six diverse EEG tasks through multi-channel autoregressive pre-training and instruction tuning, with the largest variant (NeuroLM-XL) reaching 1.7B parameters and 25,000 hours of pre-training data.

ABSTRACT

Recent advancements for large-scale pre-training with neural signals such as electroencephalogram (EEG) have shown promising results, significantly boosting the development of brain-computer interfaces (BCIs) and healthcare. However, these pre-trained models often require full fine-tuning on each downstream task to achieve substantial improvements, limiting their versatility and usability, and leading to considerable resource wastage. To tackle these challenges, we propose NeuroLM, the first multi-task foundation model that leverages the capabilities of Large Language Models (LLMs) by regarding EEG signals as a foreign language, endowing the model with multi-task learning and inference capabilities. Our approach begins with learning a text-aligned neural tokenizer through vector-quantized temporal-frequency prediction, which encodes EEG signals into discrete neural tokens. These EEG tokens, generated by the frozen vector-quantized (VQ) encoder, are then fed into an LLM that learns causal EEG information via multi-channel autoregression. Consequently, NeuroLM can understand both EEG and language modalities. Finally, multi-task instruction tuning adapts NeuroLM to various downstream tasks. We are the first to demonstrate that, by specific incorporation with LLMs, NeuroLM unifies diverse EEG tasks within a single model through instruction tuning. The largest variant NeuroLM-XL has record-breaking 1.7B parameters for EEG signal processing, and is pre-trained on a large-scale corpus comprising approximately 25,000-hour EEG data. When evaluated on six diverse downstream datasets, NeuroLM showcases the huge potential of this multi-task learning paradigm.

Motivation & Objective

• To address the limitations of task-specific fine-tuning in EEG-based brain-computer interfaces and healthcare applications.
• To bridge the gap between language and EEG modalities by enabling LLMs to understand neural signals as a form of 'foreign language'.
• To develop a unified, multi-task foundation model capable of generalizing across diverse EEG tasks without task-specific fine-tuning.
• To overcome challenges in EEG-text alignment, effective representation learning within LLMs, and unified multi-task learning.
• To demonstrate that large-scale pre-training and instruction tuning can enable a single model to handle multiple EEG tasks effectively.

Proposed method

• A text-aligned neural tokenizer is trained via vector-quantized temporal-frequency prediction to convert continuous EEG signals into discrete neural tokens, with adversarial training to align EEG and text embeddings.
• The frozen vector-quantized (VQ) encoder extracts compact, discrete EEG token sequences that are fed into a pre-trained Large Language Model (LLM).
• Multi-channel autoregressive pre-training enables the LLM to learn causal representations by predicting future EEG tokens across multiple channels.
• Joint multi-task instruction tuning adapts the model to various downstream EEG tasks using natural language instructions, enabling zero-shot and few-shot generalization.
• The model is pre-trained on a large-scale EEG corpus of approximately 25,000 hours of data, with variants ranging from small to 1.7B parameter (NeuroLM-XL) scale.
• The framework treats EEG signals as a sequence of tokens, allowing the LLM to perform both understanding and generation across multiple EEG modalities.

Experimental results

Research questions

• RQ1 Can EEG signals be effectively encoded as discrete tokens that align semantically with natural language, enabling LLMs to process them as a 'foreign language'?
• RQ2 Can multi-channel autoregressive pre-training within an LLM framework learn causal and generalizable representations of EEG signals?
• RQ3 Can a single unified foundation model achieve strong performance across diverse, unrelated EEG tasks without task-specific fine-tuning?
• RQ4 How robust is the model to variations in instruction format, such as shuffled options in multiple-choice tasks?
• RQ5 What is the impact of instruction data size and model scale on multi-task performance in EEG signal processing?

Key findings

• NeuroLM-XL achieves state-of-the-art performance on six diverse EEG datasets, with a macro F1 score of 0.6839 ± 0.0297 on the TUSL dataset, outperforming prior methods.
• The model maintains consistent performance across varying instruction data sizes, demonstrating robustness and scalability in multi-task learning.
• Ablation studies confirm that multi-channel autoregressive pre-training significantly improves downstream performance, highlighting its importance in learning causal EEG representations.
• The model shows strong robustness to option shuffling in multiple-choice instructions, indicating genuine understanding of linguistic context rather than pattern memorization.
• Validation perplexity decreases and training loss converges stably, indicating effective generalization to unseen EEG data across all model variants.
• Despite its large size (1.7B parameters), NeuroLM-XL does not fully exploit its capacity, suggesting that current pre-training data size may be insufficient for billion-parameter models.