[Paper Review] Evaluating the structure of cognitive tasks with transfer learning

Bruno Aristimunha, Raphael De Camargo+4 more|arXiv (Cornell University)|Jul 28, 2023

EEG and Brain-Computer Interfaces91 references4 citations

TL;DR

This study introduces transfer learning maps for EEG decoding by evaluating deep neural network representations across 11 cognitive tasks in two large EEG datasets (ERP CORE and M3 CV). It demonstrates that pre-training on one task boosts decoding performance by up to 28% on others via linear probing, revealing hierarchical task relationships and enabling data-efficient brain-computer interface development.

ABSTRACT

Electroencephalography (EEG) decoding is a challenging task due to the limited availability of labelled data. While transfer learning is a promising technique to address this challenge, it assumes that transferable data domains and task are known, which is not the case in this setting. This study investigates the transferability of deep learning representations between different EEG decoding tasks. We conduct extensive experiments using state-of-the-art decoding models on two recently released EEG datasets, ERP CORE and M$^3$CV, containing over 140 subjects and 11 distinct cognitive tasks. We measure the transferability of learned representations by pre-training deep neural networks on one task and assessing their ability to decode subsequent tasks. Our experiments demonstrate that, even with linear probing transfer, significant improvements in decoding performance can be obtained, with gains of up to 28% compare with the pure supervised approach. Additionally, we discover evidence that certain decoding paradigms elicit specific and narrow brain activities, while others benefit from pre-training on a broad range of representations. By revealing which tasks transfer well and demonstrating the benefits of transfer learning for EEG decoding, our findings have practical implications for mitigating data scarcity in this setting. The transfer maps generated also provide insights into the hierarchical relations between cognitive tasks, hence enhancing our understanding of how these tasks are connected from a neuroscientific standpoint.

Motivation & Objective

• Address data scarcity in EEG decoding by investigating transferability of deep learning representations across cognitive tasks.
• Understand the hierarchical relationships between cognitive tasks from a neuroscientific perspective using EEG data.
• Develop transfer maps to guide efficient transfer learning in EEG-based brain-computer interfaces and clinical applications.
• Evaluate the effectiveness of linear probing transfer across diverse EEG decoding paradigms to identify transferable representations.
• Provide practical and neuroscientific insights into which tasks benefit from pre-training on broad or narrow representations.

Proposed method

• Pre-train state-of-the-art deep learning models (e.g., EEGNet, EEGFormer) on one cognitive task using labeled EEG data from ERP CORE and M3 CV.
• Fine-tune the frozen feature extractor on a downstream task using only a linear classifier (linear probing) to assess transferability.
• Construct transfer maps by measuring decoding accuracy gains across all pairs of 11 distinct cognitive tasks.
• Use two large, recently released EEG datasets with over 140 subjects and diverse decoding modalities (e.g., N170, ERN, P300) to ensure generalization.
• Apply standardized data preprocessing and training protocols across all tasks to ensure fair comparison and reproducibility.
• Leverage transfer learning to explore subject-level transfer and task-level transfer for mitigating BCI illiteracy.

Figure 2: A. Data splitting and alignment. Source and target tasks correspond to different ERP and BCI paradigms; B. EEG decoding models as a representer network and a classification head; C. Standard EEG decoding training and evaluation; D. Transfer with linear probing. Only the classification head

Experimental results

Research questions

• RQ1 Which cognitive tasks in EEG decoding exhibit strong transferability to others when using pre-trained deep representations?
• RQ2 To what extent can linear probing transfer improve decoding performance compared to pure supervised learning in low-data regimes?
• RQ3 What hierarchical relationships exist between cognitive tasks based on the transferability of neural representations?
• RQ4 Do certain decoding paradigms elicit highly specific brain activity patterns that limit transferability?
• RQ5 Can transfer learning maps inform source localization and functional network modeling in EEG by identifying shared evoked components?

Key findings

• Transfer learning via linear probing improved decoding performance by up to 28% compared to pure supervised learning, demonstrating significant gains in data-scarce settings.
• Tasks like N170 and ERN showed low transferability to other tasks, indicating they elicit highly specific and narrow brain activity patterns.
• Other tasks, such as P300 and motor imagery, benefited from pre-training on a broad range of representations, suggesting reliance on generalizable neural features.
• Transfer maps revealed a complex, asymmetric hierarchy among cognitive tasks, with some tasks serving as effective pre-training sources for multiple downstream tasks.
• The results suggest that shared functional networks underlie closely related tasks, as evidenced by similar evoked components and transfer patterns.
• Transfer learning maps provide practical guidance for mitigating BCI illiteracy by enabling transfer from high-performing to low-performing tasks at the subject level.

Figure 3: Illustration of experiments recorded in ERP CORE ( A to F ) and M 3 CVdatasets ( G to J ). Labels under the stimuli correspond to what subjects are supposed to answer during the trials. They do not necessarily correspond to decoding labels. A. In the active visual oddball task P3b, partici