[Paper Review] An Open-Access Multi-modal Dataset for Cognitive, Motor, and Cognitive-Motor Tasks

Zaineb Ajra, Grégoire Vergotte+6 more|arXiv (Cornell University)|Mar 24, 2026

Optical Imaging and Spectroscopy Techniques0 citations

TL;DR

Presents an open-access multimodal dataset (EEG, fNIRS, ECG, behavioral, and subjective measures) collected from 30 healthy participants across seven cognitive, motor, and cognitive-motor tasks, stored in BIDS format and available on OpenNeuro. Includes initial validation showing task difficulty relates to neural patterns, especially in EEG channel-level data.

ABSTRACT

The incorporation of neuroimaging techniques such as electroenchephalography (EEG) and functional near-infrared spectroscopy (fNIRS) has provided new opportunities for the analysis of dynamic brain processes involved in cognitive and motor functions. Despite the great contribution of the open-access neuroimaging datasets to neuroscience studies, they have mainly remained on a single modality and isolated task paradigms performed in a controlled environments. These limitations restrict the analysis of multi-task effects in real-world applications, thus creating a gap in the understanding of how cognitive and motor processes interact in daily life activities. To address these limitations, we present a multi-modal dataset containing neurophysiological (EEG, fNIRS), physiological (ECG), behavioral, and subjective measures collected from 30 healthy participants over three sessions. This dataset includes a hierarchical series of seven tasks ranging from single cognitive and motor activities, such as N-back, motor, passive motor, mental arithmetic and motor imagery, to combined cognitive-motor interactions simulating real life scenarios. This raw dataset provides a resource for developing advanced preprocessing methods and analysis pipelines, with potential applications in brain-computer interfaces, neurorehabilitation, and other fields requiring an understanding of multi-tasks brain dynamics. https://doi.org/10.18112/openneuro.ds007554.v1.0.0

Motivation & Objective

• Motivate ecologically valid multimodal neuroimaging by coupling cognitive and motor tasks in a hierarchical task structure.
• Provide a raw, openly accessible dataset to enable development of preprocessing pipelines and multimodal analysis methods.
• Enable research on brain-behavior interactions across cognitive, motor, and combined task conditions in real-world-like scenarios.

Proposed method

• Simultaneous EEG (32 channels) and fNIRS (prefrontal and sensorimotor cortices) recordings across three sessions.
• Collect physiological (ECG) and behavioral (push-button, Biodex torque) data plus subjective measures (KSS, task difficulty).
• Design seven tasks arranged hierarchically: Mental Arithmetic, N-back, Motor Imagery, Passive Motor, Active Motor, NB-MA, and Full NB-MA-Act-Mot.
• Synchronize multimodal streams using Lab Streaming Layer to produce aligned XDF data with event markers.
• Provide data in BIDS-structured format (EEG .edf, fNIRS .snirf, physiologic/behavioral .tsv/.json) without preprocessing.
• Offer validation analyses (EEG/fNIRS preprocessing, RSA) as benchmarks, not prescriptive pipelines.

Figure 1: Experimental paradigm overview

Experimental results

Research questions

• RQ1 How do cognitive and motor tasks, alone and in combination, modulate neural signals across EEG and fNIRS modalities?
• RQ2 Do EEG and fNIRS patterns reflect subjective task difficulty, and is this reflected at whole-head or channel-specific levels?
• RQ3 Can a raw, multimodal BIDS dataset support development of preprocessing pipelines and multimodal fusion methods for cognitive-m motor-task analysis?
• RQ4 What is the feasibility of decoding or characterizing cognitive-motor interactions using EEG, fNIRS, and physiological data?

Key findings

• Subjective task difficulty systematically varied across seven conditions and was higher for multitask than single-task conditions.
• EEG channel-level RSA showed significant correlations between neural patterns and task difficulty in several frontal, central, and temporal channels; whole-head EEG did not show this relation.
• fNIRS RSA did not yield significant correlations after multiple-comparison correction at the channel level for O2Hb or HHb signals in a group Analysis, though data quality supported channel-level interpretations.
• Preprocessing and validation pipelines (EEG PREP, clean_rawdata, Homer3, qt-nirs) can reproduce reported effects, illustrating usefulness of raw data for diverse analyses.
• The dataset enables exploration of multivariate analyses, decoding, and BCI benchmarking in more realistic cognitive-motor contexts.

Figure 2: Configuration of the experimental fNIRS-EEG system. Left: picture of the head cap (rear view). Right: spatial layout of the 32-channel EEG electrodes arranged according to the international 10–20 system.