Skip to main content
QUICK REVIEW

[论文解读] The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark

Sylvain Chevallier, Igor Carrara|arXiv (Cornell University)|Apr 3, 2024
EEG and Brain-Computer InterfacesNeuroscience被引用 10
一句话总结

本文在 MOABB 框架内,对 36 个数据集和 30 个 ML 流水线进行有史以来最大的开放、可复现的 EEG-based BCI 管道基准测试,分析 MI、P300/ERP 和 SSVEP 范式。

ABSTRACT

Objective. This study conduct an extensive Brain-computer interfaces (BCI) reproducibility analysis on open electroencephalography datasets, aiming to assess existing solutions and establish open and reproducible benchmarks for effective comparison within the field. The need for such benchmark lies in the rapid industrial progress that has given rise to undisclosed proprietary solutions. Furthermore, the scientific literature is dense, often featuring challenging-to-reproduce evaluations, making comparisons between existing approaches arduous. Approach. Within an open framework, 30 machine learning pipelines (separated into raw signal: 11, Riemannian: 13, deep learning: 6) are meticulously re-implemented and evaluated across 36 publicly available datasets, including motor imagery (14), P300 (15), and SSVEP (7). The analysis incorporates statistical meta-analysis techniques for results assessment, encompassing execution time and environmental impact considerations. Main results. The study yields principled and robust results applicable to various BCI paradigms, emphasizing motor imagery, P300, and SSVEP. Notably, Riemannian approaches utilizing spatial covariance matrices exhibit superior performance, underscoring the necessity for significant data volumes to achieve competitive outcomes with deep learning techniques. The comprehensive results are openly accessible, paving the way for future research to further enhance reproducibility in the BCI domain. Significance. The significance of this study lies in its contribution to establishing a rigorous and transparent benchmark for BCI research, offering insights into optimal methodologies and highlighting the importance of reproducibility in driving advancements within the field.

研究动机与目标

  • 推动在 EEG 基于 BCI 领域实现开放、可重复的基准测试,以反制专有且不可重复的解决方案。
  • 更新并扩展 MOABB 基准测试,涵盖现代管道(包括深度学习),跨多个 BCI 范式。
  • 提供指南和一个开放框架,用于在开放数据集上对管道进行公平比较。

提出的方法

  • 在一个开放框架中重新实现并评估 30 种机器学习管道(原始信号:11 种,黎曼:13 种,深度学习:6 种)。
  • 使用 36 个公开可用的 EEG 数据集,涵盖运动想象 (MI)、P300 和 SSVEP 范式。
  • 使用标准的基于 Python 的工具处理数据(MNE、scikit-learn、pyriemann)以及一个 4 阶 Butterworth 带通滤波器。
  • 采用会话内评估,结合分层 k 折交叉验证,并考虑保留一个被试/会话的情景以获得更广泛的背景。
  • 执行嵌套网格搜索,带有内部交叉验证以选择超参数。
  • 应用统计元分析(效应量、p 值)并通过 Code Carbon 进行环境影响评估。
Figure 1 : Within-session evaluation, small rectangles indicate a sample or EEG trial, pastel colors on the two top lines shows the chronological order, bright color on the last three lines indicates training and testing samples/trials.
Figure 1 : Within-session evaluation, small rectangles indicate a sample or EEG trial, pastel colors on the two top lines shows the chronological order, bright color on the last three lines indicates training and testing samples/trials.

实验结果

研究问题

  • RQ1在 MI、P300/ERP 和 SSVEP 范式下,最有效的 EEG 分类方法是什么?
  • RQ2跨管道和数据集的计算时间和能量消耗如何比较?
  • RQ3在开放数据条件下,最好的深度学习方法与最佳黎曼-BCI 管道之间的比较如何?
  • RQ4在这些范式中,需要多少次试验或通道才能实现可靠的性能?
  • RQ5在 MOABB 数据集中,哪些 MI 任务或 ERP/SSVEP 配置获得最高准确度?

主要发现

  • 使用空间协方差矩阵的黎曼方法在所有范式中显示出更高的性能。
  • 深度学习需要大量数据才能在 EEG BCI 中匹配非深度方法,凸显被试层面数据稀缺。
  • MOABB 结果是开放可访问的,以实现持续、可重复的比较。
  • 该基准测试提供有原则、鲁棒的结果,为 MI、P300 和 SSVEP BCI 的方法选择提供信息。
Figure 2 : Visualization of the MOABB datasets, with Motor Imagery in green, Event Related Potential in pink/purple and Steady State Visually Evoked Potential in yellow/brown. The size of the circle is proportional to the number of subjects and the contrast depends on the number of electrodes.
Figure 2 : Visualization of the MOABB datasets, with Motor Imagery in green, Event Related Potential in pink/purple and Steady State Visually Evoked Potential in yellow/brown. The size of the circle is proportional to the number of subjects and the contrast depends on the number of electrodes.

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。