[论文解读] Training on the test set? An analysis of Spampinato et al. [31]

Li Ren, Jared Sigurd Johansen+5 more|arXiv (Cornell University)|Dec 18, 2018

Neural dynamics and brain function参考文献 23被引用 16

一句话总结

本文质疑了Spampinato等人关于使用块设计从EEG信号中分类40类物体的主张的有效性，表明其高准确率结果源于数据污染，即测试样本与训练样本来自同一区块，而非真正的脑信号解码。当改用快速事件设计时，性能下降至随机水平，且随机码书的表现优于脑信号衍生的表征，从而动摇了脑启发计算机视觉进展的核心主张。

ABSTRACT

A recent paper [31] claims to classify brain processing evoked in subjects watching ImageNet stimuli as measured with EEG and to use a representation derived from this processing to create a novel object classifier. That paper, together with a series of subsequent papers [8, 15, 17, 20, 21, 30, 35], claims to revolutionize the field by achieving extremely successful results on several computer-vision tasks, including object classification, transfer learning, and generation of images depicting human perception and thought using brain-derived representations measured through EEG. Our novel experiments and analyses demonstrate that their results crucially depend on the block design that they use, where all stimuli of a given class are presented together, and fail with a rapid-event design, where stimuli of different classes are randomly intermixed. The block design leads to classification of arbitrary brain states based on block-level temporal correlations that tend to exist in all EEG data, rather than stimulus-related activity. Because every trial in their test sets comes from the same block as many trials in the corresponding training sets, their block design thus leads to surreptitiously training on the test set. This invalidates all subsequent analyses performed on this data in multiple published papers and calls into question all of the purported results. We further show that a novel object classifier constructed with a random codebook performs as well as or better than a novel object classifier constructed with the representation extracted from EEG data, suggesting that the performance of their classifier constructed with a representation extracted from EEG data does not benefit at all from the brain-derived representation. Our results calibrate the underlying difficulty of the tasks involved and caution against sensational and overly optimistic, but false, claims to the contrary.

研究动机与目标

• 调查Spampinato等人使用块设计从EEG信号中分类40类物体的主张是否有效。
• 检验报告的高分类准确率是否源于真实的刺激相关脑活动，还是块设计带来的虚假时间相关性。
• 评估EEG中提取的脑源表征是否真正提升计算机视觉模型性能，或随机码书表现同样优异或更优。
• 挑战依赖块设计和静态脑活动的EEG基计算机视觉方法的广泛方法论基础。
• 提醒研究界警惕基于污染数据和有缺陷实验协议的过度乐观主张。

提出的方法

• 采用快速事件设计重新实现分类流程，将不同类别的刺激随机交错呈现，以消除块级时间相关性。
• 分析EEG数据中的长期静态活动，特别是DC和极低频（VLF）分量，这些分量在块设计中主导信号。
• 比较基于脑源表征的分类器与使用随机码书的分类器性能，以隔离神经数据的贡献。
• 实施随机数据攻击，检验分类模型是否依赖数据污染而非特定刺激的神经编码。
• 通过激活图分析，评估移除特征图组件对分类准确率的影响，且不依赖脑信号。
• 对LSTM-全连接-ReLU架构进行消融研究，评估其对静态与动态神经模式的依赖程度。

实验结果

研究问题

• RQ1 Spampinato等人报告的高分类准确率是否依赖于将同一类的所有刺激集中在一起的块设计？
• RQ2 在随机化类别顺序并打破块级相关性的快速事件设计下，能否复现基于EEG的物体分类？
• RQ3 DC和极低频EEG分量在块设计中的分类性能中贡献有多大？
• RQ4 EEG中提取的脑源表征是否真正提升物体分类性能，还是随机码书表现同样优异？
• RQ5 所宣称的脑启发计算机视觉进展是否有效，还是数据污染和实验设计缺陷的产物？

主要发现

• 当采用快速事件设计时，分类准确率下降至随机水平，表明原始结果依赖于块级时间相关性，而非特定刺激的神经编码。
• 分类器性能主要由DC和极低频EEG分量驱动，这些分量反映的是块内任意的长期心理状态，而非动态的刺激诱发反应。
• 使用随机码书训练的新物体分类器表现与基于脑源表征的分类器相当或更优，表明神经数据无实际增益。
• 块设计导致对测试集的隐蔽训练，因为测试试验与训练试验来自同一区块，这使得多篇已发表论文的后续分析全部失效。
• 依赖块设计的EEG基计算机视觉方法的整个方法论基础受到质疑，因为数据污染破坏了所有脑信号解码声称的有效性。
• 结果表明，从EEG中分类物体类别的基础任务远比宣称的更困难，当前方法尚无法实现所声称的最先进性能。