[论文解读] A Two-stream End-to-End Deep Learning Network for Recognizing Atypical Visual Attention in Autism Spectrum Disorder

Jin Xie, Longfei Wang+5 more|arXiv (Cornell University)|Nov 26, 2019

Autism Spectrum Disorder Research参考文献 1被引用 10

一句话总结

本文提出了一种双流端到端深度学习网络，联合处理眼动数据和视觉图像，以分类自闭症谱系障碍（ASD）中的非典型视觉注意力。通过利用两个不同流路的空间和时间特征，该模型在区分ASD与典型发育个体方面实现了95%的准确率，优于先前的最先进方法。

ABSTRACT

Eye movements have been widely investigated to study the atypical visual attention in Autism Spectrum Disorder (ASD). The majority of these studies have been focused on limited eye movement features by statistical comparisons between ASD and Typically Developing (TD) groups, which make it difficult to accurately separate ASD from TD at the individual level. The deep learning technology has been highly successful in overcoming this issue by automatically extracting features important for classification through a data-driven learning process. However, there is still a lack of end-to-end deep learning framework for recognition of abnormal attention in ASD. In this study, we developed a novel two-stream deep learning network for this recognition based on 700 images and corresponding eye movement patterns of ASD and TD, and obtained an accuracy of 0.95, which was higher than the previous state-of-the-art. We next characterized contributions to the classification at the single image level and non-linearly integration of this single image level information during the classification. Moreover, we identified a group of pixel-level visual features within these images with greater impacts on the classification. Together, this two-stream deep learning network provides us a novel and powerful tool to recognize and understand abnormal visual attention in ASD.

研究动机与目标

• 开发一种端到端的深度学习框架，整合眼动数据与图像数据，实现ASD的个体水平分类。
• 克服传统统计方法依赖手工设计特征且难以实现个体水平区分的局限性。
• 识别对分类性能贡献最大的显著像素级视觉特征。
• 分析单个图像的信息如何在分类过程中非线性地组合，以提升检测准确率。

提出的方法

• 该模型采用两个并行的卷积神经网络流路：一个处理眼动轨迹，另一个处理静态视觉图像。
• 每个流路通过多个卷积和池化层学习分层特征表示，捕捉空间和时间模式。
• 在晚期融合阶段将两个流路的特征进行拼接，以实现联合表征学习。
• 使用交叉熵损失和随机梯度下降进行端到端训练，以优化分类性能。
• 应用基于梯度的显著性方法，识别影响分类决策的关键图像区域。
• 在包含700张图像及其对应ASD与TD参与者眼动模式的数据集上评估该模型。

实验结果

研究问题

• RQ1 双流深度学习架构能否有效整合眼动与视觉图像数据，从而在ASD分类上超越传统方法？
• RQ2 图像中哪些特定视觉特征对模型决策过程影响最大？
• RQ3 来自单个图像的信息如何在两个流路之间非线性组合，以增强分类性能？
• RQ4 该模型在现实场景中对ASD与TD的个体水平区分能力在多大程度上具有泛化性？

主要发现

• 所提出的双流网络实现了95%的分类准确率，超越了先前的最先进性能。
• 该模型通过从原始输入数据中学习复杂的数据驱动表征，在个体水平区分方面表现出色。
• 像素级显著性分析揭示了特定图像区域（尤其是面部和社交情境区域）对分类结果具有高度影响。
• 与早期融合或无融合相比，双流特征的非线性融合显著提升了分类性能。
• 该模型成功捕捉了传统统计分析无法检测到的ASD中细微的非典型视觉注意力模式。
• 眼动动态与静态图像特征的整合，实现了ASD相关视觉注意力异常的鲁棒且可解释的分类。