[论文解读] Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition

Xiangbo Shu, Jiawen Yang+2 more|arXiv (Cornell University)|Dec 21, 2021

Human Pose and Action Recognition被引用 7

一句话总结

本文提出了一种扩展-挤压-激励融合网络（ESE-FN），通过学习模态和通道级注意力机制来融合RGB与骨骼特征，从而提升老年人活动识别性能。该方法在大规模ETRI-Activity3D数据集上实现了96.7%的最先进准确率，通过新颖的多模态损失函数和双注意力融合策略优于现有方法。

ABSTRACT

This work focuses on the task of elderly activity recognition, which is a challenging task due to the existence of individual actions and human-object interactions in elderly activities. Thus, we attempt to effectively aggregate the discriminative information of actions and interactions from both RGB videos and skeleton sequences by attentively fusing multi-modal features. Recently, some nonlinear multi-modal fusion approaches are proposed by utilizing nonlinear attention mechanism that is extended from Squeeze-and-Excitation Networks (SENet). Inspired by this, we propose a novel Expansion-Squeeze-Excitation Fusion Network (ESE-FN) to effectively address the problem of elderly activity recognition, which learns modal and channel-wise Expansion-Squeeze-Excitation (ESE) attentions for attentively fusing the multi-modal features in the modal and channel-wise ways. Furthermore, we design a new Multi-modal Loss (ML) to keep the consistency between the single-modal features and the fused multi-modal features by adding the penalty of difference between the minimum prediction losses on single modalities and the prediction loss on the fused modality. Finally, we conduct experiments on a largest-scale elderly activity dataset, i.e., ETRI-Activity3D (including 110,000+ videos, and 50+ categories), to demonstrate that the proposed ESE-FN achieves the best accuracy compared with the state-of-the-art methods. In addition, more extensive experimental results show that the proposed ESE-FN is also comparable to the other methods in terms of normal action recognition task.

研究动机与目标

• 为解决识别涉及人机交互和低幅运动的细微、相似老年人活动的挑战。
• 通过捕捉跨模态的判别性时空模式，提升RGB视频与骨骼序列之间的特征融合效果。
• 设计一种可学习的注意力机制，以建模模态间与通道间依赖关系，增强特征表示能力。
• 通过新颖的多模态损失函数，确保单模态与融合多模态预测之间的一致性。
• 通过在标准正常动作识别基准（NTU RGB+D）上评估性能，验证模型的泛化能力。

提出的方法

• 提出双注意力机制：模态级ESE注意力（M-ESEA），用于融合RGB与骨骼模态的特征。
• 引入通道级ESE注意力（C-ESEA），通过建模融合特征中的通道依赖关系，增强判别能力。
• 采用两阶段融合流程：首先通过M-Net实现模态级融合，再通过C-Net实现通道级融合。
• 设计多模态损失（ML），通过惩罚单模态预测损失最小值与融合模态预测损失之间的差异，以强制保持一致性。
• 使用双分支主干网络分别独立提取RGB与骨骼特征，再进行融合。
• 应用扩展与挤压操作，以建模局部与全局特征交互，扩展SENet家族的上采样与下采样行为。

Figure 1: Idea of our method for fusing multi-modal features in this work. Our method fuses multi-modal features with the modal and channel-wise Expansion-Squeeze-Excitation (ESE) attentions in the modal and channel-wise ways.

实验结果

研究问题

• RQ1 一种在模态与通道维度上同时运作的双注意力融合机制，能否提升老年人活动识别性能？
• RQ2 所提出的多模态损失在保持单模态与融合预测一致性方面效果如何？
• RQ3 ESE-FN是否能泛化到老年人特定活动之外的正常动作识别任务？
• RQ4 与标准注意力机制相比，扩展与挤压操作在多大程度上增强了特征表示？
• RQ5 该模型在高度具有挑战性、视觉上相似的老年人活动（如“吹头发”与“梳头”）上的表现如何？

主要发现

• ESE-FN在ETRI-Activity3D数据集上实现了96.7%的最先进准确率，优于现有方法。
• 在NTU RGB+D数据集上，分别在Cross-Subject与Cross-View协议下，较SOTA方法提升0.7%与0.2%。
• NTU RGB+D上的混淆矩阵显示优异的类别级识别性能，主对角线始终较浅，表明所有60个动作类别均具有高识别准确率。
• 消融实验证实，M-ESEA与C-ESEA组件均对性能提升有显著贡献，验证了双注意力融合的有效性。
• 所提出的多模态损失有效稳定了训练过程，并通过强制单模态与融合预测之间的一致性，提升了泛化能力。
• 模型在细粒度、相似的老年人活动（如“打电话”与“玩手机”）上表现出鲁棒性，其中仅手部动作或人机交互存在细微差异。

Figure 2: Examples of elderly activities. The elderly activities, “Blowing hair” vs “Combing hair”, as well as “Making call” vs “Playing mobile” shows most of similar motions in spatiotemporal space, where only the interacting objects (e.g., comb and blower are marked by yellow circle) are different