[论文解读] Meta-Auxiliary Learning for Micro-Expression Recognition

Jingyao Wang, Yunhan Tian+4 more|arXiv (Cornell University)|Apr 18, 2024

Speech Recognition and Synthesis被引用 5

一句话总结

LightmanNet 引入了一个双分支元辅助学习框架，利用辅助的图像对齐任务来引导微表情识别，使在数据有限的情况下实现快速适应并提升泛化。

ABSTRACT

Micro-expressions (MEs) are involuntary movements revealing people's hidden feelings, which has attracted numerous interests for its objectivity in emotion detection. However, despite its wide applications in various scenarios, micro-expression recognition (MER) remains a challenging problem in real life due to three reasons, including (i) data-level: lack of data and imbalanced classes, (ii) feature-level: subtle, rapid changing, and complex features of MEs, and (iii) decision-making-level: impact of individual differences. To address these issues, we propose a dual-branch meta-auxiliary learning method, called LightmanNet, for fast and robust micro-expression recognition. Specifically, LightmanNet learns general MER knowledge from limited data through a dual-branch bi-level optimization process: (i) In the first level, it obtains task-specific MER knowledge by learning in two branches, where the first branch is for learning MER features via primary MER tasks, while the other branch is for guiding the model obtain discriminative features via auxiliary tasks, i.e., image alignment between micro-expressions and macro-expressions since their resemblance in both spatial and temporal behavioral patterns. The two branches of learning jointly constrain the model of learning meaningful task-specific MER knowledge while avoiding learning noise or superficial connections between MEs and emotions that may damage its generalization ability. (ii) In the second level, LightmanNet further refines the learned task-specific knowledge, improving model generalization and efficiency. Extensive experiments on various benchmark datasets demonstrate the superior robustness and efficiency of LightmanNet.

研究动机与目标

• 解决微表情识别（MER）中的数据稀缺和类别不平衡问题。
• 捕捉微妙、快速的 ME 特征，同时避免噪声和表层相关性。
• 利用辅助任务引导学习，使 MER 表征更加判别性。
• 引入双层优化框架，以提取任务特定的和通用的 MER 知识。
• 在多个 MER 基准上展示鲁棒性和效率。

提出的方法

• 双分支架构，共享一个 2D CNN 编码器并配备一个 3D 时序编码模块。
• 主分支以交叉熵损失进行微表情识别训练。
• 辅助分支通过最大均值差异（MMD）量化的微表情与宏表情之间的图像对齐。
• 辅助任务损失将特征学习引导至具有判别性且几何一致的表征。
• 双层优化：第一层通过两种损失优化任务特定的 MER 知识；第二层在任务之间蒸馏知识以获得通用 MER 知识。

实验结果

研究问题

• RQ1 在数据有限的情况下，辅助的图像对齐任务是否能提升判别性 MER 特征？
• RQ2 双层元辅助优化是否在跨数据集的 MER 中实现更好的泛化和效率？
• RQ3 与其他元学习和 MER 基线相比，LightmanNet 在小样本 MER 设置中的表现如何？
• RQ4 辅助任务引导和骨干网络选择对 MER 性能有何影响？
• RQ5 所提组件是否提高了对数据稀缺和类别间差异的鲁棒性？

主要发现

方法	SAMM ACC	SAMM F1	SMIC ACC	SMIC F1	CASME ACC	CASME F1	CASME II ACC	CASME II F1	CAS(ME)² ACC	CAS(ME)² F1
LightmanNet	81.83 ± 0.44	82.93 ± 0.35	85.19 ± 0.34	83.83 ± 0.34	87.83 ± 0.60	89.78 ± 0.59	93.48 ± 0.42	90.49 ± 0.47	85.23 ± 0.26	82.22 ± 0.29

• LightmanNet 在多项 MER 基准测试中，在标准和小样本设置下均达到最先进或具竞争力的准确率和 F1。
• 在小样本 MER 中，LightmanNet 的表现优于 MAML、Reptile、ANIL、ProtoNet 以及其他元学习基线，以及若干深度 MER 方法。
• 辅助图像对齐任务（由 MMD 指导）在不损害 MER 性能的前提下提升了学习表示和泛化。
• 双层优化使对未见 MER 任务在数据有限的情况下快速适应，同时保持模型效率。
• 实验结果表明对数据稀缺和类别间变异具有鲁棒性，与多数基线相比训练时间和模型规模更具优势。