[论文解读] Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study

Dimitrios Kollias, Viktoriia Sharmanska+1 more|arXiv (Cornell University)|May 8, 2021

Emotion and Mood Recognition参考文献 61被引用 99

一句话总结

本文提出 FaceBehaviorNet，一种面向大规模人脸分析的整体性、异质性多任务学习框架，利用分布匹配和共注释，在 10 个真实场景数据集中联合学习面部情绪、动作单元、价态-唤醒值，以及人脸身份与属性，减少负迁移。

ABSTRACT

Multi-Task Learning has emerged as a methodology in which multiple tasks are jointly learned by a shared learning algorithm, such as a DNN. MTL is based on the assumption that the tasks under consideration are related; therefore it exploits shared knowledge for improving performance on each individual task. Tasks are generally considered to be homogeneous, i.e., to refer to the same type of problem. Moreover, MTL is usually based on ground truth annotations with full, or partial overlap across tasks. In this work, we deal with heterogeneous MTL, simultaneously addressing detection, classification & regression problems. We explore task-relatedness as a means for co-training, in a weakly-supervised way, tasks that contain little, or even non-overlapping annotations. Task-relatedness is introduced in MTL, either explicitly through prior expert knowledge, or through data-driven studies. We propose a novel distribution matching approach, in which knowledge exchange is enabled between tasks, via matching of their predictions' distributions. Based on this approach, we build FaceBehaviorNet, the first framework for large-scale face analysis, by jointly learning all facial behavior tasks. We develop case studies for: i) continuous affect estimation, action unit detection, basic emotion recognition; ii) attribute detection, face identification. We illustrate that co-training via task relatedness alleviates negative transfer. Since FaceBehaviorNet learns features that encapsulate all aspects of facial behavior, we conduct zero-/few-shot learning to perform tasks beyond the ones that it has been trained for, such as compound emotion recognition. By conducting a very large experimental study, utilizing 10 databases, we illustrate that our approach outperforms, by large margins, the state-of-the-art in all tasks and in all databases, even in these which have not been used in its training.

研究动机与目标

• 在检测、分类和回归任务中，激发并解决面部行为分析的异质多任务学习问题。
• 开发一种基于分布匹配的耦合机制，以在注释不完整或不重叠的任务之间实现知识交流。
• 提出共注释和分布匹配损失，以缓解负迁移。
• 将 FaceBehaviorNet 打造成首个面向大规模人脸分析的整体性框架。
• 展示跨数据库的强大性能，以及零样本/少样本泛化能力。

提出的方法

• 将异质多任务学习形式化为任务 T_i 与分布 D_i 的组，目标是在任务间最小化平均期望损失。
• 在训练中引入任务相关性（领域知识或来自数据集的经验）以耦合任务。
• 提出共注释，在跨任务有注释时约束相关任务标签。
• 提出分布匹配（蒸馏式）损失 L_DM，通过对情绪的混合分布 q(y_au|x) 来对齐任务预测。
• 定义软共注释变体和软目标（L_SCA），在注释不完整时加强耦合。
• 将该方法扩展到第二个案例研究，通过分布匹配将身份与 40 个属性结合起来。
• 通过利用学习到的面部行为特征展示零样本与少样本的综合表情识别。

实验结果

研究问题

• RQ1 如何将异质任务（分类、检测、回归）进行联合学习，以提升面部分析不同领域的性能？
• RQ2 是否能通过领域知识或经验数据集注释有效编码任务相关性，以实现知识迁移？
• RQ3 基于分布匹配的耦合是否能缓解面部分析多任务学习中的负迁移？
• RQ4 在大型野外数据集上，单一整体模型在情感计算和人脸识别任务上的表现如何？
• RQ5 所学习的特征是否能支持零样本和少样本的复合表情识别？

主要发现

• FaceBehaviorNet 在所有任务和所研究的全部 10 个数据库上均优于单任务网络。
• 基于分布匹配的跨异质任务知识蒸馏成功降低了负迁移。
• 该框架利用学习到的整体表征，支持复合情绪识别的零样本和少样本学习。
• 通过共注释和/或分布匹配进行的任务耦合，即使在训练中未见过的数据库上也提高了性能。
• 该方法在情感计算任务（情绪、AU、价态-唤醒）和人脸识别属性（身份、属性）方面达到最先进水平。