[Paper Review] Decoupled Multimodal Distilling for Emotion Recognition

Yong Li, Yuanzhi Wang+1 more|arXiv (Cornell University)|Mar 24, 2023

Emotion and Mood Recognition9 citations

TL;DR

DMD decouples multimodal features into modality-irrelevant and exclusive spaces and uses dual graph distillation units to adaptively transfer knowledge across language, vision, and audio for emotion recognition, achieving state-of-the-art results on MOSI and MOSEI.

ABSTRACT

Human multimodal emotion recognition (MER) aims to perceive human emotions via language, visual and acoustic modalities. Despite the impressive performance of previous MER approaches, the inherent multimodal heterogeneities still haunt and the contribution of different modalities varies significantly. In this work, we mitigate this issue by proposing a decoupled multimodal distillation (DMD) approach that facilitates flexible and adaptive crossmodal knowledge distillation, aiming to enhance the discriminative features of each modality. Specially, the representation of each modality is decoupled into two parts, i.e., modality-irrelevant/-exclusive spaces, in a self-regression manner. DMD utilizes a graph distillation unit (GD-Unit) for each decoupled part so that each GD can be performed in a more specialized and effective manner. A GD-Unit consists of a dynamic graph where each vertice represents a modality and each edge indicates a dynamic knowledge distillation. Such GD paradigm provides a flexible knowledge transfer manner where the distillation weights can be automatically learned, thus enabling diverse crossmodal knowledge transfer patterns. Experimental results show DMD consistently obtains superior performance than state-of-the-art MER methods. Visualization results show the graph edges in DMD exhibit meaningful distributional patterns w.r.t. the modality-irrelevant/-exclusive feature spaces. Codes are released at \url{https://github.com/mdswyz/DMD}.

Motivation & Objective

• Motivate robust multimodal emotion recognition (MER) under strong modality heterogeneity.
• Propose a decoupled feature framework that splits modalities into common (irrelevant) and private (exclusive) components.
• Develop two graph distillation units (HomoGD and HeteroGD) to enable adaptive cross-modal knowledge transfer.
• Leverage self-regression based decoupling with a margin and orthogonality constraints to reinforce disentanglement.
• Demonstrate superior MER performance on public datasets and provide visualization of learned cross-modal interactions.

Proposed method

• Decouple each modality into a common (completely or largely modality-irrelevant) space and a private (modality-exclusive) space via a shared encoder and modality-specific encoders.
• Use self-regression to reconstruct coupled features and enforce decoupling with reconstruction and cycle losses.
• Apply a margin loss to enforce meaningful separation between homogeneous features across emotions and modalities.
• Introduce a graph distillation unit (GD-Unit) to perform adaptive cross-modal knowledge distillation within each decoupled space.
• HomoGD distills knowledge among homogeneous features using a learned distillation graph with dynamic weights.
• HeteroGD uses a multimodal transformer to align heterogeneous features across modalities before distillation, bridging distribution gaps.
• Fuse refined homogeneous and reinforced heterogeneous features for final MER predictions.
• Objective combines a task loss, decoupling loss, and distillation loss: L_total = L_task + λ1 L_dec + λ2 L_dtl.

Figure 1 : (a) illustrates the significant emotion recognition discrepancies using unimodality, adapted from Mult [ 28 ] . (b) shows the conventional cross-modal distillation. (c) shows our proposed decoupled multimodal distillation (DMD) method. DMD consists of two graph distillation (GD) units: ho

Experimental results

Research questions

• RQ1 Can decoupling modal representations into modality-irrelevant and modality-exclusive spaces improve MER performance under cross-modal heterogeneity?
• RQ2 Do dynamic, learned distillation graphs (HomoGD/HeteroGD) provide better cross-modal knowledge transfer than fixed or naive fusion strategies?
• RQ3 How do decoupled features evolve across modalities and emotions, and what patterns do the learned graph edges reveal?
• RQ4 What is the impact of graph distillation and crossmodal attention on unimodal vs. multimodal fusion effectiveness in MER?

Key findings

• DMD achieves superior or competitive accuracy on CMU-MOSI and CMU-MOSEI compared with state-of-the-art methods under both aligned and unaligned settings.
• Decoupling into homogeneous and heterogeneous spaces, together with graph distillation, consistently improves MER performance over baselines.
• HomoGD primarily leverages language as a strong contributor, with L→A and L→V edges dominating in many cases, reflecting modality contributions.
• HeteroGD with crossmodal attention bridges distribution gaps and reveals meaningful inter-modality interactions (e.g., V→A gains).
• Ablation studies show that feature decoupling (FD) plus both GD units yields the best results, and removing CA or one GD unit degrades performance.
• Visualization indicates decoupled spaces improve class separability, with homogeneous space clustering by emotion and heterogeneous space by modality.

Figure 2 : The framework of DMD. Given the input multimodal data, DMD encodes their respective shallow features $\widetilde{\mathbf{X}}_{m}$ , where $m\in\{L,V,A\}$ . In feature decoupling , DMD exploits the decoupled homo-/heterogeneous multimodal features $\mathbf{X}^{\text{com}}_{m}$ / $\mathbf{X