[论文解读] Boosting Multi-modal Model Performance with Adaptive Gradient Modulation

Hong Li, Xingyu Li+4 more|arXiv (Cornell University)|Aug 15, 2023

Domain Adaptation and Few-Shot LearningComputer Science被引用 3

一句话总结

本文提出自适应梯度调制（AGM），一种新颖的方法，通过基于Shapley值的归因动态调整多模态模型中的梯度更新，以减少模态间竞争并提升各类融合策略下的性能。实验表明，AGM优于现有方法，并定量揭示调制通过降低竞争强度增强了模型对更富信息量模态的依赖。

ABSTRACT

While the field of multi-modal learning keeps growing fast, the deficiency of the standard joint training paradigm has become clear through recent studies. They attribute the sub-optimal performance of the jointly trained model to the modality competition phenomenon. Existing works attempt to improve the jointly trained model by modulating the training process. Despite their effectiveness, those methods can only apply to late fusion models. More importantly, the mechanism of the modality competition remains unexplored. In this paper, we first propose an adaptive gradient modulation method that can boost the performance of multi-modal models with various fusion strategies. Extensive experiments show that our method surpasses all existing modulation methods. Furthermore, to have a quantitative understanding of the modality competition and the mechanism behind the effectiveness of our modulation method, we introduce a novel metric to measure the competition strength. This metric is built on the mono-modal concept, a function that is designed to represent the competition-less state of a modality. Through systematic investigation, our results confirm the intuition that the modulation encourages the model to rely on the more informative modality. In addition, we find that the jointly trained model typically has a preferred modality on which the competition is weaker than other modalities. However, this preferred modality need not dominate others. Our code will be available at https://github.com/lihong2303/AGM_ICCV2023.

研究动机与目标

• 解决联合训练的多模态模型因模态竞争导致的次优性能问题。
• 开发一种适用于多种融合策略的梯度调制方法，而不仅限于晚期融合。
• 基于单模态概念提出一种新度量方法，以定量衡量模态竞争强度。
• 理解梯度调制在减少模态竞争并提升模型效率方面的内在机制。

提出的方法

• 该方法使用基于Shapley值的模块，从完整输入和静音输入中计算单模态输出，从而隔离每个模态的贡献。
• 计算单模态交叉熵损失（$s^m$），以量化每个模态的信息含量。
• 利用这些损失的运行平均值（$\hat{s}^m$）计算差异比率，生成调制系数（$\kappa^m$）。
• 这些系数在反向传播过程中自适应地缩放梯度信号，从而降低对信息量较少模态的竞争。
• 该方法与模型架构无关，适用于任何融合策略，包括早期、晚期和混合融合。
• 基于单模态概念构建了一种新型竞争强度度量，用于表示无竞争状态。

Figure 1: Schematic diagram of the adaptive gradient modulation (AGM) method. Firstly, based on the full input and corresponding muted inputs, the Shapley module produces mono-modal outputs $\phi^{m}$ , which disentangle the responses of the multi-modal model to individual modalities. Next, $\phi^{m

实验结果

研究问题

• RQ1 能否设计一种适用于多种融合策略（而不仅限于晚期融合）的梯度调制方法？
• RQ2 联合训练的多模态模型中模态竞争的内在机制是什么？
• RQ3 自适应梯度调制如何影响模型对更富信息量模态的依赖？
• RQ4 是否存在一种定量方法来衡量多模态学习中的模态竞争强度？
• RQ5 模型偏好的模态是否与信息含量或任务相关性相关？

主要发现

• AGM在多个数据集（包括CMU-MOSEI、AVE、CREMA-D和UR-Funny）上持续优于现有调制方法。
• 在CMU-MOSEI上，文本模态的竞争强度（$d^t$）在调制后下降，表明更富信息量的模态得到更优利用。
• 在CREMA-D上，视觉模态的$d^v$下降而$d^a$上升，表明调制增强了对更富信息量视觉模态的依赖。
• 在AVE上，音频模态的竞争强度在调制后下降，证实模型更倾向于依赖更富信息量的模态。
• 在UR-Funny上，调制平衡了音频与文本模态的信息使用，尽管文本模态的$d^t$上升，表明多模态组合中存在复杂交互。
• 原始联合训练中模型偏好的模态不一定是信息量最多的模态，而调制有效打破了这种偏差。

Figure 2: Accuracy ( $Acc$ , $Acc_{a}$ , $Acc_{v}$ ) and competition strength ( $d^{a}$ , $d^{v}$ ) of joint-training multimodal model and multimodal model with AGM using addition fusion method on the validation set of the AV-MNIST dataset. The left is the joint-training multimodal model and the rig