[论文解读] Quantifying and Enhancing Multi-modal Robustness with Modality Preference

Zequn Yang, Yake Wei+2 more|arXiv (Cornell University)|Feb 9, 2024

Natural Language Processing Techniques被引用 4

一句话总结

本文提出了一种新型训练方法——可认证鲁棒多模态训练（Certifiable Robust Multi-modal Training, CRMT），通过正交化正则化与显式控制单模态边界和模态融合，缓解模型对某一模态的偏好问题，从而提升多模态模型的鲁棒性。CRMT在Kinetics Sounds和VGGS等基准数据集上，显著提升了对单模态和多模态攻击的可认证鲁棒性，优于现有方法。

ABSTRACT

Multi-modal models have shown a promising capability to effectively integrate information from various sources, yet meanwhile, they are found vulnerable to pervasive perturbations, such as uni-modal attacks and missing conditions. To counter these perturbations, robust multi-modal representations are highly expected, which are positioned well away from the discriminative multi-modal decision boundary. In this paper, different from conventional empirical studies, we focus on a commonly used joint multi-modal framework and theoretically discover that larger uni-modal representation margins and more reliable integration for modalities are essential components for achieving higher robustness. This discovery can further explain the limitation of multi-modal robustness and the phenomenon that multi-modal models are often vulnerable to attacks on the specific modality. Moreover, our analysis reveals how the widespread issue, that the model has different preferences for modalities, limits the multi-modal robustness by influencing the essential components and could lead to attacks on the specific modality highly effective. Inspired by our theoretical finding, we introduce a training procedure called Certifiable Robust Multi-modal Training (CRMT), which can alleviate this influence from modality preference and explicitly regulate essential components to significantly improve robustness in a certifiable manner. Our method demonstrates substantial improvements in performance and robustness compared with existing methods. Furthermore, our training procedure can be easily extended to enhance other robust training strategies, highlighting its credibility and flexibility.

研究动机与目标

• 识别实现鲁棒多模态学习的基本组件，特别是在面对单模态和多模态扰动时。
• 探究为何多模态模型在存在模态偏好时，往往对特定模态的攻击更加脆弱。
• 开发一种训练流程，显式调节单模态表征边界与模态融合，以提升可认证鲁棒性。
• 在不同架构（包括Transformer）上验证该方法的有效性与可扩展性。
• 提供一种理论基础扎实、可认证的多模态鲁棒性方法，以解决模态偏好导致的不平衡问题。

提出的方法

• 提出一种基于正交性的框架，解耦单模态边界与模态融合之间的相互依赖，实现独立优化。
• 引入两步训练流程：(1) 通过正交投影进行模态偏好正则化，以平衡各模态的贡献；(2) 显式最大化每种模态的边界。
• 采用扰动半径的下界分析，该下界由单模态边界与融合权重推导得出。
• 使用脆弱性指标 η^(m) 量化并监控模态失衡，指导训练过程以减少模态间的差异。
• 以即插即用方式应用于现有鲁棒训练策略，可扩展至包括Transformer在内的多种架构。
• 采用FGM和ℓ₂-PGD攻击进行评估，通过在单个模态及组合模态上施加扰动后的准确率衡量鲁棒性。

实验结果

研究问题

• RQ1 决定多模态鲁棒性的基本理论组件是什么？它们之间如何相互作用？
• RQ2 为何多模态模型对特定模态的攻击更脆弱？模态偏好如何加剧这一问题？
• RQ3 模态偏好如何影响单模态边界的平衡与融合权重的分配？其对鲁棒性有何影响？
• RQ4 能否设计一种训练流程，显式调节边界与融合方式，以保证可认证鲁棒性？
• RQ5 CRMT在多大程度上可扩展至并提升现代架构（如Transformer）的鲁棒性？

主要发现

• 更大的单模态表征边界与更均衡的融合是实现更高多模态鲁棒性的关键，这一结论由理论下界分析支持。
• 模态偏好导致单模态脆弱性指标失衡，使得对偏好模态的攻击更为有效，如图1所示，模态#a的准确率下降更显著。
• 基于CRMT的方法降低了脆弱性指标的不平衡性（η^(v)/η^(a)），并在不同攻击类型与规模下持续提升鲁棒性，如图4所示。
• 在Kinetics Sounds数据集上，CRMT在ℓ₂-PGD攻击下优于基线联合训练（Joint Training）及三种鲁棒训练方法，尤其在更易受攻击的模态#a上表现更优。
• 当扩展至VGGS上的MMT Transformer模型时，CRMT在单模态与多模态攻击下均提升了对抗性准确率，证明了其可扩展性。
• 消融实验表明，CRMT的两个步骤（偏好正则化与边界控制）对鲁棒性均有独立贡献，完整CRMT-JT方法达到最佳性能。