[Paper Review] Quantifying and Enhancing Multi-modal Robustness with Modality Preference

Zequn Yang, Yake Wei+2 more|arXiv (Cornell University)|Feb 9, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes Certifiable Robust Multi-modal Training (CRMT), a novel training procedure that enhances multi-modal model robustness by addressing modality preference—where models overly rely on one modality—through orthogonal-based regularization and explicit control of uni-modal margins and integration. CRMT achieves significantly improved certified robustness against uni-modal and multi-modal attacks, outperforming existing methods across benchmarks like Kinetics Sounds and VGGS.

ABSTRACT

Multi-modal models have shown a promising capability to effectively integrate information from various sources, yet meanwhile, they are found vulnerable to pervasive perturbations, such as uni-modal attacks and missing conditions. To counter these perturbations, robust multi-modal representations are highly expected, which are positioned well away from the discriminative multi-modal decision boundary. In this paper, different from conventional empirical studies, we focus on a commonly used joint multi-modal framework and theoretically discover that larger uni-modal representation margins and more reliable integration for modalities are essential components for achieving higher robustness. This discovery can further explain the limitation of multi-modal robustness and the phenomenon that multi-modal models are often vulnerable to attacks on the specific modality. Moreover, our analysis reveals how the widespread issue, that the model has different preferences for modalities, limits the multi-modal robustness by influencing the essential components and could lead to attacks on the specific modality highly effective. Inspired by our theoretical finding, we introduce a training procedure called Certifiable Robust Multi-modal Training (CRMT), which can alleviate this influence from modality preference and explicitly regulate essential components to significantly improve robustness in a certifiable manner. Our method demonstrates substantial improvements in performance and robustness compared with existing methods. Furthermore, our training procedure can be easily extended to enhance other robust training strategies, highlighting its credibility and flexibility.

Motivation & Objective

• To identify the fundamental components enabling robust multi-modal learning, particularly in the face of uni-modal and multi-modal perturbations.
• To investigate why multi-modal models are often more vulnerable to attacks on specific modalities, especially when modality preference exists.
• To develop a training procedure that explicitly regulates uni-modal representation margins and modality integration to improve certified robustness.
• To validate the method's effectiveness and scalability across different architectures, including Transformers.
• To provide a theoretically grounded, certifiable approach to multi-modal robustness that addresses the imbalance caused by modality preference.

Proposed method

• Proposes an orthogonal-based framework to decouple the interdependence between uni-modal margins and modality integration, enabling independent optimization.
• Introduces a two-step training procedure: (1) modality preference regularization via orthogonal projection to balance modality contributions, and (2) explicit margin maximization for each modality.
• Employs a lower-bound analysis of the perturbation radius that the model can defend against, derived from uni-modal margins and integration weights.
• Uses the vulnerability indicator η^(m) to quantify and monitor modality imbalance, guiding the training to reduce disparity between modalities.
• Applies the method in a plug-in fashion to existing robust training strategies, enabling extension to various architectures including Transformers.
• Employs FGM and ℓ₂-PGD attacks for evaluation, with robustness measured via accuracy under perturbations on individual and combined modalities.

Experimental results

Research questions

• RQ1 What are the essential theoretical components that determine multi-modal robustness, and how do they interact?
• RQ2 Why are multi-modal models more vulnerable to attacks on specific modalities, and how does modality preference exacerbate this?
• RQ3 How does modality preference influence the balance of uni-modal margins and integration weights, and what is its impact on robustness?
• RQ4 Can a training procedure be designed to explicitly regulate margins and integration in a way that guarantees certified robustness?
• RQ5 To what extent can CRMT be extended to and improve robustness in modern architectures like Transformers?

Key findings

• Larger uni-modal representation margins and more balanced integration are essential for achieving higher multi-modal robustness, as shown by theoretical lower-bound analysis.
• Modality preference leads to imbalanced uni-modal vulnerability indicators, making attacks on the preferred modality significantly more effective, as evidenced by higher accuracy drops on modality #a in Figure 1.
• CRMT-based methods reduce the imbalance in vulnerability indicators (η^(v)/η^(a)) and consistently improve robustness across different attack types and sizes, as shown in Figure 4.
• On the Kinetics Sounds dataset, CRMT outperforms baseline Joint Training and three robust training methods under ℓ₂-PGD attacks, especially on the more vulnerable modality #a.
• When extended to the MMT Transformer-based model on VGGS, CRMT improves adversarial accuracy under both uni-modal and multi-modal attacks, demonstrating scalability.
• Abllation studies confirm that both steps of CRMT (preference regularization and margin control) contribute independently to robustness, with the full CRMT-JT method achieving the best performance.