[Paper Review] Provable Dynamic Fusion for Low-Quality Multimodal Data

Qingyang Zhang, Haitao Wu+5 more|arXiv (Cornell University)|Jun 3, 2023

Remote-Sensing Image Classification20 citations

TL;DR

The paper provides a theoretical framework for robust dynamic multimodal fusion and proposes Quality-aware Multimodal Fusion (QMF) that uses uncertainty estimates to weight modalities, with theoretical generalization guarantees and extensive experiments showing improved performance on low-quality data.

ABSTRACT

The inherent challenge of multimodal fusion is to precisely capture the cross-modal correlation and flexibly conduct cross-modal interaction. To fully release the value of each modality and mitigate the influence of low-quality multimodal data, dynamic multimodal fusion emerges as a promising learning paradigm. Despite its widespread use, theoretical justifications in this field are still notably lacking. Can we design a provably robust multimodal fusion method? This paper provides theoretical understandings to answer this question under a most popular multimodal fusion framework from the generalization perspective. We proceed to reveal that several uncertainty estimation solutions are naturally available to achieve robust multimodal fusion. Then a novel multimodal fusion framework termed Quality-aware Multimodal Fusion (QMF) is proposed, which can improve the performance in terms of classification accuracy and model robustness. Extensive experimental results on multiple benchmarks can support our findings.

Motivation & Objective

• Motivate robust multimodal fusion under varying modality quality and data noise.
• Characterize generalization error bounds for dynamic versus static fusion using Rademacher complexity.
• Propose a Quality-aware Multimodal Fusion (QMF) framework that leverages uncertainty estimates to weight modalities.
• Demonstrate that dynamic fusion can provably outperform static fusion under specific conditions.
• Provide practical training methods and regularization to align fusion weights with modality uncertainty.

Proposed method

• Formalize late fusion with dynamic weights w^m(x) and derive a generalization bound for multimodal fusion (Theorem 1).
• Show conditions under which dynamic fusion outperforms static fusion (Theorem 2).
• Propose Quality-aware Multimodal Fusion (QMF) where fusion weights depend on modality uncertainty and satisfy Correlation Conditions (Eq. 6).
• Define uncertainty-based weights w^m(x) = α^m u^m(x) + β^m with negative α^m to couple weights to uncertainty (Eq. 9).
• Use energy score as the uncertainty cue for each modality and introduce sampling-based regularization to improve correlation between uncertainty and loss (Equations 12-18).
• Provide Algorithm 1 for training QMF and describe the overall loss including a regularization term (Eq. 18).

Experimental results

Research questions

• RQ1 When does dynamic multimodal fusion provably outperform static fusion from a generalization perspective?
• RQ2 How can uncertainty estimation be integrated into fusion weights to achieve provable robustness in multimodal learning?
• RQ3 What is the relationship between dynamic fusion performance and unimodal generalization errors and uncertainties?
• RQ4 Can a practical framework (QMF) realize the theoretical advantages with reliable uncertainty-based weighting?
• RQ5 Does energy-score based uncertainty estimation, with proposed regularization, yield better correlation between uncertainty and loss for fusion weighting?

Key findings

• Dynamic fusion offers a provable generalization advantage when fusion weights negatively correlate with unimodal generalization errors (Cov(w^m, l^m) ≤ 0).
• The dynamic fusion generalization bound comprises average empirical loss, average complexity, covariance between weights and losses, and a sampling-based term, linking robustness to uncertainty alignment (Theorem 1).
• Under certain conditions, dynamic fusion upper bounds are no worse than static fusion, and can be strictly better if Pearson correlation between weights and unimodal losses is nonpositive (Theorem 2).
• The generalization ability of dynamic fusion coincides with the performance of uncertainty estimation (uncertainty quality directly informs fusion weighting).
• QMF is instantiated with uncertainty-aware weights derived from modality uncertainties, using energy score-based uncertainty and a regularization term to align weights with training difficulty (loss trajectory).
• Empirical results on benchmarks show that QMF maintains or improves accuracy on low-quality data and often outperforms unimodal baselines when data is noisy or modalities are imbalanced (as illustrated in Table 1 and Figure 3).