[Paper Review] Predictive Dynamic Fusion

Bing Cao, Yinan Xia+3 more|arXiv (Cornell University)|Jun 7, 2024

Reservoir Engineering and Simulation Methods4 citations

TL;DR

This paper proposes Predictive Dynamic Fusion (PDF), a theoretically grounded multimodal learning framework that reduces generalization error by predicting Collaborative Belief (Co-Belief) using Mono- and Holo-Confidence, which encode intra- and inter-modal loss covariances. The method achieves state-of-the-art performance across multiple benchmarks under noisy and imbalanced conditions, with significant improvements in stability and reliability.

ABSTRACT

Multimodal fusion is crucial in joint decision-making systems for rendering holistic judgments. Since multimodal data changes in open environments, dynamic fusion has emerged and achieved remarkable progress in numerous applications. However, most existing dynamic multimodal fusion methods lack theoretical guarantees and easily fall into suboptimal problems, yielding unreliability and instability. To address this issue, we propose a Predictive Dynamic Fusion (PDF) framework for multimodal learning. We proceed to reveal the multimodal fusion from a generalization perspective and theoretically derive the predictable Collaborative Belief (Co-Belief) with Mono- and Holo-Confidence, which provably reduces the upper bound of generalization error. Accordingly, we further propose a relative calibration strategy to calibrate the predicted Co-Belief for potential uncertainty. Extensive experiments on multiple benchmarks confirm our superiority. Our code is available at https://github.com/Yinan-Xia/PDF.

Motivation & Objective

• To address the lack of theoretical guarantees in existing dynamic multimodal fusion methods, which often lead to suboptimal and unstable performance.
• To improve fusion reliability by modeling the dynamic, environment-dependent reliability of modalities through a generalization error-bound perspective.
• To reduce the upper bound of generalization error by leveraging the negative covariance between fusion weights and modality-specific losses, and the positive covariance with other modalities’ losses.
• To handle prediction uncertainty in dynamic fusion by introducing a relative calibration strategy that adapts to changing modality dominance.
• To develop a robust, stable, and theoretically justified fusion mechanism that outperforms existing methods under noisy and imbalanced data settings.

Proposed method

• The framework derives from the upper bound of generalization error, identifying that optimal fusion requires negative covariance between fusion weight and its own modality loss, and positive covariance with other modalities’ losses.
• It introduces Collaborative Belief (Co-Belief) as a proxy for loss prediction, which is more stable than direct loss estimation and naturally satisfies the required covariance relationships.
• Mono-Confidence captures intra-modal reliability by modeling the negative correlation between fusion weight and its own modality’s loss.
• Holo-Confidence models inter-modal cooperation by encoding the positive correlation between a modality’s fusion weight and the losses of other modalities.
• A relative calibration strategy is proposed to adjust predicted Co-Belief based on the dynamic dominance of modalities, improving robustness to uncertainty.
• The method is implemented without additional computational cost and is validated across multiple multimodal benchmarks under varying noise levels.

Experimental results

Research questions

• RQ1 Can a theoretically grounded dynamic fusion framework reduce the upper bound of generalization error in multimodal learning?
• RQ2 How can fusion weights be dynamically adjusted based on both intra- and inter-modal loss relationships to improve stability and reliability?
• RQ3 What is the impact of modality-specific uncertainty and changing data quality on fusion performance in open environments?
• RQ4 Can a relative calibration strategy effectively handle prediction uncertainty in dynamic multimodal fusion?
• RQ5 Does the proposed Predictive Dynamic Fusion framework outperform existing state-of-the-art methods under noisy and imbalanced data conditions?

Key findings

• On the MVSA dataset, PDF achieved 79.94% ± 0.95 accuracy under clean conditions, outperforming the next best method (92.59% ± 0.07 for dynMM*) in the same setting.
• Under 10% salt-and-pepper noise, PDF achieved 61.97% ± 1.14 accuracy on MVSA, significantly outperforming QMF (61.60% ± 0.20) and TMC (60.22% ± 0.43).
• On NYU Depth V2 with 10% noise, PDF achieved 53.62% ± 2.15 accuracy, surpassing QMF (45.02% ± 2.28) and dynMM* (42.49% ± 0.43).
• On CREMA-D with 10% noise, PDF achieved 48.40% ± 2.85 accuracy, outperforming QMF (60.41% ± 2.63) and TMC (56.62% ± 3.67) under the same noise level.
• On UMPC FOOD101, PDF achieved 61.76% ± 0.33 accuracy under 10% noise, significantly outperforming QMF (51.87% ± 0.91) and dynMM* (38.17% ± 1.17).
• The proposed relative calibration strategy effectively improved robustness, particularly in high-noise scenarios, demonstrating the method’s adaptability to dynamic environmental changes.