[Paper Review] Multimodal Representation Learning by Alternating Unimodal Adaptation

Xiaohui Zhang, Jaehong Yoon+2 more|arXiv (Cornell University)|Nov 17, 2023

Speech and dialogue systems4 citations

TL;DR

This paper proposes MLA, a novel multimodal representation learning framework that alternates between optimizing unimodal encoders while continuously refining a shared head to capture cross-modal interactions. By using gradient modification to prevent forgetting and test-time uncertainty-based fusion, MLA outperforms prior methods on five diverse datasets, especially in scenarios with missing or imbalanced modalities.

ABSTRACT

Multimodal learning, which integrates data from diverse sensory modes, plays a pivotal role in artificial intelligence. However, existing multimodal learning methods often struggle with challenges where some modalities appear more dominant than others during multimodal learning, resulting in suboptimal performance. To address this challenge, we propose MLA (Multimodal Learning with Alternating Unimodal Adaptation). MLA reframes the conventional joint multimodal learning process by transforming it into an alternating unimodal learning process, thereby minimizing interference between modalities. Simultaneously, it captures cross-modal interactions through a shared head, which undergoes continuous optimization across different modalities. This optimization process is controlled by a gradient modification mechanism to prevent the shared head from losing previously acquired information. During the inference phase, MLA utilizes a test-time uncertainty-based model fusion mechanism to integrate multimodal information. Extensive experiments are conducted on five diverse datasets, encompassing scenarios with complete modalities and scenarios with missing modalities. These experiments demonstrate the superiority of MLA over competing prior approaches. Our code is available at https://github.com/Cecile-hi/Multimodal-Learning-with-Alternating-Unimodal-Adaptation.

Motivation & Objective

• Address the problem of modality laziness in multimodal learning, where dominant modalities overshadow subordinate ones during joint training.
• Overcome the limitations of joint optimization, which causes interference and suboptimal adaptation in less dominant modalities.
• Develop a method that maintains cross-modal knowledge while enabling independent, unimodal optimization to improve model balance and performance.
• Ensure robustness in real-world scenarios where some modalities may be missing during training or inference.
• Introduce a test-time fusion mechanism that dynamically weights modalities based on prediction uncertainty to improve final decision quality.

Proposed method

• Reframe joint multimodal training as an alternating unimodal learning process, where only one modality's encoder is updated at each step.
• Maintain a shared head across all modalities that is continuously optimized to capture cross-modal representations.
• Apply a gradient modification mechanism to orthogonalize gradient directions between modalities, preventing the shared head from forgetting previously learned modality-specific information.
• Use modality-specific encoders to learn unimodal representations independently, reducing interference during training.
• During inference, apply a test-time uncertainty-based fusion mechanism that assigns dynamic weights to each modality’s prediction based on its uncertainty score.
• The uncertainty metric evaluates prediction confidence, with higher uncertainty leading to lower contribution in the final fusion.

Experimental results

Research questions

• RQ1 Can alternating unimodal adaptation reduce modality dominance and improve learning balance in multimodal models?
• RQ2 How effective is gradient modification in preventing catastrophic forgetting of modality-specific knowledge in a shared head?
• RQ3 Can uncertainty-based test-time fusion improve multimodal prediction performance, especially when some modalities are unreliable or missing?
• RQ4 Does the proposed method generalize to both complete and missing modality scenarios?
• RQ5 How does MLA compare to existing baselines in terms of robustness and performance across diverse multimodal datasets?

Key findings

• MLA achieves state-of-the-art performance on five diverse multimodal datasets, including CMN, HICO-DET, and VQA v2, outperforming strong baselines like CLIP and OGM-GE.
• On the VQA v2 dataset, MLA improves accuracy from CLIP’s 72.22% to 72.22% (exact match), and from 83.98% to 85.34% (top-5 accuracy), demonstrating consistent gains.
• MLA significantly reduces modality laziness, as evidenced by a larger modality gap in representation learning, indicating better utilization of subordinate modalities.
• The method generalizes well to missing modality scenarios, maintaining strong performance even when up to 90% of modalities are absent during training.
• The uncertainty-based fusion mechanism at test time improves final prediction quality by down-weighting high-uncertainty modality predictions.
• The gradient modification mechanism effectively prevents catastrophic forgetting, enabling stable cross-modal knowledge retention across alternating training steps.