[Paper Review] A Theory of Multimodal Learning

Lu Zhou|arXiv (Cornell University)|Sep 21, 2023

Neural Networks and Applications4 citations

TL;DR

This paper proposes a theoretical framework explaining why multimodal learning outperforms unimodal learning in generalization, even on unimodal tasks. By decoupling the learning of a modality connection and a predictor, it proves a generalization bound improvement of up to $O(\sqrt{n})$ over unimodal methods when both connection and heterogeneity exist between modalities.

ABSTRACT

Human perception of the empirical world involves recognizing the diverse appearances, or 'modalities', of underlying objects. Despite the longstanding consideration of this perspective in philosophy and cognitive science, the study of multimodality remains relatively under-explored within the field of machine learning. Nevertheless, current studies of multimodal machine learning are limited to empirical practices, lacking theoretical foundations beyond heuristic arguments. An intriguing finding from the practice of multimodal learning is that a model trained on multiple modalities can outperform a finely-tuned unimodal model, even on unimodal tasks. This paper provides a theoretical framework that explains this phenomenon, by studying generalization properties of multimodal learning algorithms. We demonstrate that multimodal learning allows for a superior generalization bound compared to unimodal learning, up to a factor of $O(\sqrt{n})$, where $n$ represents the sample size. Such advantage occurs when both connection and heterogeneity exist between the modalities.

Motivation & Objective

• To provide a theoretical foundation for the empirical success of multimodal learning, which currently lacks rigorous justification beyond heuristics.
• To explain why multimodal models can outperform finely-tuned unimodal models even on unimodal tasks.
• To formalize the roles of 'connection' and 'heterogeneity' between modalities in enabling improved generalization.
• To establish provable upper and lower bounds on generalization error for a two-stage multimodal ERM algorithm.
• To demonstrate a separation between multimodal and unimodal learning, showing that multimodal learning can solve problems unimodal learning cannot.

Proposed method

• Proposes a two-stage empirical risk minimization (ERM) algorithm: first learn a connection function $\hat{g} \in \mathcal{G}$ mapping one modality to another using potentially unlabeled data, then learn a predictor $\hat{f} \in \mathcal{F}$ on the composed data.
• Uses statistical learning theory tools, including Gaussian complexity and VC-dimension, to analyze generalization bounds.
• Decouples the learning of $\hat{f}$ and $\hat{g}$, allowing the generalization error to depend on the complexities of $\mathcal{F}$ and $\mathcal{G}$ separately, rather than on worst-case compositions.
• Introduces a connection measure $\mathcal{R}(\mathcal{G}, S)$ and a heterogeneity measure $H(\mu, \mathcal{G})$ to formalize the conditions under which multimodal learning gains an advantage.
• Employs a lower bound construction to show a hard instance where unimodal learning fails regardless of hypothesis class choice, while multimodal learning succeeds.
• Analyzes the role of explicit representations in reducing the Gaussian complexity term, showing it can be smaller by up to $O(\sqrt{n})$ compared to standard composition-based approaches.

Experimental results

Research questions

• RQ1 Why does multimodal learning generalize better than unimodal learning, even when evaluating on unimodal data?
• RQ2 Under what conditions does multimodal learning achieve a provable generalization advantage over unimodal learning?
• RQ3 Can the heuristic concept of 'connection' between modalities be formalized in a way that enables theoretical guarantees?
• RQ4 Is there a fundamental separation between multimodal and unimodal learning in terms of sample complexity and generalization error?
• RQ5 How does decoupling the learning of the connection and predictor functions affect the generalization bound?

Key findings

• The generalization error of the multimodal ERM algorithm is bounded by a term that depends on the complexities of $\mathcal{F}$ and $\mathcal{G}$ separately, improving over unimodal baselines by up to $O(\sqrt{n})$.
• The improvement arises when both connection (via $\mathcal{G}$) and heterogeneity (via $\mathcal{F}$) are present between modalities.
• A hard instance is constructed where unimodal learning incurs constant error regardless of the hypothesis class, while multimodal learning succeeds, proving a strict separation.
• The Gaussian complexity term $G(\mathcal{F}(\hat{X}_t, \hat{Y}_t))$ is shown to be smaller than worst-case composition terms by up to $O(\sqrt{n})$, due to decoupling.
• The theory justifies the empirical observation that multimodal models can outperform fine-tuned unimodal models on unimodal tasks.
• The framework provides a provable link between the heuristic concepts of connection and heterogeneity and improved statistical generalization.