[论文解读] Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Paul Pu Liang, Amir Zadeh+1 more|arXiv (Cornell University)|Sep 7, 2022

Speech and dialogue systems被引用 36

一句话总结

一份全面的综述，界定多模态学习的基础原则，并提出六大核心挑战（表示、对齐、推理、生成、迁移、量化）的分类，并附带相关子问题和方法。

ABSTRACT

Multimodal machine learning is a vibrant multi-disciplinary research field that aims to design computer agents with intelligent capabilities such as understanding, reasoning, and learning through integrating multiple communicative modalities, including linguistic, acoustic, visual, tactile, and physiological messages. With the recent interest in video understanding, embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, multimodal machine learning has brought unique computational and theoretical challenges to the machine learning community given the heterogeneity of data sources and the interconnections often found between modalities. However, the breadth of progress in multimodal research has made it difficult to identify the common themes and open questions in the field. By synthesizing a broad range of application domains and theoretical frameworks from both historical and recent perspectives, this paper is designed to provide an overview of the computational and theoretical foundations of multimodal machine learning. We start by defining three key principles of modality heterogeneity, connections, and interactions that have driven subsequent innovations, and propose a taxonomy of six core technical challenges: representation, alignment, reasoning, generation, transference, and quantification covering historical and recent trends. Recent technical achievements will be presented through the lens of this taxonomy, allowing researchers to understand the similarities and differences across new approaches. We end by motivating several open problems for future research as identified by our taxonomy.

研究动机与目标

• 界定多模态学习的基础原则（异质性、连接性、交互作用）。
• 提出多模态机器学习六大核心技术挑战的分类。
• 综合表示、对齐、推理、生成、迁移和量化等维度的历史与近期方法。
• 突出多模态学习中的未解决问题和未来研究方向。

提出的方法

• 提出六大核心挑战的分类及其子类别和代表性方法。
• 评审并将现有方法分门别类地归入表示、对齐、推理、生成、迁移与量化。
• 讨论模态异质性、连接性与交互作用的原则，以及它们如何推动每一项挑战。
• 调查跨模态表示与交互，包括融合、协调与分裂等技术。
• 检视该分类所揭示的未解问题与未来方向。

实验结果

研究问题

• RQ1 驱动多模态学习的核心原则是什么，它们如何影响方法选择？
• RQ2 多模态机器学习的六大基本技术挑战是什么，如何有效地对其进行分类和解决？
• RQ3 在表示、对齐、推理、生成、迁移和量化等子挑战中，主要方法和具有代表性的示例有哪些？
• RQ4 根据此分类，多模态机器学习中仍存在哪些未解决的问题？
• RQ5 异质性、连接性与交互作用如何影响多模态系统中的学习与评估？

主要发现

• 一个有原则的分类法确定了六大核心挑战：表示、对齐、推理、生成、迁移与量化。
• 模态是异质的、相互连接并且具有交互性，这促成了每一核心挑战下的专门子领域。
• 表示的子挑战包括融合、协调和分裂；对齐包括离散和连续对齐及情境化；推理包括结构建模和外部知识；生成包括摘要、翻译和创建；迁移包括跨模态迁移、共同学习和模型诱导；量化包括异质性、互连性和学习。
• 本文综合了历史与近期工作，以映射跨应用领域与理论框架的共同主题和未解问题。
• 它将基础原理与具体的方法学问题及未来研究方向联系起来。