[Paper Review] Memory Fusion Network for Multi-view Sequential Learning

Amir Zadeh, Paul Pu Liang+4 more|arXiv (Cornell University)|Feb 3, 2018

Domain Adaptation and Few-Shot LearningComputer Science120 citations

TL;DR

MFN introduces a three-component neural architecture that separately models view-specific dynamics, identifies cross-view interactions with Delta-memory Attention, and stores cross-view information over time in a Multi-view Gated Memory to achieve state-of-the-art results on multiple multi-view sequence benchmarks.

ABSTRACT

Multi-view sequential learning is a fundamental problem in machine learning dealing with multi-view sequences. In a multi-view sequence, there exists two forms of interactions between different views: view-specific interactions and cross-view interactions. In this paper, we present a new neural architecture for multi-view sequential learning called the Memory Fusion Network (MFN) that explicitly accounts for both interactions in a neural architecture and continuously models them through time. The first component of the MFN is called the System of LSTMs, where view-specific interactions are learned in isolation through assigning an LSTM function to each view. The cross-view interactions are then identified using a special attention mechanism called the Delta-memory Attention Network (DMAN) and summarized through time with a Multi-view Gated Memory. Through extensive experimentation, MFN is compared to various proposed approaches for multi-view sequential learning on multiple publicly available benchmark datasets. MFN outperforms all the existing multi-view approaches. Furthermore, MFN outperforms all current state-of-the-art models, setting new state-of-the-art results for these multi-view datasets.

Motivation & Objective

• Motivate and address multi-view sequential learning where data from different views have view-specific and cross-view interactions.
• Propose MFN architecture to model both interaction types through time.
• Demonstrate MFN's effectiveness on diverse multimodal datasets and compare to state-of-the-art methods.

Proposed method

• Implement a System of LSTMs where each view has its own LSTM to capture view-specific dynamics.
• Use Delta-memory Attention Network (DMAN) to assign cross-view interaction relevance by attending to consecutive memory states (t-1 and t) across views.
• Introduce a Multi-view Gated Memory that updates with DMAN outputs to store and summarize cross-view interactions over time.
• Combine outputs from all view-specific LSTMs and the cross-view memory for final predictions.
• Conduct ablation studies to assess the contributions of Delta memory and the cross-view memory.

Experimental results

Research questions

• RQ1 How can one explicitly model both view-specific and cross-view interactions in multi-view sequential data?
• RQ2 Does incorporating a Delta-memory Attention mechanism improve discovery of cross-view interactions over time?
• RQ3 What is the impact of a dedicated Multi-view Gated Memory on capturing long-term cross-view information?
• RQ4 How does MFN perform compared to state-of-the-art multi-view sequential models across diverse datasets?

Key findings

• MFN achieves state-of-the-art performance across all evaluated datasets and metrics in multimodal sentiment analysis, emotion recognition, and speaker trait analysis.
• Ablation studies show MFN (with both Delta memory and Multi-view Gated Memory) outperforms MFN variants lacking these components.
• MFN offers better performance with a substantially smaller parameter count (~5e5) and faster runtime (~2858 inferences/second) compared to notable baselines.
• Using multiple views consistently improves results over single-view MFN variants, underscoring the value of cross-view modeling.
• Delta-memory (t-1, t) contributes crucial temporal context, as shown by performance drops in MFN (no Δ) ablations.