[Paper Review] M<sup>2</sup>Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis

Xingbo Wang, Jianben He+4 more|arXiv (Cornell University)|Jan 1, 2022

Topic Modeling83 references82 citations

TL;DR

M2Lens is an interactive visual analytics system that explains multimodal sentiment analysis models by visualizing intra- and inter-modal interactions at global, subset, and local levels. It uses post-hoc explainability techniques like SHAP to identify influential feature patterns and interaction types (dominance, complement, conflict), enabling users to diagnose model behavior across text, audio, and video modalities with high interpretability and multi-faceted exploration capabilities.

ABSTRACT

Multimodal sentiment analysis aims to recognize people's attitudes from multiple communication channels such as verbal content (i.e., text), voice, and facial expressions. It has become a vibrant and important research topic in natural language processing. Much research focuses on modeling the complex intra- and inter-modal interactions between different communication channels. However, current multimodal models with strong performance are often deep-learning-based techniques and work like black boxes. It is not clear how models utilize multimodal information for sentiment predictions. Despite recent advances in techniques for enhancing the explainability of machine learning models, they often target unimodal scenarios (e.g., images, sentences), and little research has been done on explaining multimodal models. In this paper, we present an interactive visual analytics system, M2 Lens, to visualize and explain multimodal models for sentiment analysis. M2 Lens provides explanations on intra- and inter-modal interactions at the global, subset, and local levels. Specifically, it summarizes the influence of three typical interaction types (i.e., dominance, complement, and conflict) on the model predictions. Moreover, M2 Lens identifies frequent and influential multimodal features and supports the multi-faceted exploration of model behaviors from language, acoustic, and visual modalities. Through two case studies and expert interviews, we demonstrate our system can help users gain deep insights into the multimodal models for sentiment analysis.

Motivation & Objective

• Address the lack of explainable, interactive tools for diagnosing deep learning-based multimodal sentiment analysis models that operate as black boxes.
• Provide multi-level explanations—global, subset, and local—of how different modalities (text, audio, video) and their interactions influence model predictions.
• Enable users to explore and understand complex interaction patterns such as dominance, complementarity, and conflict between modalities in sentiment decisions.
• Support efficient, human-friendly exploration of frequent and influential multimodal feature templates and their impact on model behavior.
• Facilitate model diagnosis and insight generation through integrated visual components and expert-informed design.

Proposed method

• Integrates post-hoc explainability methods (e.g., SHAP) to compute feature importance scores across text, audio, and visual modalities.
• Employs an augmented tree-like layout in the Summary View to visualize global modality influences and interaction types (dominance, complement, conflict).
• Generates compact, human-readable feature templates in the Template View to summarize recurring and influential multimodal feature sets.
• Uses a Projection View with customizable glyphs to enable multi-faceted exploration of instances based on feature importance, sentiment, and modality interactions.
• Visualizes local explanations in the Instance View by highlighting key features and their context across modalities for individual predictions.
• Supports interactive exploration via lasso selection, zoom, video playback, and real-time highlighting of facial regions for video data.

Experimental results

Research questions

• RQ1 How can we effectively visualize and explain intra- and inter-modal interactions in multimodal sentiment analysis models?
• RQ2 What are the most influential interaction types (e.g., dominance, complement, conflict) between modalities, and how do they affect model predictions?
• RQ3 How can we summarize frequent and influential multimodal feature patterns in a human-readable and actionable way?
• RQ4 To what extent can interactive visual analytics help users gain deep, actionable insights into model behavior and error patterns?
• RQ5 How do users perceive the usability and effectiveness of a system that supports multi-faceted, multi-level explanation of multimodal models?

Key findings

• Experts found M2Lens effective for diagnosing model behavior, with one noting it helped identify that EF-LSTM largely ignored textual sentiment cues.
• The Template View enabled users to generalize error patterns across instances, with one expert highlighting its utility in identifying recurring model failures.
• The Projection View’s heatmap and glyph-based design were praised for revealing error and importance patterns across modalities, especially for detecting conflicting signals.
• The Summary View was most valued for its global overview of modality dominance and interaction types, aiding rapid model assessment.
• Experts reported a moderate learning curve (approx. 20 minutes) but found the system highly useful for model understanding and future diagnostic tasks.
• Expert feedback led to actionable improvements, including bookmarking interactions and model comparison features, underscoring the system’s practical utility in real workflows.