[Paper Review] Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment

Jielin Qiu, Jiacheng Zhu+7 more|arXiv (Cornell University)|Oct 10, 2022

Video Analysis and Summarization4 citations

TL;DR

This paper proposes SCCS, a semantics-consistent cross-domain summarization model that uses optimal transport alignment to match segment-level visual and textual representations for multimodal summarization. By decomposing videos and articles into semantic segments and optimizing cross-domain alignment via optimal transport, SCCS achieves state-of-the-art performance on three multimodal datasets with improved interpretability through sparse transport plans.

ABSTRACT

Multimedia summarization with multimodal output (MSMO) is a recently explored application in language grounding. It plays an essential role in real-world applications, i.e., automatically generating cover images and titles for news articles or providing introductions to online videos. However, existing methods extract features from the whole video and article and use fusion methods to select the representative one, thus usually ignoring the critical structure and varying semantics. In this work, we propose a Semantics-Consistent Cross-domain Summarization (SCCS) model based on optimal transport alignment with visual and textual segmentation. In specific, our method first decomposes both video and article into segments in order to capture the structural semantics, respectively. Then SCCS follows a cross-domain alignment objective with optimal transport distance, which leverages multimodal interaction to match and select the visual and textual summary. We evaluated our method on three recent multimodal datasets and demonstrated the effectiveness of our method in producing high-quality multimodal summaries.

Motivation & Objective

• To address the limitation of existing multimodal summarization methods that treat entire videos and articles as single units, ignoring structural semantics.
• To preserve and exploit the underlying structural semantics in both visual and textual modalities for more accurate and consistent summarization.
• To develop a computationally efficient, hierarchical framework that enables interpretable cross-modal alignment at the segment level.
• To improve multimodal summary quality by jointly optimizing visual and textual summary selection through optimal transport-based matching.
• To provide interpretability of learned alignments via visualization of optimal transport coupling matrices.

Proposed method

• Decomposes input videos and text articles into semantic segments using dedicated encoders to preserve structural semantics.
• Applies a cross-domain alignment objective based on optimal transport (OT) to compute the Wasserstein distance between visual and textual segment embeddings.
• Uses the optimal transport coupling matrix to identify the most semantically aligned image-sentence pairs, favoring sparse and structured correspondences.
• Integrates a multimodal alignment module that jointly optimizes visual and textual summary selection by minimizing OT-based cross-domain distance.
• Employs K-means clustering and image histogram features for unsupervised visual summarization when only video input is available.
• Leverages BERT for text embedding and centroid-based sentence selection in unimodal settings for ablation and comparison.

Experimental results

Research questions

• RQ1 Can segment-level cross-domain alignment improve the quality and consistency of multimodal summaries compared to whole-input methods?
• RQ2 How does optimal transport-based alignment enhance the interpretability and structural coherence of visual-textual summary matching?
• RQ3 To what extent does preserving structural semantics in both modalities improve summary quality on multimodal summarization benchmarks?
• RQ4 Does the proposed method outperform existing unimodal and multimodal baselines in both textual and visual summarization quality?
• RQ5 Can the learned optimal transport coupling matrix provide meaningful, sparse, and interpretable alignments between visual and textual segments?

Key findings

• SCCS achieves state-of-the-art performance on the CNN and Daily Mail datasets, with a multimodal ROUGE-L score of 35.79, outperforming the best baseline ($\rm M^{2}$SM) by 4.11 points.
• The proposed method achieves a Cosine similarity of 73.19% on the Daily Mail dataset, significantly outperforming the second-best method (M$^{2}$SM) at 69.22%.
• On the VMSMO dataset, the multimodal approach outperforms unimodal baselines, demonstrating the effectiveness of cross-modal interaction in summary generation.
• Ablation studies confirm that multimodal learning consistently improves performance over unimodal baselines, with the best results achieved when both visual and textual features are used.
• Visualization of the optimal transport coupling matrix reveals sparse, structured patterns for matched image-text pairs, while non-matched pairs show dense, uninformative couplings, validating interpretability.