[Paper Review] Graph Optimal Transport for Cross-Domain Alignment

Li‐Qun Chen, Zhe Gan+4 more|arXiv (Cornell University)|Jun 26, 2020

Advanced Graph Neural Networks22 citations

TL;DR

This paper proposes Graph Optimal Transport (GOT), a principled framework for cross-domain alignment that formulates entity matching as a graph matching problem using Wasserstein Distance (WD) for node alignment and Gromov-Wasserstein Distance (GWD) for structural alignment. GOT acts as a drop-in regularizer in neural networks, yielding sparse, interpretable transport plans and consistently improving performance across image-text retrieval, VQA, captioning, machine translation, and summarization tasks.

ABSTRACT

Cross-domain alignment between two sets of entities (e.g., objects in an image, words in a sentence) is fundamental to both computer vision and natural language processing. Existing methods mainly focus on designing advanced attention mechanisms to simulate soft alignment, with no training signals to explicitly encourage alignment. The learned attention matrices are also dense and lacks interpretability. We propose Graph Optimal Transport (GOT), a principled framework that germinates from recent advances in Optimal Transport (OT). In GOT, cross-domain alignment is formulated as a graph matching problem, by representing entities into a dynamically-constructed graph. Two types of OT distances are considered: (i) Wasserstein distance (WD) for node (entity) matching; and (ii) Gromov-Wasserstein distance (GWD) for edge (structure) matching. Both WD and GWD can be incorporated into existing neural network models, effectively acting as a drop-in regularizer. The inferred transport plan also yields sparse and self-normalized alignment, enhancing the interpretability of the learned model. Experiments show consistent outperformance of GOT over baselines across a wide range of tasks, including image-text retrieval, visual question answering, image captioning, machine translation, and text summarization.

Motivation & Objective

• To address the lack of explicit training signals for cross-domain alignment in existing attention-based models.
• To improve interpretability and sparsity in learned alignments by replacing dense attention matrices with optimal transport plans.
• To unify node-level and structure-level matching in cross-domain alignment using optimal transport theory.
• To develop a generic, differentiable framework compatible with existing neural architectures as a regularizer.
• To empirically validate the framework across diverse multimodal and NLP tasks.

Proposed method

• Represent entities in each domain as nodes in a dynamically constructed graph using learned feature embeddings.
• Apply Wasserstein Distance (WD) to match nodes (entities) across domains based on their feature embeddings.
• Use Gromov-Wasserstein Distance (GWD) to match structural relations (edges) between entities by comparing their relative distances.
• Formulate the overall alignment as a joint optimization of WD and GWD, minimizing the transport cost between domain distributions.
• Integrate the GOT loss as a differentiable regularizer into existing neural network objectives without architectural changes.
• Use a shared Sinkhorn-regularized transport plan to ensure efficiency and consistency during training.

Experimental results

Research questions

• RQ1 Can optimal transport be effectively adapted to model cross-domain alignment in a differentiable, interpretable, and scalable way?
• RQ2 Does combining node-level (WD) and structure-level (GWD) matching improve alignment performance compared to using either alone?
• RQ3 Can GOT serve as a generic, plug-in regularizer that enhances performance across diverse vision and language tasks?
• RQ4 How do hyperparameters like λ (weighting between WD and GWD) affect model performance?
• RQ5 Is the transport plan learned by GOT sparse and interpretable, as claimed?

Key findings

• GOT consistently outperforms strong baselines across five diverse tasks: image-text retrieval, VQA, image captioning, machine translation, and abstractive summarization.
• On the EN-VI translation dataset, GOT achieved a BLEU score of 29.92 with λ=0.8, outperforming the baseline (28.65) and the unshared transport plan variant (29.77).
• The ablation study showed that a shared transport plan reduces training time and improves performance over unshared plans, with gains of 0.15 BLEU on EN-VI and 0.16 on EN-DE.
• The optimal λ value was found to be 0.8 on EN-VI, indicating that node matching (WD) contributes more than structure matching (GWD) in translation tasks.
• Visualizations of the transport plan revealed sparse, self-normalized alignments, such as 'largest' aligning with 'more' and 'projects' with 'investment', confirming interpretability.
• In text summarization, both WD and GWD improved ROUGE scores, demonstrating the framework’s generalization to abstractive generation tasks.