[Paper Review] MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

Zhuo Chen, Jiaoyan Chen+9 more|arXiv (Cornell University)|Dec 29, 2022

Advanced Graph Neural Networks5 citations

TL;DR

MEAformer proposes a dynamic, entity-level modality fusion approach for multi-modal entity alignment using a meta-learning-based Transformer architecture that adaptively predicts cross-modal weights. It achieves state-of-the-art performance across supervised, unsupervised, iterative, and low-resource settings with high efficiency, interpretability, and robustness to noisy modalities like blurry images.

ABSTRACT

Multi-modal entity alignment (MMEA) aims to discover identical entities across different knowledge graphs (KGs) whose entities are associated with relevant images. However, current MMEA algorithms rely on KG-level modality fusion strategies for multi-modal entity representation, which ignores the variations of modality preferences of different entities, thus compromising robustness against noise in modalities such as blurry images and relations. This paper introduces MEAformer, a multi-modal entity alignment transformer approach for meta modality hybrid, which dynamically predicts the mutual correlation coefficients among modalities for more fine-grained entity-level modality fusion and alignment. Experimental results demonstrate that our model not only achieves SOTA performance in multiple training scenarios, including supervised, unsupervised, iterative, and low-resource settings, but also has a limited number of parameters, efficient runtime, and interpretability. Our code is available at https://github.com/zjukg/MEAformer.

Motivation & Objective

• To address the limitation of existing MMEA methods that use static, KG-level modality fusion, which fails to account for entity-specific modality preferences and is sensitive to noisy modalities.
• To develop a more adaptive and robust MMEA framework that dynamically adjusts modality weights at the entity level based on intrinsic modality preferences and data quality.
• To enhance model robustness against noisy or missing modalities—especially visual data—through modal-aware hard entity replay and contrastive learning.
• To achieve high performance with minimal parameters and efficient inference, enabling practical deployment in low-resource and real-world settings.

Proposed method

• Introduces a dynamic cross-modal weighted (DCMW) module that learns entity-specific, mutual modality correlation coefficients via a shallow cross-attention network.
• Employs a meta-learning paradigm to predict meta modality weights that correct modality representations based on entity-level context and modality reliability.
• Applies a modal-adaptive contrastive learning objective to disentangle modality-specific features using limited pre-aligned entities.
• Uses a modal-aware hard entity replay strategy to retrain on ambiguous or noisy examples, improving robustness to vague or inconsistent visual inputs.
• Integrates vision, attribute, relation, and graph modalities into a unified Transformer-based encoder with learnable modality fusion at the entity level.
• Leverages a multi-head attention mechanism within the Transformer to aggregate multi-modal representations while preserving modality-specific semantics.

Experimental results

Research questions

• RQ1 How can multi-modal entity alignment be improved by adapting modality fusion strategies to individual entity characteristics rather than applying global, KG-level weights?
• RQ2 To what extent can dynamic, entity-level modality weighting enhance robustness against noisy or missing modalities such as blurry images or incomplete relations?
• RQ3 Can a meta-learning-based approach for modality weight prediction improve alignment performance across diverse training regimes, including unsupervised and low-resource settings?
• RQ4 How does the proposed modal-aware hard entity replay strategy improve model generalization on ambiguous or visually noisy entities?
• RQ5 What patterns in modality preference emerge across different entity types, and can they be effectively captured by a learnable, dynamic fusion mechanism?

Key findings

• MEAformer achieves state-of-the-art performance across all evaluated training scenarios, including supervised, unsupervised, iterative, and low-resource settings.
• The model demonstrates strong robustness to noisy visual inputs, such as blurry or ambiguous images, due to its dynamic modality weighting and contrastive learning.
• The average modality weight distribution across entities is more uniform than in baseline models like EVA, indicating better balance in modality utilization.
• Entities with high graph modality preference—such as football players, constellations, and Swiss municipalities—tend to have higher node degrees and simpler, non-hierarchical relation structures.
• The model’s meta modality weights show strong interpretability, with clear patterns in modality reliance (e.g., athletes depend on vision and graph, political figures on relations).
• MEAformer achieves high performance with a limited number of parameters and efficient runtime, making it suitable for real-world deployment.