[論文レビュー] Vision, Deduction and Alignment: An Empirical Study on Multi-modal Knowledge Graph Alignment

Yangning Li, Jiaoyan Chen+4 more|arXiv (Cornell University)|Feb 17, 2023

Advanced Graph Neural Networks被引用数 7

ひとこと要約

The paper builds eight large-scale Multi-OpenEA benchmarks with images for entity alignment, analyzes Vision-based signals on embedding models, and introduces LODEME, a self-supervised, multi-modal alignment method that achieves state-of-the-art results.

ABSTRACT

Entity alignment (EA) for knowledge graphs (KGs) plays a critical role in knowledge engineering. Existing EA methods mostly focus on utilizing the graph structures and entity attributes (including literals), but ignore images that are common in modern multi-modal KGs. In this study we first constructed Multi-OpenEA -- eight large-scale, image-equipped EA benchmarks, and then evaluated some existing embedding-based methods for utilizing images. In view of the complementary nature of visual modal information and logical deduction, we further developed a new multi-modal EA method named LODEME using logical deduction and multi-modal KG embedding, with state-of-the-art performance achieved on Multi-OpenEA and other existing multi-modal EA benchmarks.

研究の動機と目的

• Motivate the use of visual modality to supplement structure and literals in entity alignment (EA).
• Create large-scale, image-equipped EA benchmarks to reflect real-world multi-modal KGs.
• Evaluate existing embedding-based EA models with visual extensions on Multi-OpenEA.
• Propose a self-supervised, multi-modal EA method (LODEME) that combines logical deduction with multi-modal embeddings.

提案手法

• Extend four embedding-based EA models (BootEA, MultiKE, RDGCN, IMUSE) with image modalities (suffix -V).
• Construct Multi-OpenEA benchmarks by adding multiple images per entity to eight OpenEA baselines.
• Develop LODEME with a probabilistic reasoning (PR) module inspired by PARIS and a multi-modal semantic embedding (SE) module with structure-aware image attention.
• In SE, encode structure (GCN), relations/attributes, name embeddings (M-BERT), and image embeddings (CLIP) with a weighted fusion of modalities.
• Train with a margin-based alignment loss and use hard negative sampling; inference via greedy search and CSLS.

実験結果

リサーチクエスチョン

• RQ1 Do visual modalities improve embedding-based EA methods on large-scale, image-equipped KGs?
• RQ2 How does integrating logical deduction with multi-modal embeddings affect EA performance?
• RQ3 What is the impact of different modalities (structure, literals, names, images) and the number of images on EA accuracy?
• RQ4 How does LODEME compare to existing multi-modal EA methods on diverse benchmarks?
• RQ5 Can a structure-aware attention mechanism effectively utilize multiple images per entity?

主な発見

• Visual modality improves embedding-based EA models by an average Hit@1 gain around 12%.
• LODEME achieves state-of-the-art results on Multi-OpenEA benchmarks with Hit@1 over 95% (D-W and EN-FR variants shown).
• Modified embedding-based models with images (suffix -V) show notable performance gains across datasets; highest average improvement observed for BootEA-V, MultiKE-V, and IMUSE-V.
• Ablation shows structure information remains the most important modality; visual data provides stronger gains for sparser KGs, and removing all images degrades performance more than removing names or relations/attributes.
• The structure-aware attention over multiple images outperforms mean-pooling and single best-image strategies, highlighting effective multi-image utilization.