[论文解读] Vision, Deduction and Alignment: An Empirical Study on Multi-modal Knowledge Graph Alignment

Yangning Li, Jiaoyan Chen+4 more|arXiv (Cornell University)|Feb 17, 2023

Advanced Graph Neural Networks被引用 7

一句话总结

论文构建了八个大规模的多开放实体对齐基准（Multi-OpenEA）并包含图像信息，分析基于视觉信号对嵌入模型的影响，并提出自监督、多模态对齐方法LODEME，在现有基线上取得了最先进的结果。

ABSTRACT

Entity alignment (EA) for knowledge graphs (KGs) plays a critical role in knowledge engineering. Existing EA methods mostly focus on utilizing the graph structures and entity attributes (including literals), but ignore images that are common in modern multi-modal KGs. In this study we first constructed Multi-OpenEA -- eight large-scale, image-equipped EA benchmarks, and then evaluated some existing embedding-based methods for utilizing images. In view of the complementary nature of visual modal information and logical deduction, we further developed a new multi-modal EA method named LODEME using logical deduction and multi-modal KG embedding, with state-of-the-art performance achieved on Multi-OpenEA and other existing multi-modal EA benchmarks.

研究动机与目标

• Motivate the use of visual modality to supplement structure and literals in entity alignment (EA).
• Create large-scale, image-equipped EA benchmarks to reflect real-world multi-modal KGs.
• Evaluate existing embedding-based EA models with visual extensions on Multi-OpenEA.
• Propose a self-supervised, multi-modal EA method (LODEME) that combines logical deduction with multi-modal embeddings.

提出的方法

• Extend four embedding-based EA models (BootEA, MultiKE, RDGCN, IMUSE) with image modalities (suffix -V).
• Construct Multi-OpenEA benchmarks by adding multiple images per entity to eight OpenEA baselines.
• Develop LODEME with a probabilistic reasoning (PR) module inspired by PARIS and a multi-modal semantic embedding (SE) module with structure-aware image attention.
• In SE, encode structure (GCN), relations/attributes, name embeddings (M-BERT), and image embeddings (CLIP) with a weighted fusion of modalities.
• Train with a margin-based alignment loss and use hard negative sampling; inference via greedy search and CSLS.

实验结果

研究问题

• RQ1 Do visual modalities improve embedding-based EA methods on large-scale, image-equipped KGs?
• RQ2 How does integrating logical deduction with multi-modal embeddings affect EA performance?
• RQ3 What is the impact of different modalities (structure, literals, names, images) and the number of images on EA accuracy?
• RQ4 How does LODEME compare to existing multi-modal EA methods on diverse benchmarks?
• RQ5 Can a structure-aware attention mechanism effectively utilize multiple images per entity?

主要发现

• Visual modality improves embedding-based EA models by an average Hit@1 gain around 12%.
• LODEME achieves state-of-the-art results on Multi-OpenEA benchmarks with Hit@1 over 95% (D-W and EN-FR variants shown).
• Modified embedding-based models with images (suffix -V) show notable performance gains across datasets; highest average improvement observed for BootEA-V, MultiKE-V, and IMUSE-V.
• Ablation shows structure information remains the most important modality; visual data provides stronger gains for sparser KGs, and removing all images degrades performance more than removing names or relations/attributes.
• The structure-aware attention over multiple images outperforms mean-pooling and single best-image strategies, highlighting effective multi-image utilization.