[Paper Review] Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis

Xutan Peng, Guanyi Chen+2 more|arXiv (Cornell University)|Apr 9, 2021

Advanced Graph Neural Networks51 references17 citations

TL;DR

This paper proposes PROCRUSTES, a highly efficient knowledge graph embedding framework that reduces training time and carbon emissions by up to 98.4% and 99.3%, respectively, through full-batch learning via relational matrices, closed-form Orthogonal Procrustes Analysis, and non-negative sampling. It uniquely encodes both entity and relation information in a single vector space, yielding highly interpretable, semantics-rich embeddings with competitive performance.

ABSTRACT

Knowledge Graph Embeddings (KGEs) have been intensively explored in recent years due to their promise for a wide range of applications. However, existing studies focus on improving the final model performance without acknowledging the computational cost of the proposed approaches, in terms of execution time and environmental impact. This paper proposes a simple yet effective KGE framework which can reduce the training time and carbon footprint by orders of magnitudes compared with state-of-the-art approaches, while producing competitive performance. We highlight three technical innovations: full batch learning via relational matrices, closed-form Orthogonal Procrustes Analysis for KGEs, and non-negative-sampling training. In addition, as the first KGE method whose entity embeddings also store full relation information, our trained models encode rich semantics and are highly interpretable. Comprehensive experiments and ablation studies involving 13 strong baselines and two standard datasets verify the effectiveness and efficiency of our algorithm.

Motivation & Objective

• To address the high computational cost and environmental impact of existing knowledge graph embedding (KGE) methods.
• To develop a KGE framework that significantly reduces training time and CO2 emissions without sacrificing performance.
• To enable full-batch learning with efficient parallelization by grouping tuples by relation.
• To introduce a closed-form solution for Orthogonal Procrustes Analysis in KGEs for faster optimization.
• To demonstrate that entity embeddings can simultaneously encode rich semantics of both entities and relations, enhancing interpretability.

Proposed method

• Full-batch learning via relational matrices: tuples are grouped by relation to enable efficient, parallelized computation across batches.
• Closed-form Orthogonal Procrustes Analysis: a novel application of the Procrustes problem to KGEs, replacing iterative optimization with a direct matrix solution to accelerate training.
• Non-negative sampling training: eliminates the need for negative sampling, reducing computational overhead and bandwidth usage.
• Segmented embeddings: entity representations are split into independent sub-vectors (d/ds) to enable parallel processing and reduce matrix computation complexity.
• Joint encoding of entity and relation semantics: entity embeddings are designed to store full relation information, enhancing interpretability and expressiveness.
• Relational matrix partitioning: as shown in Figure 1, matrices are divided by relation to support efficient batch processing and reduce synchronization conflicts.

Experimental results

Research questions

• RQ1 Can a KGE framework achieve state-of-the-art performance while reducing training time and carbon emissions by orders of magnitude?
• RQ2 Can closed-form Orthogonal Procrustes Analysis be effectively applied to KGE training to replace iterative optimization?
• RQ3 Does non-negative sampling training maintain model performance while eliminating costly negative sampling?
• RQ4 Can entity embeddings be designed to encode both entity and relation semantics in a single vector space, improving interpretability?
• RQ5 How does the proposed relational matrix batching strategy compare to standard random batching in terms of speed and scalability?

Key findings

• PROCRUSTES reduces training time by up to 98.4% compared to state-of-the-art KGE methods on standard benchmarks.
• The method reduces carbon emissions by up to 99.3%, equivalent to less than two cups of coffee per training run.
• Entity embeddings produced by PROCRUSTES are highly interpretable, with 3D PCA visualizations revealing distinct semantic clusters corresponding to categories like locations, biochemical terms, and occupations.
• The model achieves competitive performance on WN18RR and FB15k-237, matching or exceeding state-of-the-art results in link prediction tasks.
• The framework successfully encodes both entity and relation semantics in a single vector space, enabling richer and more interpretable representations than prior approaches.
• Ablation studies confirm that each component—relational batching, closed-form Procrustes, and non-negative sampling—contributes significantly to efficiency and performance.