[Paper Review] Discrete Knowledge Graph Embedding based on Discrete Optimization

Yunqi Li, Shuyuan Xu+5 more|arXiv (Cornell University)|Jan 13, 2021

Advanced Graph Neural Networks16 references4 citations

TL;DR

This paper proposes Discrete Knowledge Graph Embedding (DKGE), a method that directly learns binary entity and relation codes in Hamming space via discrete optimization, using Hadamard product for translation modeling. DKGE achieves comparable link prediction accuracy to state-of-the-art continuous models with significantly lower storage and computational costs, while outperforming methods that quantize continuous embeddings post-training.

ABSTRACT

This paper proposes a discrete knowledge graph (KG) embedding (DKGE) method, which projects KG entities and relations into the Hamming space based on a computationally tractable discrete optimization algorithm, to solve the formidable storage and computation cost challenges in traditional continuous graph embedding methods. The convergence of DKGE can be guaranteed theoretically. Extensive experiments demonstrate that DKGE achieves superior accuracy than classical hashing functions that map the effective continuous embeddings into discrete codes. Besides, DKGE reaches comparable accuracy with much lower computational complexity and storage compared to many continuous graph embedding methods.

Motivation & Objective

• To address the high storage and computational costs of continuous knowledge graph embeddings in large-scale applications.
• To overcome the performance degradation caused by two-stage hashing methods that first optimize in continuous space and then quantize to discrete codes.
• To develop a computationally tractable discrete optimization framework that directly learns binary embeddings for entities and relations.
• To achieve comparable or better link prediction accuracy than continuous models while drastically reducing memory footprint and inference cost.
• To demonstrate the superiority of direct discrete optimization over post-hoc quantization of continuous embeddings.

Proposed method

• DKGE represents entities and relations as binary vectors in {±1}^k, embedded in Hamming space.
• It models entity translation using the Hadamard product of head entity and relation vectors, preserving structural semantics.
• The scoring function is based on Hamming distance between the translated head entity and the tail entity.
• A mixed-integer alternating optimization algorithm is used to solve the discrete optimization problem, ensuring computational tractability.
• Convergence of the algorithm is theoretically guaranteed through iterative optimization over discrete variables.
• The method avoids relaxation of discrete constraints, preserving embedding fidelity and minimizing quantization loss.

Experimental results

Research questions

• RQ1 Can direct discrete optimization for knowledge graph embedding achieve comparable accuracy to continuous methods while reducing storage and computation costs?
• RQ2 How does direct learning of discrete embeddings compare to post-training quantization of continuous embeddings via classical hashing functions?
• RQ3 Does discrete optimization outperform existing discrete KG embedding methods like B-CP, especially on dense datasets?
• RQ4 Can the proposed method maintain high performance with only half the memory consumption of models that use separate vectors for head and tail entities?
• RQ5 What is the impact of discrete optimization on link prediction accuracy in low-memory, high-efficiency scenarios?

Key findings

• DKGE achieves comparable or better link prediction accuracy than state-of-the-art continuous models like TransE, DistMult, ComplEx, ConvE, and R-GCN, especially on FB15k-237.
• On the same memory budget, DKGE outperforms continuous models in terms of both accuracy and efficiency, as shown in Figure 1.
• DKGE significantly outperforms three classical hashing functions (sign, equal-interval, Lloyd-Max) when applied to continuous embeddings from TransE and DistMult.
• DKGE achieves better Hit@10 on FB15k-237 than B-CP, despite using only half the memory, indicating superior performance on dense datasets.
• The method demonstrates lower computational complexity and faster inference due to bitwise operations on discrete codes, enabling deployment on edge devices.
• Theoretical convergence is guaranteed for the discrete optimization algorithm, validating its stability and reliability.