[Paper Review] Combining Embeddings and Rules for Fact Prediction (Invited Paper)

Ding, Boyang, Wang, Quan+2 more|arXiv (Cornell University)|May 7, 2018

Advanced Graph Neural Networks43 references30 citations

TL;DR

This paper proposes ComplEx-NNE+AER, a knowledge graph embedding method that enhances model performance and interpretability by applying non-negativity constraints on entity representations and approximate entailment constraints on relation representations. These simple, scalable constraints improve link prediction accuracy across WordNet, Freebase, and DBpedia without increasing computational cost, significantly outperforming strong baselines while yielding more structured, interpretable embeddings.

ABSTRACT

Embedding knowledge graphs (KGs) into continuous vector spaces is a focus of current research. Early works performed this task via simple models developed over KG triples. Recent attempts focused on either designing more complicated triple scoring models, or incorporating extra information beyond triples. This paper, by contrast, investigates the potential of using very simple constraints to improve KG embedding. We examine non-negativity constraints on entity representations and approximate entailment constraints on relation representations. The former help to learn compact and interpretable representations for entities. The latter further encode regularities of logical entailment between relations into their distributed representations. These constraints impose prior beliefs upon the structure of the embedding space, without negative impacts on efficiency or scalability. Evaluation on WordNet, Freebase, and DBpedia shows that our approach is simple yet surprisingly effective, significantly and consistently outperforming competitive baselines. The constraints imposed indeed improve model interpretability, leading to a substantially increased structuring of the embedding space. Code and data are available at https://github.com/iieir-km/ComplEx-NNE_AER.

Motivation & Objective

• To improve knowledge graph embedding by incorporating simple, interpretable constraints without increasing computational cost.
• To learn more compact and semantically meaningful entity representations through non-negativity constraints.
• To encode logical regularities of relation entailment into distributed representations using approximate entailment constraints.
• To enhance model interpretability and structural organization of the embedding space.
• To achieve consistent performance gains across diverse benchmark knowledge graphs.

Proposed method

• Apply non-negativity constraints to entity embeddings to promote sparsity and interpretability.
• Learn approximate entailment constraints from statistical properties of relations, avoiding manual rule creation.
• Integrate constraints into the ComplEx model framework, preserving its efficiency and scalability.
• Use real and imaginary components of complex-valued embeddings to represent entities and relations.
• Optimize the model end-to-end with constrained optimization, maintaining low time and space complexity.
• Automatically identify equivalence and inversion relations using heuristics on relation representations.

Experimental results

Research questions

• RQ1 Can simple, automatically derived constraints improve knowledge graph embedding performance?
• RQ2 How do non-negativity constraints affect the sparsity and interpretability of entity representations?
• RQ3 Can approximate entailment constraints effectively encode logical regularities between relations?
• RQ4 Does the integration of such constraints improve link prediction accuracy without sacrificing efficiency?
• RQ5 To what extent do the constraints enhance the structural organization of the embedding space?

Key findings

• The proposed method, ComplEx-NNE+AER, achieves significant and consistent improvements over strong baselines on WordNet, Freebase, and DBpedia in link prediction tasks.
• Non-negativity constraints lead to more compact and interpretable entity representations, with each entity activated in only a few dimensions.
• The average entropy of entity type distributions over active dimensions decreases significantly, indicating higher semantic purity in learned representations.
• Approximate entailment constraints successfully encode logical regularities: equivalent relations have similar representations, and inverse relations have conjugate representations.
• The embedding space becomes substantially more structured, with entities of the same type clustering in shared active dimensions.
• The method maintains high efficiency and scalability, with no negative impact on training or inference time.