[Paper Review] Compositional Vector Space Models for Knowledge Base Completion

Arvind Neelakantan, Benjamin Roth+1 more|arXiv (Cornell University)|Apr 24, 2015

Topic ModelingComputer Science45 references92 citations

TL;DR

This paper proposes a compositional vector space model for knowledge base completion using recurrent neural networks (RNNs) to non-atomically compose multi-hop relational paths into distributed vector representations. By learning a single high-capacity RNN composition function, the model enables zero-shot inference for unseen relation types and achieves 11% improvement over a traditional classifier and 7% over a pre-trained embedding method on a 52M-triple dataset.

ABSTRACT

Knowledge base (KB) completion adds new facts to a KB by making inferences from existing facts, for example by inferring with high likelihood nationality(X,Y) from bornIn(X,Y). Most previous methods infer simple one-hop relational synonyms like this, or use as evidence a multi-hop relational path treated as an atomic feature, like bornIn(X,Z) -> containedIn(Z,Y). This paper presents an approach that reasons about conjunctions of multi-hop relations non-atomically, composing the implications of a path using a recursive neural network (RNN) that takes as inputs vector embeddings of the binary relation in the path. Not only does this allow us to generalize to paths unseen at training time, but also, with a single high-capacity RNN, to predict new relation types not seen when the compositional model was trained (zero-shot learning). We assemble a new dataset of over 52M relational triples, and show that our method improves over a traditional classifier by 11%, and a method leveraging pre-trained embeddings by 7%.

Motivation & Objective

• To address the scalability and generalization limitations of symbolic path-based methods in knowledge base completion.
• To enable zero-shot inference for unseen relation types by learning a compositional RNN that generalizes across relational paths.
• To overcome the feature explosion problem of path ranking algorithms by using distributed vector representations instead of atomic path features.
• To improve generalization and performance on large-scale KBs by composing relation embeddings through a recurrent architecture.
• To demonstrate that RNN-based composition of relational paths yields better generalization than traditional classifiers or pre-trained embeddings.

Proposed method

• The model uses a recurrent neural network (RNN) that takes as input the vector embeddings of relations in a path and the hidden state representing the path-so-far, producing a composed vector at each step.
• The RNN composes relation vectors sequentially, with each step updating the hidden state to reflect the semantic meaning of the growing path.
• After processing the full path, the final RNN hidden state is used to predict the relation between the first and last entities in the path.
• The model learns a separate RNN per relation type for standard prediction, or a single shared RNN for zero-shot learning across unseen relations.
• The method leverages distributed vector representations of relations to enable generalization through semantic neighborhoods, avoiding the need to explicitly enumerate all paths.
• Ensemble predictions from multiple RNNs are used to improve robustness and mitigate local optima issues.

Experimental results

Research questions

• RQ1 Can a compositional RNN model generalize to multi-hop relational paths unseen during training?
• RQ2 Can a single RNN composition function enable zero-shot prediction for relation types not seen during training?
• RQ3 Does non-atomic composition of relation vectors outperform atomic path feature models in knowledge base completion?
• RQ4 To what extent does using RNNs with vector embeddings improve generalization compared to traditional classifiers or pre-trained embeddings?
• RQ5 How does the model's performance scale with path length and relation diversity in large KBs?

Key findings

• The RNN-based compositional model improves performance by 11% over a traditional path-ranking classifier on a 52M-triple dataset.
• The model achieves a 7% improvement over a method leveraging pre-trained relation embeddings.
• Ensemble prediction from five RNNs increases performance to 59.16, indicating robustness to local optima but still underperforming the best baseline.
• The zero-shot model, trained without explicit examples of target relations, significantly outperforms a random baseline (p < 0.05), demonstrating generalization to unseen relations.
• The model's performance is limited by its inability to fully capture local structural patterns compared to a classifier using bigram path features, suggesting room for improvement with memory-enhanced architectures.
• The results indicate that RNN composition enables effective generalization across complex relational paths, supporting its use in large-scale KB completion.