[Paper Review] Ranking vs. Classifying: Measuring Knowledge Base Completion Quality

Marina Speranskaya, Martin Schmitt+1 more|arXiv (Cornell University)|Feb 2, 2021

Topic Modeling4 citations

TL;DR

This paper proposes a novel classification-based evaluation paradigm for knowledge base completion (KBC) that addresses the limitations of traditional ranking-based metrics. By constructing the FB14k-QAQ dataset with queries missing true answers due to entity removal or type constraints, the authors demonstrate that models excelling in ranking do not necessarily generalize well to real-world fact prediction. A modified TransE model with relation-specific regions improves F1 score by over 30% compared to standard TransE, highlighting the need for better prediction separability in KBC models.

ABSTRACT

Knowledge base completion (KBC) methods aim at inferring missing facts from the information present in a knowledge base (KB) by estimating the likelihood of candidate facts. In the prevailing evaluation paradigm, models do not actually decide whether a new fact should be accepted or not but are solely judged on the position of true facts in a likelihood ranking with other candidates. We argue that consideration of binary predictions is essential to reflect the actual KBC quality, and propose a novel evaluation paradigm, designed to provide more transparent model selection criteria for a realistic scenario. We construct the data set FB14k-QAQ where instead of single facts, we use KB queries, i.e., facts where one entity is replaced with a variable, and construct corresponding sets of entities that are correct answers. We randomly remove some of these correct answers from the data set, simulating the realistic scenario of real-world entities missing from a KB. This way, we can explicitly measure a model's ability to handle queries that have more correct answers in the real world than in the KB, including the special case of queries without any valid answer. The latter especially contrasts the ranking setting. We evaluate a number of state-of-the-art KB embeddings models on our new benchmark. The differences in relative performance between ranking-based and classification-based evaluation that we observe in our experiments confirm our hypothesis that good performance on the ranking task does not necessarily translate to good performance on the actual completion task. Our results motivate future work on KB embedding models with better prediction separability and, as a first step in that direction, we propose a simple variant of TransE that encourages thresholding and achieves a significant improvement in classification F1 score relative to the original TransE.

Motivation & Objective

• To address the disconnect between ranking-based KBC evaluation and real-world fact prediction, where models must decide whether to accept or reject new facts.
• To identify shortcomings in current evaluation paradigms that prioritize ranking performance over binary classification accuracy.
• To develop a realistic benchmark that includes queries with no valid answers due to missing entities or type violations.
• To motivate research on KBC models with better prediction separability and intrinsic thresholding capability.
• To evaluate state-of-the-art models under a new classification-focused metric that reflects actual deployment conditions.

Proposed method

• Construct the FB14k-QAQ dataset by replacing entities in KB queries with variables and removing a subset of correct answers to simulate real-world missing data.
• Generate two types of negative queries: (1) queries with no valid answers due to removed entities, and (2) queries violating type constraints.
• Use binary classification metrics—specifically F1 score—on the resulting test sets to evaluate model performance in distinguishing true from false facts.
• Introduce a modified TransE model, called Region, which uses relation-specific diagonal matrices to define elliptical regions in embedding space for better prediction separation.
• Apply a thresholding mechanism based on the Region model’s distance function to convert scores into binary predictions.
• Evaluate models using both global and multiple relation-specific thresholds to assess robustness and generalization across relations.

Experimental results

Research questions

• RQ1 Does high performance on ranking-based metrics like MRR reliably indicate strong performance in real-world knowledge base completion, where models must make binary accept/reject decisions?
• RQ2 How do state-of-the-art KBC models perform when evaluated on a classification-based benchmark that includes queries with no valid answers?
• RQ3 Can a simple modification to TransE that encourages better prediction separability significantly improve classification performance?
• RQ4 To what extent do current scoring functions fail to support consistent thresholding across relations, and how does this affect model reliability?
• RQ5 How does the performance gap between global and multiple threshold settings reflect the scalability and robustness of KBC models?

Key findings

• The Region model, a modified TransE with relation-specific regions, achieves a 32.4% relative improvement in F1 score with a global threshold and a 36.8% improvement with multiple thresholds compared to standard TransE.
• The performance gap between ranking and classification metrics is substantial: models with high MRR values often show poor F1 scores, indicating that ranking success does not imply reliable fact prediction.
• The largest performance gains occur on queries with no valid answers (e.g., missing entities), demonstrating that the new benchmark effectively exposes weaknesses in generalization.
• Models trained under the ranking paradigm often fail to produce well-calibrated scores, making global thresholding difficult and unreliable across relations.
• The results confirm that improving prediction separability—through architectural modifications like the Region model—is crucial for building robust, deployable KBC systems.
• The new evaluation framework reveals that current models are not optimized for the actual task of fact completion, which requires reliable binary decisions rather than mere ranking.