[Paper Review] Analysis of the Impact of Negative Sampling on Link Prediction in Knowledge Graphs

Bhushan Kotnis, Vivi Năstase|arXiv (Cornell University)|Aug 22, 2017

Advanced Graph Neural Networks28 references63 citations

TL;DR

This paper empirically studies how different negative sampling strategies affect link prediction accuracy across four KG embedding models (ComplEx, DistMult, Rescal, TransE) on FB15k and WN18, proposing two new sampling methods and showing dataset-dependent effects.

ABSTRACT

Knowledge graphs are large, useful, but incomplete knowledge repositories. They encode knowledge through entities and relations which define each other through the connective structure of the graph. This has inspired methods for the joint embedding of entities and relations in continuous low-dimensional vector spaces, that can be used to induce new edges in the graph, i.e., link prediction in knowledge graphs. Learning these representations relies on contrasting positive instances with negative ones. Knowledge graphs include only positive relation instances, leaving the door open for a variety of methods for selecting negative examples. In this paper we present an empirical study on the impact of negative sampling on the learned embeddings, assessed through the task of link prediction. We use state-of-the-art knowledge graph embeddings -- escal , TransE, DistMult and ComplEX -- and evaluate on benchmark datasets -- FB15k and WN18. We compare well known methods for negative sampling and additionally propose embedding based sampling methods. We note a marked difference in the impact of these sampling methods on the two datasets, with the "traditional" corrupting positives method leading to best results on WN18, while embedding based methods benefiting the task on FB15k.

Motivation & Objective

• Motivate and understand how negative sampling choices impact KG embedding quality for link prediction.
• Compare state-of-the-art embedding models under diverse negative sampling regimes.
• Propose two new negative sampling strategies and evaluate their effectiveness.
• Assess how dataset characteristics (FB15k vs. WN18) influence method performance.
• Provide guidance on choosing negative sampling strategies for a given KG and model.

Proposed method

• Reimplemented Rescal, TransE, DistMult, and ComplEx in PyTorch with a shared max-margin loss and unit-norm constraints.
• Evaluated multiple negative sampling schemes: Random, Corrupting positives, Typed, Relational, Nearest Neighbor, Near Miss, and a pre-trained embedding based sampler.
• Introduced two novel sampling methods (Nearest Neighbor and Near Miss) that use a pre-trained negative-sampling model to generate difficult negatives.
• Used dataset-specific training setups with FB15k and WN18, including hyperparameter tuning via grid search and early stopping.
• Assessed performance using MRR and Hits@K (Hits@10 for FB15k, Hits@1 for WN18).
• Compared results against prior work using max-margin loss and analyzed relation-frequency slices (OOM) to explain performance differences.

Experimental results

Research questions

• RQ1 How do different negative sampling strategies affect link prediction performance across KG embedding models?
• RQ2 Do dataset characteristics (FB15k vs. WN18) change which negative sampling method is most effective?
• RQ3 Do newly proposed sampling methods (Nearest Neighbor, Near Miss) outperform traditional corrupting positives on these datasets?
• RQ4 How do model type and data sparsity interact with negative sampling choices to influence MRR and Hits@K?
• RQ5 What guidelines emerge for selecting negative sampling methods given a KG's properties and chosen embedding model?

Key findings

• Near Miss sampling generally yields strong results on FB15k across several models.
• Embedding-based sampling methods (Nearest Neighbor, Near Miss) are particularly effective on FB15k and help combat data sparsity.
• Corrupting positive triples works best on WN18, especially with Rescal; random sampling can be competitive for some models (e.g., TransE).
• TransE benefits from random negatives due to its 1:1-relations assumption, whereas other models gain from harder negatives.
• FB15k’s data sparsity (many relations with few instances) reduces the effectiveness of corrupt sampling, while WN18’s dense, intrinsic relations favor corruption-based negatives.
• Overall, the best negative sampling strategy is dataset- and model-dependent, with near-miss/nearest-neighbor performing well on Freebase-like data and corruption excelling on WordNet.