[Paper Review] Negative Sampling in Knowledge Graph Representation Learning: A Review

Tiroshan Madushanka, Ryutaro Ichise|arXiv (Cornell University)|Feb 29, 2024

Artificial Intelligence in Healthcare7 citations

TL;DR

A systematic survey of negative sampling methods for knowledge graph representation learning, categorizing static and dynamic approaches into five groups and outlining open research challenges.

ABSTRACT

Knowledge Graph Representation Learning (KGRL), or Knowledge Graph Embedding (KGE), is essential for AI applications such as knowledge construction and information retrieval. These models encode entities and relations into lower-dimensional vectors, supporting tasks like link prediction and recommendation systems. Training KGE models relies on both positive and negative samples for effective learning, but generating high-quality negative samples from existing knowledge graphs is challenging. The quality of these samples significantly impacts the model's accuracy. This comprehensive survey paper systematically reviews various negative sampling (NS) methods and their contributions to the success of KGRL. Their respective advantages and disadvantages are outlined by categorizing existing NS methods into six distinct categories. Moreover, this survey identifies open research questions that serve as potential directions for future investigations. By offering a generalization and alignment of fundamental NS concepts, this survey provides valuable insights for designing effective NS methods in the context of KGRL and serves as a motivating force for further advancements in the field.

Motivation & Objective

• Explain why negative sampling is critical for KGRL and how sample quality affects embeddings.
• Provide a structured taxonomy of negative sampling methods across static, dynamic, and auxiliary data-based categories.
• Analyze the properties (efficiency, effectiveness, stability, independence, quality) of NS methods.
• Summarize open research questions and prospective directions in NS for KGRL.

Proposed method

• Conduct a PRISMA-guided systematic literature review of NS methods in KGRL across multiple databases and ArXiv.
• Propose a five-category taxonomy of NS methods (Static NS, Dynamic NS, External Model-Based, Auxiliary Data-Based, Mixing NS).
• Detail the KGRL framework with positive/negative sampling grounded in Noise Contrastive Estimation (NCE) and typical scoring functions.
• Define standard notations and concepts for quality negatives and NS notations.
• Evaluate NS methods along five dimensions: efficiency, effectiveness, stability, independence, and quality.

Experimental results

Research questions

• RQ1 What are the main categories and subcategories of negative sampling methods used in KGRL?
• RQ2 How do static and dynamic NS approaches compare in terms of efficiency, effectiveness, and quality?
• RQ3 What are the open research challenges and future directions in negative sampling for knowledge graph embeddings?
• RQ4 How does the NCE framework apply to KGRL and influence negative sample generation?

Key findings

• The paper provides a comprehensive taxonomy of NS methods, grouped into five categories and multiple subcategories.
• Negative sampling quality significantly influences KGE training and downstream tasks, with trade-offs among efficiency, stability, and independence.
• A formal framework and notations standardize NS discussion, including quality negatives and sampling from S̄(h,r,t).
• The PRISMA-based review process yielded 47 relevant papers across venues and arXiv, outlining historical and contemporary NS techniques.
• The paper discusses the General KGRL framework with objective formulations for both translational distance and semantic matching models.
• It identifies that no single NS method fully satisfies all desirable properties (efficiency, effectiveness, stability, independence, quality), underscoring trade-offs.