[Paper Review] Adaptive Margin Ranking Loss for Knowledge Graph Embeddings via a Correntropy Objective Function

Mojtaba Nayyeri, Xiaotian Zhou+3 more|arXiv (Cornell University)|Jul 9, 2019

Advanced Graph Neural Networks20 references4 citations

TL;DR

This paper proposes Adaptive Margin Loss (AML), a novel loss function for knowledge graph embedding models that automatically adjusts the margin during training using a correntropy-based objective. By dynamically expanding the margin around a learnable center, AML eliminates the need for manual hyperparameter tuning of upper and lower bounds, significantly improving link prediction performance on FB15k and WN18, with Hit@10 scores of 77.7% and 95.2%, respectively.

ABSTRACT

Translation-based embedding models have gained significant attention in link prediction tasks for knowledge graphs. TransE is the primary model among translation-based embeddings and is well-known for its low complexity and high efficiency. Therefore, most of the earlier works have modified the score function of the TransE approach in order to improve the performance of link prediction tasks. Nevertheless, proven theoretically and experimentally, the performance of TransE strongly depends on the loss function. Margin Ranking Loss (MRL) has been one of the earlier loss functions which is widely used for training TransE. However, the scores of positive triples are not necessarily enforced to be sufficiently small to fulfill the translation from head to tail by using relation vector (original assumption of TransE). To tackle this problem, several loss functions have been proposed recently by adding upper bounds and lower bounds to the scores of positive and negative samples. Although highly effective, previously developed models suffer from an expansion in search space for a selection of the hyperparameters (in particular the upper and lower bounds of scores) on which the performance of the translation-based models is highly dependent. In this paper, we propose a new loss function dubbed Adaptive Margin Loss (AML) for training translation-based embedding models. The formulation of the proposed loss function enables an adaptive and automated adjustment of the margin during the learning process. Therefore, instead of obtaining two values (upper bound and lower bound), only the center of a margin needs to be determined. During learning, the margin is expanded automatically until it converges. In our experiments on a set of standard benchmark datasets including Freebase and WordNet, the effectiveness of AML is confirmed for training TransE on link prediction tasks.

Motivation & Objective

• To address the sensitivity of translation-based knowledge graph embedding models to manually tuned margin hyperparameters.
• To eliminate the need for extensive hyperparameter search in setting upper and lower bounds for positive and negative triple scores.
• To develop a loss function that automatically adapts the margin during training to improve model generalization and performance.
• To enhance the robustness and effectiveness of TransE and similar models in link prediction tasks through a more adaptive optimization process.

Proposed method

• Proposes a correntropy-based objective function that models the margin as a probabilistic distribution centered on a learnable parameter.
• Introduces a slack variable that dynamically expands the margin during training, ensuring positive triples are scored sufficiently low.
• Replaces fixed margin bounds in traditional loss functions with a single learnable center parameter, reducing hyperparameter search space.
• Uses a differentiable formulation of the loss that enables end-to-end training with standard optimizers like Adam and Adagrad.
• Applies the loss function to TransE, resulting in TransEAML, a variant that automatically adjusts margin width during optimization.
• Employs a symmetric margin adjustment strategy, adding the same slack to both positive and negative sample scores to maintain robustness.

Experimental results

Research questions

• RQ1 Can a loss function be designed to automatically adapt the margin during training, reducing reliance on manual hyperparameter tuning?
• RQ2 Does an adaptive margin mechanism improve the performance of translation-based knowledge graph embedding models on standard benchmarks?
• RQ3 How does the proposed correntropy-based objective compare to fixed-margin and bounded-score loss functions in terms of link prediction accuracy?
• RQ4 To what extent does the reduction in hyperparameter search space improve training efficiency and model stability?

Key findings

• TransEAML achieves a Hit@10 score of 95.2% on the WN18 benchmark, significantly outperforming TransE with Margin Ranking Loss (89.2%) and Limited-Score Loss (93.7%).
• On FB15k, TransEAML reaches 77.7% Hit@10, compared to 47.1% for TransE with MRL and 72.3% for Limited-Score Loss, demonstrating a substantial improvement.
• The proposed AML loss function reduces the need for manual tuning of upper and lower score bounds, simplifying hyperparameter selection.
• The use of a correntropy-based objective enables automatic margin expansion during training, leading to better separation of positive and negative triples.
• Empirical results confirm that the adaptive margin mechanism enhances model generalization and robustness across multiple standard knowledge graph benchmarks.
• The method maintains high performance with minimal hyperparameter search, using only the center of the margin as a tunable parameter.