[Paper Review] Leveraging Lexical Resources for Learning Entity Embeddings in Multi-Relational Data

Teng Long, Ryan Lowe+2 more|arXiv (Cornell University)|May 18, 2016

Topic Modeling21 references3 citations

TL;DR

This paper proposes initializing entity embeddings in multi-relational models using lexical descriptions from external resources, combined with pre-trained word embeddings, to improve performance and convergence. Applied to TransE, this method achieves a new state-of-the-art mean rank of 51 on WordNet—down from 212—while also accelerating training convergence.

ABSTRACT

Recent work in learning vector-space embeddings for multi-relational data has focused on combining relational information derived from knowledge bases with distributional information derived from large text corpora. We propose a simple approach that leverages the descriptions of entities or phrases available in lexical resources, in conjunction with distributional semantics, in order to derive a better initialization for training relational models. Applying this initialization to the TransE model results in significant new state-of-the-art performances on the WordNet dataset, decreasing the mean rank from the previous best of 212 to 51. It also results in faster convergence of the entity representations. We find that there is a trade-off between improving the mean rank and the hits@10 with this approach. This illustrates that much remains to be understood regarding performance improvements in relational models.

Motivation & Objective

• To address the limitation of existing relational models that rely on distributional word embeddings which often lack coverage for rare or domain-specific entities.
• To improve initialization of entity embeddings in relational models by incorporating descriptive text from lexical resources such as WordNet and Freebase.
• To evaluate whether lexical descriptions can enhance both model performance and training speed in multi-relational knowledge graph embedding tasks.
• To investigate the trade-off between mean rank and hits@10 as evaluation metrics in relational model performance.
• To demonstrate that even common entities benefit from better initialization using lexical semantics, not just rare ones.

Proposed method

• Obtain entity descriptions from lexical resources such as WordNet and Freebase, which typically include short definitions or descriptive sentences.
• Use pre-trained word embeddings (e.g., Word2Vec, GloVe) to convert each word in the entity description into a vector representation.
• Aggregate the word vectors in each description using simple averaging to form a dense vector representation for the entity.
• Use these aggregated vectors as the initial entity embeddings in the TransE model, replacing random initialization.
• Train the TransE model end-to-end using the initialized embeddings, leveraging both structural relations and lexical semantics.
• Apply filtering during evaluation to ensure fair comparison with prior work, especially on hits@10 and mean rank metrics.

Experimental results

Research questions

• RQ1 Can entity descriptions from lexical resources improve the quality of learned entity embeddings in multi-relational knowledge graphs?
• RQ2 Does using lexical descriptions as initialization lead to faster convergence during training of relational models?
• RQ3 How does the performance of the model vary between mean rank and hits@10 when using lexical-based initialization?
• RQ4 Why does the method show strong gains on WordNet but only marginal improvements on Freebase, despite similar initialization?
• RQ5 To what extent do lexical descriptions benefit both rare and common entities in relational modeling?

Key findings

• The proposed method achieves a new state-of-the-art mean rank of 51 on the WordNet dataset, a significant improvement from the previous best of 212.
• The model converges faster during training when initialized with lexical descriptions, especially on the WordNet dataset.
• On the Freebase dataset, the method leads to faster convergence but only marginal improvements in mean rank, suggesting differences in optimization landscape or model capacity.
• A trade-off is observed between mean rank and hits@10: improvements in mean rank often come at the cost of reduced hits@10, indicating conflicting optimization objectives.
• The method reduces the occurrence of 'disaster' cases where correct entities are ranked very low, which disproportionately affects mean rank.
• The results suggest that mean rank is a more reliable metric than hits@10 for evaluating relational models on raw data, especially when relations are sparse.