[Paper Review] TransG : A Generative Mixture Model for Knowledge Graph Embedding

Han Xiao, Minlie Huang+2 more|arXiv (Cornell University)|Sep 18, 2015

Advanced Graph Neural NetworksComputer Science23 references81 citations

TL;DR

TransG proposes a generative Bayesian non-parametric mixture model for knowledge graph embedding that addresses multiple relation semantics by learning multiple relation-specific components per relation. By using a Chinese Restaurant Process to dynamically discover latent semantics, TransG achieves state-of-the-art performance, improving by 1.7% on WN11 and 5.8% on FB13 over TransR.

ABSTRACT

Recently, knowledge graph embedding, which projects symbolic entities and relations into continuous vector space, has become a new, hot topic in artificial intelligence. This paper addresses a new issue of multiple relation semantics that a relation may have multiple meanings revealed by the entity pairs associated with the corresponding triples, and proposes a novel Gaussian mixture model for embedding, TransG. The new model can discover latent semantics for a relation and leverage a mixture of relation component vectors for embedding a fact triple. To the best of our knowledge, this is the first generative model for knowledge graph embedding, which is able to deal with multiple relation semantics. Extensive experiments show that the proposed model achieves substantial improvements against the state-of-the-art baselines.

Motivation & Objective

• To formally identify and address the issue of multiple relation semantics in knowledge graphs, where a single relation can express different meanings across entity pairs.
• To develop a generative model capable of automatically discovering and representing these latent semantic components without prior knowledge of the number of components.
• To improve knowledge graph embedding by modeling relations as mixtures of relation-specific vectors rather than single fixed vectors.
• To demonstrate that modeling multiple semantics leads to significant performance gains in link prediction and triple classification tasks.

Proposed method

• Employs a Bayesian non-parametric infinite mixture model using the Chinese Restaurant Process (CRP) to allow an unbounded number of semantic components per relation.
• Models each triple (h, r, t) using a mixture of relation-specific component vectors, where the embedding satisfies h_r + r_i ≈ t_r for component i.
• Uses stochastic gradient ascent (SGA) for efficient optimization of mixture weights π and variance σ, replacing computationally expensive likelihood counting.
• Fixes the variance σ as a constant for numerical stability, while initializing new cluster centers with random vectors rather than ground-truth differences (t - h) to improve generalization.
• Applies a Bernoulli sampling strategy for negative sampling during training to enhance efficiency and convergence.
• Introduces a hierarchical clustering mechanism to group entity pairs with similar semantics under the same component, enabling semantic disambiguation.

Experimental results

Research questions

• RQ1 Can a generative model effectively capture multiple latent semantics for a single relation in knowledge graphs?
• RQ2 Does modeling relations as a mixture of components improve performance in knowledge graph embedding compared to single-vector models?
• RQ3 How many semantic components do relations typically have, and do they correspond to interpretable meanings?
• RQ4 Can the model automatically discover the correct number of semantic components without manual specification?
• RQ5 To what extent does capturing multiple semantics improve link prediction and triple classification accuracy?

Key findings

• TransG achieves 87.4% accuracy on the WN11 dataset and 87.3% on FB13, outperforming all baselines, including TransR (85.9% and 82.5%) and TransG-Hierarchical (85.4% and 85.3%).
• The model improves by 1.7 percentage points on WN11 and 5.8 percentage points on FB13 compared to TransR, demonstrating the benefit of modeling multiple semantics.
• Relations with more semantic components—such as "Profession" and "Type Of"—show greater performance gains, while relations with only one component (e.g., "Similar") show minimal improvement.
• The number of semantic components per relation varies: WN11 has an average of 2.63 components, FB13 has 4.53, with most relations (except "Also See", "Synset Usage", "Gender") having more than one component.
• Visualizations confirm that different components correspond to distinct semantic meanings, such as "HasPart.1" for composition and "HasPart.2" for location.
• The model converges faster in the final version, training for only 2,000 epochs instead of 10,000, due to improved optimization and learning rate scheduling.