[Paper Review] Analogical Inference for Multi-Relational Embeddings

Hanxiao Liu, Yuexin Wu+1 more|arXiv (Cornell University)|May 6, 2017

Advanced Graph Neural Networks42 references193 citations

TL;DR

Introduces ANALOGY, a framework that explicitly models analogical structures in multi-relational embeddings using commuting normal matrices, yielding scalable, differentiable optimization and improved performance over baselines on benchmark datasets.

ABSTRACT

Large-scale multi-relational embedding refers to the task of learning the latent representations for entities and relations in large knowledge graphs. An effective and scalable solution for this problem is crucial for the true success of knowledge-based inference in a broad range of applications. This paper proposes a novel framework for optimizing the latent representations with respect to the extit{analogical} properties of the embedded entities and relations. By formulating the learning objective in a differentiable fashion, our model enjoys both theoretical power and computational scalability, and significantly outperformed a large number of representative baseline methods on benchmark datasets. Furthermore, the model offers an elegant unification of several well-known methods in multi-relational embedding, which can be proven to be special instantiations of our framework.

Motivation & Objective

• Motivate and formalize analogical reasoning in large-scale knowledge graph embedding.
• Define a tractable objective that enforces analogical structures via commutativity and normality of relation maps.
• Develop a scalable algorithm for learning entity and relation representations under these constraints.
• Show how existing embedding models are special cases within the ANALOGY framework.

Proposed method

• Model each relation r as a linear map W_r acting on entity embeddings v_e, scoring triples via φ(s,r,o)=v_s^T W_r v_o.
• Impose that relation maps form a commuting family and are normal, i.e., W_r W_r^T = W_r^T W_r and W_r W_{r'} = W_{r'} W_r for all r,r'.
• Represent commuting normal matrices via joint block-diagonalization with an orthogonal basis Q, enabling an equivalent formulation with sparse almost-diagonal blocks B_r.
• Prove that any solution under the original formulation has an equivalent solution with B_r in a reduced set, reducing per-triple computation to O(m).
• Demonstrate that DistMult, ComplEx, and HolE arise as special cases within the framework.
• Use logistic loss with asynchronous AdaGrad SGD, and generate negative samples by corrupting s, r, or o.

Experimental results

Research questions

• RQ1 How can analogical reasoning be formally incorporated into multi-relational embeddings?
• RQ2 Can the commutativity and normality constraints on relation matrices improve knowledge graph completion?
• RQ3 Is there an equivalent, more efficient formulation of ANALOGY that preserves performance while reducing computation?
• RQ4 How do existing embedding models relate to ANALOGY as special cases, and what does this imply about their empirical effectiveness?

Key findings

• ANALOGY achieves strong performance improvements over a broad set of baselines on FB15K and WN18.
• The framework provides a unified view showing several representative methods as special cases.
• The alternative, block-diagonal formulation reduces computation to O(m) per triple and scales linearly with embedding size.
• HolE and ComplEx can be seen as related constructions within the ANALOGY framework, with HolE equivalent to a complex-valued relaxation of the real-valued formulation.