[Paper Review] Gaussian Attention Model and Its Application to Knowledge Base Embedding and Question Answering

Liwen Zhang, John Winn+1 more|arXiv (Cornell University)|Nov 7, 2016

Topic ModelingComputer Science21 references20 citations

TL;DR

This paper proposes the Gaussian Attention Model, a differentiable content-based attention mechanism that uses multivariate Gaussian likelihood to score memory items, enabling flexible attention control from sharp to broad focus. It improves knowledge base embedding and question answering by naturally modeling uncertainty in relation composition and conjunction queries, achieving state-of-the-art performance on path and conjunctive queries in a FIFA World Cup 2014 knowledge base.

ABSTRACT

We propose the Gaussian attention model for content-based neural memory access. With the proposed attention model, a neural network has the additional degree of freedom to control the focus of its attention from a laser sharp attention to a broad attention. It is applicable whenever we can assume that the distance in the latent space reflects some notion of semantics. We use the proposed attention model as a scoring function for the embedding of a knowledge base into a continuous vector space and then train a model that performs question answering about the entities in the knowledge base. The proposed attention model can handle both the propagation of uncertainty when following a series of relations and also the conjunction of conditions in a natural way. On a dataset of soccer players who participated in the FIFA World Cup 2014, we demonstrate that our model can handle both path queries and conjunctive queries well.

Motivation & Objective

• To develop a more flexible attention mechanism that can control focus from sharp to broad, improving neural memory access in knowledge base reasoning.
• To enable effective modeling of uncertainty when traversing multiple relations in knowledge graphs.
• To support natural handling of conjunctive queries (e.g., 'who is from Germany and plays forward?') in a differentiable, compositional framework.
• To improve question answering over knowledge bases by integrating Gaussian scoring into entity embedding and query reasoning.
• To demonstrate the model's superiority on complex queries involving path traversal and logical conjunctions.

Proposed method

• The model uses a multivariate Gaussian likelihood as a scoring function: score = -0.5*(v - μ)ᵀΣ⁻¹(v - μ) + const, where μ and Σ are context-dependent mean and covariance.
• The attention mechanism is content-based, with the context (e.g., query or relation) determining the Gaussian parameters (μ, Σ), allowing dynamic focus control.
• Knowledge base entities are embedded in a continuous vector space using this scoring function, forming a TransGaussian model analogous to TransE but with uncertainty-aware relations.
• Relations are modeled as Gaussian transformations: mean shift and variance addition, enabling compositional reasoning through Gaussian convolution.
• The model supports both path queries (e.g., 'who plays for a club that has a player from Germany?') and conjunctive queries (e.g., 'who is from Germany and plays forward?') via Gaussian-based conjunction and composition.
• Training uses end-to-end optimization with differentiable attention, allowing gradient-based learning of both entity embeddings and relation parameters.

Experimental results

Research questions

• RQ1 Can a differentiable attention mechanism that models uncertainty via Gaussian distributions improve reasoning over knowledge bases?
• RQ2 How well can a Gaussian attention model handle complex queries involving composition of relations (path queries) and conjunction of conditions?
• RQ3 Does the ability to modulate attention spread (from sharp to broad) enhance generalization and robustness in knowledge base question answering?
• RQ4 Can Gaussian-based scoring outperform standard inner-product attention in modeling semantic proximity and uncertainty in sparse knowledge bases?
• RQ5 To what extent does the model benefit from compositional training and end-to-end learning in complex reasoning tasks?

Key findings

• The Gaussian Attention Model achieved 98.81% accuracy on conjunctive queries, significantly outperforming baseline models on this complex query type.
• On path queries, the model reached 85.94% accuracy, demonstrating strong performance in multi-hop reasoning over knowledge graphs.
• For simple questions, the model achieved 98.77% accuracy on 'who plays forward for FC Barcelona?', showing high precision on atomic facts.
• The model handled uncertainty naturally during relation composition, with performance remaining robust even in complex, multi-step queries.
• The model outperformed strong baselines on all query types, particularly excelling in conjunction and path queries where inner-product attention struggled.
• The ablation study confirmed that the Gaussian attention mechanism's ability to control attention spread was critical for handling diverse query types effectively.