[Paper Review] Question Answering with Subgraph Embeddings

Antoine Bordes, Sumit Chopra+1 more|arXiv (Cornell University)|Jun 14, 2014

Topic ModelingComputer Science12 references119 citations

TL;DR

This paper proposes a question answering system that uses subgraph embeddings to represent knowledge base paths and answer candidates, enabling accurate, end-to-end learning without hand-crafted features. By learning low-dimensional vector representations of words and KB entities and scoring questions against structured subgraph paths, the model achieves state-of-the-art performance on WebQuestions, outperforming prior methods with F1 scores of 43.2% (Yao) and 41.8% (ensemble).

ABSTRACT

This paper presents a system which learns to answer questions on a broad range of topics from a knowledge base using few hand-crafted features. Our model learns low-dimensional embeddings of words and knowledge base constituents; these representations are used to score natural language questions against candidate answers. Training our system using pairs of questions and structured representations of their answers, and pairs of question paraphrases, yields competitive results on a competitive benchmark of the literature.

Motivation & Objective

• To develop a scalable, end-to-end question answering system that requires minimal human-designed components such as lexicons, grammars, or part-of-speech taggers.
• To improve upon prior embedding-based models by incorporating richer structural information from knowledge base subgraphs around candidate answers.
• To enable the model to handle complex questions involving multi-hop reasoning paths, not just direct entity connections.
• To achieve competitive performance on open-domain QA benchmarks using only question-answer pairs and a structured knowledge base for supervision.
• To demonstrate that subgraph-level representations significantly improve answer ranking accuracy compared to simpler path or single-entity representations.

Proposed method

• The model learns joint low-dimensional embeddings for words and knowledge base constituents (entities and relations) using a siamese neural network architecture.
• For each question, candidate answers are generated using a heuristic set selection strategy (C2), which prioritizes relation types likely to appear in the question and includes top 10 relation types and 2-hop paths.
• The answer representation is enriched by encoding the local subgraph structure around the candidate answer, including the path from the question’s entity to the answer and surrounding relations.
• The model scores each candidate answer by computing a similarity score between the question embedding and the answer’s subgraph embedding using a learned scoring function.
• Multiple answers are handled by averaging the embeddings of all entities on the same path, enabling the model to predict sets of answers for questions like 'Who are David Beckham’s children?'
• Training is performed using weak supervision: question-answer pairs and question paraphrase pairs, avoiding the need for expensive semantic parsing annotations.

Experimental results

Research questions

• RQ1 Can a neural network model learn to answer complex open-domain questions using only question-answer pairs and a knowledge base, without relying on hand-crafted rules or linguistic features?
• RQ2 Does incorporating subgraph structure around candidate answers improve answer selection performance compared to simple path or entity-only representations?
• RQ3 Can a model trained with weak supervision (paraphrase pairs and QA pairs) achieve competitive results on a benchmark like WebQuestions without full supervision?
• RQ4 How does the choice of candidate answer set (e.g., 1-hop vs. 2-hop vs. all 2-hops) affect the model’s performance and inference efficiency?
• RQ5 Can subgraph embeddings enable the model to generalize to questions requiring multi-hop reasoning beyond direct entity connections?

Key findings

• The proposed subgraph embedding model achieves an F1 score of 43.2% on the WebQuestions test set using the F1 (Yao) metric, outperforming prior state-of-the-art methods including [14] and [1].
• The model’s performance improves significantly when using the C2 candidate selection strategy (top 10 relation types and 2-hop paths), compared to using only 1-hop answers (C1), which drops F1 to 31.3%.
• Using all 2-hop candidates as the answer set degrades performance to 37.1% F1 (Yao), indicating that excessive candidate diversity harms ranking accuracy and model confidence.
• The model with subgraph representation and C2 strategy achieves 40.4% P@1 and 39.2% F1 (Berant), outperforming the baseline model of [5] by over 10 percentage points in F1.
• An ensemble model combining this approach with [2] achieves 45.7% F1 (Yao), indicating complementary strengths and validating the model’s distinct design and generalization capability.
• The ablation study confirms that the subgraph representation is critical: replacing it with a simple path representation reduces performance to 36.2% F1 (Yao), showing that local context enhances reasoning.