[Paper Review] Neural Multi-Step Reasoning for Question Answering on Semi-Structured Tables

Till Haug, Octavian-Eugen Ganea+1 more|arXiv (Cornell University)|Feb 21, 2017

Topic Modeling19 references4 citations

TL;DR

This paper proposes a weakly supervised neural network model for multi-step reasoning on semi-structured tables, translating natural language questions into interpretable paraphrases of logical forms and ranking them via joint embedding with CNNs and bilinear layers. The method achieves a state-of-the-art precision@1 of 38.7% on the WikiTableQuestions dataset using an ensemble of 15 models.

ABSTRACT

Advances in natural language processing tasks have gained momentum in recent years due to the increasingly popular neural network methods. In this paper, we explore deep learning techniques for answering multi-step reasoning questions that operate on semi-structured tables. Challenges here arise from the level of logical compositionality expressed by questions, as well as the domain openness. Our approach is weakly supervised, trained on question-answer-table triples without requiring intermediate strong supervision. It performs two phases: first, machine understandable logical forms (programs) are generated from natural language questions following the work of [Pasupat and Liang, 2015]. Second, paraphrases of logical forms and questions are embedded in a jointly learned vector space using word and character convolutional neural networks. A neural scoring function is further used to rank and retrieve the most probable logical form (interpretation) of a question. Our best single model achieves 34.8% accuracy on the WikiTableQuestions dataset, while the best ensemble of our models pushes the state-of-the-art score on this task to 38.7%, thus slightly surpassing both the engineered feature scoring baseline, as well as the Neural Programmer model of [Neelakantan et al., 2016].

Motivation & Objective

• To address complex, multi-compositional questions on semi-structured tables without expensive gold logical form annotations.
• To improve reasoning over tables by generating human-readable paraphrases of logical forms for better interpretability and model performance.
• To eliminate the need for hand-crafted features in candidate ranking by learning end-to-end neural representations.
• To achieve state-of-the-art performance on the WikiTableQuestions benchmark using weak supervision from question-answer-table triples.
• To investigate the impact of different neural architectures and components on reasoning accuracy in table-based QA.

Proposed method

• Candidate logical forms are generated using the method of [8], representing questions as executable programs in Lambda DCS format.
• Each logical form is recursively converted into a natural language paraphrase for interpretability and embedding compatibility.
• Joint sentence embeddings are learned using two parallel CNNs over word and character-level embeddings for questions and paraphrases.
• A bilinear interaction layer combines question and paraphrase embeddings, followed by a fully connected layer for similarity scoring.
• The highest-scoring paraphrase is selected and executed on the input table to retrieve the answer.
• Model ensembles and ablation studies are used to optimize performance and analyze component contributions.

Experimental results

Research questions

• RQ1 Can a weakly supervised neural model achieve strong performance on multi-step reasoning over semi-structured tables without gold logical form annotations?
• RQ2 Does paraphrasing logical forms improve model performance and interpretability compared to direct use of formal expressions?
• RQ3 How do different neural architectures (CNNs vs RNNs) and interaction mechanisms (bilinear vs fully connected) affect reasoning accuracy?
• RQ4 What is the contribution of key components such as character embeddings, dropout, and pre-trained word vectors to model performance?
• RQ5 What types of questions are most challenging for the model, and why?

Key findings

• An ensemble of 15 CNN-FC-BILIN models achieved a state-of-the-art precision@1 of 38.7% on the WikiTableQuestions dataset at the time of publication.
• The best single model, CNN-FC-BILIN, achieved 34.8% precision@1, demonstrating the effectiveness of combining bilinear and fully connected layers.
• Ablation studies showed that removing paraphrasing reduced performance to 33.1%, confirming its importance for model accuracy.
• The use of pre-trained GloVe embeddings improved performance compared to random initialization, which dropped to 32.4%.
• RNN-based sentence encoders underperformed CNNs, with RNN-FC-BILIN achieving only 29.6% precision@1.
• Common errors arose from incorrect logical form generation and poor ranking, especially for questions relying on table structure or subtle semantic differences like 'smaller than' vs 'smaller equals'.