[Paper Review] Contextual Aware Joint Probability Model Towards Question Answering System

Yang Liu, Lijing Song|arXiv (Cornell University)|Apr 17, 2019

Topic Modeling9 references4 citations

TL;DR

This paper proposes a hybrid BERT-BiDAF model with a novel joint probability predictor to improve question answering on SQuAD 2.0, explicitly modeling the joint posterior of answerability and span prediction. The approach achieves an F1 score of 75.842% and EM score of 72.24% on the test leaderboard, demonstrating improved reasoning over standard methods by avoiding artificial 'no-answer' tokens.

ABSTRACT

In this paper, we address the question answering challenge with the SQuAD 2.0 dataset. We design a model architecture which leverages BERT's capability of context-aware word embeddings and BiDAF's context interactive exploration mechanism. By integrating these two state-of-the-art architectures, our system tries to extract the contextual word representation at word and character levels, for better comprehension of both question and context and their correlations. We also propose our original joint posterior probability predictor module and its associated loss functions. Our best model so far obtains F1 score of 75.842% and EM score of 72.24% on the test PCE leaderboad.

Motivation & Objective

• To address the challenge of distinguishing answerable from unanswerable questions in SQuAD 2.0, which contains 50,000 unanswerable questions designed to mimic answerable ones.
• To overcome limitations of existing models that rely on artificial 'no-answer' tokens and implicit assumptions about answer span prediction.
• To develop a joint probability framework that models the causal relationship between answerability and span prediction in a more natural, probabilistically consistent way.
• To investigate whether combining BERT’s word-level contextual embeddings with BiDAF’s character-level attention mechanisms enhances representation learning and model performance.

Proposed method

• Integrates BERT’s contextual word embeddings with BiDAF’s bidirectional attention mechanism to create a hybrid encoder that captures both word-level and character-level context-aware representations.
• Proposes a joint posterior probability predictor that models the joint distribution of the binary answerable variable A and the start/end span variables X₁ and X₂.
• Designs a custom loss function that aligns with the joint probability structure, avoiding the limitations of standard span prediction loss functions.
• Uses the attention flow layer from BiDAF to generate question-aware context representations at the character level, enhancing interaction between question and context.
• Combines BERT’s word-level representations with character-level features from BiDAF to enable mutual enhancement of representations at different granularities.
• Avoids inserting a special 'no-answer' token; instead, the model learns to predict answerability and span simultaneously through probabilistic inference.

Experimental results

Research questions

• RQ1 Can a joint probability model that explicitly models the causal dependency of answerability on span existence outperform models that treat answerability as a separate classification head?
• RQ2 How does combining BERT’s word-level contextual embeddings with BiDAF’s character-level attention improve representation learning for question answering?
• RQ3 Can a custom loss function designed for joint probability prediction better reflect model performance than standard entropy-based loss functions in SQuAD 2.0?
• RQ4 To what extent do mutual enhancements between word-level and character-level representations improve generalization on unanswerable questions?
• RQ5 Does avoiding artificial 'no-answer' tokens lead to more robust and interpretable predictions in open-domain question answering?

Key findings

• The proposed model achieves an F1 score of 75.842% and an EM score of 72.24% on the SQuAD 2.0 test leaderboard, demonstrating strong performance on a challenging benchmark.
• The joint probability predictor with a custom loss function enables more coherent and causally consistent reasoning between answerability and span prediction compared to standard approaches.
• Ablation studies confirm that the integration of BERT and BiDAF features leads to improved performance, validating the mutual enhancement of multi-granularity representations.
• The model reduces false positives on unanswerable questions by better modeling the absence of a valid span, especially in cases where distractors like 'The Men in Black' mislead attention mechanisms.
• The model exhibits fewer false negatives compared to baseline models, particularly when key terms like 'status' are absent but contextually implied.
• Despite strong performance, the model still faces challenges with ambiguous contexts and imprecise answer boundaries, indicating room for improvement in span localization and context understanding.