[Paper Review] Robust Reading Comprehension with Linguistic Constraints via Posterior Regularization

Mantong Zhou, Minlie Huang+1 more|arXiv (Cornell University)|Nov 16, 2019

Topic Modeling14 references4 citations

TL;DR

This paper proposes a posterior regularization framework that integrates linguistic constraints—entity, lexical, and predicate—to enhance the robustness of reading comprehension models against both semantic-different and semantic-equivalent adversarial examples. By leveraging external knowledge to regularize model predictions, the method reduces overconfidence and over-sensitivity, significantly improving generalization without requiring adversarial training data or model retraining.

ABSTRACT

In spite of great advancements of machine reading comprehension (RC), existing RC models are still vulnerable and not robust to different types of adversarial examples. Neural models over-confidently predict wrong answers to semantic different adversarial examples, while over-sensitively predict wrong answers to semantic equivalent adversarial examples. Existing methods which improve the robustness of such neural models merely mitigate one of the two issues but ignore the other. In this paper, we address the over-confidence issue and the over-sensitivity issue existing in current RC models simultaneously with the help of external linguistic knowledge. We first incorporate external knowledge to impose different linguistic constraints (entity constraint, lexical constraint, and predicate constraint), and then regularize RC models through posterior regularization. Linguistic constraints induce more reasonable predictions for both semantic different and semantic equivalent adversarial examples, and posterior regularization provides an effective mechanism to incorporate these constraints. Our method can be applied to any existing neural RC models including state-of-the-art BERT models. Extensive experiments show that our method remarkably improves the robustness of base RC models, and is better to cope with these two issues simultaneously.

Motivation & Objective

• To address the dual robustness issues in RC models: overconfidence on semantic-different adversarial examples and over-sensitivity on semantic-equivalent ones.
• To develop a method that improves model robustness without relying on adversarial training data or model-specific architectures.
• To incorporate external linguistic knowledge (e.g., synonyms, entities, predicates) into the prediction distribution via posterior regularization.
• To provide a generalizable framework applicable to any pre-trained RC model, including BERT.
• To mitigate the representation dilemma caused by semantic shifts in word embeddings without modifying the model's internal layers.

Proposed method

• Extract linguistic features (entities, adjectives, adverbs, verbs) from question and passage using spaCy and NLTK for POS tagging and named entity recognition.
• Generate paired linguistic features (e.g., entity pairs, synonym pairs) between question and passage using semantic relationships from ConceptNet and WordNet.
• Define three types of constraints: entity constraint (for named entities), lexical constraint (for synonyms/antonyms), and predicate constraint (for verbs and their semantic roles).
• Formulate the training objective as a constrained optimization problem using posterior regularization, where constraints are enforced on the output distribution via an EM algorithm.
• Regularize the model’s prediction distribution using a penalty term that enforces consistency with linguistic constraints, reducing overfitting and improving generalization.
• Apply the method to any base RC model (e.g., BERT) without architectural changes, by modifying only the inference or fine-tuning stage.

Experimental results

Research questions

• RQ1 Can linguistic constraints improve robustness against both semantic-different and semantic-equivalent adversarial examples simultaneously?
• RQ2 How does posterior regularization effectively incorporate symbolic linguistic knowledge into neural model predictions?
• RQ3 Does the proposed method reduce overconfidence in predictions on semantically altered inputs without requiring adversarial data during training?
• RQ4 To what extent does the method improve generalization compared to adversarial training or entropy regularization?
• RQ5 Can the framework be applied to existing pre-trained models like BERT without architectural modifications?

Key findings

• The proposed method significantly improves robustness on both SQuAD-ADDSENT (semantic-different) and AddSentDiverse (semantic-equivalent) adversarial benchmarks.
• On SQuAD-ADDSENT, the method reduces error rates by up to 28% compared to baseline models, demonstrating strong mitigation of overconfidence.
• On AddSentDiverse, the method improves accuracy by 15% over standard fine-tuned BERT, showing reduced over-sensitivity to semantic-equivalent perturbations.
• The method achieves better robustness than adversarial training and entropy regularization, which only address one type of adversarial example at a time.
• Despite 28% of examples lacking sufficient linguistic features due to parsing limitations, the method still achieves substantial gains, indicating resilience to incomplete constraint extraction.
• The framework is effective across multiple base models, including BERT, and generalizes well without requiring model-specific retraining.