[Paper Review] Probing What Different NLP Tasks Teach Machines about Function Word Comprehension

Najoung Kim, Roma Patel+10 more|arXiv (Cornell University)|Apr 25, 2019

Topic Modeling45 references4 citations

TL;DR

This paper introduces nine targeted probing tasks to evaluate how well NLP models understand function words—such as prepositions, wh-words, and negations—by structurally mutating sentences from existing datasets. Using these tasks, it compares pretraining objectives (e.g., language modeling, NLI, CCG supertagging) and finds that language modeling performs best on average, while NLI enhances negation understanding and CCG supertagging improves sentence boundary detection, revealing distinct linguistic inductive biases across objectives.

ABSTRACT

We introduce a set of nine challenge tasks that test for the understanding of function words. These tasks are created by structurally mutating sentences from existing datasets to target the comprehension of specific types of function words (e.g., prepositions, wh-words). Using these probing tasks, we explore the effects of various pretraining objectives for sentence encoders (e.g., language modeling, CCG supertagging and natural language inference (NLI)) on the learned representations. Our results show that pretraining on language modeling performs the best on average across our probing tasks, supporting its widespread use for pretraining state-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably. Overall, no pretraining objective dominates across the board, and our function word probing tasks highlight several intuitive differences between pretraining objectives, e.g., that NLI helps the comprehension of negation.

Motivation & Objective

• To investigate how different pretraining objectives shape sentence representations with respect to function word comprehension.
• To isolate the effects of pretraining objectives by holding model architecture constant, enabling controlled comparison.
• To address the underrepresentation of function words in probing literature despite their critical role in compositional meaning.
• To release high-quality, linguistically curated probing datasets for future research on function word understanding.

Proposed method

• The authors create nine probing tasks by applying targeted structural mutations to sentences from existing datasets, focusing on specific function word types (e.g., prepositions, negation, quantifiers).
• Each task evaluates whether a model can correctly identify semantic changes induced by minimal syntactic or lexical modifications.
• The probing tasks are designed as sentence pair classification problems, where models must judge whether a modified sentence preserves the meaning of the original.
• The study evaluates multiple pretraining objectives—language modeling, CCG supertagging, and natural language inference—on the same model architecture.
• Performance is measured across all nine tasks to compare the linguistic knowledge encoded by each pretraining objective.
• The authors conduct ablation studies to assess the impact of data size and vocabulary overlap between pretraining and probing datasets.

Experimental results

Research questions

• RQ1 How do different pretraining objectives affect a model’s ability to comprehend function words such as prepositions, wh-words, and negations?
• RQ2 Which pretraining objective leads to the most robust function word understanding across diverse linguistic phenomena?
• RQ3 To what extent do linguistic inductive biases from pretraining objectives shape the representation of function words in sentence encoders?
• RQ4 Are there systematic differences in how models handle specific function word types (e.g., spatial vs. negation) depending on the pretraining objective?
• RQ5 Does increasing pretraining data size or vocabulary overlap consistently improve function word probing performance?

Key findings

• Language modeling pretraining achieves the best average performance across all nine probing tasks, supporting its dominance in current NLP practice.
• Natural language inference (NLI) pretraining significantly enhances models’ ability to comprehend negation, particularly in complex negation patterns.
• CCG supertagging pretraining improves the detection of meaningful sentence boundaries, suggesting it captures syntactic structure more effectively.
• No single pretraining objective outperforms all others across all function word types, indicating complementary strengths among objectives.
• Vocabulary overlap between pretraining and probing datasets is not a significant predictor of performance, suggesting that performance differences are due to linguistic inductive biases rather than lexical exposure.
• Data size during pretraining does not consistently improve probing performance, and in some cases, larger datasets can lead to worse performance, indicating potential negative inductive biases.