[Paper Review] Neural Language Priors

Joseph Enguehard, Dan Busbridge+2 more|arXiv (Cornell University)|Oct 4, 2019

Topic Modeling36 references4 citations

TL;DR

This paper investigates the impact of neural network encoder architectures on sentence representation quality, independent of training procedures. It finds that even architectures with strong inductive biases—such as self-attention or graph-based models—yield minimal performance gains over simpler models with weak priors, suggesting architectural inductive bias plays a limited role in downstream task success when training signals are controlled.

ABSTRACT

The choice of sentence encoder architecture reflects assumptions about how a sentence's meaning is composed from its constituent words. We examine the contribution of these architectures by holding them randomly initialised and fixed, effectively treating them as as hand-crafted language priors, and evaluating the resulting sentence encoders on downstream language tasks. We find that even when encoders are presented with additional information that can be used to solve tasks, the corresponding priors do not leverage this information, except in an isolated case. We also find that apparently uninformative priors are just as good as seemingly informative priors on almost all tasks, indicating that learning is a necessary component to leverage information provided by architecture choice.

Motivation & Objective

• To isolate the contribution of encoder architecture to sentence representation quality, independent of training signals.
• To evaluate whether architectures with strong language priors (e.g., self-attention, graph-based models) outperform those with weak or uninformative priors.
• To assess whether architectural inductive biases significantly influence downstream task performance when training procedures are held constant.
• To challenge the assumption that strong inductive biases in neural architectures inherently lead to better generalization in NLP tasks.

Proposed method

• Trained sentence encoders using identical downstream training signals across diverse architectures to isolate architectural effects.
• Compared architectures with strong language priors (e.g., transformers, graph networks) against those with minimal inductive bias (e.g., bag-of-words, simple feedforwards).
• Used fixed training objectives and data splits to ensure performance differences stemmed solely from architecture, not optimization or data.
• Evaluated models on standard NLP benchmarks to measure downstream performance across tasks.
• Applied controlled training protocols to eliminate confounding factors from fine-tuning, data augmentation, or task-specific adaptation.
• Measured performance differences under identical training conditions to assess the relative contribution of architectural bias.

Experimental results

Research questions

• RQ1 Does the choice of encoder architecture significantly affect sentence representation quality when training signals are held constant?
• RQ2 Do architectures with strong inductive biases (e.g., self-attention, graph structures) outperform those with weak priors in downstream NLP tasks?
• RQ3 To what extent does architectural inductive bias contribute to generalization, independent of learning dynamics?
• RQ4 Can models with minimal inductive bias achieve performance comparable to those with strong priors under identical training conditions?

Key findings

• Architectures with strong inductive biases, such as self-attention or graph-based models, showed almost no performance gains over models with weak or uninformative priors.
• Even models with explicit structural priors—like those leveraging syntactic or dependency structures—performed comparably to simpler architectures when training signals were held constant.
• The results suggest that architectural inductive bias contributes little to downstream performance when training procedures are controlled.
• The observed performance differences between architectures were minimal and not statistically significant under controlled training conditions.
• This challenges the prevailing assumption that strong inductive biases in architecture inherently improve generalization in NLP.
• The findings imply that other factors—such as data quality, model scale, or optimization—may dominate over architectural design in determining performance.