[Paper Review] Syntax-Aware Language Modeling with Recurrent Neural Networks

Duncan A. J. Blythe, Alan Akbik+1 more|arXiv (Cornell University)|Mar 2, 2018

Natural Language Processing Techniques19 references3 citations

TL;DR

This paper proposes Syntax-Aware Language Models (SALMs), which integrate syntactic features—such as part-of-speech tags—into character-level RNN language models by prefixing tokens with syntactic labels. Using a sequential Monte Carlo method called SYNSIR, SALMs can infer probable syntactic structures for unparsed text, significantly reducing perplexity in character-level modeling while maintaining parity with standard models at the word level.

ABSTRACT

Neural language models (LMs) are typically trained using only lexical features, such as surface forms of words. In this paper, we argue this deprives the LM of crucial syntactic signals that can be detected at high confidence using existing parsers. We present a simple but highly effective approach for training neural LMs using both lexical and syntactic information, and a novel approach for applying such LMs to unparsed text using sequential Monte Carlo sampling. In experiments on a range of corpora and corpus sizes, we show our approach consistently outperforms standard lexical LMs in character-level language modeling; on the other hand, for word-level models the models are on a par with standard language models. These results indicate potential for expanding LMs beyond lexical surface features to higher-level NLP features for character-level models.

Motivation & Objective

• To investigate whether incorporating explicit syntactic features improves neural language modeling performance.
• To address the challenge of applying syntax-augmented language models to unparsed, real-world text.
• To develop a method that enables language models to reason about syntactic structure during inference without requiring pre-parsed input.
• To evaluate the impact of syntax-aware modeling on both character-level and word-level language modeling.
• To demonstrate that syntactic information can enhance generalization and data efficiency in low-level language modeling.

Proposed method

• The paper introduces SALMs by transforming each sentence into a sequence where each token is prefixed with its syntactic tag (e.g., 'NOUN-look' instead of 'look').
• The model is trained as a standard character-level LSTM language model over these syntactically annotated sequences, with syntax tags treated as part of the input token.
• A novel sequential Monte Carlo inference algorithm, SYNSIR, is proposed to estimate the most probable syntactic prefix distributions for unparsed text fragments during inference.
• SYNSIR performs online resampling of syntax prefixes based on incoming character sequences, allowing dynamic updating of syntactic uncertainty as the input grows.
• The method uses a particle-based approach to estimate the posterior distribution over syntax prefixes given a partial input, enabling scoring or generation under uncertainty.
• The approach is evaluated across multiple corpora and sample sizes using perplexity as the primary metric, with comparisons to standard LSTM language models at both word and character levels.

Experimental results

Research questions

• RQ1 Can explicit syntactic features improve the performance of character-level neural language models?
• RQ2 How can syntax-aware language models be applied to unparsed text, where syntactic structure is not available at inference time?
• RQ3 Does incorporating syntax lead to better generalization or data efficiency in language modeling?
• RQ4 Why do syntax-aware models show gains at the character level but not the word level?
• RQ5 Can the model infer and update syntactic structure dynamically as new text is processed?

Key findings

• SALMs significantly reduce perplexity compared to standard LSTM language models in character-level modeling across all tested corpora and sample sizes.
• The improvement in perplexity is consistent and statistically meaningful, indicating that syntax enhances generalization and data efficiency at the character level.
• At the word level, SALMs perform on par with standard language models, suggesting that word-level models may implicitly learn syntactic patterns without explicit supervision.
• SYNSIR successfully infers and updates syntactic structure online: for example, the model shifts from treating 'display' as a noun to a verb as more context is revealed.
• The model demonstrates dynamic syntactic reasoning, with posterior probabilities over syntax prefixes updating in real time as new characters are processed.
• The results suggest that syntax-aware modeling holds strong potential for improving low-level language modeling, particularly when data is limited.