[Paper Review] The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations

Felix Hill, Antoine Bordes+2 more|arXiv (Cornell University)|Jan 1, 2016

Topic Modeling307 citations

TL;DR

This paper proposes the Goldilocks Principle for language models, demonstrating that storing explicit, window-based memory representations of intermediate text lengths (neither too short nor too long) optimizes performance on semantic content word prediction in children's books. Models using such memories outperform standard neural language models, especially for meaningful words, and achieve state-of-the-art results on CNN QA through self-supervised training of attention over these memories.

ABSTRACT

Abstract: We introduce a new test of how well language models capture meaning in children's books. Unlike standard language modelling benchmarks, it distinguishes the task of predicting syntactic function words from that of predicting lower-frequency words, which carry greater semantic content. We compare a range of state-of-the-art models, each with a different way of encoding what has been previously read. We show that models which store explicit representations of long-term contexts outperform state-of-the-art neural language models at predicting semantic content words, although this advantage is not observed for syntactic function words. Interestingly, we find that the amount of text encoded in a single memory representation is highly influential to the performance: there is a sweet-spot, not too big and not too small, between single words and full sentences that allows the most meaningful information in a text to be effectively retained and recalled. Further, the attention over such window-based memories can be trained effectively through self-supervision. We then assess the generality of this principle by applying it to the CNN QA benchmark, which involves identifying named entities in paraphrased summaries of news articles, and achieve state-of-the-art performance.

Motivation & Objective

• To evaluate how well language models capture meaning in children's books by distinguishing syntactic function word prediction from semantic content word prediction.
• To investigate whether explicit long-term memory representations improve performance over standard autoregressive modeling in semantic prediction tasks.
• To determine the optimal size of memory windows for retaining and recalling meaningful textual information.
• To assess the generalization of the proposed memory mechanism to non-narrative, fact-based tasks such as named entity recognition in news summaries.
• To explore whether attention over memory windows can be effectively trained through self-supervision.

Proposed method

• The authors design a new benchmark that separates prediction of syntactic function words (e.g., 'the', 'and') from lower-frequency semantic content words (e.g., 'dog', 'happy') in children's books.
• They compare state-of-the-art language models that use different mechanisms for encoding prior context, including models with explicit memory representations.
• The memory mechanism stores intermediate text spans (windows) of variable length, with a fixed-size memory vector updated via a sliding window over the input.
• Attention over these memory windows is trained end-to-end using self-supervision, allowing the model to dynamically attend to relevant past content.
• The method is evaluated on a children's book prediction task and then transferred to the CNN Question Answering benchmark for named entity identification.
• Performance is measured by accuracy on predicting semantic content words and named entities, with ablation studies on memory window size.

Experimental results

Research questions

• RQ1 Does storing explicit representations of long-term context improve language model performance on predicting semantic content words in children's books compared to standard autoregressive models?
• RQ2 Is there an optimal window size for memory representations that maximizes retention of meaningful information, and if so, what is its characteristic scale?
• RQ3 Can attention over window-based memory representations be effectively trained through self-supervision without external supervision?
• RQ4 Does the proposed memory mechanism generalize beyond narrative text to fact-based, paraphrased news summaries, as evidenced by performance on the CNN QA benchmark?
• RQ5 Does the performance advantage of explicit memory representations hold equally for syntactic function words and semantic content words?

Key findings

• Models with explicit memory representations outperform state-of-the-art neural language models in predicting semantic content words in children's books.
• The performance gain from explicit memory is not observed for syntactic function words, indicating a selective benefit for semantic content.
• There exists a 'sweet spot' in memory window size—neither too small nor too large—where performance on semantic prediction is maximized.
• Attention over memory windows can be effectively trained through self-supervision, enabling dynamic and context-aware recall of past information.
• The Goldilocks Principle generalizes to non-narrative text, as the method achieves state-of-the-art performance on the CNN Question Answering benchmark for named entity recognition.
• The optimal memory window size is empirically found to be intermediate, suggesting that overly granular or overly compressed representations degrade performance.