[Paper Review] Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

Yikang Shen, Shawn W. Tan+2 more|arXiv (Cornell University)|Oct 22, 2018

Natural Language Processing TechniquesComputer Science158 citations

TL;DR

ON-LSTM introduces a hierarchical inductive bias by ordering neurons with a cumax-based gating mechanism, enabling tree-like constituency composition within recurrent networks. It improves language modeling and unsupervised parsing, and enhances long-range syntactic and logical reasoning tasks.

ABSTRACT

Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such an inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

Motivation & Objective

• Motivate a bias for learning latent hierarchical (constituent) structure in language.
• Propose a neural unit that allocates neurons to long- vs. short-term information via ordered gating.
• Develop ON-LSTM with cumax activation to enforce monotonic master gates and structured updates.
• Demonstrate improvements on language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

Proposed method

• Introduce cumax() as cummax(softmax(...)) followed by cumsum, enabling a soft binary gate split across neuron blocks.
• Define master forget and master input gates using cumax to deterministically control high-level update granularity.
• Compute updated cell state c_t using a combination of master gates and standard LSTM gates, enabling hierarchical information retention.
• Downsize master gates to D_m = D/C dimensional vectors to share gating across chunks and reduce parameters.
• Train a three-layer ON-LSTM on PTB for language modeling and evaluate on perplexity; infer latent trees for unsupervised parsing using expected split points; evaluate on syntactic tasks and a logical inference dataset.

Experimental results

Research questions

• RQ1 Can an inductive bias that enforces a hierarchy of neuron updates improve learning of latent tree-structured representations in RNNs?
• RQ2 Does cumax-based structured gating enable better long-range dependency modeling and constituency parsing without supervised trees?
• RQ3 How does ON-LSTM perform on language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference relative to standard LSTMs?

Key findings

• ON-LSTM achieves better language modeling perplexities than standard LSTM with similar capacity (PTB, three-layer model: 25M parameters; Validation 58.29, Test 56.17).
• In unsupervised constituency parsing, the second layer of ON-LSTM yields state-of-the-art results on WSJ test with the best F1 among reported layers.
• ON-LSTM demonstrates improved generalization to longer sequences and stronger performance on long-range dependencies in targeted syntactic evaluation.
• ON-LSTM shows superior performance on a logical inference task for longer sequences compared to a standard LSTM, indicating better handling of structured data.
• The model’s inductive bias aligns with human syntactic structure, and provides potential benefits for downstream tasks requiring hierarchical representations.