[Paper Review] Compositional generalization in a deep seq2seq model by separating syntax and semantics

Jacob Russin, Jason Jo+2 more|arXiv (Cornell University)|Apr 22, 2019

Natural Language Processing TechniquesComputer Science28 references86 citations

TL;DR

The paper introduces Syntactic Attention, a two-stream seq2seq model that separates syntax (alignment) from semantics (word-to-output mappings), achieving strong compositional generalization on SCAN beyond prior models without extra supervision.

ABSTRACT

Standard methods in deep learning for natural language processing fail to capture the compositional structure of human language that allows for systematic generalization outside of the training distribution. However, human learners readily generalize in this way, e.g. by applying known grammatical rules to novel words. Inspired by work in neuroscience suggesting separate brain systems for syntactic and semantic processing, we implement a modification to standard approaches in neural machine translation, imposing an analogous separation. The novel model, which we call Syntactic Attention, substantially outperforms standard methods in deep learning on the SCAN dataset, a compositional generalization task, without any hand-engineered features or additional supervision. Our work suggests that separating syntactic from semantic learning may be a useful heuristic for capturing compositional structure.

Motivation & Objective

• Motivate separation of syntax and semantics as a neural mechanism for compositional generalization.
• Propose and implement a Syntactic Attention architecture that encodes syntax and semantics in separate streams.
• Evaluate on SCAN, focusing on the add-jump compositional generalization split.
• Compare to prior models (GRU with attention, CNN) and analyze robustness across seeds.

Proposed method

• Introduce two parallel input encodings: semantic representations m_j = W_m x_j (linear transform per word) and syntactic annotations h_j from a biRNN over the sentence.
• Compute attention using syntactic representations to align inputs to outputs via a standard attention mechanism (e_ij = s_i • h_j).
• Decode by forming d_i as a weighted sum of semantic representations (d_i = sum_j α_ij m_j) and generating y_i from d_i.
• Maintain separation: semantics do not depend on other words, while syntax encodes temporal dependencies affecting attention.
• Use dot-product attention for e_ij and standard LSTM/Adam optimization; encoder: 2-layer, 200 hidden units; decoder: 1-layer, 400 hidden units; semantic vector size 120; dropout 0.5.
• Experimentally validate on SCAN, especially add-jump split where 'jump' is held out except in its basic form.

Experimental results

Research questions

• RQ1 Can separating syntax and semantics improve compositional generalization in seq2seq models?
• RQ2 Does forcing attention to operate via syntactic representations enhance extrapolation to unseen constructions?
• RQ3 How do variations that relax or alter the separation affect generalization?

Key findings

• Syntactic Attention achieves state-of-the-art compositional generalization on SCAN add-jump with mean accuracy 91.0% (median 91.0%) in the best run, outperforming prior models.
• Across seeds, the add-jump performance shows variance (mean 78.4%, median 91.0%) indicating sensitivity to initialization.
• On simple and length splits, Syntactic Attention reaches 100.0% and 15.2% (respectively) in the reported results, outperforming GRU+attn and CNN baselines.
• Additional experiments show that sequential semantics reduces generalization (e.g., sequential semantics: Add-jump 42.3±32.7), while syntax-action (allowing syntax to influence outputs) preserves strong performance (Add-jump 88.7±14.2).
• Overall, enforcing a separation between syntax and semantics improves systematic generalization without hand-engineering or extra supervision.