[Paper Review] Disentangled Sequence to Sequence Learning for Compositional Generalization

Hao Zheng, Mirella Lapata|arXiv (Cornell University)|Oct 9, 2021

Natural Language Processing Techniques9 citations

TL;DR

This paper proposes Dangle, a sequence-to-sequence model extension that improves compositional generalization by adaptively re-encoding the source input at each decoding step, conditioned on the current target context. This mechanism reduces representation entanglement, enabling models to better generalize to unseen compositions in semantic parsing and machine translation tasks, outperforming strong baselines on COGS, CFQ, and CoGnition benchmarks.

ABSTRACT

There is mounting evidence that existing neural network models, in particular the very popular sequence-to-sequence architecture, struggle to systematically generalize to unseen compositions of seen components. We demonstrate that one of the reasons hindering compositional generalization relates to representations being entangled. We propose an extension to sequence-to-sequence models which encourages disentanglement by adaptively re-encoding (at each time step) the source input. Specifically, we condition the source representations on the newly decoded target context which makes it easier for the encoder to exploit specialized information for each prediction rather than capturing it all in a single forward pass. Experimental results on semantic parsing and machine translation empirically show that our proposal delivers more disentangled representations and better generalization.

Motivation & Objective

• Address the failure of standard sequence-to-sequence models in compositional generalization, particularly their struggle with unseen combinations of known components.
• Identify representation entanglement—where semantic factors like meaning and relations are interdependent in hidden states—as a key cause of poor generalization.
• Develop a method that enables disentangled representations by dynamically adapting source encoding during decoding based on the current target context.
• Demonstrate that this approach improves zero-shot generalization without task-specific modifications or explicit supervision on disentanglement.
• Validate the method across multiple benchmarks, including COGS, CFQ, and CoGnition, to show robustness and generalizability.

Proposed method

• Introduce an adaptive re-encoding mechanism that re-encodes the source input at each decoding time step, conditioned on the current hidden state of the decoder.
• Condition the source representation on the newly generated target context, allowing the encoder to focus on relevant semantic factors for the current prediction.
• Modify the standard sequence-to-sequence architecture by injecting decoder state information into the encoder’s input representation at each step.
• Use a context-aware encoder that maintains specialized representations for different components (e.g., relations and entities) by leveraging dynamic context conditioning.
• Train the model end-to-end with standard sequence-to-sequence objectives, without additional regularization or supervision for disentanglement.
• Apply the method to both semantic parsing and machine translation, showing consistent gains in zero-shot and out-of-distribution generalization.

Experimental results

Research questions

• RQ1 To what extent does representation entanglement in standard sequence-to-sequence models hinder compositional generalization?
• RQ2 Can adaptive re-encoding of the source input, conditioned on the current target context, lead to more disentangled representations?
• RQ3 Does this dynamic re-encoding mechanism improve generalization on unseen compositional structures in semantic parsing and machine translation?
• RQ4 How does the proposed method compare to strong baselines and specialized techniques in zero-shot and out-of-distribution settings?
• RQ5 Can the model generalize to novel combinations of familiar syntactic and semantic components without explicit supervision on disentanglement?

Key findings

• The proposed Dangle model achieves state-of-the-art performance on the COGS benchmark, outperforming strong baselines in both lexical and structural generalization tasks.
• On the CFQ benchmark, Dangle shows significant improvements in zero-shot generalization, particularly on complex, nested compositional queries.
• In machine translation, Dangle correctly translates rare compound phrases like "behind the small doctor on the floor" in all five test contexts, while the baseline fails consistently.
• The model reduces representation entanglement, as evidenced by better disentanglement in attention and representation analysis, especially for relations and entities.
• The adaptive re-encoding mechanism leads to more robust and context-specific source representations, improving generalization without architectural or dataset-specific modifications.
• The method generalizes across different tasks and domains, demonstrating consistent gains in compositional generalization across semantic parsing, machine translation, and logical reasoning benchmarks.