[Paper Review] Systematic Generalization on gSCAN with Language Conditioned Embedding

Tong Gao, Qi Huang+1 more|arXiv (Cornell University)|Sep 11, 2020

Multimodal Machine Learning Applications22 references4 citations

TL;DR

This paper proposes a language-conditioned message passing mechanism to learn contextualized object embeddings for systematic generalization in grounded language understanding. By dynamically aggregating relational information between objects based on input natural language, the model achieves state-of-the-art performance on gSCAN, significantly outperforming prior methods on multiple test splits, particularly those requiring composition of novel action-attribute combinations.

ABSTRACT

Systematic Generalization refers to a learning algorithm's ability to extrapolate learned behavior to unseen situations that are distinct but semantically similar to its training data. As shown in recent work, state-of-the-art deep learning models fail dramatically even on tasks for which they are designed when the test set is systematically different from the training data. We hypothesize that explicitly modeling the relations between objects in their contexts while learning their representations will help achieve systematic generalization. Therefore, we propose a novel method that learns objects' contextualized embeddings with dynamic message passing conditioned on the input natural language and end-to-end trainable with other downstream deep learning modules. To our knowledge, this model is the first one that significantly outperforms the provided baseline and reaches state-of-the-art performance on grounded-SCAN (gSCAN), a grounded natural language navigation dataset designed to require systematic generalization in its test splits.

Motivation & Objective

• To address the failure of deep learning models in systematic generalization, especially on tasks requiring composition of novel concept combinations.
• To improve object representation learning by explicitly modeling relations between objects in context, grounded in natural language input.
• To develop an end-to-end trainable architecture that enhances zero-shot generalization to unseen semantic combinations in grounded navigation tasks.
• To evaluate the method on gSCAN, a benchmark designed to test systematic generalization through systematic distributional shift in test sets.

Proposed method

• The model uses dynamic message passing between all object pairs in a grid world to aggregate relational information.
• Message passing weights are conditioned on the input natural language sentence, enabling context-aware aggregation of object features.
• Each object’s contextualized embedding is computed as a weighted sum of messages from all other objects, with weights determined by the input sentence.
• The method is integrated into an end-to-end training pipeline with a downstream action-sequence decoder for navigation.
• The model treats the agent and all objects as homogeneous nodes in a message-passing graph, with no distinction in edge types during message passing.
• Ablation studies evaluate the contribution of language-conditioned message passing by removing it and comparing performance.

Experimental results

Research questions

• RQ1 Can language-conditioned message passing improve systematic generalization in vision-language navigation tasks?
• RQ2 Does learning contextualized object embeddings based on relational structure and input language lead to better zero-shot composition of novel action-attribute pairs?
• RQ3 How does the proposed method compare to strong baselines on gSCAN, especially on test splits designed to probe systematic generalization?
• RQ4 Why does the model still fail on certain test splits, such as split B, despite improvements on others?

Key findings

• The proposed model achieves state-of-the-art performance on gSCAN, significantly outperforming the baseline on multiple test splits, including split G (93.02% exact match) and split E (99.08%).
• Ablation studies confirm that language-conditioned message passing improves performance on most splits, though it slightly degrades performance on split F, suggesting trade-offs in learning adverb semantics versus action semantics.
• Failure on split B is attributed to the model's inability to generalize action sequences to novel directions, as it generates actions blindly without modeling the agent's changing state.
• The model correctly identifies the target location via attention but fails to generate a valid path, indicating a lack of dynamic state modeling during navigation.
• The results suggest that current models still struggle with perspective-taking and state transition modeling, especially when actions involve unfamiliar directions.
• The authors identify the need for model-based reinforcement learning or type-specific message passing to better capture environmental dynamics and agent-environment interactions.