[Paper Review] Environmental drivers of systematicity and generalization in a situated agent

Felix Hill, Andrew K. Lampinen+5 more|arXiv (Cornell University)|Oct 1, 2019

Language and cultural evolution39 references53 citations

TL;DR

The paper shows that an embodied, multimodal agent can generalize compositional language-grounded behaviors zero-shot, and that generalization is shaped by training data diversity, viewpoint constraints, and perceptual richness in realistic environments.

ABSTRACT

The question of whether deep neural networks are good at generalising beyond their immediate training experience is of critical importance for learning-based approaches to AI. Here, we consider tests of out-of-sample generalisation that require an agent to respond to never-seen-before instructions by manipulating and positioning objects in a 3D Unity simulated room. We first describe a comparatively generic agent architecture that exhibits strong performance on these tests. We then identify three aspects of the training regime and environment that make a significant difference to its performance: (a) the number of object/word experiences in the training set; (b) the visual invariances afforded by the agent's perspective, or frame of reference; and (c) the variety of visual input inherent in the perceptual aspect of the agent's perception. Our findings indicate that the degree of generalisation that networks exhibit can depend critically on particulars of the environment in which a given task is instantiated. They further suggest that the propensity for neural networks to generalise in systematic ways may increase if, like human children, those networks have access to many frames of richly varying, multi-modal observations as they learn.

Motivation & Objective

• Investigate whether standard neural architectures can achieve systematic generalization in a multimodal, situated setting.
• Determine how environmental factors influence the emergence of compositional understanding (verbs/nouns) in grounded language tasks.
• Identify key training regime factors that enhance zero-shot generalization to unseen objects and actions.

Proposed method

• A multimodal agent with visual (pixels) and language inputs processes observations via a 3-layer CNN and an LSTM-based language module.
• An LSTM-based policy and value network operate within an actor-critic framework trained with distributed actors (IMPALA-style).
• Experiments test zero-shot generalization for lifting and putting objects, bind predicates to arguments, and evaluate generalization under varying environmental conditions.
• Comparative analyses between 3D Unity environments and 2D grid worlds, with egocentric versus allocentric perspectives, and with/without language supervision.
• Control conditions compare a vision-language classifier to a situated agent on color-shape generalization tasks.

Experimental results

Research questions

• RQ1 Can a standard neural architecture ground and generalize verbs and nouns to novel objects in a 3D interactive environment?
• RQ2 What environmental and perceptual factors promote or hinder systematic generalization in situated agents?
• RQ3 Does increasing training diversity (words/objects), bounding the frame of reference, and richer temporal perception boost zero-shot generalization?
• RQ4 To what extent does language supervision contribute to systematic generalization in grounded tasks?

Key findings

• The agent generalizes to novel objects and novel word-object pairings in zero-shot lifting and putting tasks.
• Generalization improves when increasing the variety of words/objects experienced during training (negation experiments).
• Egocentric/bounded visual perspective enhances generalization compared to allocentric viewpoints.
• Temporal, rich perceptual input (moving through varied views) boosts generalization beyond single-frame perception.
• In 3D environments, generalization is stronger than in comparable 2D grid-world settings, and language contributes modestly to training performance but is not strictly essential for generalization.
• A vision-language classifier on static images underperforms the situated agent in test generalization, highlighting the benefit of embodied perception for systematic generalization.