[Paper Review] Emergent Systematic Generalization In a Situated Agent

Felix Hill, Andrew K. Lampinen+5 more|arXiv (Cornell University)|Oct 1, 2019

Multimodal Machine Learning Applications22 citations

TL;DR

This paper investigates systematic generalization in a situated agent using a 3D simulated environment, demonstrating that performance on out-of-distribution instructions improves significantly when agents are trained with diverse, multi-modal observations. Key factors include high object/word exposure, perspective-based visual invariances, and perceptual input variety, suggesting that neural networks generalize better when trained with rich, varied sensory experiences akin to human learning.

ABSTRACT

The question of whether deep neural networks are good at generalising beyond their immediate training experience is of critical importance for learning-based approaches to AI. Here, we consider tests of out-of-sample generalisation that require an agent to respond to never-seen-before instructions by manipulating and positioning objects in a 3D Unity simulated room. We first describe a comparatively generic agent architecture that exhibits strong performance on these tests. We then identify three aspects of the training regime and environment that make a significant difference to its performance: (a) the number of object/word experiences in the training set; (b) the visual invariances afforded by the agent's perspective, or frame of reference; and (c) the variety of visual input inherent in the perceptual aspect of the agent's perception. Our findings indicate that the degree of generalisation that networks exhibit can depend critically on particulars of the environment in which a given task is instantiated. They further suggest that the propensity for neural networks to generalise in systematic ways may increase if, like human children, those networks have access to many frames of richly varying, multi-modal observations as they learn.

Motivation & Objective

• To investigate whether deep neural networks can systematically generalize to never-before-seen instructions in a situated, 3D environment.
• To identify specific training and environmental factors that influence generalization performance in vision-language agents.
• To explore how multi-modal, perceptually rich observations affect the emergence of systematic generalization in neural networks.

Proposed method

• A generic agent architecture was trained in a 3D Unity simulation to perform object manipulation tasks based on natural language instructions.
• The training regime varied the number of object/word experience pairs to assess their impact on generalization.
• The agent's visual perspective and frame of reference were manipulated to evaluate the effect of visual invariance on performance.
• Perceptual diversity was increased by varying visual input, such as object positions, lighting, and viewpoints, to test its influence on learning.

Experimental results

Research questions

• RQ1 How does the number of object/word experiences during training affect generalization to unseen instructions?
• RQ2 To what extent does the agent's frame of reference or visual perspective influence systematic generalization?
• RQ3 How does perceptual variability in visual input affect the agent's ability to generalize beyond training distribution?

Key findings

• Increasing the number of object/word experiences in the training set significantly improved the agent's ability to generalize to unseen instructions.
• Visual invariances introduced by the agent's perspective played a crucial role in enabling systematic generalization.
• Greater perceptual diversity in visual input led to stronger generalization performance, suggesting richer sensory input enhances learning.
• The results indicate that systematic generalization in neural networks is highly sensitive to environmental and training-specific design choices.