[Paper Review] Zero-Shot Compositional Policy Learning via Language Grounding

Tianshi Cao, Jingkang Wang+2 more|arXiv (Cornell University)|Apr 15, 2020

Multimodal Machine Learning Applications27 references4 citations

TL;DR

This paper proposes a novel zero-shot compositional policy learning framework that enables agents to generalize across unseen environment dynamics using language grounding. By introducing BabyAI++, a new environment with disentangled vision-dynamics pairings and descriptive text, the authors demonstrate that a multi-modal attention-based fusion method (MMA) significantly improves zero-shot generalization in both RL and IL settings, outperforming baselines by up to 20% in success rate on unseen combinations.

ABSTRACT

Despite recent breakthroughs in reinforcement learning (RL) and imitation learning (IL), existing algorithms fail to generalize beyond the training environments. In reality, humans can adapt to new tasks quickly by leveraging prior knowledge about the world such as language descriptions. To facilitate the research on language-guided agents with domain adaption, we propose a novel zero-shot compositional policy learning task, where the environments are characterized as a composition of different attributes. Since there are no public environments supporting this study, we introduce a new research platform BabyAI++ in which the dynamics of environments are disentangled from visual appearance. At each episode, BabyAI++ provides varied vision-dynamics combinations along with corresponding descriptive texts. To evaluate the adaption capability of learned agents, a set of vision-dynamics pairings are held-out for testing on BabyAI++. Unsurprisingly, we find that current language-guided RL/IL techniques overfit to the training environments and suffer from a huge performance drop when facing unseen combinations. In response, we propose a multi-modal fusion method with an attention mechanism to perform visual language-grounding. Extensive experiments show strong evidence that language grounding is able to improve the generalization of agents across environments with varied dynamics.

Motivation & Objective

• To address the lack of generalization in existing RL/IL algorithms when faced with unseen environment dynamics.
• To enable agents to transfer knowledge across novel combinations of visual appearance and dynamics using natural language descriptions.
• To develop a benchmark environment that supports zero-shot compositional policy learning with disentangled vision-dynamics pairings.
• To evaluate the effectiveness of language grounding in improving zero-shot generalization beyond training distributions.
• To propose a multi-modal fusion method that aligns language descriptions with visual features for improved policy generalization.

Proposed method

• Proposes BabyAI++, a new environment platform that disentangles visual appearance from dynamics, enabling diverse vision-dynamics pairings.
• Introduces procedural generation of environment dynamics and corresponding descriptive natural language texts for each pairing.
• Develops a multi-modal attention fusion mechanism (MMA) that assigns language embeddings to spatial locations in visual feature maps.
• Uses cross-attention to ground language descriptions to specific visual regions, enabling dynamic policy adaptation.
• Trains policies via both reinforcement learning and imitation learning on the generated environment configurations.
• Evaluates generalization by holding out unseen vision-dynamics combinations during testing.

Experimental results

Research questions

• RQ1 Can language grounding enable agents to generalize to unseen combinations of visual appearance and dynamics in zero-shot settings?
• RQ2 How does the performance of existing language-guided RL/IL methods degrade when tested on unseen vision-dynamics pairings?
• RQ3 To what extent does meaningful descriptive text improve zero-shot generalization compared to random or shuffled text?
• RQ4 Does multi-modal fusion with spatially aligned attention improve policy generalization across compositional environment changes?
• RQ5 How does the proposed MMA method compare to standard fusion baselines like concatenation and FiLM in zero-shot compositional generalization?

Key findings

• The proposed MMA method achieves a success rate of 0.575 ± 0.016 on GoTo-v1 under RL, significantly outperforming the baseline (0.471 ± 0.016) and concat (0.323 ± 0.015) baselines.
• In IL, MMA achieves 0.635 ± 0.015 success rate on GoTo-v1, surpassing the best baseline FiLM (0.692 ± 0.015) and outperforming the next-best method concat (0.475 ± 0.016).
• On the more complex GoTo-v2 environment, MMA achieves 0.405 ± 0.016 success rate under RL, outperforming the baseline (0.283 ± 0.014) and concat (0.216 ± 0.013) methods.
• Ablation studies confirm that meaningful descriptive text is essential for performance, as random and shuffled text lead to significantly worse results.
• Visualization of attention weights confirms that the model learns to ground language descriptions to specific visual locations, such as the target object.
• The method generalizes effectively to unseen attributes and dynamics, with performance degradation being substantially lower than for baselines when testing on held-out vision-dynamics combinations.