[论文解读] BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning

Maxime Chevalier-Boisvert, Dzmitry Bahdanau+5 more|arXiv (Cornell University)|Oct 18, 2018

Natural Language Processing Techniques被引用 68

一句话总结

BabyAI 提供一个 2D 网格世界平台，具备 19 个逐步增加难度的等级和一个模拟人类教师，用于研究在接地语言学习中的样本效率，显示当前方法需要大量数据，课程学习和互动教学可以有所帮助，但可扩展性仍然具有挑战性。

ABSTRACT

Allowing humans to interactively train artificial agents to understand language instructions is desirable for both practical and scientific reasons, but given the poor data efficiency of the current learning methods, this goal may require substantial research efforts. Here, we introduce the BabyAI research platform to support investigations towards including humans in the loop for grounded language learning. The BabyAI platform comprises an extensible suite of 19 levels of increasing difficulty. The levels gradually lead the agent towards acquiring a combinatorially rich synthetic language which is a proper subset of English. The platform also provides a heuristic expert agent for the purpose of simulating a human teacher. We report baseline results and estimate the amount of human involvement that would be required to train a neural network-based agent on some of the BabyAI levels. We put forward strong evidence that current deep learning methods are not yet sufficiently sample efficient when it comes to learning a language with compositional properties.

研究动机与目标

• Motivate research on human-in-the-loop grounded language learning and sample efficiency.
• Provide an extensible platform with a compositional synthetic language and evaluation suite.
• Establish baseline sample-efficiency benchmarks for imitation learning and reinforcement learning on progressive levels.
• Investigate curriculum learning and interactive teaching as strategies to reduce data requirements.

提出的方法

• Introduce MiniGrid-based 2D gridworld with partial observability and a formal Baby Language (BNF grammar).
• Define 19 levels with competency-based progressions and a bot agent to simulate human demonstrations.
• Train neural models with imitation learning (IL) and reinforcement learning (RL) baselines using demonstrations and PPO respectively.
• Use Gaussian Process modeling to interpolate sample-efficiency and report 99% credible intervals for k_min (minimum required demonstrations/episodes).
• Evaluate curriculum pretraining and interactive learning as methods to improve data efficiency.

实验结果

研究问题

• RQ1 How much data is required for neural agents to learn compositional language-grounded tasks in BabyAI under IL and RL?
• RQ2 Do curriculum learning and interactive teaching significantly reduce the data requirements for solving BabyAI levels?
• RQ3 How does imitation learning compare to reinforcement learning in terms of sample efficiency on BabyAI levels?
• RQ4 Does pretraining on base levels or using RL demonstrations improve IL sample efficiency?
• RQ5 Can interactive imitation learning substantially reduce demonstrations needed for success?

主要发现

• Baseline IL on 6 levels requires hundreds of thousands of demonstrations; RL requires substantially more episodes to reach similar performance.
• Sample efficiency IL from Bot estimates range between roughly 8.4k and 408k demonstrations across levels, while RL demands roughly 16k to 1.7M episodes for the same levels (values in thousands).
• RL demonstrations can improve IL efficiency on some levels by 1.5–2x, particularly when the RL expert shares architecture with the learner.
• Curriculum pretraining helps in several target levels (e.g., GoToLocal and related pairs) but not universally; GoToObjMaze as base level often yields no benefit.
• Interactive IL can substantially reduce demonstrations needed (up to ~4x on some levels) compared to vanilla IL.