[Paper Review] Learning to Generalize from Sparse and Underspecified Rewards

Rishabh Agarwal, Liang Chen+2 more|arXiv (Cornell University)|Feb 19, 2019

Topic Modeling70 references46 citations

TL;DR

The paper introduces Meta Reward Learning (MeRL) and MAPOX to improve generalization when learning from underspecified, sparse rewards, validated on weakly supervised semantic parsing and instruction-following tasks.

ABSTRACT

We consider the problem of learning from sparse and underspecified rewards, where an agent receives a complex input, such as a natural language instruction, and needs to generate a complex response, such as an action sequence, while only receiving binary success-failure feedback. Such success-failure rewards are often underspecified: they do not distinguish between purposeful and accidental success. Generalization from underspecified rewards hinges on discounting spurious trajectories that attain accidental success, while learning from sparse feedback requires effective exploration. We address exploration by using a mode covering direction of KL divergence to collect a diverse set of successful trajectories, followed by a mode seeking KL divergence to train a robust policy. We propose Meta Reward Learning (MeRL) to construct an auxiliary reward function that provides more refined feedback for learning. The parameters of the auxiliary reward function are optimized with respect to the validation performance of a trained policy. The MeRL approach outperforms our alternative reward learning technique based on Bayesian Optimization, and achieves the state-of-the-art on weakly-supervised semantic parsing. It improves previous work by 1.2% and 2.4% on WikiTableQuestions and WikiSQL datasets respectively.

Motivation & Objective

• Address learning from sparse and underspecified success/failure rewards in language-conditioned tasks.
• Disentangle exploration from robust policy optimization using mode covering and mode seeking KL directions.
• Learn an auxiliary reward function (MeRL) whose parameters are optimized by validation performance.
• Compare MeRL with Bayesian Optimization-based reward learning (BoRL) and baseline methods.
• Demonstrate improved generalization on WikiTableQuestions, WikiSQL, and an instruction-following maze task.

Proposed method

• Use mode covering exploration (MAPOX) to collect diverse successful trajectories.
• Train a policy with a robust objective that includes mode seeking KL divergence on discovered trajectories.
• Define a parametric auxiliary reward R_phi based on trajectory features plus underspecified reward R(a|x,y).
• Optimize phi via meta-learning (MeRL) by differentiating through the policy update to maximize validation performance.
• Alternative: optimize phi with Bayesian Optimization (BoRL) to maximize non-differentiable validation metrics.
• Employ MAPO-based estimators to handle memory buffers and gradient estimates for sparse rewards.

Experimental results

Research questions

• RQ1 How can auxiliary reward learning improve generalization when rewards are underspecified?
• RQ2 Does a mode covering exploration strategy combined with robust policy optimization outperform mode seeking methods on weakly supervised tasks?
• RQ3 Can gradient-based (MeRL) and gradient-free (BoRL) reward learning approaches improve validation performance in sparse reward settings?
• RQ4 What is the impact of learned auxiliary rewards on generalization in semantic parsing benchmarks?
• RQ5 How close can exploration-heavy methods come to oracle (fully supervised) performance in instruction-following and semantic parsing tasks?

Key findings

• MeRL and BoRL improve generalization over baseline underspecified rewards.
• MAPOX exploration discovers more diverse successful trajectories than MAPO alone.
• MeRL substantially closes the gap to Oracle rewards in the instruction-following task (Dev 83.0 vs 95.7; Test 74.5 vs 92.6).
• Auxiliary rewards learned via MeRL/BoRL yield higher validation-driven performance than underspecified rewards alone.
• MAPOX with MeRL achieves state-of-the-art results on weakly supervised semantic parsing benchmarks (WikiTableQuestions, WikiSQL).