[Paper Review] Environment Generation for Zero-Shot Compositional Reinforcement Learning

İzzeddin Gür, Natasha Jaques+5 more|arXiv (Cornell University)|Jan 21, 2022

Multimodal Machine Learning Applications6 citations

TL;DR

This paper proposes Compositional Design of Environments (CoDE), a method that uses a generator agent to automatically create a curriculum of increasingly complex compositional tasks tailored to an RL agent's skill level. By leveraging a multi-objective reward function with regret-based stabilization and task difficulty adjustment, CoDE achieves 4x higher success rate than baselines and enables zero-shot generalization to unseen real-world web navigation tasks.

ABSTRACT

Many real-world problems are compositional - solving them requires completing interdependent sub-tasks, either in series or in parallel, that can be represented as a dependency graph. Deep reinforcement learning (RL) agents often struggle to learn such complex tasks due to the long time horizons and sparse rewards. To address this problem, we present Compositional Design of Environments (CoDE), which trains a Generator agent to automatically build a series of compositional tasks tailored to the RL agent's current skill level. This automatic curriculum not only enables the agent to learn more complex tasks than it could have otherwise, but also selects tasks where the agent's performance is weak, enhancing its robustness and ability to generalize zero-shot to unseen tasks at test-time. We analyze why current environment generation techniques are insufficient for the problem of generating compositional tasks, and propose a new algorithm that addresses these issues. Our results assess learning and generalization across multiple compositional tasks, including the real-world problem of learning to navigate and interact with web pages. We learn to generate environments composed of multiple pages or rooms, and train RL agents capable of completing wide-range of complex tasks in those environments. We contribute two new benchmark frameworks for generating compositional tasks, compositional MiniGrid and gMiniWoB for web navigation.CoDE yields 4x higher success rate than the strongest baseline, and demonstrates strong performance of real websites learned on 3500 primitive tasks.

Motivation & Objective

• To address the challenge of training RL agents to generalize across complex, compositional real-world tasks like web navigation and form-filling.
• To overcome limitations of existing curriculum learning and domain randomization in compositional task settings, where task difficulty is not aligned with agent proficiency.
• To develop a domain-agnostic, scalable environment generation framework that supports hierarchical, dependency-based task structures.
• To enable zero-shot generalization to unseen tasks by training agents on a curriculum that targets performance weaknesses.
• To introduce two open-source benchmarks—compositional MiniGrid and gMiniWoB—for evaluating compositional task learning.

Proposed method

• A generator agent is trained using a multi-objective reward function that maximizes regret between agents in a population to stabilize learning and avoid local minima.
• The generator incorporates a difficulty incentive that dynamically adjusts task complexity based on the learners' current performance, increasing difficulty when agents succeed and reducing it when they fail.
• The method uses Petri Nets formalism to model compositional tasks as dependency graphs, enabling structured, hierarchical task generation from primitives.
• A domain-agnostic generator architecture composes environments from reusable primitives such as navigation bars, form fields, and product carousels, enabling scalable and diverse environment creation.
• The framework supports curriculum learning by gradually increasing the number of active and passive primitives, with a shift toward more complex and relevant primitives over time.
• The generator is trained on a large space of possible environments (e.g., ~10^31 for real websites), enabling exposure to diverse, realistic task topologies.

Experimental results

Research questions

• RQ1 Can a generator agent automatically construct a curriculum of compositional tasks that adapts to the agent’s evolving skill level?
• RQ2 How does a multi-objective reward function combining regret estimation and difficulty adjustment improve learning stability and performance in compositional RL?
• RQ3 To what extent can RL agents trained via CoDE generalize zero-shot to unseen, real-world web navigation tasks?
• RQ4 What role does task topology, modeled via Petri Nets, play in enabling structured and generalizable task learning?
• RQ5 How do the proposed benchmarks—compositional MiniGrid and gMiniWoB—compare to existing benchmarks in complexity and realism?

Key findings

• CoDE achieves a 4x higher success rate than the strongest baseline across multiple compositional tasks, demonstrating significant performance gains.
• The CoDE agent maintains consistent performance even as environment complexity increases, while baselines like curriculum learning (CL) and domain randomization (DR) degrade due to misaligned difficulty.
• On real-world websites, CoDE achieves 62% success on the Login task and 44% on Shopping, with performance dropping to 9% on Payment, reflecting distributional skew in primitives.
• The generator successfully scales to a space of approximately 10^31 possible environments using a 100x larger primitive set than prior benchmarks, enabling realistic web navigation training.
• Despite distributional shifts, CoDE agents generalize to hand-engineered primitives in MiniWoB, achieving 90% success on Address tasks and 0% on simple enter-text tasks, indicating selective generalization.
• The method enables a single RL agent to master a wide range of tasks across multiple pages and complex workflows, including form-filling and navigation, with zero-shot deployment on unseen websites.