[Paper Review] Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learning

Wenhao Li, Dan Qiao+4 more|arXiv (Cornell University)|May 18, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

SAMA proposes a novel semantically aligned task decomposition framework in multi-agent reinforcement learning that leverages chain-of-thought prompting in pretrained language models to generate, decompose, and allocate sub-goals while enabling self-reflection-based replanning. It achieves state-of-the-art sample efficiency, requiring only ~10% of the training data needed by prior methods on Overcooked and MiniRTS.

ABSTRACT

The difficulty of appropriately assigning credit is particularly heightened in cooperative MARL with sparse reward, due to the concurrent time and structural scales involved. Automatic subgoal generation (ASG) has recently emerged as a viable MARL approach inspired by utilizing subgoals in intrinsically motivated reinforcement learning. However, end-to-end learning of complex task planning from sparse rewards without prior knowledge, undoubtedly requires massive training samples. Moreover, the diversity-promoting nature of existing ASG methods can lead to the "over-representation" of subgoals, generating numerous spurious subgoals of limited relevance to the actual task reward and thus decreasing the sample efficiency of the algorithm. To address this problem and inspired by the disentangled representation learning, we propose a novel "disentangled" decision-making method, Semantically Aligned task decomposition in MARL (SAMA), that prompts pretrained language models with chain-of-thought that can suggest potential goals, provide suitable goal decomposition and subgoal allocation as well as self-reflection-based replanning. Additionally, SAMA incorporates language-grounded RL to train each agent's subgoal-conditioned policy. SAMA demonstrates considerable advantages in sample efficiency compared to state-of-the-art ASG methods, as evidenced by its performance on two challenging sparse-reward tasks, Overcooked and MiniRTS.

Motivation & Objective

• To address the challenge of sparse credit assignment in cooperative multi-agent reinforcement learning (MARL) with sparse rewards.
• To overcome the limitations of existing automatic subgoal generation (ASG) methods, including over-representation of irrelevant sub-goals and poor sample efficiency.
• To enable dynamic, semantically aligned task decomposition and subgoal allocation through pretrained language models (PLMs) with chain-of-thought prompting.
• To improve sample efficiency by integrating language-grounded reinforcement learning with self-reflection for replanning.
• To reduce reliance on handcrafted rules or domain-specific reward functions through semantic grounding of sub-goals.

Proposed method

• SAMA uses a pretrained language model (PLM) to generate high-level goals and decompose them into semantically meaningful sub-goals based on the current environment state.
• It employs chain-of-thought prompting to guide the PLM in reasoning about goal decomposition, subgoal allocation to agents, and self-reflection when sub-goals fail.
• Sub-goals are grounded in environmental state through semantic parsing, enabling the construction of language-grounded reward functions for policy training.
• Each agent is trained with a subgoal-conditioned policy using language-grounded intrinsic rewards to encourage sub-goal completion.
• The system supports dynamic replanning by re-prompting the PLM upon sub-goal failure, incorporating self-reflection to revise strategy.
• A preprocessing step extracts interactive objects from the environment state to constrain the goal space and reduce semantic ambiguity in PLM input.

Experimental results

Research questions

• RQ1 Can a pretrained language model effectively generate semantically aligned sub-goals and decompose tasks in cooperative MARL?
• RQ2 Does chain-of-thought prompting with self-reflection improve sub-goal planning and replanning in sparse-reward environments?
• RQ3 To what extent does language-grounded intrinsic reward design enhance sample efficiency in MARL?
• RQ4 How does the inclusion of interactive object extraction affect the quality and efficiency of PLM-generated sub-goals?
• RQ5 What is the impact of PLM version (e.g., GPT-3.5 vs. GPT-4) and few-shot prompting on sub-goal generation and overall performance?

Key findings

• SAMA achieves state-of-the-art performance on Overcooked and MiniRTS with approximately 10% of the training data required by prior SOTA methods.
• GPT-4 outperforms GPT-3.5 significantly, with a large performance gap observed in both Overcooked and MiniRTS environments.
• The language model-generated reward function performs comparably to hand-designed functions in MiniRTS but is slightly weaker in Overcooked due to higher semantic complexity.
• Excluding interactive object extraction leads to a significant performance drop, indicating that constrained input space improves PLM reasoning and policy learning.
• Using one few-shot example instead of three achieves better performance and cost efficiency, suggesting diminishing returns with more examples.
• The method demonstrates robustness to bias in PLM outputs, though societal risks from stereotypes in goal generation are acknowledged and require mitigation in real-world deployment.