[Paper Review] Opponent Modeling in Deep Reinforcement Learning

He He, Jordan Boyd‐Graber+2 more|arXiv (Cornell University)|Sep 18, 2016

Reinforcement Learning in Robotics18 references108 citations

TL;DR

This paper introduces DRON (Deep Reinforcement Opponent Network) with two architectures to jointly learn a policy and opponent behavior in multi-agent reinforcement learning, and shows it outperforms DQN baselines on soccer and quiz bowl while handling non-stationary opponents. It also explores multitask supervision and Mixture-of-Experts for opponent strategies.

ABSTRACT

Opponent modeling is necessary in multi-agent settings where secondary agents with competing goals also adapt their strategies, yet it remains challenging because strategies interact with each other and change. Most previous work focuses on developing probabilistic models or parameterized strategies for specific applications. Inspired by the recent success of deep reinforcement learning, we present neural-based models that jointly learn a policy and the behavior of opponents. Instead of explicitly predicting the opponent's action, we encode observation of the opponents into a deep Q-Network (DQN); however, we retain explicit modeling (if desired) using multitasking. By using a Mixture-of-Experts architecture, our model automatically discovers different strategy patterns of opponents without extra supervision. We evaluate our models on a simulated soccer game and a popular trivia game, showing superior performance over DQN and its variants.

Motivation & Objective

• Motivate a general opponent modeling framework in reinforcement learning that does not rely on domain knowledge.
• Jointly learn a policy and a probabilistic model of opponent behavior to handle non-stationary strategies.
• Explore architectures that combine opponent representation with Q-learning, including multitask supervision.
• Evaluate robustness and performance against varied opponents in two tasks (soccer and quiz bowl).

Proposed method

• Propose DRON, a Deep Q-Network that also learns an opponent network to model opponent behavior.
• Compare two architectures: dron-concat, which concatenates state and opponent representations to predict Q-values, and dron-moE, which uses a Mixture-of-Experts to marginalize over opponent strategies.
• Optionally apply multitask supervision to provide direct information about the opponent (e.g., action or type) to shape the opponent representation (h^o).
• Extend the Q-learning update to account for opponent policy by modeling Q^{π|π^o} and incorporating opponent information.
• Evaluate on two tasks: a grid-based two-player soccer game and a Quiz Bowl question-answering game, comparing against DQN baselines (dqn-world and dqn-self).
• Investigate the effect of the number of experts (K) in dron-moE and the impact of multitask supervision on performance.

Experimental results

Research questions

• RQ1 Can implicit opponent representations learned within DRL improve policy against adaptive opponents?
• RQ2 Do DRON architectures (concat vs. Mixture-of-Experts) better capture opponent strategy variations than standard DQN?
• RQ3 Does multitask supervision about opponents provide additional benefits for learning robust policies?
• RQ4 How do DRON models perform against diverse and non-stationary opponents in different tasks?
• RQ5 What is the impact of the number of experts on DRON-MoE performance across tasks?

Key findings

• DRON variants outperform the DQN baseline in both soccer and quiz bowl.
• Dron-concat yields strong performance when augmented with opponent-related supervision, while dron-moE shows robustness to multiple opponent strategies through expert specialization.
• Dron-moE, with an appropriate number of experts, often achieves the best or near-best results and reduces learning variance compared to DQN.
• Multitask supervision can help in some settings (e.g., opponent type in Quiz Bowl) but may not always improve performance for all DRON variants (e.g., action supervision not always beneficial).
• DQN-world (treating opponents as part of the world) struggles against mixed opponent behaviors, whereas DRON provides better trade-offs against both aggressive and cautious opponents.
• In Quiz Bowl, dron-moE outperforms dron-concat; increasing the number of experts improves performance, and opponent-type supervision is particularly helpful when more experts are used.
• Overall, DRON demonstrates improved robustness to non-stationary opponents and can learn adaptive responses without domain-specific opponent models.