[Paper Review] Non-myopic learning in repeated stochastic games

Jacob W. Crandall|arXiv (Cornell University)|Sep 30, 2014

Advanced Bandit Algorithms Research19 references3 citations

TL;DR

This paper introduces mega, a meta-gaming technique that reduces the strategy space of two-player general-sum repeated stochastic games (RSGs) to a small set of expert strategies—effectively transforming the problem into a multi-armed bandit. By preserving key game-theoretic properties like security and best response, mega enables fast, robust learning using standard bandit algorithms such as S++, outperforming existing methods in self-play and against diverse opponents across three RSGs.

ABSTRACT

In repeated stochastic games (RSGs), an agent must quickly adapt to the behavior of previously unknown associates, who may themselves be learning. This machine-learning problem is particularly challenging due, in part, to the presence of multiple (even infinite) equilibria and inherently large strategy spaces. In this paper, we introduce a method to reduce the strategy space of two-player general-sum RSGs to a handful of expert strategies. This process, called Mega, effectually reduces an RSG to a bandit problem. We show that the resulting strategy space preserves several important properties of the original RSG, thus enabling a learner to produce robust strategies within a reasonably small number of interactions. To better establish strengths and weaknesses of this approach, we empirically evaluate the resulting learning system against other algorithms in three different RSGs.

Motivation & Objective

• Address the challenge of learning robust strategies in two-player general-sum repeated stochastic games (RSGs) with unknown, learning opponents.
• Overcome the limitations of traditional game abstraction in general-sum RSGs, where multiple equilibria and large strategy spaces hinder effective learning.
• Develop a strategy reduction method that preserves critical game-theoretic properties such as security, best response, and Nash equilibria.
• Enable fast, non-myopic learning in realistic time scales by transforming the RSG into a tractable bandit problem.
• Empirically evaluate the robustness and efficiency of the resulting learning system against existing algorithms in diverse RSG environments.

Proposed method

• Apply game abstraction via meta-gaming to identify a finite set of expert strategies (Φi) in two-player general-sum RSGs, reducing the original strategy space.
• Compute expert strategies using equilibrium computation (e.g., Nash equilibria), maximin strategies, and attack strategies to ensure robustness.
• Construct leader strategies by solving MDPs for various weighting parameters (ω), ensuring coverage of key strategic behaviors.
• Maintain strategy set Φi using polynomial-time computations; update guilt and other metrics incrementally after each round.
• Use standard bandit algorithms (e.g., S++, Exp3, UCB) on the reduced strategy space to learn effective policies quickly.
• Leverage the fact that follower strategies can reuse precomputed expert computations, minimizing per-round overhead.

Experimental results

Research questions

• RQ1 Can a reduced strategy space derived from expert strategies preserve essential game-theoretic properties such as security and best response in general-sum RSGs?
• RQ2 To what extent does the mega method enable faster and more robust learning compared to existing algorithms in repeated stochastic games with unknown opponents?
• RQ3 How does the performance of mega compare to model-based reinforcement learning (MBRL) and Counterfactual Regret Minimization (CFR) in terms of payoff, computation time, and convergence?
• RQ4 Does the strategy reduction via mega maintain effectiveness across diverse opponent behaviors, including top-performing and low-performing associates?
• RQ5 Can the meta-gaming approach be efficiently scaled to real-world RSGs with complex dynamics and large state spaces?

Key findings

• In the Microgrid Scenario, mega-S++ achieved an average self-play payoff of 32.2, significantly outperforming CFR (17.6) and MBRL (0.1) in initialization time and runtime.
• Against Top-5 opponents in the Microgrid Scenario, mega-S++ earned 26.9 average payoff, outperforming CFR (23.4) and MBRL (24.1) in the same metric.
• In the Block Game, mega-S++ achieved 28.0 average payoff against Low-5 opponents, surpassing CFR (24.4) and MBRL (14.6).
• mega-S++ had the lowest total execution time (10.2 seconds) in the Microgrid Scenario, with only 0.9 seconds in runtime, compared to CFR’s 930.0 seconds and MBRL’s 91.4 seconds.
• The initialization time for mega-S++ was dominated by unoptimized minimax computation, suggesting that performance can be further improved with code optimization.
• The method maintains critical game-theoretic properties such as security and best response, enabling robust performance even when opponents change strategies dynamically.