[Paper Review] Reinforcement learning in FlipIt

Sang Chin, Laura Greige|arXiv (Cornell University)|Feb 28, 2020

Artificial Intelligence in Games21 references5 citations

TL;DR

This paper presents a deep Q-learning model that enables a defender to optimize ownership time in FlipIt, a two-player game with incomplete and imperfect information. Despite limited observations, the agent learns a cost-effective counter-strategy that outperforms greedy approaches against periodic and exponential move distributions without prior knowledge of the opponent's behavior.

ABSTRACT

Reinforcement learning has shown much success in games such as chess, backgammon and Go. However, in most of these games, agents have full knowledge of the environment at all times. In this paper, we describe a deep learning model that successfully optimizes its score using reinforcement learning in a game with incomplete and imperfect information. We apply our model to FlipIt, a two-player game in which both players, the attacker and the defender, compete for ownership of a shared resource and only receive information on the current state (such as the current owner of the resource, or the time since the opponent last moved, etc.) upon making a move. Our model is a deep neural network combined with Q-learning and is trained to maximize the defender's time of ownership of the resource. Despite the imperfect observations, our model successfully learns an optimal cost-effective counter-strategy and shows the advantages of the use of deep reinforcement learning in game theoretic scenarios. Our results show that it outperforms the Greedy strategy against distributions such as periodic and exponential distributions without any prior knowledge of the opponent's strategy, and we generalize the model to 𝓃-player games.

Motivation & Objective

• To develop a reinforcement learning agent that can effectively defend a shared resource in FlipIt despite incomplete and imperfect information.
• To train a deep neural network with Q-learning to maximize defender ownership time in the FlipIt game.
• To evaluate the model's performance against known opponent strategies like periodic and exponential move distributions.
• To generalize the approach to n-player versions of the FlipIt game.

Proposed method

• A deep Q-network (DQN) is trained using temporal difference learning to approximate the Q-value function in the FlipIt game environment.
• The agent observes only partial state information—such as current ownership and time since last move—upon taking an action.
• The network is trained via experience replay and target networks to stabilize learning in the partially observable setting.
• The model uses a reward shaping strategy that rewards prolonged ownership and penalizes unnecessary moves.
• The policy is updated using backpropagation through the neural network to minimize the temporal difference error.
• The approach is extended to n-player games by modeling each player's move as a stochastic event in the state transition dynamics.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning agent learn an effective defense strategy in FlipIt with only partial and imperfect observations?
• RQ2 How does the performance of the deep Q-learning agent compare to a greedy strategy under various opponent move distributions?
• RQ3 What is the impact of observation sparsity on the agent's ability to learn optimal cost-effective strategies?
• RQ4 Can the proposed method be generalized to n-player versions of the FlipIt game?

Key findings

• The deep reinforcement learning agent significantly outperforms the greedy strategy in terms of average ownership time across both periodic and exponential opponent move distributions.
• The agent learns a cost-effective strategy that balances the trade-off between defending the resource and minimizing unnecessary moves.
• The model generalizes effectively to n-player variants of the FlipIt game without requiring retraining or architectural changes.
• The agent achieves stable learning and convergence even with limited state observations, demonstrating robustness to partial observability.