[Paper Review] Playing 20 Question Game with Policy-Based Reinforcement Learning

Huang Hu, Xianchao Wu+5 more|arXiv (Cornell University)|Aug 23, 2018

Multimodal Machine Learning Applications9 references4 citations

TL;DR

This paper proposes a policy-based reinforcement learning framework for the 20 Questions game that enables an agent to learn optimal question selection without relying on a knowledge base or engineered heuristics. By using a reward network to estimate informative immediate rewards and a policy network to sample diverse, robust questions, the method outperforms entropy-based baselines in noisy real-world settings while maintaining competitive performance in noise-free simulations.

ABSTRACT

The 20 Questions (Q20) game is a well known game which encourages deductive reasoning and creativity. In the game, the answerer first thinks of an object such as a famous person or a kind of animal. Then the questioner tries to guess the object by asking 20 questions. In a Q20 game system, the user is considered as the answerer while the system itself acts as the questioner which requires a good strategy of question selection to figure out the correct object and win the game. However, the optimal policy of question selection is hard to be derived due to the complexity and volatility of the game environment. In this paper, we propose a novel policy-based Reinforcement Learning (RL) method, which enables the questioner agent to learn the optimal policy of question selection through continuous interactions with users. To facilitate training, we also propose to use a reward network to estimate the more informative reward. Compared to previous methods, our RL method is robust to noisy answers and does not rely on the Knowledge Base of objects. Experimental results show that our RL method clearly outperforms an entropy-based engineering system and has competitive performance in a noisy-free simulation environment.

Motivation & Objective

• To develop a question selection strategy for the 20 Questions game that does not depend on a pre-existing knowledge base of objects.
• To improve robustness against noisy user answers, which commonly occur in real-world interactions.
• To enable the agent to explore diverse question strategies through stochastic policy sampling, enhancing generalization and avoiding local optima.
• To design a meaningful reward signal for long-horizon decision making, where only final guesses yield sparse rewards.
• To create a scalable, end-to-end RL framework that can be trained via interaction with real users.

Proposed method

• The game is modeled as a Markov Decision Process (MDP), with the agent maintaining a belief state (confidence vector) over possible target objects.
• A policy network πθ(a|s) outputs a question distribution based on the current belief state, enabling stochastic sampling of questions rather than greedy selection.
• A novel RewardNet is introduced to estimate non-zero, informative rewards at each time step, improving credit assignment for long-term returns.
• The agent uses experience replay to store and sample historical trajectories (s, a, r, s') for stable training.
• The policy is trained via policy gradient methods using the estimated returns from the RewardNet, allowing end-to-end optimization of question selection.
• The framework is trained through interaction with real users, making it robust to answer noise and adaptable to diverse object distributions.

Experimental results

Research questions

• RQ1 Can a policy-based reinforcement learning agent learn effective question selection in the 20 Questions game without access to a knowledge base?
• RQ2 How can meaningful intermediate rewards be designed to improve training efficiency when only final win/loss feedback is available?
• RQ3 To what extent can an RL agent generalize and remain robust to noisy or incorrect answers from real users?
• RQ4 Does stochastic question sampling via a policy network improve diversity and generalization compared to greedy rule-based methods?
• RQ5 How does the performance of the RL agent compare to highly engineered baselines in both noisy and noise-free environments?

Key findings

• The proposed RL method significantly outperforms an entropy-based engineering system in real-world human evaluations where noisy answers are common.
• In a noise-free simulation environment, the RL agent achieves competitive win rates compared to the entropy-based baseline, demonstrating strong generalization capability.
• The agent maintains high performance across both frequent and infrequent objects, indicating robustness to data sparsity and object popularity bias.
• The use of a RewardNet leads to more informative training signals, enabling faster convergence and better policy learning despite sparse final rewards.
• The policy-based approach with stochastic sampling produces more diverse and natural-sounding questions, improving user experience.
• The method generalizes well across different object sets, as evidenced by consistent performance under both uniform and popularity-based object sampling distributions.