[Paper Review] Reinforcement Learning: A Survey

Leslie Pack Kaelbling, Michael L. Littman+1 more|arXiv (Cornell University)|May 1, 1996

Evolutionary Algorithms and ApplicationsComputer Science102 references99 citations

TL;DR

This seminal survey provides a comprehensive overview of reinforcement learning (RL) from a computer science perspective, covering foundational concepts like Markov decision processes, exploration-exploitation trade-offs, temporal credit assignment, and function approximation. It synthesizes key algorithms such as Q-learning and TD(λ), emphasizes the role of generalization and hierarchical learning, and evaluates practical challenges and applications, establishing a critical reference for researchers in RL and AI.

ABSTRACT

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word ``reinforcement.'' The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

Motivation & Objective

• To provide a comprehensive, accessible overview of reinforcement learning for researchers familiar with machine learning.
• To trace the historical development of RL and its theoretical foundations in Markov decision processes.
• To identify and analyze core challenges in RL, including delayed rewards, partial observability, and scalability.
• To evaluate the practical utility of current RL methods through case studies and implemented systems.
• To highlight open problems and future research directions in scaling and biasing RL for complex tasks.

Proposed method

• Uses the standard RL model with an agent interacting with a dynamic environment through perception, action, and scalar reward signals.
• Applies Markov decision process (MDP) theory as the formal foundation for RL, modeling states, actions, and rewards.
• Introduces and analyzes key algorithms such as Q-learning, TD(λ), and adaptive heuristic critic for learning from delayed rewards.
• Emphasizes function approximation and generalization to handle large or continuous state-action spaces.
• Proposes hierarchical and shaped reward approaches to improve sample efficiency and learning in complex environments.
• Discusses model-based methods that build empirical models to accelerate learning and improve planning.

Experimental results

Research questions

• RQ1 How can an agent balance exploration of unknown actions with exploitation of known high-reward actions in a dynamic environment?
• RQ2 What mechanisms enable effective learning when rewards are delayed, and how can credit assignment be resolved?
• RQ3 How can generalization and function approximation be leveraged to scale RL to large or continuous state spaces?
• RQ4 What strategies mitigate the challenges of partial observability and hidden state in real-world RL applications?
• RQ5 How can bias—through shaping, local rewards, or problem decomposition—improve sample efficiency and scalability in RL?

Key findings

• Reinforcement learning is best framed as a class of problems rather than a fixed set of techniques, with two main approaches: search-based and statistical/dynamic programming-based.
• Q-learning and TD(λ) are effective model-free algorithms for learning optimal policies from delayed rewards, even without a full environment model.
• Function approximation and generalization are essential for scaling RL to large or continuous state spaces, though they introduce challenges in convergence and stability.
• Partial observability and hidden state significantly complicate learning, requiring techniques like belief states or recurrent models.
• Practical success in complex domains (e.g., robotic control) often depends on incorporating human-designed biases such as shaping, local rewards, or reflexes.
• Despite strong theoretical foundations, most current RL techniques do not scale well to large problems without significant inductive bias or architectural innovation.