[Paper Review] Deep Reinforcement Learning for Adaptive Learning Systems

Xiao Li, Hanchen Xu+2 more|arXiv (Cornell University)|Apr 17, 2020

Reinforcement Learning in Robotics46 references4 citations

TL;DR

This paper formulates adaptive learning as a Markov decision process with continuous latent traits and proposes a model-free deep Q-learning algorithm enhanced by a neural network-based transition model estimator to learn optimal, personalized learning policies from limited data. The method significantly improves sample efficiency, enabling effective policy learning even with as few as 200 learners.

ABSTRACT

In this paper, we formulate the adaptive learning problem---the problem of how to find an individualized learning plan (called policy) that chooses the most appropriate learning materials based on learner's latent traits---faced in adaptive learning systems as a Markov decision process (MDP). We assume latent traits to be continuous with an unknown transition model. We apply a model-free deep reinforcement learning algorithm---the deep Q-learning algorithm---that can effectively find the optimal learning policy from data on learners' learning process without knowing the actual transition model of the learners' continuous latent traits. To efficiently utilize available data, we also develop a transition model estimator that emulates the learner's learning process using neural networks. The transition model estimator can be used in the deep Q-learning algorithm so that it can more efficiently discover the optimal learning policy for a learner. Numerical simulation studies verify that the proposed algorithm is very efficient in finding a good learning policy, especially with the aid of a transition model estimator, it can find the optimal learning policy after training using a small number of learners.

Motivation & Objective

• To address the challenge of learning optimal individualized learning policies in adaptive systems with continuous latent traits.
• To overcome the limitation of model-free DRL in low-data regimes by introducing a data-efficient transition model estimator.
• To improve sample efficiency in training deep Q-networks for adaptive learning using synthetic rollouts from a learned transition model.
• To demonstrate that the proposed method outperforms heuristic and random policies in minimizing learning steps to reach target proficiency levels.
• To enable scalable, personalized learning without requiring teacher supervision or large historical datasets.

Proposed method

• Formulates the adaptive learning problem as a Markov decision process (MDP) with continuous latent traits as states and learning materials as actions.
• Applies a model-free deep Q-learning (DQN) algorithm to learn the optimal policy without prior knowledge of the state transition dynamics.
• Develops a neural network-based transition model estimator that learns to simulate learner progression from historical data.
• Uses the trained transition model estimator to generate synthetic rollouts, enabling more efficient DQN training through data augmentation.
• Trains the DQN on both real interactions and synthetic trajectories from the transition model, improving policy generalization and sample efficiency.
• Employs measurement models (e.g., IRT-inspired) to estimate continuous latent traits from learner responses, forming the state representation.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning effectively learn optimal learning policies for adaptive systems with continuous latent traits?
• RQ2 How does incorporating a learned transition model estimator improve sample efficiency in DRL-based adaptive learning?
• RQ3 To what extent does the proposed method outperform heuristic and random learning policies in terms of convergence speed and final performance?
• RQ4 How does the performance of the DQN policy scale with increasing numbers of real learners?
• RQ5 Can the transition model estimator enable effective policy learning even when only a small number of real learners are available?

Key findings

• The proposed method with a transition model estimator achieves significantly higher mean rewards than the baseline DQN without it, especially with fewer than 200 real learners.
• With up to 200 actual learners, the virtual DQN (trained using the transition model) outperformed the actual DQN (trained only on real data), demonstrating superior sample efficiency.
• When the number of real learners exceeds 200, both the virtual and actual DQN policies converge to similar optimal performance levels.
• The DQN-based policy consistently outperformed heuristic and random strategies, reducing the number of learning steps required to reach target proficiency levels.
• The transition model estimator effectively captures learner progression dynamics, enabling high-quality synthetic rollouts that enhance DQN training.
• The integration of the transition model estimator allows the system to discover a near-optimal learning policy after training on a small number of learners.