[Paper Review] Optimal Hierarchical Learning Path Design with Reinforcement Learning

Xiao Li, Hanchen Xu+2 more|arXiv (Cornell University)|Oct 12, 2018

Online Learning and Analytics4 citations

TL;DR

This paper proposes a model-free reinforcement learning framework for optimizing personalized learning paths in e-learning systems by modeling hierarchical skills and proficiency levels using cognitive diagnosis models (CDMs) and hidden Markov models (HMMs). The method learns optimal learning strategies without prior knowledge of transition dynamics, outperforming heuristic methods in reward and convergence speed, even under estimation errors.

ABSTRACT

E-learning systems are capable of providing more adaptive and efficient learning experiences for students than the traditional classroom setting. A key component of such systems is the learning strategy, the algorithm that designs the learning paths for students based on information such as the students' current progresses, their skills, learning materials, and etc. In this paper, we address the problem of finding the optimal learning strategy for an E-learning system. To this end, we first develop a model for students' hierarchical skills in the E-learning system. Based on the hierarchical skill model and the classical cognitive diagnosis model, we further develop a framework to model various proficiency levels of hierarchical skills. The optimal learning strategy on top of the hierarchical structure is found by applying a model-free reinforcement learning method, which does not require information on students' learning transition process. The effectiveness of the proposed framework is demonstrated via numerical experiments.

Motivation & Objective

• To address the lack of integration between skill hierarchy and proficiency levels in existing e-learning systems.
• To develop a hierarchical learning model that explicitly represents skill structures and mastery levels using CDMs and HMMs.
• To design an optimal learning strategy using model-free reinforcement learning that does not require prior knowledge of student transition dynamics.
• To evaluate the performance of the proposed RL-based strategy against heuristic methods under realistic conditions, including estimation errors.
• To demonstrate the robustness and generalization of the RL method across diverse initial learner proficiency levels.

Proposed method

• A hierarchical skill model is constructed based on the attribute hierarchy model, representing skills as binary attributes with proficiency levels.
• The learning process is modeled as a Markov decision process (MDP), where student states are defined by their attribute profiles and updated via HMMs based on response data.
• Cognitive diagnosis models (CDMs) are used to estimate students’ hidden attribute profiles from their responses to assessment items.
• A model-free reinforcement learning algorithm is applied to learn the optimal policy for selecting learning materials without requiring knowledge of the underlying transition kernel.
• The system uses response data to iteratively update state estimates and optimize the learning path in a data-driven manner.
• The framework is trained using simulated episodes, with performance evaluated via cumulative rewards and episode length.

Experimental results

Research questions

• RQ1 How can skill hierarchy and proficiency levels be effectively modeled in an e-learning system to support personalized learning?
• RQ2 Can a model-free reinforcement learning approach learn an optimal learning strategy without prior knowledge of student transition dynamics?
• RQ3 How does the performance of the RL-based strategy compare to heuristic strategies in terms of reward and convergence speed?
• RQ4 How robust is the RL-based strategy under estimation errors in student attribute profiling?
• RQ5 Can the learned strategy generalize across learners with diverse initial skill profiles?

Key findings

• The model-free reinforcement learning method achieved a mean reward of 6.43 under no estimation error, significantly outperforming the heuristic method’s mean reward of 3.99.
• The RL method reduced the mean episode length to 7.34, compared to 8.57 for the heuristic method, indicating faster learning progression.
• The standard deviation of rewards was lower under the RL method (3.61) than the heuristic (5.34), indicating more consistent performance.
• Even with a 5% estimation error, the RL method maintained a mean reward of 6.41, showing robustness to real-world measurement noise.
• The RL method converged to the optimal strategy within 200 episodes across all tested initial states, demonstrating fast adaptation to diverse learner profiles.
• The box plot in Figure 9 confirms that the RL method consistently outperformed the heuristic method across 11 different estimation error levels.