[Paper Review] CT-DQN: Control-Tutored Deep Reinforcement Learning

Francesco De Lellis, Marco Coraggio+3 more|arXiv (Cornell University)|Dec 2, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

CT-DQN introduces a control-tutored deep reinforcement learning framework that integrates a simple, model-based control tutor into Deep Q-Networks to accelerate learning. By occasionally guiding exploration with heuristic control actions derived from approximate system dynamics, CT-DQN achieves significantly faster convergence and improved data efficiency across three OpenAI Gym environments—achieving better or comparable performance with fewer training episodes than standard DQN.

ABSTRACT

One of the major challenges in Deep Reinforcement Learning for control is the need for extensive training to learn the policy. Motivated by this, we present the design of the Control-Tutored Deep Q-Networks (CT-DQN) algorithm, a Deep Reinforcement Learning algorithm that leverages a control tutor, i.e., an exogenous control law, to reduce learning time. The tutor can be designed using an approximate model of the system, without any assumption about the knowledge of the system's dynamics. There is no expectation that it will be able to achieve the control objective if used stand-alone. During learning, the tutor occasionally suggests an action, thus partially guiding exploration. We validate our approach on three scenarios from OpenAI Gym: the inverted pendulum, lunar lander, and car racing. We demonstrate that CT-DQN is able to achieve better or equivalent data efficiency with respect to the classic function approximation solutions.

Motivation & Objective

• To address the long training times typical in deep reinforcement learning for control by introducing a mechanism to accelerate policy learning.
• To design a framework that leverages a control tutor—based on approximate system models—without requiring full knowledge of system dynamics.
• To improve data efficiency in deep Q-learning by guiding exploration with heuristic control suggestions during training.
• To validate the approach across increasingly complex control tasks, demonstrating robustness and scalability.
• To establish a foundation for formal analysis of tutor design and performance bounds in DRL.

Proposed method

• Introduces CT-DQN, an extension of Deep Q-Networks (DQN) that incorporates a control tutor to suggest actions during training.
• The tutor is a simple feedback control law derived from an approximate model of the system dynamics, not requiring exact knowledge of the true dynamics.
• During learning, the tutor suggests actions with a fixed probability $\omega$, blending its suggestions with the DQN policy's exploration.
• The control tutor uses state-based heuristics: e.g., accelerating if speed is low and angle is small, braking if angle exceeds a threshold.
• The method uses experience replay, target networks, and standard DQN components, with hyperparameters tuned per environment.
• The framework is evaluated on three environments: inverted pendulum, lunar lander, and car racing, using both tabular and deep neural network approximators.

Experimental results

Research questions

• RQ1 Can a simple, model-based control tutor significantly reduce learning time in deep reinforcement learning without prior knowledge of the system's exact dynamics?
• RQ2 How does the integration of a control tutor affect data efficiency and final policy performance in complex control tasks?
• RQ3 Does the quality of the tutor's suggestions correlate with the degree of learning acceleration observed in training?
• RQ4 Can CT-DQN outperform standard DQN in terms of cumulative reward and convergence speed across diverse control scenarios?
• RQ5 What is the impact of tutor design—based on limited system information—on the stability and effectiveness of the final policy?

Key findings

• CT-DQN achieved faster learning convergence than standard DQN across all three environments, with cumulative rewards increasing more rapidly over episodes.
• In the car racing environment, CT-DQN achieved significantly higher greedy policy performance ($J^{\pi_{\text{greedy}}}$) than DQN after 250 training episodes on randomly generated tracks.
• Even with a very simple tutor based on basic heuristics (e.g., angle and speed thresholds), CT-DQN reduced learning time substantially, demonstrating robustness to tutor quality.
• The improvement was most pronounced in complex tasks like lunar lander and car racing, where the tutor provided meaningful guidance despite limited dynamics knowledge.
• The method showed consistent performance gains without compromising policy effectiveness, even when the tutor alone could not solve the task.
• Hyperparameter tuning confirmed that the method is stable and effective across different network architectures and training configurations.