[Paper Review] Introduction to Quantum Reinforcement Learning: Theory and PennyLane-based Implementation

Yunseok Kwak, Won Joon Yun+3 more|arXiv (Cornell University)|Aug 16, 2021

Quantum Computing Algorithms and Architecture8 citations

TL;DR

This paper introduces a variational quantum circuit-based quantum reinforcement learning (QRL) model using PennyLane, implementing a proximal policy optimization (PPO) framework in the CartPole environment. It demonstrates that quantum policies can learn effective control policies despite high reward variance, highlighting the potential and challenges of QRL with small-scale quantum circuits.

ABSTRACT

The emergence of quantum computing enables for researchers to apply quantum circuit on many existing studies. Utilizing quantum circuit and quantum differential programming, many research are conducted such as extit{Quantum Machine Learning} (QML). In particular, quantum reinforcement learning is a good field to test the possibility of quantum machine learning, and a lot of research is being done. This work will introduce the concept of quantum reinforcement learning using a variational quantum circuit, and confirm its possibility through implementation and experimentation. We will first present the background knowledge and working principle of quantum reinforcement learning, and then guide the implementation method using the PennyLane library. We will also discuss the power and possibility of quantum reinforcement learning from the experimental results obtained through this work.

Motivation & Objective

• To introduce quantum reinforcement learning (QRL) using variational quantum circuits (VQCs) as a quantum machine learning extension of deep reinforcement learning.
• To provide a practical implementation guide for QRL using the PennyLane quantum machine learning library.
• To evaluate the feasibility and performance of QRL in a classical simulation environment using the CartPole-v0 environment.
• To compare the learning behavior of the quantum policy against a random baseline, highlighting stability and exploration trade-offs.

Proposed method

• The QRL model replaces the deep neural network policy in PPO with a parameterized quantum circuit (VQC) implemented in PennyLane.
• The VQC processes state observations (position, velocity, angle, angular velocity) as continuous inputs mapped to the range [−π, π], encoded via single-qubit rotations.
• The quantum circuit outputs measurement probabilities for two actions (left/right), which are used to sample actions via a softmax function.
• The policy parameters are optimized using the Adam optimizer with a learning rate of 1e-3 for the quantum policy and 1e-5 for the classical critic.
• The training uses a reward function with discount factor γ = 0.98 and baseline λ = 0.95, with experience replay and target network techniques adapted from DQN.
• The implementation leverages PyTorch tensors and automatic differentiation in PennyLane for end-to-end gradient-based optimization.

Figure 1: A policy-VQC for deep-Q learning with parameter $\theta$ .

Experimental results

Research questions

• RQ1 Can a variational quantum circuit effectively serve as a policy network in a reinforcement learning framework?
• RQ2 How does the performance of a quantum policy compare to a random policy in the CartPole environment?
• RQ3 What are the stability and convergence characteristics of quantum reinforcement learning under noisy and probabilistic quantum measurements?
• RQ4 To what extent does the quantum policy enable effective exploration with fewer parameters compared to classical deep networks?
• RQ5 What are the key challenges in maintaining high performance in QRL due to quantum system uncertainty?

Key findings

• The quantum reinforcement learning model successfully learns to balance the pole in the CartPole environment, achieving significantly higher average cumulative rewards than the random baseline.
• The model exhibits high variance in episode rewards during training, indicating instability due to quantum measurement uncertainty and probabilistic outcomes.
• Despite high variance, the policy converges toward effective control strategies, demonstrating that quantum circuits can encode meaningful decision-making policies.
• The use of a small 4-qubit VQC enables learning with relatively few parameters, suggesting potential for efficient exploration in quantum policy spaces.
• The performance of the quantum policy remains below state-of-the-art classical DRL baselines, but the results indicate promising potential for future improvements with better error mitigation and hardware.

Figure 2: Quantum Reinforcement Learning System