[Paper Review] Quantum Reinforcement Learning for Solving a Stochastic Frozen Lake Environment and the Impact of Quantum Architecture Choices

Theodora-Augustina Drăgan, Maureen Monnet+2 more|arXiv (Cornell University)|Dec 15, 2022

Quantum Computing Algorithms and Architecture4 citations

TL;DR

This paper proposes a hybrid quantum-classical Proximal Policy Optimization (PPO) agent that replaces classical neural networks with parametrized quantum circuits (PQCs) to solve a stochastic, slippery 4x4 Frozen Lake environment. It evaluates multiple quantum circuit architectures and finds that while all PQCs achieve strong performance—reducing training steps by up to a third and using only one-third the parameters—no direct correlation exists between quantum circuit metrics (expressibility, entanglement capability, effective dimension) and reinforcement learning performance, indicating that circuit design must be determined empirically.

ABSTRACT

Quantum reinforcement learning (QRL) models augment classical reinforcement learning schemes with quantum-enhanced kernels. Different proposals on how to construct such models empirically show a promising performance. In particular, these models might offer a reduced parameter count and shorter times to reach a solution than classical models. It is however presently unclear how these quantum-enhanced kernels as subroutines within a reinforcement learning pipeline need to be constructed to indeed result in an improved performance in comparison to classical models. In this work we exactly address this question. First, we propose a hybrid quantum-classical reinforcement learning model that solves a slippery stochastic frozen lake, an environment considerably more difficult than the deterministic frozen lake. Secondly, different quantum architectures are studied as options for this hybrid quantum-classical reinforcement learning model, all of them well-motivated by the literature. They all show very promising performances with respect to similar classical variants. We further characterize these choices by metrics that are relevant to benchmark the power of quantum circuits, such as the entanglement capability, the expressibility, and the information density of the circuits. However, we find that these typical metrics do not directly predict the performance of a QRL model.

Motivation & Objective

• To develop a hybrid quantum-classical reinforcement learning model capable of solving a challenging stochastic variant of the Frozen Lake environment.
• To evaluate the impact of different parametrized quantum circuit (PQC) architectures on the performance of quantum reinforcement learning (QRL).
• To investigate whether established quantum circuit metrics—expressibility, entanglement capability, and effective dimension—predict QRL performance.
• To provide empirical guidance on quantum circuit design for QRL, given the lack of theoretical predictability from standard metrics.

Proposed method

• A hybrid quantum-classical PPO algorithm is implemented, where the policy network is replaced by a parametrized quantum circuit (PQC).
• The PPO agent is trained on a stochastic 4x4 Frozen Lake environment with 80% probability of moving in the intended direction and 20% in orthogonal directions.
• Six distinct PQC architectures are evaluated, each based on literature-suggested designs for hardware efficiency and problem-specific subroutines.
• Quantum circuit performance is quantified using three metrics: expressibility, entanglement capability, and effective dimension (ED), which allows comparison with classical models.
• Training is conducted via simulation, with performance measured by maximum return (MR) and training convergence time (TTC).
• Results are analyzed to assess correlations between quantum circuit metrics and RL performance across different PQC designs.

Experimental results

Research questions

• RQ1 Does replacing the classical policy network with a parametrized quantum circuit improve performance in a stochastic reinforcement learning environment?
• RQ2 Which quantum circuit architectures yield the best performance in solving the stochastic Frozen Lake environment?
• RQ3 To what extent do standard quantum circuit metrics—expressibility, entanglement capability, and effective dimension—predict the performance of quantum reinforcement learning agents?
• RQ4 Why do some quantum circuits with lower theoretical metrics outperform others with higher metrics in practice?
• RQ5 Can quantum-enhanced kernels reduce the number of trainable parameters and accelerate convergence in reinforcement learning?

Key findings

• The hybrid quantum-classical PPO model successfully solves the stochastic 4x4 Frozen Lake environment, achieving a maximum return comparable to classical models.
• The quantum-enhanced agent converges in approximately one-third the number of training steps compared to classical baselines.
• The quantum circuits reduce the number of trainable parameters to about one-third of those in classical models.
• Circuit 6 achieves the highest maximum return and also ranks best in expressibility and effective dimension, but no consistent correlation is found across all metrics and performance measures.
• Circuits with similar or better expressibility and entanglement capability (e.g., circuit 13) perform worse than others (e.g., circuit 6), indicating that metrics alone do not predict performance.
• The position of entangling gates significantly affects performance, a factor not captured by the three metrics, suggesting additional architectural design factors are at play.