[Paper Review] A Survey on Quantum Reinforcement Learning

Nico Meyer, Christian Ufrecht+4 more|arXiv (Cornell University)|Nov 7, 2022

Quantum Computing Algorithms and Architecture30 citations

TL;DR

A comprehensive survey of quantum reinforcement learning (QRL), surveying quantum-inspired, variational quantum circuit (VQC) based, projective simulation, Boltzmann machines, and post-NISQ approaches, with emphasis on near-term devices and fault-tolerant prospects.

ABSTRACT

Quantum reinforcement learning is an emerging field at the intersection of quantum computing and machine learning. While we intend to provide a broad overview of the literature on quantum reinforcement learning - our interpretation of this term will be clarified below - we put particular emphasis on recent developments. With a focus on already available noisy intermediate-scale quantum devices, these include variational quantum circuits acting as function approximators in an otherwise classical reinforcement learning setting. In addition, we survey quantum reinforcement learning algorithms based on future fault-tolerant hardware, some of which come with a provable quantum advantage. We provide both a birds-eye-view of the field, as well as summaries and reviews for selected parts of the literature.

Motivation & Objective

• Provide a broad overview of QRL literature and its variants.
• Clarify the interpretation of QRL and distinguish purely classical, NISQ-era, and post-NISQ approaches.
• Summarize key algorithmic families and their hardware requirements.

Proposed method

• Classify QRL algorithms by level of quantum-classical hybridization (from classical to quantum).
• Present detailed summaries of quantum-inspired RL, VQC-based QRL, projective simulation, Boltzmann machines, and quantum subroutine approaches.
• Discuss both near-term NISQ implementations and future fault-tolerant paradigms with possible quantum advantage.
• Explain foundational RL and QC concepts to contextualize QRL approaches.

Experimental results

Research questions

• RQ1 What are the main families of QRL algorithms and how do they differ in quantum-classical hybridization?
• RQ2 What hardware regimes (NISQ vs fault-tolerant) are required for representative QRL approaches and where is quantum advantage expected?
• RQ3 How do variational quantum circuits serve as function approximators in QRL and what are their advantages/limitations?
• RQ4 What is the role of quantum-inspired methods in RL and do they offer intrinsic quantum advantages?
• RQ5 How might QRL be organized and reviewed to guide future research directions?

Key findings

• QRL encompasses quantum-inspired methods and fully quantum approaches using post-NISQ resources and potential quantum advantage.
• VQC-based QRL uses quantum circuits as function approximators within hybrid quantum-classical RL pipelines.
• Several QRL variants rely on subroutines or components that assume future fault-tolerant quantum hardware.
• Near-term work focuses on exploiting NISQ devices, while some post-NISQ approaches aim for provable quantum advantages.
• The survey provides a structured overview and detailed summaries of prominent QRL contributions across multiple subareas.