[Paper Review] Quantum Policy Iteration via Amplitude Estimation and Grover Search -- Towards Quantum Advantage for Reinforcement Learning

Simon Wiedemann, Daniel Hein+2 more|arXiv (Cornell University)|Jun 9, 2022

Quantum Computing Algorithms and Architecture4 citations

TL;DR

This paper proposes a quantum reinforcement learning algorithm that combines amplitude estimation and Grover search to achieve quadratic sample complexity improvement over classical Monte Carlo methods in policy evaluation. By constructing a unitary-based quantum realization of a finite Markov decision process, the method enables quantum policy iteration that provably reduces the number of agent-environment interactions needed to find an optimal policy.

ABSTRACT

We present a full implementation and simulation of a novel quantum reinforcement learning method. Our work is a detailed and formal proof of concept for how quantum algorithms can be used to solve reinforcement learning problems and shows that, given access to error-free, efficient quantum realizations of the agent and environment, quantum methods can yield provable improvements over classical Monte-Carlo based methods in terms of sample complexity. Our approach shows in detail how to combine amplitude estimation and Grover search into a policy evaluation and improvement scheme. We first develop quantum policy evaluation (QPE) which is quadratically more efficient compared to an analogous classical Monte Carlo estimation and is based on a quantum mechanical realization of a finite Markov decision process (MDP). Building on QPE, we derive a quantum policy iteration that repeatedly improves an initial policy using Grover search until the optimum is reached. Finally, we present an implementation of our algorithm for a two-armed bandit MDP which we then simulate.

Motivation & Objective

• To demonstrate a provable quantum advantage in reinforcement learning by reducing sample complexity through quantum algorithms.
• To develop a fully quantum policy iteration framework that replaces classical subroutines with quantum primitives.
• To formalize a quantum mechanical realization of a finite MDP using unitary operators for states, actions, and rewards in superposition.
• To show that quantum policy evaluation (QPE) using amplitude estimation achieves quadratic speedup over classical Monte Carlo estimation.
• To validate the convergence and efficiency of quantum policy iteration through simulation on a two-armed bandit MDP.

Proposed method

• Construct a unitary representation of a finite MDP where agent-environment interactions are encoded as quantum operations on qubit states.
• Implement quantum policy evaluation (QPE) using amplitude estimation to estimate the value function of a policy with quadratically fewer quantum samples than classical Monte Carlo.
• Embed QPE into a policy improvement loop using Grover search to iteratively find better policies, forming a quantum approximate policy iteration scheme.
• Design explicit quantum circuits using single- and multi-qubit gates to realize the policy evaluation and improvement steps on a digital quantum computer.
• Use amplitude estimation with controlled state preparation to estimate expected cumulative rewards in superposition, leveraging quantum phase estimation techniques.
• Apply Grover-like amplification to the policy space to search for ǫ-optimal policies, with the number of iterations growing as O(√N) for N policies.

Experimental results

Research questions

• RQ1 Can quantum amplitude estimation provide a provable sample complexity advantage over classical Monte Carlo policy evaluation in reinforcement learning?
• RQ2 How can Grover search be integrated into a quantum policy iteration framework to improve policies more efficiently than classical methods?
• RQ3 What is the empirical convergence behavior of quantum policy iteration when applied to a finite MDP with a small policy space?
• RQ4 Does the runtime of quantum policy iteration scale as O(√N) with respect to the number of policies N, as predicted by theory?
• RQ5 Under what conditions does the quantum advantage in QPE survive when considering the cost of state preparation for quantum samples?

Key findings

• Quantum policy evaluation (QPE) achieves a quadratic speedup over classical Monte Carlo policy evaluation, requiring quadratically fewer quantum samples to estimate the value function within a given precision.
• Simulations on a two-armed bandit MDP confirm the theoretical quantum advantage of QPE, with QPE converging faster and with fewer required interactions than classical MC methods.
• Quantum policy iteration converges to the optimal policy in a finite number of iterations, with the final success probability reaching 1 after the optimal policy is found.
• The average number of Grover rotations required for policy improvement scales linearly with √N, confirming the predicted O(√N) complexity for N policies.
• Over 99% of simulation runs successfully found an ǫ-optimal policy, indicating high reliability of the quantum policy iteration procedure despite noise and state preparation challenges.
• The runtime distribution of quantum policy iteration is symmetric with a large interquartile range, reflecting stochastic convergence due to random sampling of near-optimal policies in early iterations.