[Paper Review] Hybrid actor-critic algorithm for quantum reinforcement learning at CERN beam lines

Michael Schenk, Elías F. Combarro+5 more|arXiv (Cornell University)|Sep 22, 2022

Advancements in Semiconductor Devices and Circuit Design4 citations

TL;DR

This paper proposes a hybrid actor-critic reinforcement learning algorithm that integrates a classical deep deterministic policy gradient (DDPG) actor with a quantum Boltzmann machine (QBM)-based critic for continuous state-action space control in particle accelerator beam lines. It demonstrates improved sample efficiency over classical DQN in discrete-action environments and successfully deploys a trained agent on the real AWAKE electron beam line using both simulated and D-Wave quantum annealing hardware, with the hybrid model enabling inference using only the classical actor network.

ABSTRACT

Free energy-based reinforcement learning (FERL) with clamped quantum Boltzmann machines (QBM) was shown to significantly improve the learning efficiency compared to classical Q-learning with the restriction, however, to discrete state-action space environments. In this paper, the FERL approach is extended to multi-dimensional continuous state-action space environments to open the doors for a broader range of real-world applications. First, free energy-based Q-learning is studied for discrete action spaces, but continuous state spaces and the impact of experience replay on sample efficiency is assessed. In a second step, a hybrid actor-critic scheme for continuous state-action spaces is developed based on the Deep Deterministic Policy Gradient algorithm combining a classical actor network with a QBM-based critic. The results obtained with quantum annealing, both simulated and with D-Wave quantum annealing hardware, are discussed, and the performance is compared to classical reinforcement learning methods. The environments used throughout represent existing particle accelerator beam lines at the European Organisation for Nuclear Research (CERN). Among others, the hybrid actor-critic agent is evaluated on the actual electron beam line of the Advanced Plasma Wakefield Experiment (AWAKE).

Motivation & Objective

• To extend free energy-based reinforcement learning (FERL) with quantum Boltzmann machines (QBM) from discrete to continuous state-action spaces, enabling broader real-world application in accelerator control.
• To develop a hybrid actor-critic architecture combining classical deep reinforcement learning with QBM-based critics for continuous control tasks in complex beamline environments.
• To evaluate the sample efficiency and performance of the hybrid quantum-classical RL agent against classical baselines using both simulated quantum annealing and real D-Wave quantum annealing hardware.
• To validate the sim-to-real transfer capability of the trained agent on the actual AWAKE electron beam line at CERN.
• To demonstrate practical deployment feasibility by showing that only the classical actor network is needed at inference, simplifying real-world integration.

Proposed method

• The hybrid actor-critic framework replaces the classical critic in DDPG with a clamped quantum Boltzmann machine (QBM) to estimate Q-values using free energy minimization.
• The QBM critic is trained via free energy-based reinforcement learning (FERL), leveraging quantum annealing to compute the Q-function approximation.
• Experience replay is incorporated in the discrete-action phase to enhance sample efficiency, particularly in continuous state spaces.
• The algorithm is first validated in a one-dimensional beam steering task with continuous states and discrete actions before scaling to a ten-dimensional continuous state-action space.
• Training is performed using both simulated quantum annealing (SQA) and real D-Wave Advantage quantum annealing hardware to assess performance differences.
• At inference, only the classical actor network is used, enabling practical deployment in accelerator control rooms without requiring quantum hardware.

Experimental results

Research questions

• RQ1 Can free energy-based reinforcement learning (FERL) with QBM critics be extended to continuous state-action spaces, overcoming the prior limitation to discrete action spaces?
• RQ2 How does the inclusion of experience replay affect sample efficiency in FERL for continuous state spaces with discrete actions?
• RQ3 Does the hybrid quantum-classical actor-critic model outperform classical DDPG in terms of sample efficiency and convergence speed in continuous control tasks?
• RQ4 What is the performance gap between simulated quantum annealing and real D-Wave quantum annealing hardware in training the hybrid agent?
• RQ5 Can a hybrid agent trained in simulation successfully transfer to and perform reliably in the real-world AWAKE electron beam line environment?

Key findings

• FERL with QBM critics achieves significantly higher sample efficiency than classical DQN in the discrete-action, continuous-state TT24-T4 proton beam line task, with or without experience replay.
• Experience replay further improves sample efficiency in the discrete-action setting, reducing the number of interactions needed to reach the reward objective.
• The hybrid actor-critic model successfully solves the ten-dimensional continuous state-action control problem on the AWAKE electron beam line in both simulation and real-world deployment.
• The agent trained on D-Wave quantum annealing hardware shows marginally better average performance than the SQA-trained agent, particularly in final reward distributions and step count efficiency.
• Both SQA- and D-Wave-trained agents generalize from simulation to the real AWAKE beam line without additional fine-tuning, demonstrating robust sim-to-real transfer.
• The trained hybrid agent fails to solve the task for one initial state in the real environment when using D-Wave hardware, likely due to incomplete training convergence from limited QPU time.