[Paper Review] Reinforcement learning for optimization of variational quantum circuit architectures

Mateusz Ostaszewski, Lea M. Trenkwalder+3 more|arXiv (Cornell University)|Mar 30, 2021

Quantum Computing Algorithms and ArchitectureComputer Science40 references54 citations

TL;DR

The paper presents a deep reinforcement learning framework with intrinsic curriculum learning to automatically construct shallow, gate-efficient variational quantum circuit (VQE) ansätze, achieving chemical accuracy for LiH while minimizing circuit depth. It compares favorably to HE and UCCSD baselines and scales to larger qubit counts via a moving-threshold training strategy.

ABSTRACT

The study of Variational Quantum Eigensolvers (VQEs) has been in the spotlight in recent times as they may lead to real-world applications of near-term quantum devices. However, their performance depends on the structure of the used variational ansatz, which requires balancing the depth and expressivity of the corresponding circuit. In recent years, various methods for VQE structure optimization have been introduced but the capacities of machine learning to aid with this problem has not yet been fully investigated. In this work, we propose a reinforcement learning algorithm that autonomously explores the space of possible ans{ä}tze, identifying economic circuits which still yield accurate ground energy estimates. The algorithm is intrinsically motivated, and it incrementally improves the accuracy of the result while minimizing the circuit depth. We showcase the performance of our algorithm on the problem of estimating the ground-state energy of lithium hydride (LiH). In this well-known benchmark problem, we achieve chemical accuracy, as well as state-of-the-art results in terms of circuit depth.

Motivation & Objective

• Motivate and address the need for architecture optimization in variational quantum eigensolvers (VQEs under NISQ constraints.

Proposed method

• Formulate ansatz construction as a reinforcement learning task with DDQN and a discrete action space over gate insertions.
• Represent circuit state as an ordered list of gate-layer entries; optimize rotation angles via a classical subroutine (COBYLA or Rotosolve) separate from the RL agent.
• Use a reward structure that strongly incentivizes reaching chemical accuracy while penalizing excessive depth (max layers L) and rewarding energy improvement.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent autonomously construct compact, accurate VQE ansätze for LiH under NISQ constraints?
• RQ2 How does intrinsic curriculum learning influence the agent's ability to reach chemical accuracy with shallow circuits?
• RQ3 What is the impact of global versus local angle optimization on circuit depth and gate count?
• RQ4 How does the moving-threshold approach perform when exact ground-state energy is unavailable or approximated?
• RQ5 How does the RL-derived architecture compare to HE and UCCSD baselines in depth and gate efficiency?

Key findings

• The RL approach yields chemical accuracy with shallower circuits than HE and UCCSD in most cases for 4-qubit LiH across bond distances 1.2Å, 2.2Å, and 3.4Å.
• With 6-qubit LiH at 2.2Å, the moving-threshold curriculum RL achieves chemical accuracy in 2 of 10 trials, with avg depth 14 and min depth 12, avg gates 36 and min gates 29 under global COBYLA optimization.
• For the 6-qubit case, the RL method generates circuits roughly five times shallower than the UCCSD baseline reported in the comparison, and shallower than HE in average cases.
• COBYLA generally outperforms Rotosolve in producing shallower circuits that reach chemical accuracy under the RL framework.
• The intrinsically motivated moving-threshold curriculum enables learning without requiring prior knowledge of the exact ground-state energy, even when using a lower bound proxy.