[Paper Review] Deep reinforcement learning for quantum Hamiltonian engineering

Pai Peng, Xiaoyang Huang+4 more|arXiv (Cornell University)|Feb 25, 2021

Quantum many-body systems4 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL) framework to discover optimal quantum Hamiltonian engineering sequences for decoupling spin-1/2 systems, outperforming conventional sequences like Cory48 in both simulations and experiments on solid-state NMR platforms. The method learns robust pulse sequences under realistic imperfections, revealing a previously unknown symmetric control pattern that enables systematic sequence optimization and improved fidelity, especially at long timescales.

ABSTRACT

Engineering desired Hamiltonian in quantum many-body systems is essential for applications such as quantum simulation, computation and sensing. Conventional quantum Hamiltonian engineering sequences are designed using human intuition based on perturbation theory, which may not describe the optimal solution and is unable to accommodate complex experimental imperfections. Here we numerically search for Hamiltonian engineering sequences using deep reinforcement learning (DRL) techniques and experimentally demonstrate that they outperform celebrated sequences on a solid-state nuclear magnetic resonance quantum simulator. As an example, we aim at decoupling strongly-interacting spin-1/2 systems. We train DRL agents in the presence of different experimental imperfections and verify robustness of the output sequences both in simulations and experiments. Surprisingly, many of the learned sequences exhibit a common pattern that had not been discovered before, to our knowledge, but has an meaningful analytical description. We can thus restrict the searching space based on this control pattern, allowing to search for longer sequences, ultimately leading to sequences that are robust against dominant imperfections in our experiments. Our results not only demonstrate a general method for quantum Hamiltonian engineering, but also highlight the importance of combining black-box artificial intelligence with understanding of physical system in order to realize experimentally feasible applications.

Motivation & Objective

• To overcome the limitations of intuition-based and perturbative methods in designing optimal quantum control sequences for many-body systems.
• To develop a model-free, data-driven approach that can handle complex experimental imperfections and non-integrable dynamics.
• To demonstrate the experimental feasibility and superiority of DRL-learned sequences in real solid-state NMR systems.
• To uncover hidden structural patterns in learned sequences that enable systematic extension and robustness to dominant imperfections.

Proposed method

• A deep reinforcement learning agent is trained to generate pulse sequences that maximize the fidelity of the target evolution (e.g., decoupling) in a simulated spin-1/2 system with dipolar interactions.
• The DRL agent operates in a Markov decision process, where the state encodes the current spin system evolution, and actions correspond to applying specific pulses (e.g., π/2 rotations along x, y, etc.).
• The environment simulator computes the time-evolution operator (propagator) after each action, and the agent receives a dense reward based on the fidelity to the desired identity evolution.
• The agent is trained under various experimental imperfections (e.g., angle errors, drifts) to ensure robustness, and the resulting sequences are validated in both simulations and on a 300 MHz NMR spectrometer.
• A symmetry-based analysis is applied post-training to identify and exploit a common structural pattern (e.g., symmetric pulse sequences with z-axis rotation invariance), enabling sequence extension and higher-order error cancellation.
• The method leverages average Hamiltonian theory (AHT) to analytically verify and refine the learned sequences, particularly by canceling zeroth- and first-order error terms through sequence symmetry and phase shifts.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning discover quantum control sequences that outperform classically designed sequences in decoupling strongly interacting spin systems?
• RQ2 How can DRL be trained to produce sequences robust against realistic experimental imperfections such as angle errors and drifts?
• RQ3 Do DRL-learned sequences exhibit hidden structural patterns that can be exploited for analytical understanding and sequence extension?
• RQ4 Can the combination of black-box AI and physical insight lead to more efficient and experimentally feasible quantum control protocols?

Key findings

• The DRL-learned yxx48 sequence outperforms the benchmark Cory48 sequence in both simulations and experiments on calcium fluoride and fluorapatite, showing superior long-term fidelity.
• In experiments on powdered adamantane, the yxx48 sequence matches the performance of Cory48 at short times and surpasses it at longer times, demonstrating improved robustness.
• The yxx24 and yxx48 sequences are experimentally verified to maintain high fidelity over 128 cycles, with the yxx48 sequence showing stable performance despite initial lower fidelity.
• A previously unknown symmetric control pattern—characterized by z-axis rotational invariance and mirror symmetry—was discovered in the DRL-learned sequences, enabling analytical sequence extension.
• By applying a π phase shift to specific pulses in the Angle12 sequence, the zeroth-order average Hamiltonian is canceled, leading to the modified Angle12 sequence with improved error suppression.
• The symmetrized and modified Angle12 sequence, when doubled with a π phase shift, yields the yxx24 sequence with vanishing zeroth- and first-order average Hamiltonians, confirming its high-order error suppression.