[Paper Review] Transformer-QEC: Quantum Error Correction Code Decoding with Transferable Transformers

Hanrui Wang, Pengyu Liu+6 more|arXiv (Cornell University)|Nov 27, 2023

Quantum Computing Algorithms and Architecture4 citations

TL;DR

This paper proposes Transformer-QEC, a transferable transformer-based quantum error correction decoder that uses self-attention for global syndrome analysis and mixed loss training to achieve superior logical error rates across multiple code distances. It reduces training cost by over 10× through transfer learning, outperforming non-ML and existing ML decoders like Union Find, MWPM, and MLPs.

ABSTRACT

Quantum computing has the potential to solve problems that are intractable for classical systems, yet the high error rates in contemporary quantum devices often exceed tolerable limits for useful algorithm execution. Quantum Error Correction (QEC) mitigates this by employing redundancy, distributing quantum information across multiple data qubits and utilizing syndrome qubits to monitor their states for errors. The syndromes are subsequently interpreted by a decoding algorithm to identify and correct errors in the data qubits. This task is complex due to the multiplicity of error sources affecting both data and syndrome qubits as well as syndrome extraction operations. Additionally, identical syndromes can emanate from different error sources, necessitating a decoding algorithm that evaluates syndromes collectively. Although machine learning (ML) decoders such as multi-layer perceptrons (MLPs) and convolutional neural networks (CNNs) have been proposed, they often focus on local syndrome regions and require retraining when adjusting for different code distances. We introduce a transformer-based QEC decoder which employs self-attention to achieve a global receptive field across all input syndromes. It incorporates a mixed loss training approach, combining both local physical error and global parity label losses. Moreover, the transformer architecture's inherent adaptability to variable-length inputs allows for efficient transfer learning, enabling the decoder to adapt to varying code distances without retraining. Evaluation on six code distances and ten different error configurations demonstrates that our model consistently outperforms non-ML decoders, such as Union Find (UF) and Minimum Weight Perfect Matching (MWPM), and other ML decoders, thereby achieving best logical error rates. Moreover, the transfer learning can save over 10x of training cost.

Motivation & Objective

• To address the limitations of existing quantum error correction (QEC) decoders, such as restricted receptive fields and high retraining costs when changing code distances.
• To develop a machine learning-based QEC decoder that achieves high accuracy across varying code distances without retraining from scratch.
• To enable efficient transfer learning in QEC decoding by leveraging the transformer architecture’s flexibility for variable-length inputs.
• To improve decoding performance by modeling long-range error correlations through self-attention mechanisms.
• To reduce training overhead while maintaining or improving logical error rate performance compared to state-of-the-art decoders.

Proposed method

• The model uses a transformer encoder-decoder architecture to process 3D syndrome inputs (spatial x, y, and temporal rounds) with 3D sinusoidal positional encoding to preserve spatial and temporal context.
• Input syndrome bits are embedded into learnable token embeddings, enabling the model to process variable-length syndrome sequences across different code distances.
• A mixed loss function combines local physical error loss (per-qubit prediction) and global parity label loss (logical outcome correctness) to improve generalization and error detection reliability.
• The model is pre-trained on distance-5 surface code data and fine-tuned for other distances using transfer learning with a fixed, low learning rate.
• Self-attention mechanisms allow the model to capture long-range correlations between distant syndrome qubits, crucial for detecting error chains.
• The decoder is evaluated on six code distances and ten error configurations, with performance compared against Union Find, MWPM, and MLP-based decoders.

Figure 1. Transformer for error correction decoding overview. The Transformer takes syndrome inputs and processes them through both the transformer model encoder and decoder. The output of this process consists of error predictions. One notable advantage is its ability to be seamlessly applied to di

Experimental results

Research questions

• RQ1 Can a transformer-based decoder achieve better logical error rates than classical decoders like Union Find and MWPM across multiple code distances?
• RQ2 Can a single pre-trained model be effectively fine-tuned for different code distances without full retraining, reducing training cost?
• RQ3 Does the use of a mixed loss function—combining local and global supervision—improve decoding accuracy and generalization?
• RQ4 How does the global receptive field of self-attention compare to local models like CNNs and MLPs in detecting complex error patterns?
• RQ5 What is the impact of model size and transfer learning on decoding performance and generalization across code distances?

Key findings

• Transformer-QEC achieves the lowest logical error rates across all tested code distances (from 3 to 10) and physical error rates, outperforming Union Find, MWPM, and MLP decoders.
• The model reduces training cost by over 10× through transfer learning, with fine-tuning on new code distances requiring only 10 epochs and a fixed learning rate.
• The ablation study shows that combining local and global loss improves performance in nine out of ten cases, with the global loss enhancing logical outcome accuracy.
• The larger model (7.9M parameters) outperforms the smaller one (503K parameters), with no sign of overfitting due to the large training dataset.
• The model achieves 43% higher true positive accuracy (class 1) than baseline MLPs, indicating better detection of actual errors in syndrome patterns.
• The threshold for logical error rate reduction is approximately 0.038, indicating that Transformer-QEC maintains fault tolerance up to this physical error rate.

Figure 2. The surface code contains data qubits and two kinds of syndrome qubits. X-type syndrome qubits in green checks Z errors while Z-type syndrome qubits in yellow check X errors. When error occurs on data qubits, the nearby syndromes may be flipped depending on the parity of data qubits. When