[Paper Review] Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attention

Ethan N. Evans, Matthew Cook+2 more|arXiv (Cornell University)|Mar 21, 2024

Neural Networks and Reservoir Computing4 citations

TL;DR

This paper proposes SASQuaTCh, a novel variational quantum transformer architecture that leverages the quantum Fourier transform (QFT) to implement kernel-based self-attention, enabling efficient sequence modeling. By encoding data into quantum amplitudes and applying multi-qubit QFTs, the model achieves exponential speedup potential in kernel computation, demonstrating competitive performance on image classification tasks with reduced parameter complexity compared to classical transformers.

ABSTRACT

The recent exploding growth in size of state-of-the-art machine learning models highlights a well-known issue where exponential parameter growth, which has grown to trillions as in the case of the Generative Pre-trained Transformer (GPT), leads to training time and memory requirements which limit their advancement in the near term. The predominant models use the so-called transformer network and have a large field of applicability, including predicting text and images, classification, and even predicting solutions to the dynamics of physical systems. Here we present a variational quantum circuit architecture named Self-Attention Sequential Quantum Transformer Channel (SASQuaTCh), which builds networks of qubits that perform analogous operations of the transformer network, namely the keystone self-attention operation, and leads to an exponential improvement in parameter complexity and run-time complexity over its classical counterpart. Our approach leverages recent insights from kernel-based operator learning in the context of predicting spatiotemporal systems to represent deep layers of a vision transformer network using simple gate operations and a set of multi-dimensional quantum Fourier transforms. To validate our approach, we consider image classification tasks in simulation and with hardware, where with only 9 qubits and a handful of parameters we are able to simultaneously embed and classify a grayscale image of handwritten digits with high accuracy.

Motivation & Objective

• To develop a quantum neural network architecture that efficiently implements self-attention using quantum Fourier transforms.
• To reduce the number of trainable parameters in attention mechanisms by replacing learnable weight matrices with fixed QFT operations.
• To explore the feasibility of using quantum circuits to emulate the kernel-based nature of self-attention in classical transformers.
• To evaluate the model's performance on simplified image classification tasks using variational quantum circuits.
• To analyze the computational and parameter complexity of the proposed quantum architecture in comparison to classical counterparts.

Proposed method

• The model uses a variational quantum circuit where the self-attention mechanism is implemented via a quantum Fourier transform (QFT) acting on qubit amplitudes.
• Input data is embedded into the amplitudes of an n-qubit quantum state, enabling superposition-based processing.
• The QFT performs channel mixing across tokens, effectively computing attention scores in the Fourier domain without learnable query/key/value matrices.
• A parameterized ansatz U_p(θ) is applied to the encoded state, with parameters θ updated via classical optimization to minimize loss.
• A single readout qubit is used to measure the output, with conditional rotations encoding the final classification decision.
• The approach exploits the fact that self-attention can be viewed as a convolution with a stationary kernel, which the QFT naturally computes.

Figure 1 : Visual representation of a transformer model with self-attention. This figure is taken with permission from [ 6 ] .

Experimental results

Research questions

• RQ1 Can the quantum Fourier transform be used to efficiently implement the self-attention mechanism in a quantum transformer without learnable attention weights?
• RQ2 What is the computational and parameter complexity of a QFT-based quantum attention layer compared to classical self-attention?
• RQ3 How does the performance of a QFT-based quantum transformer compare to classical transformers on small-scale classification tasks?
• RQ4 Can the kernel-based interpretation of self-attention be naturally realized in a quantum circuit using QFT?
• RQ5 What are the limitations of state preparation and measurement in realizing such a quantum attention mechanism?

Key findings

• The SASQuaTCh architecture achieves 92–97% of the accuracy of state-of-the-art BERT models on the GLUE benchmark when using unparameterized Fourier transforms, suggesting that QFT-based attention can be highly effective.
• The model reduces the number of trainable parameters significantly by replacing learnable query, key, and value matrices with fixed QFT operations.
• The QFT enables exponential speedup in kernel computation, with O(n²) gate operations on n qubits, compared to O(n·2ⁿ) in classical FFT.
• The model demonstrates feasibility on simplified image classification tasks, showing that quantum circuits can emulate attention mechanisms with minimal parameter overhead.
• The approach leverages the kernel nature of self-attention, where the QFT acts as a natural kernel operator in the frequency domain.
• Despite the potential speedup, state preparation and measurement remain non-trivial bottlenecks, as amplitudes cannot be directly accessed without repeated measurements.