[Paper Review] Quixer: A Quantum Transformer Model

Nikhil Khatri, Gabriel Matos+2 more|arXiv (Cornell University)|Jun 6, 2024

Quantum Computing Algorithms and Architecture4 citations

TL;DR

Quixer proposes a novel quantum transformer model that leverages the Linear Combination of Unitaries (LCU) and Quantum Singular Value Transform (QSVT) to create a trainable, non-linear attention mechanism. It achieves competitive language modeling performance on the Penn Treebank dataset in classical simulation, marking the first application of a quantum transformer to a practical NLP task.

ABSTRACT

Progress in the realisation of reliable large-scale quantum computers has motivated research into the design of quantum machine learning models. We present Quixer: a novel quantum transformer model which utilises the Linear Combination of Unitaries and Quantum Singular Value Transform primitives as building blocks. Quixer operates by preparing a superposition of tokens and applying a trainable non-linear transformation to this mix. We present the first results for a quantum transformer model applied to a practical language modelling task, obtaining results competitive with an equivalent classical baseline. In addition, we include resource estimates for evaluating the model on quantum hardware, and provide an open-source implementation for classical simulation. We conclude by highlighting the generality of Quixer, showing that its parameterised components can be substituted with fixed structures to yield new classes of quantum transformers.

Motivation & Objective

• To design a quantum transformer architecture that leverages quantum algorithmic primitives for efficient token mixing.
• To demonstrate the feasibility of applying quantum models to real-world language modeling tasks, such as on the Penn Treebank dataset.
• To provide resource estimates and an open-source implementation for classical simulation of the model.
• To establish Quixer as a flexible framework by showing that its parameterized components can be replaced with fixed structures to generate new quantum transformer classes.

Proposed method

• The model uses a Linear Combination of Unitaries (LCU) procedure to prepare a superposition of token unitaries, encoding input tokens into quantum amplitudes.
• A Quantum Singular Value Transform (QSVT) is applied to this superposition using a trainable polynomial function encoded in phase rotations, enabling non-linear transformations.
• Token embeddings are mapped to angles that parameterize parameterized quantum circuits (PQCs), which are then unitarily transformed via the LCU and QSVT procedures.
• The final quantum state is measured using multiple observables, and the resulting expectation values are classically processed to produce the model output.
• The model's architecture supports multi-layer extensions by repeating the QSVT and LCU circuits with different data encodings.
• The framework allows for substituting trainable components (e.g., polynomial coefficients or unitary structures) with fixed ones to generate new quantum transformer variants.

Experimental results

Research questions

• RQ1 Can a quantum transformer model based on LCU and QSVT primitives achieve competitive performance on a standard language modeling benchmark?
• RQ2 How does the performance of Quixer compare to classical transformer baselines on the Penn Treebank dataset in a classical simulation setting?
• RQ3 What are the resource requirements and success probabilities for evaluating Quixer on near-term quantum hardware?
• RQ4 Can Quixer be generalized into a flexible framework by replacing trainable components with fixed structures to yield new classes of quantum transformers?
• RQ5 What trade-offs exist between model expressivity, gradient magnitudes, and classical simulability in quantum transformer models?

Key findings

• Quixer achieves language modeling performance competitive with classical transformer baselines on the Penn Treebank dataset when trained and evaluated in classical simulation.
• The model demonstrates the first successful application of a quantum transformer to a practical language modeling task, establishing a proof-of-concept for quantum NLP.
• Resource estimates show that the model's success probability scales with system size, and postselection success probabilities were analyzed across 10 random seeds, with mean and extremal values reported.
• The model's architecture is highly extensible: by fixing polynomial coefficients or unitary structures, Quixer can be adapted to generate new quantum transformer variants, such as recovering the quantum Boltzmann machine with specific choices.
• The framework supports multi-layer extensions by repeating the QSVT and LCU circuits with different data encodings, enabling deeper architectures.
• Despite promising results, the model faces challenges related to gradient vanishing and high parameter count, which may hinder scalability on near-term devices.