[Paper Review] Quantum Large Language Models via Tensor Network Disentanglers

Borja Aizpurua, Saeed S. Jahromi+2 more|arXiv (Cornell University)|Oct 22, 2024

Topic Modeling4 citations

TL;DR

This paper proposes a hybrid quantum-classical architecture for Large Language Models (LLMs) that replaces traditional dense weight matrices in self-attention and feed-forward layers with a combination of variational quantum circuits and tensor networks, specifically Matrix Product Operators (MPOs). By using quantum disentanglers to decompose weight matrices and retain only the non-entangled residual MPO, the method achieves higher model accuracy than classical LLMs while maintaining low memory overhead, enabling scalable quantum-enhanced language modeling.

ABSTRACT

We propose a method to enhance the performance of Large Language Models (LLMs) by integrating quantum computing and quantum-inspired techniques. Specifically, our approach involves replacing the weight matrices in the Self-Attention and Multi-layer Perceptron layers with a combination of two variational quantum circuits and a quantum-inspired tensor network, such as a Matrix Product Operator (MPO). This substitution enables the reproduction of classical LLM functionality by decomposing weight matrices through the application of tensor network disentanglers and MPOs, leveraging well-established tensor network techniques. By incorporating more complex and deeper quantum circuits, along with increasing the bond dimensions of the MPOs, our method captures additional correlations within the quantum-enhanced LLM, leading to improved accuracy beyond classical models while maintaining low memory overhead.

Motivation & Objective

• To address the high energy consumption and memory demands of classical Large Language Models (LLMs) by integrating quantum computing techniques.
• To overcome the limitations of purely classical tensor network compression by introducing quantum circuits that capture richer correlations.
• To develop a hybrid quantum-classical LLM architecture that maintains compatibility with existing transformer-based models while enabling quantum-enhanced performance.
• To enable practical deployment of quantum-enhanced LLMs on Noisy Intermediate-Scale Quantum (NISQ) hardware through efficient parameterization and error-mitigation strategies.

Proposed method

• Replace dense weight matrices in LLM layers with a three-part structure: a variational quantum circuit (VQC), a Matrix Product Operator (MPO), and a second VQC, forming a unitary disentangler framework.
• Apply tensor network disentanglement to decompose the original weight matrix W into U × MPO_new × V†, where U and V† are unitary quantum circuits that remove entanglement.
• Use the Time-Evolving Block Decimation (TEBD) algorithm to efficiently compute the residual MPO_new from the original MPO_old, ensuring low bond dimension χ_new ≤ χ_old.
• Optimize the quantum circuits variationally using a self-consistent method analogous to the Variational Quantum Eigensolver (VQE) algorithm.
• Encode classical inputs as quantum states using tensor network or quantum GAN-based methods, and extract outputs via qubit measurements.
• Retrain MPO tensors using distributed tensor network retraining techniques, allowing for hybrid optimization of classical and quantum components.

Experimental results

Research questions

• RQ1 Can variational quantum circuits be effectively integrated into the attention and feed-forward layers of LLMs to enhance model expressivity beyond classical limits?
• RQ2 How can quantum circuits be combined with tensor networks to maintain low memory overhead while increasing model accuracy?
• RQ3 What is the optimal decomposition strategy for representing classical LLM weight matrices using quantum circuits and MPOs, and how does it compare to alternative methods like polar decomposition?
• RQ4 Can the resulting hybrid quantum-classical LLM architecture be trained efficiently on NISQ devices with realistic qubit counts and error rates?
• RQ5 To what extent can quantum-enhanced LLMs outperform classical models in accuracy while remaining scalable and deployable on near-term quantum hardware?

Key findings

• The proposed method enables the compression of classical LLM weight matrices into a hybrid quantum-classical representation with polynomial scaling in circuit depth and bond dimension.
• By using unitary disentanglers (U and V†), the method ensures that the quantum circuits remain shallow and the residual MPO has reduced bond dimension, minimizing memory overhead.
• The approach generalizes prior tensor network compression methods by introducing quantum correlations through variational quantum circuits, leading to higher model capacity.
• Numerical results suggest that quantum circuits with hundreds of qubits and multiple layers are sufficient to improve performance on open-source LLMs like LlaMA.
• The method is compatible with classical compression techniques such as quantization, knowledge distillation, and low-rank approximation, enabling synergistic performance gains.
• The framework is designed for deployment on NISQ devices, with input encoding and measurement sampling that can be controlled and compensated via increased circuit depth and MPO bond dimensions.