[Paper Review] Is attention all you need to solve the correlated electron problem?

Max Geier, Khachatur G. Nazaryan+2 more|ArXiv.org|Feb 7, 2025

Electron and X-Ray Spectroscopy Techniques3 citations

TL;DR

The paper uses a large-parameter self-attention neural network as a variational wavefunction within VMC to solve interacting electron problems in moiré solids, finding parameter counts scale roughly as N^2 and achieving accurate results beyond Hartree-Fock and BP-ED.

ABSTRACT

The attention mechanism has transformed artificial intelligence research by its ability to learn relations between objects. In this work, we explore how a many-body wavefunction ansatz constructed from a large-parameter self-attention neural network can be used to solve the interacting electron problem in solids. By a systematic neural-network variational Monte Carlo study on a moiré quantum material, we demonstrate that the self-attention ansatz provides an accurate and efficient solution without human bias. Moreover, our numerical study finds that the required number of variational parameters scales roughly as $N^2$ with the number of electrons, which opens a path towards efficient large-scale simulations.

Motivation & Objective

• Motivate the use of neural network variational Monte Carlo (NN-VMC) for solving the many-electron problem in solids.
• Introduce a self-attention based neural network wavefunction that generates generalized Slater determinants.
• Demonstrate accuracy and efficiency of the attention-based ansatz on moiré semiconductor systems.
• Assess how the number of variational parameters scales with system size and compare to traditional methods.

Proposed method

• Construct a SlaterNet to generate unrestricted Hartree-Fock-like orbitals via a deep feed-forward network.
• Incorporate self-attention to mix particle streams and capture correlations, producing correlated orbitals.
• Form a multi-determinant wavefunction as a sum of determinants built from correlated orbitals (Psi = sum det()).
• Optimize the variational parameters using variational Monte Carlo with the energy objective and natural gradient (KFAC approximation).
• Benchmark against Hartree-Fock and band-projected exact diagonalization for WSe2/WS2 moiré heterobilayers.

Figure 1: Architecture of the neural network wavefunction ansatz. SlaterNet : Multilayer perceptron neural network generates one-body orbitals to approximate general single Slater determinant wavefunctions. Psi-Solid : Self-attention neural network for solids based on Psi-Former [ 11 ] capturing cor

Experimental results

Research questions

• RQ1 Can a self-attention based neural network wavefunction serve as a universal, scalable ansatz for interacting electrons in solids?
• RQ2 How does the required number of variational parameters scale with the number of electrons in a moiré material system?
• RQ3 How does the NN-VMC approach compare to conventional methods (HF, BP-ED) in accuracy for moiré semiconductor Hamiltonians?
• RQ4 What is the performance of a self-attention NN wavefunction in capturing electron correlations in two-dimensional moiré lattices?
• RQ5 Does the attention-based ansatz remain accurate as system size grows toward large-scale simulations?

Key findings

• The self-attention NN wavefunction provides an accurate, efficient, and unbiased solution to the correlated electron problem in the studied moiré system.
• For the moiré Hamiltonian, the saturation energy can be reached with a number of parameters N_par that scales approximately as N^2 (N_par* ≈ 320 × N^2.1).
• Self-attention based wavefunctions outperform band-projected exact diagonalization when five bands are included in benchmarks.
• The approach remains robust as system size increases, suggesting scalability to larger, more realistic solids.
• The method achieves lower energies than BP-ED for small systems and captures correlation effects beyond Hartree-Fock.

Figure 2: Building blocks of variational Monte Carlo. In the Monte Carlo algorithm, the wavefunction ansatz $\Psi_{\theta}$ is constructed and sampled to efficiently evaluate the optimization goal $L[\theta]:=\langle\Psi_{\theta}|\hat{H}|\Psi_{\theta}\rangle$ of minimizing the energy. Accordingly up