[Paper Review] Is Attention Better Than Matrix Decomposition?

Zhengyang Geng, Meng-Hao Guo+4 more|arXiv (Cornell University)|Jan 1, 2021

Domain Adaptation and Few-Shot Learning73 references52 citations

TL;DR

The paper shows that a matrix-decomposition–based global context module (Hamburger) can match or exceed self-attention in vision tasks, with lower computation and memory costs, and proposes a one-step gradient for training through the MD loop.

ABSTRACT

As an essential ingredient of modern deep learning, attention mechanism, especially self-attention, plays a vital role in the global correlation discovery. However, is hand-crafted attention irreplaceable when modeling the global context? Our intriguing finding is that self-attention is not better than the matrix decomposition (MD) model developed 20 years ago regarding the performance and computational cost for encoding the long-distance dependencies. We model the global context issue as a low-rank recovery problem and show that its optimization algorithms can help design global information blocks. This paper then proposes a series of Hamburgers, in which we employ the optimization algorithms for solving MDs to factorize the input representations into sub-matrices and reconstruct a low-rank embedding. Hamburgers with different MDs can perform favorably against the popular global context module self-attention when carefully coping with gradients back-propagated through MDs. Comprehensive experiments are conducted in the vision tasks where it is crucial to learn the global context, including semantic segmentation and image generation, demonstrating significant improvements over self-attention and its variants.

Motivation & Objective

• Reevaluate whether hand-crafted attention is indispensable for modeling global context in vision and NLP tasks.
• Formulate global context as a low-rank recovery problem and solve it via matrix decomposition to design a white-box module.
• Develop Hamburger, a lightweight global context block built from MD solvers (VQ, CD, NMF) with efficient backpropagation.
• Demonstrate the effectiveness of Hamburger on semantic segmentation and image generation tasks, benchmarking against self-attention modules.

Proposed method

• Model global context as low-rank recovery of unfolded input representations and solve with matrix decomposition to produce a clean low-rank embedding.
• Introduce Hamburger, which applies a linear transform (lower bread), an MD-based ham block to recover the low-rank subspace, and a linear transform (upper bread) to produce outputs.
• Instantiate the MD ham block with differentiable variants of Vector Quantization (VQ), Concept Decomposition (CD), and Non-negative Matrix Factorization (NMF).
• Use a one-step gradient—instead of full backpropagation through time—to backpropagate through the iterative MD solver, mitigating unstable gradients.
• Maintain O(n) complexity for Hamburger and avoid large n-by-n attention matrices, reducing memory usage compared to traditional self-attention.

Experimental results

Research questions

• RQ1 Is hand-crafted attention (self-attention) necessary for modeling global context, or can matrix-decomposition–based global context be competitive?
• RQ2 Can Hamburger achieve competitive or superior performance to self-attention across segmentation and generation tasks while reducing computational and memory costs?
• RQ3 What are effective training strategies for differentiating through an iterative matrix decomposition process (e.g., one-step gradient) in neural networks?
• RQ4 How do different MD choices (VQ, CD, NMF) affect performance, efficiency, and interpretability of global context modeling?

Key findings

• Hamburger, built on matrix decomposition, can achieve competitive to state-of-the-art results versus self-attention in semantic segmentation and image generation.
• On PASCAL VOC test set, HamNet (Hamburger-based) achieves 85.9% mIoU, exceeding several attention-based models listed.
• On PASCAL Context validation, HamNet attains 55.2% mIoU, surpassing multiple attention modules.
• In image generation on ImageNet at 128x128, HamGAN variants with NMF/one-step gradient achieve strong FID improvements (e.g., HamGAN-strong FID 14.77, HamGAN-baby 16.05) compared to SAGAN.
• The one-step gradient method stabilizes training and effectively backprops through the MD loop, avoiding the instabilities of full BPTT.
• Hamburger exhibits lower memory and computation (O(ndr) and no large n×n attention matrix) than traditional self-attention modules.