[Paper Review] Limits to Depth Efficiencies of Self-Attention

Yoav Levine, Noam Wies+3 more|arXiv (Cornell University)|Jun 22, 2020

Ferroelectric and Negative Capacitance Devices22 citations

TL;DR

This paper identifies a logarithmic depth threshold, $ L_{\text{th}} = \log_3(d_x) $, beyond which self-attention networks become depth-inefficient due to width limitations. It explains why widening and deepening are empirically comparable in performance, showing that depth efficiency is double-exponential up to the threshold, after which further depth gains diminish rapidly.

ABSTRACT

Self-attention architectures, which are rapidly pushing the frontier in natural language processing, demonstrate a surprising depth-inefficient behavior: Empirical signals indicate that increasing the internal representation (network width) is just as useful as increasing the number of self-attention layers (network depth). In this paper, we theoretically study the interplay between and width in self-attention, and shed light on the root of the above phenomenon. We invalidate the seemingly plausible hypothesis by which widening is as effective as deepening for self-attention, and show that in fact stacking self-attention layers is so effective that it quickly saturates a capacity of the network width. Specifically, we pinpoint a depth that is logarithmic in $d_x$, the network width: $L_{ extrm{th}}=\log_{3}(d_x)$. For networks of that is below the threshold, we establish a double-exponential depth-efficiency of the self-attention operation, while for depths over the threshold we show that depth-inefficiency kicks in. Our predictions strongly accord with extensive empirical ablations in Kaplan et al. (2020), accounting for the different behaviors in the two depth-(in)efficiency regimes. By identifying network width as a limiting factor, our analysis indicates that solutions for dramatically increasing the width can facilitate the next leap in self-attention expressivity.

Motivation & Objective

• To understand the empirical observation that increasing width and depth yield comparable performance gains in self-attention networks.
• To investigate why self-attention networks exhibit depth-inefficiency beyond a certain depth, despite their architectural advantages.
• To identify the theoretical limits of depth efficiency in self-attention mechanisms based on network width.
• To resolve the apparent paradox that widening and deepening are equally effective, by showing that width constrains depth capacity.

Proposed method

• Derives a theoretical depth threshold $ L_{\text{th}} = \log_3(d_x) $, where $ d_x $ is the network width, based on information propagation in self-attention layers.
• Analyzes the expressivity of stacked self-attention layers using a double-exponential growth model for depth efficiency below the threshold.
• Establishes that beyond $ L_{\text{th}} $, depth gains diminish due to saturation of representational capacity constrained by width.
• Uses theoretical analysis to invalidate the hypothesis that widening and deepening are equally effective for all depths.
• Compares theoretical predictions with empirical ablation results from Kaplan et al. (2020), showing strong alignment across depth regimes.
• Identifies the role of width as a limiting factor on depth efficiency, suggesting that increasing width could unlock further expressivity gains.

Experimental results

Research questions

• RQ1 Why do width and depth yield comparable performance gains in self-attention networks despite their different architectural roles?
• RQ2 What is the theoretical depth threshold beyond which self-attention networks become depth-inefficient?
• RQ3 How does network width constrain the effective depth capacity of self-attention mechanisms?
• RQ4 Why does depth efficiency exhibit a double-exponential growth pattern before saturation?
• RQ5 To what extent do theoretical predictions align with empirical ablation studies in Kaplan et al. (2020)?

Key findings

• The depth threshold for self-attention efficiency is $ L_{\text{th}} = \log_3(d_x) $, where $ d_x $ is the network width.
• Below this threshold, depth efficiency grows double-exponentially, indicating highly effective representational capacity scaling.
• Beyond the threshold, depth-inefficiency emerges due to saturation of representational capacity constrained by width.
• Theoretical predictions align closely with empirical ablation results from Kaplan et al. (2020), validating the model across depth regimes.
• The study invalidates the hypothesis that widening and deepening are equally effective across all depths, showing that width limits depth potential.
• Increasing network width beyond current scales could enable significant future gains in self-attention expressivity.