[Paper Review] Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling

Tao Shen, Tianyi Zhou+3 more|arXiv (Cornell University)|Apr 3, 2018

Topic ModelingComputer Science59 references77 citations

TL;DR

Bi-BloSAN introduces a RNN/CNN-free sequence encoder using intra-block and inter-block self-attention to achieve SAN-like performance with memory efficiency comparable to RNNs, validated on nine NLP tasks.

ABSTRACT

Recurrent neural networks (RNN), convolutional neural networks (CNN) and self-attention networks (SAN) are commonly used to produce context-aware representations. RNN can capture long-range dependency but is hard to parallelize and not time-efficient. CNN focuses on local dependency but does not perform well on some tasks. SAN can model both such dependencies via highly parallelizable computation, but memory requirement grows rapidly in line with sequence length. In this paper, we propose a model, called "bi-directional block self-attention network (Bi-BloSAN)", for RNN/CNN-free sequence encoding. It requires as little memory as RNN but with all the merits of SAN. Bi-BloSAN splits the entire sequence into blocks, and applies an intra-block SAN to each block for modeling local context, then applies an inter-block SAN to the outputs for all blocks to capture long-range dependency. Thus, each SAN only needs to process a short sequence, and only a small amount of memory is required. Additionally, we use feature-level attention to handle the variation of contexts around the same word, and use forward/backward masks to encode temporal order information. On nine benchmark datasets for different NLP tasks, Bi-BloSAN achieves or improves upon state-of-the-art accuracy, and shows better efficiency-memory trade-off than existing RNN/CNN/SAN.

Motivation & Objective

• Motivate a sequence encoder that combines the advantages of self-attention with low memory cost similar to RNNs.
• Propose masked block self-attention (mBloSA) to capture local intra-block and global inter-block dependencies.
• Develop Bi-BloSAN as a Bi-Directional Block Self-Attention network for comprehensive sequence encoding.
• Demonstrate efficiency–memory trade-offs and competitive accuracy across NLP tasks.
• Provide ablation analyses to quantify contributions of local/global context and masking mechanisms.

Proposed method

• Split the input sequence into equal-length blocks and apply masked self-attention within each block (intra-block).
• Compute block-level representations via source2token self-attention on intra-block outputs (inter-block step).
• Apply masked self-attention across blocks to model long-range dependencies (inter-block SAN).
• Fuse local and global block-context outputs with the original input through a gating mechanism.
• Bi-BloSAN stacks two mBloSA modules with forward and backward masks to encode temporal order (Bi-BloSA).
• Reduce memory by ensuring each SAN handles only short sequences, improving efficiency over full-sequence SANs.

Experimental results

Research questions

• RQ1 Can a self-attention based encoder operate with memory footprint similar to RNNs while maintaining or improving accuracy?
• RQ2 Does a two-layer block-wise attention (intra-block + inter-block) capture both local and global dependencies effectively?
• RQ3 How do forward and backward masking and feature-level attention affect temporal ordering and context sensitivity?
• RQ4 What is the empirical efficiency and memory trade-off of Bi-BloSAN compared with RNN/CNN/SAN baselines on various NLP tasks?

Key findings

• Bi-BloSAN achieves state-of-the-art or competitive accuracy on nine NLP datasets, with favorable efficiency-memory trade-offs.
• On SNLI, Bi-BloSAN (480D) attains Test Accu 85.7% and trains faster than several baselines, with memory similar to RNNs.
• Bi-BloSAN outperforms several RNN/CNN/SAN baselines in accuracy and is more memory-efficient than DiSAN.
• Ablation shows both local (intra-block) and global (inter-block) contexts significantly contribute to performance, and mBloSA (masking) improves accuracy by up to 2.6 percentage points in tests.
• Source2token self-attention substantially boosts performance compared to vanilla attention (approx. +3.3% in SNLI).
• Bi-BloSAN demonstrates strong results on SQuAD-modified reading comprehension and SICK semantic relatedness, indicating robust generalization across tasks.