[Paper Review] ASFormer: Transformer for Action Segmentation

Fangqiu Yi, Hongyu Wen+1 more|arXiv (Cornell University)|Oct 16, 2021

Human Pose and Action Recognition52 references61 citations

TL;DR

ASFormer introduces a Transformer-based model with local connectivity, hierarchical representation, and a refined decoder for action segmentation, achieving state-of-the-art results on three public datasets.

ABSTRACT

Algorithms for the action segmentation task typically use temporal models to predict what action is occurring at each frame for a minute-long daily activity. Recent studies have shown the potential of Transformer in modeling the relations among elements in sequential data. However, there are several major concerns when directly applying the Transformer to the action segmentation task, such as the lack of inductive biases with small training sets, the deficit in processing long input sequence, and the limitation of the decoder architecture to utilize temporal relations among multiple action segments to refine the initial predictions. To address these concerns, we design an efficient Transformer-based model for action segmentation task, named ASFormer, with three distinctive characteristics: (i) We explicitly bring in the local connectivity inductive priors because of the high locality of features. It constrains the hypothesis space within a reliable scope, and is beneficial for the action segmentation task to learn a proper target function with small training sets. (ii) We apply a pre-defined hierarchical representation pattern that efficiently handles long input sequences. (iii) We carefully design the decoder to refine the initial predictions from the encoder. Extensive experiments on three public datasets demonstrate that effectiveness of our methods. Code is available at \url{https://github.com/ChinaYi/ASFormer}.

Motivation & Objective

• Motivate action segmentation as a frame-wise, long-sequence prediction task requiring effective temporal modeling on small datasets.
• Address three key challenges of applying vanilla Transformers: lack of inductive bias, difficulty handling long input sequences, and refinement of initial predictions across action segments.
• Propose ASFormer with local connectivity priors, a pre-defined hierarchical representation pattern, and a novel cross-attention decoder to iteratively refine predictions.
• Demonstrate effectiveness on multiple public datasets and analyze ablations to justify architectural choices.

Proposed method

• Incorporate a dilated temporal convolution in the encoder feed-forward path to inject local inductive bias.
• Impose a hierarchical representation pattern by restricting self-attention to progressively larger local windows across layers.
• Use a cross-attention-based decoder to refine encoder predictions without disturbing the encoder feature space.
• Employ multiple decoders to perform iterative refinement with exponentially decreasing influence of external (encoder) information.
• Train with a joint loss of frame-wise cross-entropy and a temporal smoothing term to encourage stable predictions.

Experimental results

Research questions

• RQ1 How can Transformer models be adapted to action segmentation with long video sequences and small training sets?
• RQ2 Does enforcing a local-to-global hierarchical attention pattern improve learning efficiency and performance on long sequences?
• RQ3 Can a cross-attention decoder refine encoder predictions effectively while leveraging temporal relations across action segments?
• RQ4 What is the impact of using multiple decoders and temporal convolutions on segmentation accuracy and smoothness?

Key findings

• ASFormer achieves state-of-the-art results on three public action segmentation datasets (50Salads, GTEA, and Breakfast).
• A local connectivity inductive bias via temporal convolution significantly improves performance over MLP-based feed-forward layers.
• A predefined hierarchical attention pattern markedly outperforms a non-hierarchical, freely attending transformer on long sequences.
• Multiple decoders provide notable gains through iterative refinement, with three decoders yielding best results in experiments.
• Dropping position encoding and using a single-head self-attention are effective choices in ASFormer, aided by the temporal convolution.
• The ASFormer backbone can replace MS-TCN in existing pipelines to improve performance (ASRF* results).