[Paper Review] Unsupervised Action Segmentation for Instructional Videos

AJ Piergiovanni, Anelia Angelova+2 more|arXiv (Cornell University)|Jun 7, 2021

Human Pose and Action Recognition11 references4 citations

TL;DR

This paper proposes an unsupervised method for atomic action segmentation in instructional videos using a stochastic recurrent autoregressive model with Gumbel-Softmax sampling to generate multiple candidate action sequences. By ranking these sequences via self-imposed constraints—action occurrence, consistent duration, and visual plausibility—the model iteratively self-labels videos and converges to accurate segmentations without any ground-truth annotations, outperforming prior unsupervised and even some weakly-supervised methods on benchmark datasets.

ABSTRACT

In this paper we address the problem of automatically discovering atomic actions in unsupervised manner from instructional videos, which are rarely annotated with atomic actions. We present an unsupervised approach to learn atomic actions of structured human tasks from a variety of instructional videos based on a sequential stochastic autoregressive model for temporal segmentation of videos. This learns to represent and discover the sequential relationship between different atomic actions of the task, and which provides automatic and unsupervised self-labeling.

Motivation & Objective

• To discover atomic actions in instructional videos without any human-annotated action boundaries or labels.
• To model the temporal structure of complex tasks, where actions follow predictable but variable-order sequences.
• To develop a self-supervision mechanism that generates and ranks plausible action sequences using video-level constraints.
• To enable unsupervised learning by iteratively refining action segmentations through a self-labeling EM-like procedure.
• To outperform existing unsupervised and weakly-supervised methods on standard benchmarks using only video input.

Proposed method

• A sequential stochastic autoregressive model with states, output symbols, and probabilistic transition rules, implemented via fully connected layers and Gumbel-Softmax for stochastic sampling.
• Input video frames are encoded via a CNN (e.g., VGG or AssembleNet) into feature sequences, which are processed autoregressively to generate action sequences.
• Multiple candidate action sequences are generated per video by sampling different rules via Gumbel-Softmax, introducing stochasticity across runs.
• A ranking function evaluates sequences based on three constraints: (1) each action appears at least once, (2) action durations are consistent across videos of the same task, and (3) action symbols match visual features.
• The top-ranked sequence is used as pseudo-labels to train the model, and the process is repeated in an EM-like iterative optimization loop.
• The model is trained end-to-end using a differentiable loss that combines the ranking function components, enabling gradient-based optimization.

Experimental results

Research questions

• RQ1 Can a stochastic autoregressive model learn to generate multiple plausible action sequences from unstructured instructional videos?
• RQ2 How effective is a self-labeling strategy based on structural constraints (e.g., action occurrence, duration consistency) in the absence of annotations?
• RQ3 To what extent can unsupervised action segmentation performance match or exceed weakly-supervised and supervised baselines?
• RQ4 How sensitive is the method to the number of actions (k) specified as a hyperparameter?
• RQ5 Does the use of Gumbel-Softmax sampling significantly improve self-labeling quality compared to deterministic or random selection?

Key findings

• The proposed unsupervised method achieves state-of-the-art performance on the 50-Salads, Breakfast, and NIV datasets, outperforming prior unsupervised approaches and even some weakly-supervised methods.
• On the NIV dataset, the method achieved an F1 score of 0.457 using AssembleNet features, surpassing previous methods (e.g., Alayrac et al. [1] at 0.238 and Kukleva et al. [8] at 0.283).
• Abllation studies showed that all three components of the cost function—action occurrence, duration consistency, and visual plausibility—significantly improved performance, with the full combination yielding 33.4 F1 on 50-Salads and 29.8 on Breakfast.
• Random selection of candidate sequences or disabling Gumbel-Softmax led to poor performance (12.5 and 10.5 F1 on 50-Salads), confirming the necessity of stochastic sampling for effective self-labeling.
• The model is relatively robust to the choice of k (number of actions), with performance peaking near the true number of actions, as shown in Figure 5.
• Using VGG features instead of AssembleNet still yielded strong results, with an F1 of 0.376 on NIV, indicating the method generalizes across feature extractors.