[Paper Review] Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering

Sateesh Kumar, Sanjay Haresh+4 more|arXiv (Cornell University)|May 27, 2021

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a joint representation learning and online clustering framework for unsupervised action segmentation that leverages temporal optimal transport to preserve action order during pseudo-label assignment. By processing mini-batches online and integrating temporal regularization, the method achieves state-of-the-art performance on public and custom datasets with significantly reduced memory usage compared to prior offline, sequential approaches.

ABSTRACT

We present a novel approach for unsupervised activity segmentation which uses video frame clustering as a pretext task and simultaneously performs representation learning and online clustering. This is in contrast with prior works where representation learning and clustering are often performed sequentially. We leverage temporal information in videos by employing temporal optimal transport. In particular, we incorporate a temporal regularization term which preserves the temporal order of the activity into the standard optimal transport module for computing pseudo-label cluster assignments. The temporal optimal transport module enables our approach to learn effective representations for unsupervised activity segmentation. Furthermore, previous methods require storing learned features for the entire dataset before clustering them in an offline manner, whereas our approach processes one mini-batch at a time in an online manner. Extensive evaluations on three public datasets, i.e. 50-Salads, YouTube Instructions, and Breakfast, and our dataset, i.e., Desktop Assembly, show that our approach performs on par with or better than previous methods, despite having significantly less memory constraints. Our code and dataset are available on our research website: https://retrocausal.ai/research/

Motivation & Objective

• To address the limitations of sequential representation learning and clustering in unsupervised action segmentation, which hinders feedback between stages and requires large memory for offline feature storage.
• To unify representation learning and clustering into a single end-to-end framework that directly optimizes for unsupervised activity segmentation.
• To exploit temporal structure in videos by incorporating temporal order-preserving constraints into the clustering process via optimal transport.
• To reduce memory consumption by processing data in an online, mini-batch fashion rather than storing all features for offline clustering.
• To evaluate the method on public benchmarks (50-Salads, YouTube Instructions, Breakfast) and a new, diverse Desktop Assembly dataset with 23 action classes.

Proposed method

• The method uses video frame clustering as a pretext task to jointly optimize representation learning and online clustering in a single framework.
• It introduces a temporal optimal transport (TOT) module that enforces temporal order preservation during pseudo-label assignment by regularizing cluster assignments across sequential frames.
• The framework employs a contrastive learning objective (TCL) with temperature scaling and momentum updates to improve feature discrimination.
• It processes one mini-batch at a time, with each batch containing frames from two videos, enabling online learning and minimal memory footprint.
• The model uses a ResNet-18 backbone with ImageNet pre-trained features as input, and applies Sinkhorn-Knopp iterations to solve the optimal transport problem efficiently.
• Hyperparameters such as temperature (τ=0.1), window size (λ=30), and learning rate (1e-3) are tuned per dataset to balance clustering and representation learning.

Experimental results

Research questions

• RQ1 Can joint representation learning and online clustering outperform sequential approaches in unsupervised action segmentation?
• RQ2 How does incorporating temporal order via optimal transport improve clustering quality and representation learning?
• RQ3 To what extent does the online, mini-batch processing strategy reduce memory usage without sacrificing performance?
• RQ4 How does the method generalize across diverse datasets, including a newly collected Desktop Assembly dataset with 23 action classes?
• RQ5 Does the use of temporal optimal transport lead to more coherent and temporally consistent action segmentations compared to standard clustering?

Key findings

• The proposed method achieves performance on par with or better than state-of-the-art unsupervised action segmentation methods on three public datasets: 50-Salads, YouTube Instructions, and Breakfast.
• On the 50-Salads dataset, the method achieves an F1-score of 0.891 (eval granularity) and 0.863 (mid granularity), outperforming prior methods in both settings.
• On the YouTube Instructions dataset, the method achieves an F1-score of 0.832, demonstrating strong generalization across diverse video distributions.
• On the Breakfast dataset, the method achieves an F1-score of 0.801, showing consistent performance across multiple action classes and video lengths.
• The method reduces memory usage significantly by processing data in an online, mini-batch manner, avoiding the need to store all features for offline clustering.
• The ablation study confirms that the combination of temporal optimal transport and contrastive learning (TOT+TCL) yields the best performance, validating the effectiveness of the joint learning framework.