[Paper Review] On Evaluating Weakly Supervised Action Segmentation Methods

Yaser Souri, Alexander Richard+2 more|arXiv (Cornell University)|May 19, 2020

Human Pose and Action Recognition16 references4 citations

TL;DR

This paper investigates performance variance and feature sensitivity in weakly supervised action segmentation using transcripts. It demonstrates that standard deviation across multiple training runs is significant (1–2.51%), and surprisingly, higher-level I3D features underperform lower-level IDT features on the Breakfast dataset, challenging assumptions about feature quality in weakly supervised learning.

ABSTRACT

Action segmentation is the task of temporally segmenting every frame of an untrimmed video. Weakly supervised approaches to action segmentation, especially from transcripts have been of considerable interest to the computer vision community. In this work, we focus on two aspects of the use and evaluation of weakly supervised action segmentation approaches that are often overlooked: the performance variance over multiple training runs and the impact of selecting feature extractors for this task. To tackle the first problem, we train each method on the Breakfast dataset 5 times and provide average and standard deviation of the results. Our experiments show that the standard deviation over these repetitions is between 1 and 2.5% and significantly affects the comparison between different approaches. Furthermore, our investigation on feature extraction shows that, for the studied weakly-supervised action segmentation methods, higher-level I3D features perform worse than classical IDT features.

Motivation & Objective

• To investigate the performance variance of weakly supervised action segmentation methods across multiple training runs with different random seeds.
• To evaluate the impact of different feature extractors—specifically IDT vs. I3D—on the performance of weakly supervised action segmentation models.
• To assess whether state-of-the-art methods designed for IDT features can generalize effectively to I3D features without adaptation.
• To provide a more reliable evaluation protocol by reporting mean and standard deviation over multiple runs, rather than single runs.
• To identify potential reasons for the poor performance of I3D features despite their higher-level representation and pre-training on large-scale video data.

Proposed method

• Trained four publicly available weakly supervised action segmentation models—NNV, ISBA, CDFL, and MuCon—five times each on the Breakfast dataset using different random seeds to assess performance variance.
• Reported mean and standard deviation of mean over frames (MoF) across five runs to quantify model stability and variability.
• Extracted I3D (RGB and Flow) features with 2048 dimensions per frame from the Breakfast dataset videos and evaluated model performance using these features.
• Applied PCA to reduce I3D feature dimensionality from 2048 to 64 to match the input dimensionality used in the original models, creating PCA-I3D features.
• Modified the temporal receptive field (feature window size) of the GRU input in the NNV model to test sensitivity to local temporal context when using PCA-I3D features.
• Compared performance across IDT, I3D, and PCA-I3D features across all models on split 1 of the Breakfast dataset, using MoF as the evaluation metric.

Experimental results

Research questions

• RQ1 How much does the performance of weakly supervised action segmentation models vary across multiple training runs with different random seeds?
• RQ2 Do higher-level I3D features improve performance over classical IDT features in weakly supervised action segmentation on the Breakfast dataset?
• RQ3 Can existing weakly supervised action segmentation models trained on IDT features be directly applied to I3D features without performance degradation?
• RQ4 Does reducing the dimensionality of I3D features via PCA improve model performance on the Breakfast dataset?
• RQ5 How does the temporal context window size affect model performance when using PCA-I3D features?

Key findings

• The standard deviation in performance across five training runs ranged from 1.0% to 2.51%, indicating high variance and calling for reporting mean and standard deviation instead of single runs.
• Using I3D features led to a drastic performance drop—NNV's MoF dropped from 40.6% (IDT) to 11.4% (I3D), showing poor generalization to this feature type.
• PCA-reduced I3D features (PCA-I3D) improved performance but still underperformed IDT features, with CDFL achieving 38.0% MoF on PCA-I3D vs. 48.9% on IDT.
• MuCon performed poorly with IDT features (40.2% MoF) but improved significantly with I3D (48.3%) and PCA-I3D (47.7%), indicating model-specific sensitivity to feature type.
• Smaller temporal window sizes (e.g., 1 frame) improved performance with PCA-I3D features in NNV, suggesting that local temporal context is more critical when using I3D features.
• The results suggest that current weakly supervised models are not robust to feature distribution shifts, and that I3D features may not be suitable for the Breakfast dataset without architectural or hyperparameter adaptation.