[Paper Review] Learning Semantic Segmentation with Weakly-Annotated Videos

Pavel Tokmakov, Karteek Alahari+1 more|arXiv (Cornell University)|Jul 28, 2016

Advanced Neural Network Applications11 references15 citations

TL;DR

This paper proposes motion-CNN (M-CNN), a fully convolutional neural network that leverages motion cues from weakly-annotated videos—using motion segments as soft constraints—to achieve state-of-the-art semantic segmentation performance without requiring pixel-level annotations. Trained on just 150 weak video annotations, M-CNN matches performance of methods trained on thousands of strongly annotated images and outperforms prior weakly supervised approaches on PASCAL VOC 2012 and YouTube-Objects.

ABSTRACT

Fully convolutional neural networks (FCNNs) trained on a large number of images with strong pixel-level annotations have become the new state of the art for the semantic segmentation task. While there have been recent attempts to learn FCNNs from image-level weak annotations , they need additional constraints, such as the size of an object, to obtain reasonable performance. To address this issue, we present motion-CNN (M-CNN), a novel FCNN framework which incorporates motion cues and is learned from video-level weak annotations. Our learning scheme to train the network uses motion segments as soft constraints, thereby handling noisy motion information. When trained on weakly-annotated videos, our method outperforms the state-of-the-art EM-Adapt approach on the PASCAL VOC 2012 image segmentation benchmark. We also demonstrate that the performance of M-CNN learned with 150 weak video annotations is on par with state-of-the-art weakly-supervised methods trained with thousands of images. Finally, M-CNN substantially outperforms recent approaches in a related task of video co-localization on the YouTube-Objects dataset. This is an extended version of our ECCV 2016 paper.

Motivation & Objective

• To address the limitation of weakly supervised semantic segmentation methods that require additional constraints like object size.
• To explore whether motion cues from video-level weak annotations can effectively guide semantic segmentation without pixel-level supervision.
• To develop a framework that achieves high performance using only video-level labels, reducing reliance on costly pixel-level annotations.
• To improve performance on both image-level semantic segmentation and video co-localization tasks using weakly supervised learning.

Proposed method

• M-CNN is a fully convolutional neural network trained end-to-end using video-level weak annotations.
• Motion segments extracted from videos are used as soft constraints during training to guide the network toward accurate spatial localization.
• The framework leverages temporal consistency in videos to infer plausible object boundaries and semantic layouts.
• It employs a multi-stage training strategy that progressively refines predictions using motion-aware supervision.
• The method incorporates a loss function that encourages activation consistency across motion segments, reducing noise from unreliable motion cues.
• It is trained using a weakly supervised objective that does not require bounding boxes or pixel-level masks.

Experimental results

Research questions

• RQ1 Can motion cues from weakly-annotated videos improve semantic segmentation performance without pixel-level annotations?
• RQ2 How does M-CNN compare to existing weakly supervised methods in terms of accuracy and data efficiency?
• RQ3 Can M-CNN generalize to related tasks such as video co-localization using only video-level labels?
• RQ4 To what extent can a model trained on 150 weak video annotations match the performance of models trained on thousands of strongly annotated images?

Key findings

• M-CNN outperforms the EM-Adapt approach on the PASCAL VOC 2012 image segmentation benchmark when trained on weakly-annotated videos.
• With only 150 weak video annotations, M-CNN achieves performance comparable to state-of-the-art weakly-supervised methods trained on thousands of images.
• M-CNN significantly outperforms recent approaches on the YouTube-Objects dataset for video co-localization.
• The use of motion segments as soft constraints effectively mitigates noise in motion information, improving localization accuracy.
• The model demonstrates strong generalization across both image and video-level weakly supervised tasks.