[Paper Review] Semantically-Guided Video Object Segmentation

Sergi Caelles, Yuhua Chen+2 more|arXiv (Cornell University)|Apr 6, 2017

Visual Attention and Saliency Detection24 references10 citations

TL;DR

This paper proposes Semantically-Guided Video Object Segmentation (SGV), a method that improves semi-supervised video object segmentation by fusing an appearance model from the first frame with a semantic prior derived from instance-level segmentation. By using a conditional classifier to combine appearance and semantic cues, SGV achieves state-of-the-art performance on DAVIS and YouTube-Objects, maintaining high accuracy over long sequences and running in 547 ms per frame on average.

ABSTRACT

This paper tackles the problem of semi-supervised video object segmentation, that is, segmenting an object in a sequence given its mask in the first frame. One of the main challenges in this scenario is the change of appearance of the objects of interest. Their semantics, on the other hand, do not vary. This paper investigates how to take advantage of such invariance via the introduction of a semantic prior that guides the appearance model. Specifically, given the segmentation mask of the first frame of a sequence, we estimate the semantics of the object of interest, and propagate that knowledge throughout the sequence to improve the results based on an appearance model. We present Semantically-Guided Video Object Segmentation (SGV), which improves results over previous state of the art on two different datasets using a variety of evaluation metrics, while running in half a second per frame.

Motivation & Objective

• Address the challenge of appearance variation in video object segmentation, where object appearance changes due to lighting, occlusion, or viewpoint shifts.
• Leverage semantic invariance—specifically, the consistent category and instance identity of the target object—despite changing visual appearance.
• Improve segmentation accuracy and robustness over long video sequences by incorporating semantic priors into an appearance-based model.
• Develop an end-to-end trainable framework that combines appearance modeling with instance-level semantic segmentation to guide segmentation propagation.

Proposed method

• Estimate the semantics of the target object in the first frame by identifying the most overlapping instance-level segmentation proposals from a pre-trained instance segmentation network (MNC).
• Propagate the semantic prior (instance category and identity) throughout the video sequence to guide segmentation at each frame.
• Use a conditional classifier layer—a learnable CNN module—that combines predictions from the appearance model and the semantic prior to produce the final segmentation mask.
• Train the conditional classifier end-to-end with the appearance model, allowing the network to learn how to weigh appearance and semantic cues adaptively.
• Run the appearance model and instance segmentation network in parallel for efficiency, then combine outputs via a fast bilateral solver for refinement.
• Use the first-frame mask to initialize the tracking process and propagate the semantic prior to subsequent frames using instance matching based on IoU.

Experimental results

Research questions

• RQ1 Can semantic priors derived from instance-level segmentation improve the robustness of appearance-based video object segmentation under significant appearance changes?
• RQ2 How does incorporating semantic invariance (e.g., object category and instance identity) affect segmentation accuracy and temporal stability in long video sequences?
• RQ3 Can a learnable conditional classifier effectively fuse appearance and semantic information to outperform methods relying solely on appearance or temporal coherence?
• RQ4 To what extent does the semantic prior mitigate performance degradation caused by occlusions, lighting changes, or fast motion compared to appearance-only models?
• RQ5 Can the method maintain high accuracy across diverse video sequences while operating in real-time (under 0.6s per frame) without relying on optical flow or recurrent networks?

Key findings

• SGV achieves a mean Intersection over Union (mIoU) of 82.38 on the DAVIS dataset, significantly outperforming the closest competitor (74.72) and demonstrating an 8% improvement.
• The method maintains a high performance level throughout long sequences, with the lowest point of the mIoU curve at 79.10, compared to 73.62 for the next best method.
• False negative rates are substantially reduced, indicating that SGV effectively recovers occluded or partially visible parts of the object using semantic priors.
• The method runs in 547 ms per frame on average (120 ms for appearance model, 360 ms for MNC, 187 ms for refinement), enabling real-time inference.
• Error analysis shows that FP-Close (contour inaccuracies) is slightly higher than in some baselines, but this is offset by a dramatic reduction in FN, indicating better object completion.
• Qualitative results demonstrate that SGV successfully segments objects across complex appearance changes (e.g., front, side, and back views of a car), thanks to semantic guidance.