[Paper Review] Online Video Instance Segmentation via Robust Context Fusion

Xiang Li, Jinglu Wang+3 more|arXiv (Cornell University)|Jul 12, 2022

Music and Audio Processing4 citations

TL;DR

This paper proposes a robust context fusion (RCF) network for online video instance segmentation, leveraging importance-aware compression and transformer-based fusion to efficiently integrate compact, relevant context from a few preceding frames into real-time target frame predictions. The method achieves state-of-the-art performance on YouTube-VIS 2019 and 2021, outperforming even clip-level methods, and demonstrates that audio cues can marginally improve segmentation in unconstrained scenarios.

ABSTRACT

Video instance segmentation (VIS) aims at classifying, segmenting and tracking object instances in video sequences. Recent transformer-based neural networks have demonstrated their powerful capability of modeling spatio-temporal correlations for the VIS task. Relying on video- or clip-level input, they suffer from high latency and computational cost. We propose a robust context fusion network to tackle VIS in an online fashion, which predicts instance segmentation frame-by-frame with a few preceding frames. To acquire the precise and temporal-consistent prediction for each frame efficiently, the key idea is to fuse effective and compact context from reference frames into the target frame. Considering the different effects of reference and target frames on the target prediction, we first summarize contextual features through importance-aware compression. A transformer encoder is adopted to fuse the compressed context. Then, we leverage an order-preserving instance embedding to convey the identity-aware information and correspond the identities to predicted instance masks. We demonstrate that our robust fusion network achieves the best performance among existing online VIS methods and is even better than previously published clip-level methods on the Youtube-VIS 2019 and 2021 benchmarks. In addition, visual objects often have acoustic signatures that are naturally synchronized with them in audio-bearing video recordings. By leveraging the flexibility of our context fusion network on multi-modal data, we further investigate the influence of audios on the video-dense prediction task, which has never been discussed in existing works. We build up an Audio-Visual Instance Segmentation dataset, and demonstrate that acoustic signals in the wild scenarios could benefit the VIS task.

Motivation & Objective

• To address high latency and computational cost in video instance segmentation by enabling real-time, frame-by-frame inference.
• To improve online VIS performance by fusing compact, effective context from preceding reference frames while preserving temporal consistency.
• To eliminate the need for post-processing matching by learning an order-preserving instance embedding for identity tracking.
• To investigate the role of audio modality in video-dense prediction tasks, especially in unconstrained, real-world scenarios.
• To build and evaluate a new Audio-Visual Instance Segmentation dataset to study multimodal fusion in VIS.

Proposed method

• The method employs importance-aware compression to distill relevant features from reference frames, reducing redundancy and noise before fusion.
• A transformer encoder fuses the compressed reference features with the target frame features using expressive self- and cross-attention mechanisms.
• An order-preserving instance code is learned via a fixed-length query in the transformer decoder, enabling identity tracking without additional matching.
• The model leverages Lipschitz continuity to ensure stable identity assignment, making explicit supervision for order preservation unnecessary.
• Audio modality is integrated via audio tokens that are fused with visual features using the same context fusion mechanism.
• The framework is flexible and supports multi-modal inputs, enabling joint visual-audio reasoning for dense prediction.

Experimental results

Research questions

• RQ1 Can a robust context fusion mechanism improve online video instance segmentation performance by effectively leveraging a small number of reference frames?
• RQ2 Does importance-aware compression of reference features enhance model robustness and efficiency in online inference?
• RQ3 Can an order-preserving instance code eliminate the need for complex post-processing matching in online VIS?
• RQ4 To what extent can audio signals improve video instance segmentation in real-world, unconstrained video scenarios?
• RQ5 Is the performance gain from audio fusion statistically significant in wild, uncontrolled video environments?

Key findings

• The proposed RCF network achieves 40.8 mAP on YouTube-VIS 2019 and 43.4 mAP on YouTube-VIS 2021, outperforming all existing online VIS methods and even surpassing previously published clip-level methods.
• The model is robust to redundant instance queries, maintaining strong performance even when the number of queries is much larger than the actual number of instances in a frame.
• Supervising the order of instance codes does not improve performance, as the order-preserving property emerges naturally from the network architecture.
• The pixel-wise weight in token fusion improves mAP by 0.7 points when disabled, indicating its importance in focusing on foreground regions.
• Audio integration yields a marginal gain of +1.9 mAP on average over the AVIS validation set, but the improvement is not statistically significant (p-value > 0.05).
• The benefit of audio is limited in unconstrained videos due to weak, inconsistent audio-visual correlations and short-lived sound events.