[Paper Review] Joint Modeling of Feature, Correspondence, and a Compressed Memory for Video Object Segmentation

Jiaming Zhang, Yutao Cui+2 more|arXiv (Cornell University)|Aug 25, 2023

Advanced Neural Network ApplicationsComputer Science3 citations

TL;DR

This paper proposes JointFormer, a unified video object segmentation framework that jointly models features, correspondence, and a compressed memory token within a single transformer-based architecture. By leveraging attention mechanisms in its Joint Blocks to iteratively propagate target information across features, correspondence, and memory, the method achieves state-of-the-art performance on DAVIS 2017 (89.7% J&F on validation, 87.6% on test-dev) and YouTube-VOS 2018/2019 (87.0% on both validation splits), outperforming prior methods through enhanced discriminative feature learning and long-term temporal modeling.

ABSTRACT

Current prevailing Video Object Segmentation methods follow the pipeline of extraction-then-matching, which first extracts features on current and reference frames independently, and then performs dense matching between them. This decoupled pipeline limits information propagation between frames to high-level features, hindering fine-grained details for matching. Furthermore, the pixel-wise matching lacks holistic target understanding, making it prone to disturbance by similar distractors. To address these issues, we propose a unified VOS framework, coined as JointFormer, for jointly modeling feature extraction, correspondence matching, and a compressed memory. The core Joint Modeling Block leverages attention to simultaneously extract and propagate the target information from the reference frame to the current frame and a compressed memory token. This joint scheme enables extensive multi-layer propagation beyond high-level feature space and facilitates robust instance-distinctive feature learning. To incorporate the long-term and holistic target information, we introduce a compressed memory token with a customized online updating mechanism, which aggregates target features and facilitates temporal information propagation in a frame-wise manner, enhancing global modeling consistency. Our JointFormer achieves a new state-of-the-art performance on the DAVIS 2017 val/test-dev (89.7\% and 87.6\%) benchmarks and the YouTube-VOS 2018/2019 val (87.0\% and 87.0\%) benchmarks, outperforming the existing works. To demonstrate the generalizability of our model, it is further evaluated on four new benchmarks with various difficulties, including MOSE for complex scenes, VISOR for egocentric videos, VOST for complex transformations, and LVOS for long-term videos.

Motivation & Objective

• To address the limitations of decoupled feature extraction and matching in existing VOS methods, which restrict information flow to high-level features and overlook holistic object understanding.
• To improve discriminative feature learning by jointly modeling features, pixel-wise correspondence, and instance-level compressed memory within a single architecture.
• To enhance long-term temporal modeling by designing a customized online update mechanism for the compressed memory token that propagates information across frames.
• To enable effective use of large-scale pre-training (e.g., Masked Image Modeling) by unifying the three components under a single attention-based framework.
• To achieve superior performance on standard benchmarks by enabling multi-level, iterative information propagation across features, correspondence, and memory.

Proposed method

• The framework concatenates flattened current and reference frame features with a compressed memory token into a sequence of tokens, which are processed by stacked Joint Blocks in a Vision Transformer backbone.
• Each Joint Block uses self- and cross-attention to simultaneously extract features, compute dense correspondence maps, and update the compressed memory token in an end-to-end manner.
• The compressed memory token represents each object as a single instance-level embedding, enabling holistic object modeling and reducing memory overhead.
• A customized online update mechanism for the compressed memory uses multi-level features from previous frames as temporal context, enabling long-range information flow along the temporal dimension.
• Multi-scale inference is applied during testing by evaluating at multiple scales and averaging predictions to improve robustness and accuracy.
• The model is trained with a masked image modeling pre-training objective on the backbone, enabling better feature learning and improved performance on downstream VOS tasks.

Figure 1 : The pipeline of existing VOS works (a) and ours (b). (a) Existing works perform feature extraction and matching separately. (b) Our framework jointly models features, correspondence, and the compressed memory.

Experimental results

Research questions

• RQ1 Can joint modeling of features, correspondence, and compressed memory in a single transformer block improve video object segmentation performance compared to decoupled pipelines?
• RQ2 How does the integration of a compressed memory token—representing each object as a whole—enhance holistic object understanding and reduce sensitivity to distractors?
• RQ3 To what extent does the proposed online updating mechanism for the compressed memory improve long-term temporal modeling and robustness in long video sequences?
• RQ4 Does the unified attention-based design enable better utilization of large-scale pre-training techniques like Masked Image Modeling compared to separate feature and matching modules?
• RQ5 What is the trade-off between performance, inference speed, and memory consumption when using multiple reference frames in the compressed memory?

Key findings

• JointFormer achieves a new state-of-the-art performance of 89.7% J&F on the DAVIS 2017 validation split and 87.6% on the test-dev split, significantly outperforming prior methods.
• On YouTube-VOS 2018 and 2019 validation sets, the method achieves 87.0% global score (G), demonstrating strong generalization across diverse video distributions.
• The ablation study confirms that multi-scale inference improves performance, with the best result reaching 88.5% J&F on DAVIS 2017 test-dev when using multi-scale evaluation.
• Replacing the backbone of prior SOTA models (AOT-L, DeAOT-L, XMem) with ConvMAE-Base, JointFormer still outperforms them by a large margin, even with identical training settings and no synthetic pre-training.
• The model maintains high accuracy (86.0% G on YouTube-VOS 2018) while reducing memory consumption to 5.785K parameters per frame, compared to 18.77K in XMem, showing efficiency gains.
• Reducing the number of reference frames from 3 to 1 results in only a minor performance drop (from 87.6% to 86.2% on YouTube-VOS 2018), indicating strong robustness and potential for faster inference.

Figure 2 : Overview of our JointFormer . The current and reference frames with masks are split and flattened into patches, then fed with our compressed memory into the Vision Transformer backbone consisting of Joint Blocks . Finally, we enhance the current tokens with compressed memory and sent them