[Paper Review] Fully Transformer-Equipped Architecture for End-to-End Referring Video Object Segmentation

Ping Li, Yu Zhang+2 more|arXiv (Cornell University)|Sep 21, 2023

Advanced Image and Video Retrieval TechniquesComputer Science3 citations

TL;DR

This paper proposes FTEA, a fully end-to-end Transformer-based architecture for referring video object segmentation that treats the task as object-wise mask prediction. It uses stacked transformers to decode object-specific features with diversity regularization, achieving state-of-the-art performance with 45.1% mAP on A2D Sentences, 38.7% on J-HMDB Sentences, and 56.6% J&F on Ref-YouTube-VOS.

ABSTRACT

Referring Video Object Segmentation (RVOS) requires segmenting the object in video referred by a natural language query. Existing methods mainly rely on sophisticated pipelines to tackle such cross-modal task, and do not explicitly model the object-level spatial context which plays an important role in locating the referred object. Therefore, we propose an end-to-end RVOS framework completely built upon transformers, termed extit{Fully Transformer-Equipped Architecture} (FTEA), which treats the RVOS task as a mask sequence learning problem and regards all the objects in video as candidate objects. Given a video clip with a text query, the visual-textual features are yielded by encoder, while the corresponding pixel-level and word-level features are aligned in terms of semantic similarity. To capture the object-level spatial context, we have developed the Stacked Transformer, which individually characterizes the visual appearance of each candidate object, whose feature map is decoded to the binary mask sequence in order directly. Finally, the model finds the best matching between mask sequence and text query. In addition, to diversify the generated masks for candidate objects, we impose a diversity loss on the model for capturing more accurate mask of the referred object. Empirical studies have shown the superiority of the proposed method on three benchmarks, e.g., FETA achieves 45.1% and 38.7% in terms of mAP on A2D Sentences (3782 videos) and J-HMDB Sentences (928 videos), respectively; it achieves 56.6% in terms of $\mathcal{J\&F}$ on Ref-YouTube-VOS (3975 videos and 7451 objects). Particularly, compared to the best candidate method, it has a gain of 2.1% and 3.2% in terms of P$@$0.5 on the former two, respectively, while it has a gain of 2.9% in terms of $\mathcal{J}$ on the latter one.

Motivation & Objective

• Address the limitation of existing pixel-wise classification methods in modeling object-level spatial relations in referring video object segmentation (RVOS).
• Overcome the failure of prior methods to explicitly model relative object positions and interactions in complex scenes.
• Develop a fully end-to-end framework using only Transformers to unify visual and textual feature encoding, cross-modal alignment, and mask decoding.
• Improve mask diversity and accuracy by explicitly modeling candidate objects as independent entities with a dedicated decoding strategy.
• Enable better generalization in challenging scenarios such as object appearance variation, occlusion, and low-light conditions.

Proposed method

• Propose a Fully Transformer-Equipped Architecture (FTEA) that replaces CNNs and RNNs with pure Transformers for all components: visual encoder, text encoder, cross-modal alignment, and mask decoder.
• Introduce a Stacked Transformer-based Mask Decoder that processes frame features through stacked self-attention and feed-forward layers to generate object-specific feature maps.
• Apply a progressive object learning strategy using Stacked Attention (SA) and Stacked Feed-Forward Network (SFFN) to model long-range spatiotemporal dependencies across video frames.
• Implement dynamic convolution in the decoder to enhance feature representation for each candidate object while preserving spatial context.
• Incorporate a diversity loss during training to encourage distinct mask predictions across candidate objects, improving identification of multiple objects in a scene.
• Align visual and textual features via semantic similarity matching between pixel-level and word-level representations to ensure accurate cross-modal grounding.

Experimental results

Research questions

• RQ1 Can a fully Transformer-based architecture outperform hybrid CNN-RNN pipelines in end-to-end referring video object segmentation?
• RQ2 Does modeling RVOS as an object-wise classification task—generating masks for all candidate objects—improve performance over pixel-wise classification?
• RQ3 Can stacked transformers effectively capture object-level spatial context and long-range temporal dependencies in video sequences?
• RQ4 How does enforcing diversity in predicted masks affect the identification of referred objects, especially in scenes with multiple similar or overlapping objects?
• RQ5 To what extent does the proposed method generalize to challenging visual conditions such as low lighting, object occlusion, or appearance variation?

Key findings

• FTEA achieves 45.1% mAP on A2D Sentences, outperforming the best baseline by 2.1% in P@0.5, demonstrating strong generalization on diverse referring expressions.
• On J-HMDB Sentences, FTEA reaches 38.7% mAP, a 3.2% improvement over the strongest prior method in P@0.5, indicating robustness to complex human-object interactions.
• On Ref-YouTube-VOS, FTEA achieves 56.6% J&F, a 2.9% gain over the best baseline, highlighting its effectiveness in large-scale, diverse video datasets.
• Ablation studies confirm that the diversity loss significantly improves mask quality and object identification, especially in scenes with overlapping or similar-looking objects.
• Failure cases reveal limitations in segmenting small or low-contrast objects (e.g., bird beak, cat in dim light), suggesting room for improvement in fine-grained contour prediction.
• The model’s performance is sensitive to object appearance variation and occlusion, indicating that future work should focus on relational reasoning and model compression.