[Paper Review] Self-supervised Video Retrieval Transformer Network

Xiangteng He, Yulin Pan+2 more|arXiv (Cornell University)|Apr 16, 2021

Advanced Image and Video Retrieval Techniques32 references4 citations

TL;DR

This paper proposes SVRTN, a self-supervised video retrieval framework that learns clip-level video representations using a transformer-based set encoder, avoiding costly manual annotation and reducing storage and search costs. It achieves state-of-the-art performance on FIVR-200K and SVD, improving mAP by up to 30.6% over video-level methods and reducing feature storage by 78.7% while speeding up retrieval by 25x compared to frame-level methods.

ABSTRACT

Content-based video retrieval aims to find videos from a large video database that are similar to or even near-duplicate of a given query video. Video representation and similarity search algorithms are crucial to any video retrieval system. To derive effective video representation, most video retrieval systems require a large amount of manually annotated data for training, making it costly inefficient. In addition, most retrieval systems are based on frame-level features for video similarity searching, making it expensive both storage wise and search wise. We propose a novel video retrieval system, termed SVRTN, that effectively addresses the above shortcomings. It first applies self-supervised training to effectively learn video representation from unlabeled data to avoid the expensive cost of manual annotation. Then, it exploits transformer structure to aggregate frame-level features into clip-level to reduce both storage space and search complexity. It can learn the complementary and discriminative information from the interactions among clip frames, as well as acquire the frame permutation and missing invariant ability to support more flexible retrieval manners. Comprehensive experiments on two challenging video retrieval datasets, namely FIVR-200K and SVD, verify the effectiveness of our proposed SVRTN method, which achieves the best performance of video retrieval on accuracy and efficiency.

Motivation & Objective

• To reduce the reliance on expensive manual annotation in video retrieval by leveraging self-supervised learning on unlabeled videos.
• To minimize storage and computational costs in video retrieval by aggregating frame-level features into compact clip-level representations.
• To enhance robustness and flexibility in retrieval by enabling frame permutation and missing-frame invariance through masked frame modeling in a set transformer.
• To support multiple retrieval modes, including clip-to-clip and frame-to-clip retrieval, for greater application adaptability.

Proposed method

• Self-supervised video representation learning is performed by generating positive pairs from a video and its augmented versions via temporal, spatial, and photometric transformations.
• A clip-level set transformer network aggregates frame-level features into a single clip-level representation using multi-head self-attention to model inter-frame dependencies.
• Masked frame modeling is integrated into the clip-level encoding to make the model invariant to frame permutations and missing frames, improving robustness.
• Video representations are binarized via hashing for efficient similarity search in large-scale retrieval.
• The model is trained end-to-end using contrastive learning on the self-generated positive pairs to maximize feature discrimination.
• The framework supports flexible retrieval by encoding both clips and individual frames, enabling diverse query types.

Experimental results

Research questions

• RQ1 Can self-supervised learning effectively replace manual annotation in video representation learning for retrieval?
• RQ2 Can clip-level representation learning significantly reduce storage and search complexity without sacrificing retrieval accuracy?
• RQ3 Does the use of a set transformer with masked frame modeling improve robustness to frame order and missing frames in video retrieval?
• RQ4 Can the proposed method support flexible retrieval modes such as frame-to-clip and clip-to-clip retrieval?

Key findings

• SVRTN achieves a 30.6% improvement in mAP on the DSVR task of the FIVR-200K dataset compared to video-level methods.
• On the CSVR and ISVR tasks of FIVR-200K, SVRTN improves mAP by 28.2% and 21.3%, respectively, over video-level baselines.
• On the SVD dataset, SVRTN achieves a 4.7% mAP improvement over existing methods.
• The method reduces feature storage cost by approximately 78.7% compared to frame-level retrieval methods.
• Retrieval speed is increased by about 25 times compared to frame-level approaches due to clip-level representation.
• The model demonstrates strong robustness to frame missing and permutation, with performance stability across different clip lengths (4s, 6s, 8s).