[Paper Review] CNN Retrieval based Unsupervised Metric Learning for Near-Duplicated Video Retrieval

Hao Cheng, Ping Wang+1 more|arXiv (Cornell University)|May 30, 2021

Advanced Image and Video Retrieval Techniques13 references4 citations

TL;DR

This paper proposes a CNN-based unsupervised metric learning framework for near-duplicate video retrieval (NDVR) that fuses features from intermediate convolutional layers and the fully-connected layer of pre-trained networks. By employing unsupervised metric learning and a k-nearest-neighbor-based re-ranking strategy, the method achieves state-of-the-art mAP of 0.9790 on the CC_WEB_VIDEO dataset.

ABSTRACT

As important data carriers, the drastically increasing number of multimedia videos often brings many duplicate and near-duplicate videos in the top results of search. Near-duplicate video retrieval (NDVR) can cluster and filter out the redundant contents. In this paper, the proposed NDVR approach extracts the frame-level video representation based on convolutional neural network (CNN) features from fully-connected layer and aggregated intermediate convolutional layers. Unsupervised metric learning is used for similarity measurement and feature matching. An efficient re-ranking algorithm combined with k-nearest neighborhood fuses the retrieval results from two levels of features and further improves the retrieval performance. Extensive experiments on the widely used CC\_WEB\_VIDEO dataset shows that the proposed approach exhibits superior performance over the state-of-the-art.

Motivation & Objective

• Address the challenge of near-duplicate video retrieval in large-scale video datasets with high redundancy.
• Improve video representation by fusing low-level features (LLF) from intermediate convolutional layers and upper-level features (ULF) from the fully-connected layer.
• Develop an unsupervised metric learning framework to enhance similarity measurement between videos without requiring labeled data.
• Introduce an efficient re-ranking algorithm based on k-nearest neighbors and Jaccard distance to fuse results from two feature levels and improve retrieval accuracy.

Proposed method

• Extract key frames using inter-frame differences from the last fully-connected layer output, followed by temporal correlation-based pruning to reduce redundancy.
• Extract multi-level video features: low-level features (LLF) from intermediate convolutional layers and upper-level features (ULF) from the final fully-connected layer (e.g., fc7 or pool5).
• Apply unsupervised metric learning via sparse contextual activations and Jaccard distance to compute similarity between query and candidate videos.
• Construct two sparse contextual activation vectors per query: MIN(F_q^fc, F_q^conv) for positive set and MAX(F_q^fc, F_q^conv) for negative set.
• Use a modified Randomly Projected kd-Tree for fast k-nearest neighbor retrieval in high-dimensional feature space.
• Perform rank fusion using Jaccard distance on top of k-NN results from both LLF and ULF to improve final retrieval ranking.

Experimental results

Research questions

• RQ1 Can fusing features from multiple levels of a pre-trained CNN improve near-duplicate video retrieval performance compared to single-level feature usage?
• RQ2 How effective is unsupervised metric learning using sparse contextual activations and Jaccard distance in measuring video similarity without labeled data?
• RQ3 To what extent does the proposed re-ranking strategy based on k-NN fusion enhance retrieval accuracy over baseline methods?
• RQ4 How does the proposed method compare to state-of-the-art NDVR approaches in terms of mAP and precision-recall performance on the CC_WEB_VIDEO benchmark?

Key findings

• The proposed CNN-UML method achieves a mean Average Precision (mAP) of 0.9790 on the CC_WEB_VIDEO dataset, outperforming all compared state-of-the-art methods.
• GoogLeNet-based features with rank fusion (RankF) yield the highest mAP (0.9790), demonstrating the effectiveness of multi-level feature fusion.
• Low-level features (LLF) from intermediate layers slightly outperform upper-level features (ULF) from the fully-connected layer across all CNN architectures.
• The re-ranking strategy based on k-NN and Jaccard distance significantly improves retrieval performance, especially when combining LLF and ULF representations.
• The method achieves superior performance without requiring complex training or codebook generation, unlike previous approaches such as CNN-L or MFH.
• The PR curve analysis confirms that the proposed method maintains high precision across varying recall levels, indicating robust and consistent retrieval performance.