[Paper Review] An Active Learning Based Approach For Effective Video Annotation And Retrieval

Moitreya Chatterjee, Anton Leuski|arXiv (Cornell University)|Apr 27, 2015

Video Analysis and Summarization19 references3 citations

TL;DR

This paper proposes CRMActive, an active learning framework that integrates uncertainty, density, and diversity measures into a clustering-based sample selection strategy for video annotation and retrieval. By combining these with the NormCRM generative model, CRMActive achieves superior performance over random and state-of-the-art baselines on both TRECVID and SmartBody datasets, demonstrating faster convergence and consistent gains with minimal labeling effort.

ABSTRACT

Conventional multimedia annotation/retrieval systems such as Normalized Continuous Relevance Model (NormCRM) [16] require a fully labeled training data for a good performance. Active Learning, by determining an order for labeling the training data, allows for a good performance even before the training data is fully annotated. In this work we propose an active learning algorithm, which combines a novel measure of sample uncertainty with a novel clustering-based approach for determining sample density and diversity and integrate it with NormCRM. The clusters are also iteratively refined to ensure both feature and label-level agreement among samples. We show that our approach outperforms multiple baselines both on a recent, open character animation dataset and on the popular TRECVID corpus at both the tasks of annotation and text-based retrieval of videos.

Motivation & Objective

• To address the high human annotation cost in multimedia systems by reducing the need for fully labeled training data.
• To improve video annotation and retrieval performance by selecting the most informative samples for labeling in an active learning setting.
• To integrate uncertainty, density, and diversity measures into a unified sample selection strategy that enhances model learning efficiency.
• To evaluate the effectiveness of the proposed method on real-world video datasets, including TRECVID and a character animation dataset.
• To demonstrate that joint modeling of features and labels via NormCRM, combined with active learning, yields better performance than independent classifier training.

Proposed method

• Proposes a novel uncertainty measure for unlabeled samples based on the gap between the top-k and (k+1)th most relevant label probabilities: $\text{unct}(\mathbf{x}) = \frac{1}{P(w_1|\mathbf{x}) - P(w_{k+1}|\mathbf{x})}$.
• Introduces a cluster-refinement approach to estimate sample density and diversity, ensuring selected samples are representative and distinct.
• Uses iterative clustering to refine feature and label agreement across samples, improving model robustness during active learning.
• Integrates the sample selection engine with NormCRM, a generative model that jointly models features and labels to improve annotation and retrieval.
• Employs a batch active learning strategy where the top-K most informative samples are selected per iteration based on combined informativeness scores.
• Re-estimates the density weighting parameter $\gamma_d$ from labeled data at each batch and uses equal weights ($\lambda_i = \frac{1}{3}$) for uncertainty, density, and diversity components.

Experimental results

Research questions

• RQ1 Can a novel active learning strategy that combines uncertainty, density, and diversity improve video annotation and retrieval performance with minimal labeling?
• RQ2 How does the proposed CRMActive framework compare to random labeling and state-of-the-art active learning baselines in terms of annotation and retrieval accuracy?
• RQ3 To what extent does joint modeling of features and labels via NormCRM enhance performance compared to training independent classifiers per concept?
• RQ4 Does the iterative refinement of clusters improve label and feature consistency, leading to more robust model updates during active learning?
• RQ5 Can CRMActive achieve high performance even before full annotation, particularly in early rounds of labeling?

Key findings

• CRMActive outperforms both the random baseline and the state-of-the-art method of Zha et al. in annotation performance on both TRECVID 2007 and the SmartBody dataset.
• The AP scores for annotation using CRMActive show a monotonic non-decreasing trend across rounds, indicating stable and consistent improvement with each labeling batch.
• On the SmartBody dataset, CRMActive achieves correct top-6 labels for a sample by Round 7, even before full annotation, demonstrating rapid model convergence.
• For individual concepts on SmartBody, CRMActive maintains or exceeds the performance of all other models across all concepts, including those with low positive example counts.
• The NormCRM-based models (including CRMActive and the random baseline) significantly outperform Zha et al.’s method, which trains independent classifiers per concept, due to better modeling of label correlations.
• The performance gain is particularly notable for complex concepts like 'Dance', suggesting that the joint feature-label modeling captures richer semantic patterns than isolated classifiers.