[Paper Review] Analysing domain shift factors between videos and images for object detection

Vicky Kalogeiton, Vittorio Ferrari+1 more|arXiv (Cornell University)|Jan 6, 2015

Advanced Neural Network Applications30 references4 citations

TL;DR

This paper systematically analyzes domain shift factors between still images and video frames for object detection, identifying spatial location accuracy, appearance diversity, image quality, and aspect distribution as key contributors to performance gaps. By progressively equalizing these factors, the authors reduce the mAP gap between image- and video-trained detectors to under 1.5%, demonstrating that image quality and aspect distribution are particularly critical for cross-domain generalization.

ABSTRACT

Object detection is one of the most important challenges in computer vision. Object detectors are usually trained on bounding-boxes from still images. Recently, video has been used as an alternative source of data. Yet, for a given test domain (image or video), the performance of the detector depends on the domain it was trained on. In this paper, we examine the reasons behind this performance gap. We define and evaluate different domain shift factors: spatial location accuracy, appearance diversity, image quality and aspect distribution. We examine the impact of these factors by comparing performance before and after factoring them out. The results show that all four factors affect the performance of the detectors and their combined effect explains nearly the whole performance gap.

Motivation & Objective

• To identify and quantify the domain shift factors that cause performance gaps between object detectors trained on images versus videos.
• To evaluate the impact of each factor—spatial accuracy, appearance diversity, image quality, and aspect distribution—on detector performance.
• To determine whether equalizing these factors can close the performance gap between image- and video-trained detectors.
• To guide future research on weakly supervised video-based training by revealing which factors most hinder detector generalization.

Proposed method

• The authors analyze two dataset pairs: PASCAL VOC 07 (images) and YouTube-Objects (video), and ILSVRC 2015 image and video tracks.
• They define and isolate four domain shift factors: spatial location accuracy, appearance diversity, image quality (measured via gradient energy and blur), and aspect distribution (measured via KL divergence).
• For each factor, they apply equalization procedures to make training sets comparable—e.g., reducing unique samples to match, applying Gaussian blur to match video blur levels, and reweighting aspect distributions.
• An R-CNN detector is trained on both original and equalized training sets, then tested on both image and video test sets to measure mAP performance.
• The performance gap is measured before and after each equalization step to isolate the contribution of each factor.
• The analysis uses a structured protocol to cancel out factors one by one, observing the incremental reduction in performance gap.

Experimental results

Research questions

• RQ1 What specific domain shift factors contribute to the performance gap between object detectors trained on images versus videos?
• RQ2 How much does each factor—appearance diversity, image quality, aspect distribution, and spatial accuracy—contribute to the performance gap?
• RQ3 Can equalizing these domain shift factors significantly reduce or eliminate the performance gap between image- and video-trained detectors?
• RQ4 Which factor has the largest impact on detector generalization across domains?

Key findings

• The combined effect of all four domain shift factors explains nearly the entire performance gap between detectors trained on images versus videos.
• Image quality has a strong impact: equalizing blur levels significantly reduces the gap, especially when video frames are blurrier than images.
• Appearance diversity has a smaller effect when training data is abundant; even after removing 40% of unique samples, performance remains strong on images.
• Aspect distribution has a major influence: equalizing aspect distributions closes a large portion of the performance gap on both test domains.
• After equalizing all four factors, the final mAP gap between image- and video-trained detectors is below 1.5% on both test sets.
• The study shows that deblurring algorithms and broader aspect coverage in video data are more effective than simply increasing training sample count.