[Paper Review] AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling

Ting-Wu Chin, Ruizhou Ding+1 more|arXiv (Cornell University)|Feb 8, 2019

Advanced Image and Video Retrieval Techniques23 references39 citations

TL;DR

AdaScale adaptively selects the input image scale for each video frame to improve both speed and accuracy in video object detection, achieving notable mAP gains with speedups on ImageNet VID and mini YouTube-BB datasets.

ABSTRACT

In vision-enabled autonomous systems such as robots and autonomous cars, video object detection plays a crucial role, and both its speed and accuracy are important factors to provide reliable operation. The key insight we show in this paper is that speed and accuracy are not necessarily a trade-off when it comes to image scaling. Our results show that re-scaling the image to a lower resolution will sometimes produce better accuracy. Based on this observation, we propose a novel approach, dubbed AdaScale, which adaptively selects the input image scale that improves both accuracy and speed for video object detection. To this end, our results on ImageNet VID and mini YouTube-BoundingBoxes datasets demonstrate 1.3 points and 2.7 points mAP improvement with 1.6x and 1.8x speedup, respectively. Additionally, we improve state-of-the-art video acceleration work by an extra 1.25x speedup with slightly better mAP on ImageNet VID dataset.

Motivation & Objective

• Motivate faster and more accurate video object detection for autonomous systems.
• Investigate whether down-sampling can improve detection accuracy, not just speed.
• Develop an adaptive scale mechanism that predicts the optimal input scale per frame.
• Demonstrate improvements on ImageNet VID and YouTube-BB datasets and show compatibility with existing video acceleration methods.

Proposed method

• Define a finite set of input scales S and a metric based on per-image detection loss to select an optimal scale m_opt for each image.
• Train a scale regressor that takes deep CNN features and outputs a relative scale t in [-1, 1] indicating how to adjust the input scale for the next frame.
• Use a loss L_scalereg (mean squared error) to train the regressor against ground-truth optimal scales derived from the per-image loss metric.
• Leverage temporal consistency by assuming similar optimal scales for consecutive video frames.
• Apply AdaScale during testing with a simple decoding step to map the regressed scale back to an actual image scale within S.
• Optionally combine AdaScale with existing video acceleration methods to further boost speed.
• Training uses multi-scale fine-tuning of an R-FCN-based detector across scales, and regressor weights are learned while keeping the detector fixed.

Experimental results

Research questions

• RQ1 Can adaptive image scaling improve both speed and accuracy for video object detectors beyond traditional fixed-scale testing?
• RQ2 Does down-sampling at inference time help reduce false positives and increase true positives for video object detectors?
• RQ3 How effectively can a learned regressor predict the optimal scale for the next video frame based on current frame content?
• RQ4 Is AdaScale complementary to other video acceleration techniques in achieving a better speed-accuracy Pareto frontier?

Key findings

• AdaScale yields 1.3 point mAP improvement on ImageNet VID with 1.6x speedup (MS/AdaScale vs SS/SS).
• AdaScale yields 2.7 point mAP improvement on mini YouTube-BB with 1.8x speedup (MS/AdaScale vs SS/SS).
• Combining AdaScale with state-of-the-art video acceleration can provide an additional ~25% speedup with slight mAP gains on ImageNet VID.
• The scale regressor adds about 2 ms of overhead (≈3% of R-FCN runtime).
• Adaptive scaling reduces false positives and can increase precision with modest recall changes compared to fixed-scale baselines.