[Paper Review] Speed/accuracy trade-offs for modern convolutional object detectors

Jonathan Huang, Vivek Rathod+9 more|arXiv (Cornell University)|Nov 30, 2016

Advanced Neural Network Applications42 references157 citations

TL;DR

The paper conducts a unified, apples-to-apples evaluation of Faster R-CNN, R-FCN, and SSD meta-architectures across multiple feature extractors, image resolutions, and proposals to map speed/accuracy/memory trade-offs and identify optimal configurations.

ABSTRACT

The goal of this paper is to serve as a guide for selecting a detection architecture that achieves the right speed/memory/accuracy balance for a given application and platform. To this end, we investigate various ways to trade accuracy for speed and memory usage in modern convolutional object detection systems. A number of successful systems have been proposed in recent years, but apples-to-apples comparisons are difficult due to different base feature extractors (e.g., VGG, Residual Networks), different default image resolutions, as well as different hardware and software platforms. We present a unified implementation of the Faster R-CNN [Ren et al., 2015], R-FCN [Dai et al., 2016] and SSD [Liu et al., 2015] systems, which we view as "meta-architectures" and trace out the speed/accuracy trade-off curve created by using alternative feature extractors and varying other critical parameters such as image size within each of these meta-architectures. On one extreme end of this spectrum where speed and memory are critical, we present a detector that achieves real time speeds and can be deployed on a mobile device. On the opposite end in which accuracy is critical, we present a detector that achieves state-of-the-art performance measured on the COCO detection task.

Motivation & Objective

• Provide a concise survey of modern convolutional detection systems and show their similar high-level designs.
• Create a unified TensorFlow-based implementation of Faster R-CNN, R-FCN, and SSD to enable fair speed/accuracy comparisons.
• Characterize how different feature extractors, proposal counts, and input sizes affect speed, memory, and accuracy.
• Identify sweet spots on the speed/accuracy frontier and demonstrate near state-of-the-art single-model performance.
• Explain configurations that enable real-time or high-accuracy detectors for practical applications.

Proposed method

• Implement unified, single-pass detectors for three meta-architectures (Faster R-CNN, R-FCN, SSD) in TensorFlow.
• Evaluate combinations of six feature extractors (VGG-16, ResNet-101, Inception v2, Inception v3, Inception-ResNet v2, MobileNet).
• Vary input image sizes (high: 600, low: 300) and, for Faster R-CNN/R-FCN, the number of region proposals (10–300).
• Use Argmax matching with standard ground-truth encodings and Smooth L1 localization loss.
• Train end-to-end with asynchronous SGD, freeze batchnorm parameters, and evaluate on COCO with COCO metrics (mAP across IOU thresholds).
• Post-process with non-maximum suppression and report timing/memory on GPU with 1-image batches.

Experimental results

Research questions

• RQ1 How do speed, memory, and accuracy trade off across Faster R-CNN, R-FCN, and SSD meta-architectures?
• RQ2 How do different feature extractors impact detection performance and efficiency within each meta-architecture?
• RQ3 What is the effect of input resolution and number of proposals on speed and mAP?
• RQ4 Are there identifiable sweet spots on the speed/accuracy frontier that balance real-time performance with accuracy?
• RQ5 Can single-model detectors approach state-of-the-art accuracy without ensembling or multi-crop methods?

Key findings

• Faster R-CNN tends to be slower but more accurate, unless the number of proposals is limited to reduce runtime.
• R-FCN and SSD generally offer faster inference with competitive accuracy across configurations.
• The study identifies sweet spots such as Faster R-CNN with ResNet-101 and 50–100 proposals or R-FCN with ResNet-101 and 300 proposals as strong speed/accuracy candidates.
• The most accurate single-model configuration reported is Faster R-CNN with Inception-ResNet-v2 and 300 proposals, though it is the slowest option.
• SSD configurations with MobileNet or Inception V2 provide the fastest results among the evaluated setups, with notable accuracy advantages at low resolutions.
• Increasing input resolution improves mAP but increases runtime substantially, highlighting the trade-off between accuracy and speed.