[Paper Review] Learning a Unified Sample Weighting Network for Object Detection

Qi Cai, Yingwei Pan+4 more|arXiv (Cornell University)|Jun 11, 2020

Advanced Neural Network Applications27 references4 citations

TL;DR

This paper proposes a unified sample weighting network (SWN) that dynamically predicts task-specific sample weights for object detection by modeling uncertainty in classification and regression losses, IoU, and prediction scores. The method improves detection accuracy by up to 1.8% across multiple one- and two-stage detectors without increasing inference time.

ABSTRACT

Region sampling or weighting is significantly important to the success of modern region-based object detectors. Unlike some previous works, which only focus on "hard" samples when optimizing the objective function, we argue that sample weighting should be data-dependent and task-dependent. The importance of a sample for the objective function optimization is determined by its uncertainties to both object classification and bounding box regression tasks. To this end, we devise a general loss function to cover most region-based object detectors with various sampling strategies, and then based on it we propose a unified sample weighting network to predict a sample's task weights. Our framework is simple yet effective. It leverages the samples' uncertainty distributions on classification loss, regression loss, IoU, and probability score, to predict sample weights. Our approach has several advantages: (i). It jointly learns sample weights for both classification and regression tasks, which differentiates it from most previous work. (ii). It is a data-driven process, so it avoids some manual parameter tuning. (iii). It can be effortlessly plugged into most object detectors and achieves noticeable performance improvements without affecting their inference time. Our approach has been thoroughly evaluated with recent object detection frameworks and it can consistently boost the detection accuracy. Code has been made available at \url{https://github.com/caiqi/sample-weighting-network}.

Motivation & Objective

• To address the limitations of static or hard-example-only sample weighting in object detection by making sample importance data- and task-dependent.
• To jointly optimize classification and regression weights in a unified framework, avoiding separate handling of each task.
• To reduce reliance on manual hyperparameter tuning by learning sample weights in an end-to-end, data-driven manner.
• To improve detection performance across diverse detectors (one-stage and two-stage) while preserving inference speed.

Proposed method

• The method introduces a sample weighting network (SWN) that takes classification loss, regression loss, IoU, and prediction score as inputs to predict sample-specific weights for both classification and regression tasks.
• It formulates a unified loss function combining classification and regression losses with learned uncertainty-aware weights, using a Gaussian noise model to estimate uncertainty.
• The network uses a differentiable weighting mechanism where sample weights are derived from learned log-variance of losses, with a regularization term to prevent weight collapse.
• A batch-wise smoothing strategy is applied to stabilize training by averaging weights across positive and negative samples per batch.
• The approach is plug-and-play, compatible with existing detectors like Faster R-CNN, SSD, and YOLO, with minimal architectural changes.
• Hyperparameters such as λ₁ control the influence of uncertainty on weights, enabling stable optimization without manual tuning.

Experimental results

Research questions

• RQ1 How can sample weighting in object detection be made both data-dependent and task-dependent, rather than relying on fixed or hard-example-only strategies?
• RQ2 Can a unified network jointly learn optimal weights for both classification and regression tasks in a multi-task detection framework?
• RQ3 Does a data-driven, uncertainty-based sample weighting mechanism outperform traditional hard example mining or focal loss in terms of detection accuracy?
• RQ4 To what extent does the proposed method generalize across different backbone architectures and detector frameworks?
• RQ5 How robust is the sample weighting network to different initializations and training dynamics?

Key findings

• The proposed sample weighting network achieves consistent performance gains of up to 1.8% mAP on MS COCO and Pascal VOC across multiple detectors, including ResNet-50 and ResNeXt-101-32x4d backbones.
• The method improves detection accuracy without increasing inference time, making it suitable for real-time applications.
• Ablation studies confirm that the network effectively learns to assign higher weights to samples with low uncertainty, even when they are 'easy' in terms of loss magnitude.
• The sample weights increase during training as losses decrease, indicating that the network learns to prioritize easy, low-uncertainty samples—consistent with the model's uncertainty-driven design.
• The approach is robust to different weight initializations, with averaged sample weights converging to similar values across multiple runs, indicating stable optimization.
• The regression loss distribution analysis shows that the method assigns higher weights to high-IoU samples, confirming that it prioritizes accurate localization in a data-dependent manner.