[Paper Review] Exploring Multi-Branch and High-Level Semantic Networks for Improving Pedestrian Detection

Jiale Cao, Yanwei Pang+1 more|arXiv (Cornell University)|Apr 3, 2018

Video Surveillance and Tracking Methods58 references6 citations

TL;DR

This paper proposes a multi-branch and high-level semantic network (MHN) for improved pedestrian detection by splitting a base CNN into parallel branches with identical depth, ensuring consistent high-level semantics across feature maps. By leveraging different receptive fields and skip connections, MHN enhances multi-scale detection—especially for small pedestrians—while maintaining parameter efficiency; it achieves state-of-the-art performance on KITTI, Caltech, Citypersons, and COCO benchmarks.

ABSTRACT

To better detect pedestrians of various scales, deep multi-scale methods usually detect pedestrians of different scales by different in-network layers. However, the semantic levels of features from different layers are usually inconsistent. In this paper, we propose a multi-branch and high-level semantic network by gradually splitting a base network into multiple different branches. As a result, the different branches have the same depth and the output features of different branches have similarly high-level semantics. Due to the difference of receptive fields, the different branches are suitable to detect pedestrians of different scales. Meanwhile, the multi-branch network does not introduce additional parameters by sharing convolutional weights of different branches. To further improve detection performance, skip-layer connections among different branches are used to add context to the branch of relatively small receptive filed, and dilated convolution is incorporated into part branches to enlarge the resolutions of output feature maps. When they are embedded into Faster RCNN architecture, the weighted scores of proposal generation network and proposal classification network are further proposed. Experiments on KITTI dataset, Caltech pedestrian dataset, and Citypersons dataset demonstrate the effectiveness of proposed method. On these pedestrian datasets, the proposed method achieves state-of-the-art detection performance. Moreover, experiments on COCO benchmark show the proposed method is also suitable for general object detection.

Motivation & Objective

• Address the inconsistency in semantic levels across feature maps in existing feature pyramid networks for pedestrian detection.
• Improve detection performance for small-scale pedestrians by enhancing high-resolution features with contextual information from deeper branches.
• Maintain parameter efficiency by sharing convolutional weights across parallel branches in the multi-branch architecture.
• Extend the method to general object detection by evaluating on the COCO benchmark.
• Introduce weighted scores in Faster R-CNN to further improve proposal generation and classification in the detection pipeline.

Proposed method

• Gradually split a base CNN into multiple parallel branches before pooling layers, ensuring all branches have the same depth and thus similar semantic levels.
• Leverage different receptive fields across branches to detect pedestrians of varying scales, with larger receptive fields in deeper branches.
• Apply skip-layer connections from low-resolution, large-receptive-field branches to high-resolution, small-receptive-field branches to inject contextual information.
• Incorporate dilated convolutions in part branches to preserve spatial resolution and enhance feature representation.
• Integrate MHN and MHN-D into Faster R-CNN by introducing weighted scores for both region proposal network (RPN) and Fast R-CNN classification head.
• Train and evaluate using standard protocols on KITTI, Caltech, Citypersons, and COCO datasets with detectron framework.

Experimental results

Research questions

• RQ1 Can a multi-branch network with identical depth across branches achieve more consistent high-level semantic features for multi-scale pedestrian detection?
• RQ2 Does adding skip connections from high-receptive-field to low-receptive-field branches improve detection of small pedestrians?
• RQ3 Can dilated convolutions in part branches enhance spatial resolution and detection performance without increasing parameters?
• RQ4 Does the proposed MHN architecture outperform existing feature pyramid methods like FPN and MSCNN on pedestrian detection benchmarks?
• RQ5 Is the MHN framework transferable and effective for general object detection on large-scale benchmarks like COCO?

Key findings

• On the KITTI dataset, MHN achieves 76.58% average precision (AP), surpassing FPN (75.42%) and MSCNN (74.83%) by 1.16 and 1.75 percentage points, respectively.
• On the Caltech pedestrian dataset, MHN achieves 66.76% AP, outperforming FPN (66.14%) and MSCNN (65.20%) by 0.62 and 1.56 points, respectively.
• On Citypersons, MHN achieves 37.19% AP, improving upon FPN (35.91%) and MSCNN (35.55%) by 1.28 and 1.64 points, respectively.
• On COCO test-dev, MHN with ResNet101 backbone achieves 44.1% AP, outperforming Mask R-CNN (39.8%) by 4.3 points and showing a 5.4% improvement on small-scale object detection.
• MHN-D with dilated convolutions and skip connections further boosts performance, especially on small-scale objects, with a 3.38% AP gain over FPN on ResNet101 for small pedestrians.
• The proposed weighted scoring mechanism in Faster R-CNN further improves detection by 1.30% AP on COCO with ResNet50 and 1.79% with ResNet101, demonstrating its effectiveness in end-to-end detection.