[Paper Review] Convolutional Neural Networks for Aerial Multi-Label Pedestrian Detection

Amir Soleimani, Nasser M. Nasrabadi|arXiv (Cornell University)|Jul 16, 2018

Advanced Neural Network Applications11 references4 citations

TL;DR

This paper proposes a two-stage framework for aerial multi-label pedestrian action detection using SSD for object proposal generation and a VGG-based network to learn a shared latent space between high-resolution object proposals and multi-action labels. The method achieves 28.30% mAP on the Okutama-Action dataset, significantly outperforming single-stage SSD baselines (e.g., 18.80% for SSD960), demonstrating improved accuracy through high-resolution feature fusion and multi-label learning.

ABSTRACT

The low resolution of objects of interest in aerial images makes pedestrian detection and action detection extremely challenging tasks. Furthermore, using deep convolutional neural networks to process large images can be demanding in terms of computational requirements. In order to alleviate these challenges, we propose a two-step, yes and no question answering framework to find specific individuals doing one or multiple specific actions in aerial images. First, a deep object detector, Single Shot Multibox Detector (SSD), is used to generate object proposals from small aerial images. Second, another deep network, is used to learn a latent common sub-space which associates the high resolution aerial imagery and the pedestrian action labels that are provided by the human-based sources

Motivation & Objective

• To address the challenge of low-resolution pedestrian detection in aerial imagery, where objects are often just a few pixels in size.
• To improve multi-label pedestrian action detection in aerial images by leveraging high-resolution object proposals instead of low-resolution input to detectors.
• To develop a framework that fuses image features from high-resolution object proposals with multi-action label information to enhance detection accuracy.
• To overcome the limitations of single-stage detectors like SSD, which struggle with small objects and multi-label action recognition in aerial views.

Proposed method

• First, SSD512 is used to generate object proposals from small aerial images (300x300 or 512x512), producing bounding boxes for potential pedestrians.
• High-resolution patches corresponding to these proposals are extracted from the original, full-resolution aerial images to preserve fine-grained details.
• A VGG-based convolutional network is employed to extract deep features from the high-resolution object proposals.
• A shared latent space is learned between the image features and multi-action label embeddings, enabling joint representation learning.
• The framework uses a yes/no question-answering mechanism to determine if a detected pedestrian is performing a specific action (or multiple actions).
• Multi-label learning is applied to exploit shared semantic information across actions, improving generalization and detection performance.

Experimental results

Research questions

• RQ1 Can a two-stage framework improve multi-label pedestrian action detection in low-resolution aerial images compared to single-stage detectors?
• RQ2 Does using high-resolution object proposals extracted from original images enhance action recognition accuracy despite the small object size?
• RQ3 To what extent does learning a shared latent space between image features and multi-action labels improve detection performance?
• RQ4 How does multi-label learning contribute to performance gains in aerial pedestrian action detection?

Key findings

• The proposed two-stage framework achieves 28.30% mAP on the Okutama-Action dataset for multi-label pedestrian action detection, significantly outperforming SSD512 (15.39%) and SSD960 (18.80%).
• Using high-resolution object proposals from original images improves detection accuracy compared to processing low-resolution inputs, even when the initial detector (SSD) operates on small images.
• The use of multi-label learning in the latent space enables the model to leverage shared semantic patterns across actions, enhancing generalization and performance.
• SSD512 achieves 75.3% mAP@0.35 on pedestrian detection alone, indicating strong baseline performance, but the two-stage framework further improves action detection by 10–15 percentage points.
• The framework effectively handles small object sizes in aerial images by preserving resolution in the second stage, reducing the impact of resolution degradation on action recognition.
• The results demonstrate that fusing high-resolution image features with multi-label action embeddings leads to more accurate and robust detection than relying solely on low-resolution detection heads.