[Paper Review] Hybrid Knowledge Routed Modules for Large-scale Object Detection

Chenhan Jiang, Hang Xu+2 more|arXiv (Cornell University)|Oct 30, 2018

Advanced Neural Network Applications46 references58 citations

TL;DR

HKRM introduces two knowledge-driven modules (explicit and implicit) that reason over a region-to-region graph to improve large-scale object detection, achieving substantial gains on VG and ADE datasets by leveraging attribute, relationship, and spatial knowledge.

ABSTRACT

The dominant object detection approaches treat the recognition of each region separately and overlook crucial semantic correlations between objects in one scene. This paradigm leads to substantial performance drop when facing heavy long-tail problems, where very few samples are available for rare classes and plenty of confusing categories exists. We exploit diverse human commonsense knowledge for reasoning over large-scale object categories and reaching semantic coherency within one image. Particularly, we present Hybrid Knowledge Routed Modules (HKRM) that incorporates the reasoning routed by two kinds of knowledge forms: an explicit knowledge module for structured constraints that are summarized with linguistic knowledge (e.g. shared attributes, relationships) about concepts; and an implicit knowledge module that depicts some implicit constraints (e.g. common spatial layouts). By functioning over a region-to-region graph, both modules can be individualized and adapted to coordinate with visual patterns in each image, guided by specific knowledge forms. HKRM are light-weight, general-purpose and extensible by easily incorporating multiple knowledge to endow any detection networks the ability of global semantic reasoning. Experiments on large-scale object detection benchmarks show HKRM obtains around 34.5% improvement on VisualGenome (1000 categories) and 30.4% on ADE in terms of mAP. Codes and trained model can be found in https://github.com/chanyn/HKRM.

Motivation & Objective

• Motivate robust object detection under large label spaces and long-tail distributions by leveraging commonsense knowledge.
• Develop a light-weight, plug-in module that enables global semantic reasoning over region proposals.
• Explore explicit knowledge (attributes, relationships) and implicit knowledge (spatial/layout) forms to guide reasoning.
• Demonstrate that integrating knowledge-aware graphs improves detection in occlusion, small objects, and rare categories.

Proposed method

• Define two knowledge modules: explicit (linguistic-derived constraints) and implicit (unlabeled constraints like spatial layouts).
• Construct an adaptive region-to-region graph per knowledge form and learn edges via MLPs conditioned on external knowledge graphs Q.
• Explicit module: learn edges e_ij by e_ij = MLP_Q(alpha(f_i, f_j)) supervised by ground-truth class relations in Q; propagate features using normalized edge weights to obtain f' and concatenate for detection.
• Implicit module: learn multiple region graphs with MLPs to capture implicit spatial constraints; combine graphs to produce g' with shared transformation W_g.
• Combine outputs from all modules (f'_a, f'_r, g') and feed to bounding box regression and classification layers for final results.
• Demonstrate generalizability by plugging HKRM into Faster R-CNN-like architectures and evaluating on VG (1000/3000 classes) and ADE.

Experimental results

Research questions

• RQ1 Can explicit and implicit knowledge modules improve large-scale object detection when integrated into region-based detectors?
• RQ2 Do different knowledge forms (attributes, relationships, spatial layouts) contribute differently to detection performance, especially for rare classes?
• RQ3 Can a per-image adaptive region graph guided by external knowledge lead to better global reasoning and accuracy than baseline region-wise predictions?
• RQ4 Is HKRM transferable across datasets (VG, ADE) and scalable to many classes?

Key findings

• HKRM yields large mAP gains over Faster R-CNN baselines: around 34.5% improvement on VG with 1000 categories and 30.4% on ADE.
• HKRM also achieves 26.5% improvement on VG with 3000 categories.
• Explicit knowledge (attribute/relationship) and implicit knowledge (spatial) each contribute to performance, with combined HKRM providing the strongest gains.
• For rare categories, HKRM shows notable improvements (about 1.5% average mAP on the top infrequent 150 classes).
• On PASCAL VOC and MS COCO, HKRM with all three knowledge modules improves over baselines (e.g., VOC: 78.8% mAP vs 75.1% baseline; COCO: 37.8% mAP vs 34.2% baseline).
• HKRM remains lightweight (adds roughly ~2% parameters) and is easily integrable with existing detectors.