[Paper Review] Adversarial Complementary Learning for Weakly Supervised Object Localization

Xiaolin Zhang, Yunchao Wei+3 more|arXiv (Cornell University)|Apr 19, 2018

Adversarial Robustness in Machine Learning37 references77 citations

TL;DR

ACoL introduces two adversarial classifiers that jointly locate integral object regions under weak supervision, achieving state-of-the-art WSOL results on ILSVRC 2016.

ABSTRACT

In this work, we propose Adversarial Complementary Learning (ACoL) to automatically localize integral objects of semantic interest with weak supervision. We first mathematically prove that class localization maps can be obtained by directly selecting the class-specific feature maps of the last convolutional layer, which paves a simple way to identify object regions. We then present a simple network architecture including two parallel-classifiers for object localization. Specifically, we leverage one classification branch to dynamically localize some discriminative object regions during the forward pass. Although it is usually responsive to sparse parts of the target objects, this classifier can drive the counterpart classifier to discover new and complementary object regions by erasing its discovered regions from the feature maps. With such an adversarial learning, the two parallel-classifiers are forced to leverage complementary object regions for classification and can finally generate integral object localization together. The merits of ACoL are mainly two-fold: 1) it can be trained in an end-to-end manner; 2) dynamically erasing enables the counterpart classifier to discover complementary object regions more effectively. We demonstrate the superiority of our ACoL approach in a variety of experiments. In particular, the Top-1 localization error rate on the ILSVRC dataset is 45.14%, which is the new state-of-the-art.

Motivation & Objective

• Motivate and address the limitations of CAM-based WSOL by enabling integral object localization under image-level supervision.
• Propose a theoretically grounded method to obtain localization maps from last-layer feature maps during forward pass.
• Develop an end-to-end architecture with two adversarial classifiers that discover complementary object regions.
• Show that dynamic erasing guides the second classifier to find new regions and fuse maps for improved localization.

Proposed method

• Provide a theoretical justification that class-specific maps can be obtained from last-convolutional feature maps during forward pass.
• Introduce two parallel classifiers (A and B) fed by the backbone feature maps, where Classifier B erases regions identified by Classifier A to discover complementary regions.
• Use thresholding on Classifier A’s localization map to erase discriminative regions in the backbone features.
• Fuse the normalized localization maps from both classifiers via a elementwise maximum to form the final integral localization map.
• Train the network end-to-end with cross-entropy losses for both classifiers, enabling joint optimization.

Experimental results

Research questions

• RQ1 How can integral object localization be achieved under weak supervision without post-hoc CAM steps?
• RQ2 Can two adversarial classifiers with dynamic erasing discover complementary object regions to cover entire objects?
• RQ3 Does end-to-end training with erasing improve WSOL localization accuracy over single-classifier approaches?

Key findings

• ACoL achieves state-of-the-art Top-1 localization error of 45.14% on ILSVRC 2016 (with ensemble results reaching 30.03% Top-5 localization in some setups).
• Localization maps produced by ACoL cover larger object regions than CAM-based approaches, leading to more accurate bounding boxes.
• Two classifiers discover complementary object regions (e.g., head vs. legs) and their fusion yields integral object localization.
• Erasing discriminative regions guided by Classifier A enables Classifier B to uncover new regions, and the fusion of maps improves localization robustness.
• Across datasets (ILSVRC, CUB-200-2011, Caltech-256), ACoL improves localization results and achieves competitive classification performance without relying on bounding box annotations.