[Paper Review] Interactively Transferring CNN Patterns for Part Localization

Quanshi Zhang, Ruiming Cao+4 more|arXiv (Cornell University)|Aug 5, 2017

Generative Adversarial Networks and Image Synthesis21 references17 citations

TL;DR

This paper proposes an interactive method to transfer latent patterns from a pre-trained CNN for object part localization using minimal human input. By mining activation patterns from convolutional layers and allowing users to refine them via an And-Or Graph (AOG), the approach achieves superior localization accuracy—especially in few-shot settings—by leveraging human perception to correct noisy or incorrect patterns, outperforming end-to-end learning baselines.

ABSTRACT

In the scenario of one/multi-shot learning, conventional end-to-end learning strategies without sufficient supervision are usually not powerful enough to learn correct patterns from noisy signals. Thus, given a CNN pre-trained for object classification, this paper proposes a method that first summarizes the knowledge hidden inside the CNN into a dictionary of latent activation patterns, and then builds a new model for part localization by manually assembling latent patterns related to the target part via human interactions. We use very few (e.g., three) annotations of a semantic object part to retrieve certain latent patterns from conv-layers to represent the target part. We then visualize these latent patterns and ask users to further remove incorrect patterns, in order to refine part representation. With the guidance of human interactions, our method exhibited superior performance of part localization in experiments.

Motivation & Objective

• To address the challenge of learning object part detectors with very few annotated examples (1–3), where end-to-end CNN training often overfits to noise or fails to capture semantic parts.
• To enable human-in-the-loop refinement of CNN-derived latent patterns, ensuring semantic correctness and robustness in part localization.
• To develop a generalizable framework that transfers knowledge from a pre-trained CNN to a human-interpretable AOG model for part representation.
• To improve part localization performance in weakly-supervised settings by combining pre-trained CNN features with interactive pattern selection.

Proposed method

• Mining hundreds of latent activation patterns from pre-trained CNN conv-layers using a statistical criterion that emphasizes frequent, contextually relevant, and spatially coherent patterns.
• Representing the mined patterns via an And-Or Graph (AOG) to model semantic hierarchies: from CNN units to latent patterns, part templates, and semantic parts.
• Using up-convolutional networks (up-conv-net) to visualize latent patterns at different network depths, enabling human inspection of low-level details and high-level context.
• Allowing users to manually prune irrelevant AOG nodes (i.e., patterns) based on visual inspection, effectively removing background noise and spurious activations.
• Constructing a final AOG model by combining only human-verified, semantically relevant patterns, which are then used for part localization.
• Evaluating the method using normalized distance as the metric, with part localization performed on cropped images using object bounding boxes to isolate part detection performance.

Experimental results

Research questions

• RQ1 Can human-interactive refinement of CNN-derived latent patterns improve part localization in few-shot learning scenarios?
• RQ2 Can a pre-trained CNN’s internal representations be effectively mined and transferred into a human-interpretable model for part detection?
• RQ3 Does interactive selection of patterns via AOGs lead to better performance than end-to-end training with minimal supervision?
• RQ4 How do low-level and high-level CNN features contribute to accurate part localization when guided by human perception?

Key findings

• The proposed method achieved state-of-the-art performance on the Pascal VOC Part dataset, with normalized distances of 0.1225 for bird beak, 0.1570 for bird neck, 0.1580 for bird wing, and 0.1331 for cat eye, outperforming the Mining-raw baseline.
• On the ILSVRC 2013 DET Animal-Part dataset, the method reduced average normalized distance across all parts, demonstrating consistent superiority in few-shot part localization.
• The CUB200-2011 dataset evaluation showed that the method achieved lower normalized distances than baselines, particularly for challenging parts like the head (forehead) in birds.
• Human interaction time averaged 12.3 seconds per image, with labeling a single part bounding box taking 3.4 seconds, indicating practical efficiency for interactive use.
• Visualizations showed that low-layer patterns captured fine details (e.g., beak texture), while high-layer patterns encoded contextual relationships, both of which were refined effectively by human selection.
• The AOG-based model after human pruning showed significant improvement in localization accuracy, confirming that human perception effectively corrects noisy or incorrect CNN patterns.