[Paper Review] Self-informed neural network structure learning

David Warde-Farley, Andrew Rabinovich+1 more|arXiv (Cornell University)|Dec 20, 2014

Domain Adaptation and Few-Shot Learning6 references13 citations

TL;DR

This paper proposes a method to enhance a pre-trained deep neural network's performance on large-scale multi-label visual recognition by adding auxiliary fully connected layers that specialize in clusters of visually similar classes. Using the network's own predictions to guide spectral clustering of label similarities, the approach improves mean average precision by 2.61 percentage points with under 3% increase in multiply-adds.

ABSTRACT

We study the problem of large scale, multi-label visual recognition with a large number of possible classes. We propose a method for augmenting a trained neural network classifier with auxiliary capacity in a manner designed to significantly improve upon an already well-performing model, while minimally impacting its computational footprint. Using the predictions of the network itself as a descriptor for assessing visual similarity, we define a partitioning of the label space into groups of visually similar entities. We then augment the network with auxilliary hidden layer pathways with connectivity only to these groups of label units. We report a significant improvement in mean average precision on a large-scale object recognition task with the augmented model, while increasing the number of multiply-adds by less than 3%.

Motivation & Objective

• To improve performance of a pre-trained deep neural network on large-scale, multi-label visual recognition without significantly increasing inference cost.
• To address the challenge of recognizing fine-grained and long-tail classes that are difficult for a generalist network to distinguish.
• To develop a method that intelligently augments network capacity in a way that leverages the model's own predictions to identify semantically similar label groups.
• To minimize computational overhead while maximizing performance gains through targeted, structure-aware capacity expansion.

Proposed method

• The method uses the trained network's predictions on a held-out dataset to compute a confusion matrix, capturing inter-class similarities.
• Spectral clustering is applied to the confusion matrix to partition the label space into groups of visually similar classes.
• Auxiliary fully connected network stacks are added in parallel to the original network, with each stack connected only to a specific cluster of output units.
• The pre-trained network's features are frozen, and only the new auxiliary heads and the final classifier layer are fine-tuned during training.
• The architecture is trained to leverage the auxiliary capacity for improved discrimination within each label cluster, while preserving the original model's generalization.
• The approach avoids retraining the entire network and instead focuses on learning to use the new pathways effectively.

Experimental results

Research questions

• RQ1 Can a pre-trained neural network be improved by adding specialized, low-overhead pathways that target visually similar subsets of classes?
• RQ2 Does using the model’s own predictions to define label clusters lead to better performance than random or heuristic label grouping?
• RQ3 Can such targeted capacity augmentation improve mean average precision on long-tail and fine-grained classes with minimal computational cost?
• RQ4 Is the performance gain robust across different clustering strategies and network configurations?

Key findings

• The proposed method achieved a 39.41% mean average precision (mAP) on the JFT-18B dataset, a 2.61 percentage point improvement over the base model’s 36.80%.
• The improvement was achieved with only a 2.6% increase in multiply-add operations, rising from 1.52B to 1.56B.
• The method significantly outperformed a randomized label partitioning baseline, which achieved only 32.97% mAP under the same capacity constraints.
• When using co-detection-based clustering with 13 heads, the method achieved 38.07% mAP, still outperforming the random baseline (32.13%) and the base model.
• The results demonstrate that self-informed clustering of labels based on model predictions leads to more effective capacity utilization than random or metadata-driven grouping.
• The method is general and could be applied to other domains beyond computer vision, as it relies only on model predictions and not on input modality.