[Paper Review] Learning Category Correlations for Multi-label Image Recognition with Graph Networks

Qing Li, Xiaojiang Peng+2 more|arXiv (Cornell University)|Sep 28, 2019

Text and Document Classification TechnologiesComputer Science40 references18 citations

TL;DR

This paper proposes A-GCN, a novel graph neural network framework that learns adaptive label correlation graphs for multi-label image recognition using word embeddings and 1×1 convolutions. By introducing a sparse correlation constraint and a plug-and-play label graph module, A-GCN outperforms baseline methods and achieves state-of-the-art performance on MS-COCO and Fashion550K, improving mAP by 2.8% over ResNet101 on MS-COCO.

ABSTRACT

Multi-label image recognition is a task that predicts a set of object labels in an image. As the objects co-occur in the physical world, it is desirable to model label dependencies. Previous existing methods resort to either recurrent networks or pre-defined label correlation graphs for this purpose. In this paper, instead of using a pre-defined graph which is inflexible and may be sub-optimal for multi-label classification, we propose the A-GCN, which leverages the popular Graph Convolutional Networks with an Adaptive label correlation graph to model label dependencies. Specifically, we introduce a plug-and-play Label Graph (LG) module to learn label correlations with word embeddings, and then utilize traditional GCN to map this graph into label-dependent object classifiers which are further applied to image features. The basic LG module incorporates two 1x1 convolutional layers and uses the dot product to generate label graphs. In addition, we propose a sparse correlation constraint to enhance the LG module and also explore different LG architectures. We validate our method on two diverse multi-label datasets: MS-COCO and Fashion550K. Experimental results show that our A-GCN significantly improves baseline methods and achieves performance superior or comparable to the state of the art.

Motivation & Objective

• To address the limitation of pre-defined, inflexible label correlation graphs in multi-label image recognition.
• To model semantic label dependencies in a data-driven, end-to-end trainable manner.
• To improve multi-label classification performance by learning dynamic, adaptive label correlations from word embeddings.
• To enhance generalization and robustness using a sparse correlation constraint to prevent over-smoothing.
• To evaluate the effectiveness of various label graph architectures in capturing label co-occurrence patterns.

Proposed method

• Introduces a plug-and-play Label Graph (LG) module that uses two 1×1 convolutional layers and a dot product to generate an adaptive label correlation graph from word embeddings.
• Employs Graph Convolutional Networks (GCN) to map the learned label graph into label-dependent classifiers applied to image features.
• Applies a sparse correlation constraint using L1-norm loss between the learned correlation matrix and an identity matrix to encourage sparsity and reduce over-smoothing.
• Explores multiple alternative LG architectures, including fully connected and attention-based variants, to evaluate robustness and performance.
• Trains the entire model end-to-end with a combined loss function that includes classification loss and the sparse correlation regularization.
• Uses word embeddings as input to the LG module to encode semantic relationships between labels without relying on hand-crafted graphs.

Experimental results

Research questions

• RQ1 Can an end-to-end trainable, adaptive label correlation graph improve multi-label image recognition beyond fixed or pre-defined graphs?
• RQ2 How effective is a sparse correlation constraint in preventing over-smoothing and improving generalization in label graph learning?
• RQ3 Do different label graph architectures (e.g., FC, attention-based) yield significant performance differences in multi-label recognition?
• RQ4 Can the proposed A-GCN framework generalize across diverse datasets with varying label co-occurrence patterns?
• RQ5 How much performance gain does the adaptive graph bring compared to standard GCN or baseline CNNs?

Key findings

• A-GCN improves mAP from 80.3% (ResNet101 baseline) to 83.1% on MS-COCO, outperforming the state-of-the-art ML-GCN method.
• The sparse correlation constraint (L_A) improves performance by reducing over-smoothing, with optimal α=1.0; higher values degrade performance.
• The FC-A architecture for the LG module achieves the best performance among alternatives, outperforming the default 1×1 convolution-based design.
• On Fashion550K, A-GCN achieves 66.32% mAP when fine-tuned on clean data, surpassing the baseline by 3.4% and outperforming re-implemented ML-GCN.
• Visualization shows significant performance gains on co-occurring object categories (e.g., spoon, backpack, toothbrush), confirming effective learning of label dependencies.
• The method generalizes well across datasets, demonstrating consistent improvements on both MS-COCO and Fashion550K under diverse training settings.