[Paper Review] Multi-Label Image Recognition with Graph Convolutional Networks

Zhao-Min Chen, Xiu-Shen Wei+2 more|arXiv (Cornell University)|Apr 7, 2019

Text and Document Classification TechnologiesComputer Science35 references62 citations

TL;DR

The paper introduces ML-GCN, a Graph Convolutional Network based model that maps label word embeddings into inter-dependent object classifiers and applies them to image features for end-to-end multi-label recognition, with a re-weighted label correlation matrix to improve propagation and generalization.

ABSTRACT

The task of multi-label image recognition is to predict a set of object labels that present in an image. As objects normally co-occur in an image, it is desirable to model the label dependencies to improve the recognition performance. To capture and explore such important dependencies, we propose a multi-label classification model based on Graph Convolutional Network (GCN). The model builds a directed graph over the object labels, where each node (label) is represented by word embeddings of a label, and GCN is learned to map this label graph into a set of inter-dependent object classifiers. These classifiers are applied to the image descriptors extracted by another sub-net, enabling the whole network to be end-to-end trainable. Furthermore, we propose a novel re-weighted scheme to create an effective label correlation matrix to guide information propagation among the nodes in GCN. Experiments on two multi-label image recognition datasets show that our approach obviously outperforms other existing state-of-the-art methods. In addition, visualization analyses reveal that the classifiers learned by our model maintain meaningful semantic topology.

Motivation & Objective

• Model and leverage label dependencies to improve multi-label image recognition.
• Learn inter-dependent classifiers for labels from word embeddings via Graph Convolutional Networks.
• Introduce a data-driven, re-weighted correlation matrix to guide information propagation and mitigate over-smoothing.
• Demonstrate end-to-end trainability and achieve superior performance on standard benchmarks.

Proposed method

• Represent each label as a word embedding and build a directed label graph.
• Use stacked GCNs to map label embeddings to a set of inter-dependent classifiers W.
• Apply classifiers W to image features x extracted by a CNN to obtain predictions ŷ = W x.
• Construct a data-driven label correlation matrix A from label co-occurrences; binarize with threshold τ to form A.
• Introduce a re-weighted scheme A' to balance node self-weights and neighborhood influence and reduce over-smoothing.
• Train end-to-end with standard multi-label classification loss.

Experimental results

Research questions

• RQ1 How can label dependencies be explicitly modeled to improve multi-label recognition performance?
• RQ2 Can label embeddings be transformed into inter-dependent classifiers that leverage co-occurrence patterns?
• RQ3 Does a re-weighted correlation matrix reduce overfitting and over-smoothing in GCN-based label modeling?
• RQ4 Do the learned classifiers reflect meaningful semantic topology among labels?

Key findings

• ML-GCN with the re-weighted correlation matrix consistently outperforms state-of-the-art methods on MS-COCO and VOC 2007.
• Using binary correlations leads to over-smoothing and worse performance compared to the re-weighted scheme.
• The re-weighted scheme improves key metrics such as mAP, CF1, and OF1 across datasets.
• Word embeddings are helpful but not solely responsible for gains; the GCN-based mapping and correlation modeling drive significant improvement.
• Classifier visualizations show meaningful semantic topology, with related labels forming clusters in classifier space.
• The learned image representations via ML-GCN improve image retrieval quality compared to a vanilla ResNet baseline.