[Paper Review] Multi-Label Classification with Label Graph Superimposing

Ya Wang, Dongliang He+5 more|arXiv (Cornell University)|Nov 21, 2019

Text and Document Classification Technologies32 references14 citations

TL;DR

This paper proposes KSSNet, a novel multi-label classification framework that superimposes a knowledge graph onto a statistical co-occurrence graph to model label correlations more effectively. By introducing lateral connections between GCN and CNN at multiple depths, the model enhances feature learning with label system awareness, achieving state-of-the-art performance with 6.4% and 12.0% mAP gains on MS-COCO and Charades, respectively.

ABSTRACT

Images or videos always contain multiple objects or actions. Multi-label recognition has been witnessed to achieve pretty performance attribute to the rapid development of deep learning technologies. Recently, graph convolution network (GCN) is leveraged to boost the performance of multi-label recognition. However, what is the best way for label correlation modeling and how feature learning can be improved with label system awareness are still unclear. In this paper, we propose a label graph superimposing framework to improve the conventional GCN+CNN framework developed for multi-label recognition in the following two aspects. Firstly, we model the label correlations by superimposing label graph built from statistical co-occurrence information into the graph constructed from knowledge priors of labels, and then multi-layer graph convolutions are applied on the final superimposed graph for label embedding abstraction. Secondly, we propose to leverage embedding of the whole label system for better representation learning. In detail, lateral connections between GCN and CNN are added at shallow, middle and deep layers to inject information of label system into backbone CNN for label-awareness in the feature learning process. Extensive experiments are carried out on MS-COCO and Charades datasets, showing that our proposed solution can greatly improve the recognition performance and achieves new state-of-the-art recognition performance.

Motivation & Objective

• To address the challenge of modeling label correlations in multi-label recognition by combining statistical co-occurrence patterns with external knowledge graphs.
• To improve feature representation learning by integrating label system structure into the CNN feature extraction process, rather than treating labels only at inference.
• To overcome over-smoothing in deep GCNs by introducing lateral connections and redundancy control mechanisms.
• To develop a unified framework that enhances performance on both image and video multi-label recognition tasks with minimal computational overhead.

Proposed method

• Construct two adjacency matrices: one from label co-occurrence statistics (A_S) and one from a knowledge graph (ConceptNet) (A_K), representing statistical and prior-based label relationships.
• Superimpose the two graphs into a final adjacency matrix A = λA_S + (1−λ)A_K, where λ is a learnable superimposition weight.
• Initialize label embeddings using pre-trained node representations from ConceptNet to enrich semantic priors in the label graph.
• Apply multi-layer graph convolutional networks (GCNs) on the superimposed graph to learn contextualized label embeddings that capture both statistical and prior-based correlations.
• Introduce lateral connections (LC) between CNN and GCN at shallow, middle, and deep layers, injecting label embeddings into CNN feature maps to guide label-aware feature learning.
• Regularize GCN training using gradient signals from LC operations and apply redundant edge removal to mitigate over-smoothing in deep GCNs.

Experimental results

Research questions

• RQ1 How can label correlation modeling be improved by combining statistical co-occurrence patterns with external knowledge graphs?
• RQ2 Can injecting label system information into multiple CNN layers improve feature representation learning for multi-label recognition?
• RQ3 Does the proposed superimposed graph construction outperform using only statistical or only knowledge-based graphs?
• RQ4 How does the depth of GCN affect performance, especially in the presence of over-smoothing?
• RQ5 Can the lateral connection mechanism effectively regularize feature learning and improve generalization in multi-label settings?

Key findings

• The proposed KS graph, formed by superimposing statistical and knowledge-based graphs, achieves the best performance across both MS-COCO and Charades datasets.
• KSSNet achieves a 6.4% absolute mAP improvement over a plain CNN baseline on MS-COCO, setting a new state-of-the-art.
• On the Charades dataset, KSSNet achieves a 12.0% mAP gain over the baseline, demonstrating strong generalization to video recognition tasks.
• Increasing GCN depth from 2 to 4 layers improves mAP by 0.6% on MS-COCO and 1.9% on Charades, indicating that deeper GCNs are beneficial in KSSNet due to regularization from lateral connections.
• The ablation study confirms that the knowledge graph is more effective than statistical graphs on Charades due to data sparsity, while the KS graph consistently outperforms both on both datasets.
• Hyperparameter analysis shows optimal performance at λ=0.6 and τ=0.03, with mAP peaking at 44.93 on Charades, validating the robustness of the superimposition strategy.