[Paper Review] Learning Image Conditioned Label Space for Multilabel Classification

Yinan Li, Mei-Chen Yeh|arXiv (Cornell University)|Feb 21, 2018

Image Retrieval and Classification Techniques28 references3 citations

TL;DR

This paper proposes a novel image-dependent ranking model for multilabel image classification that learns a transformation matrix from an image to map word vectors into a k-dimensional space, enabling effective ranking of relevant labels. Unlike conventional CNNs that learn image representations, this method learns a transformation to distinguish relevant from irrelevant labels, achieving state-of-the-art performance with fewer training images on the NUS-WIDE benchmark.

ABSTRACT

This work addresses the task of multilabel image classification. Inspired by the great success from deep convolutional neural networks (CNNs) for single-label visual-semantic embedding, we exploit extending these models for multilabel images. Specifically, we propose an image-dependent ranking model, which returns a ranked list of labels according to its relevance to the input image. In contrast to conventional CNN models that learn an image representation (i.e. the image embedding vector), the developed model learns a mapping (i.e. a transformation matrix) from an image in an attempt to differentiate between its relevant and irrelevant labels. Despite the conceptual simplicity of our approach, experimental results on a public benchmark dataset demonstrate that the proposed model achieves state-of-the-art performance while using fewer training images than other multilabel classification methods.

Motivation & Objective

• To address the challenge of multilabel image classification, where images can contain multiple semantically related or independent labels with complex co-occurrence dependencies.
• To overcome limitations of existing CNN-based models that struggle with scalability, semantic redundancy, and label co-occurrence in multilabel settings.
• To develop a simple yet powerful framework that learns a transformation from image to label space, rather than a fixed image embedding, to improve label relevance ranking.
• To enable efficient, end-to-end training without requiring object segments, bounding boxes, or complex architectures like RNNs.
• To explore the feasibility of zero-shot prediction and nonlinear transformations in future extensions.

Proposed method

• The model learns a k×d transformation matrix A from a convolutional neural network (CNN) that maps word vectors from the original space to a k-dimensional Euclidean space.
• Each row of the transformation matrix A represents a principal direction in the word vector space along which labels are ranked by relevance to the input image.
• The method performs linear transformation of word vectors using the learned matrix A, enabling fast and accurate label prediction via ranking in the transformed space.
• The framework is trained end-to-end using a ranking loss to optimize the separation between relevant and irrelevant labels.
• The model is interpreted as an ensemble of k complementary classifiers sharing the same CNN features, with label predictions aggregated via voting.
• The approach avoids complex subnetworks like RNNs or region proposal modules, maintaining simplicity while modeling label dependencies.

Experimental results

Research questions

• RQ1 Can a simple CNN-based transformation matrix learn to effectively rank labels according to their relevance to an input image?
• RQ2 Does learning a transformation rather than a fixed image representation improve performance in multilabel image classification?
• RQ3 How does the model handle semantic redundancy and co-occurrence dependencies among labels without explicit modeling components?
• RQ4 What is the impact of the transformed dimension k on classification performance?
• RQ5 Can the model generalize to unseen labels in a zero-shot setting using the shared semantic space?

Key findings

• The proposed method achieves state-of-the-art performance on the NUS-WIDE benchmark, outperforming existing models like WARP and CNN-RNN with significantly fewer training images.
• The model maintains stable performance across different values of k (from 10 to 100), indicating that the choice of transformed dimension has minimal impact on overall accuracy.
• The average Jaccard coefficient between label sets predicted by individual classifiers is 0.1001, indicating high diversity among the k classifiers, which contributes to improved ensemble performance.
• Using a voting strategy on the top-5 predictions from all k classifiers achieves a 63.8% O-R (overall recall) score, outperforming single-label baselines and prior methods.
• The model's performance is robust even when trained on limited data, demonstrating strong generalization capability.
• The method is conceptually simple and compact, yet more powerful than many complex deep learning models for multilabel classification.