[Paper Review] Labeling Topics with Images using Neural Networks

Νικόλαος Αλέτρας, Arpit Mittal|arXiv (Cornell University)|Aug 1, 2016

Image Retrieval and Classification Techniques22 references3 citations

TL;DR

This paper proposes a deep neural network (DNN) model that directly predicts the relevance of any unseen image to a given topic, outperforming prior methods that rely on iterative re-ranking of candidate images. By jointly encoding topic terms, image captions, and visual features (via VGG-16) into a 1600D input, the DNN learns non-linear associations between modalities, achieving state-of-the-art performance with O(n) runtime complexity and a Top-1 average rating of 2.12 on a benchmark dataset.

ABSTRACT

Topics generated by topic models are usually represented by lists of $t$ terms or alternatively using short phrases and images. The current state-of-the-art work on labeling topics using images selects images by re-ranking a small set of candidates for a given topic. In this paper, we present a more generic method that can estimate the degree of association between any arbitrary pair of an unseen topic and image using a deep neural network. Our method has better runtime performance $O(n)$ compared to $O(n^2)$ for the current state-of-the-art method, and is also significantly more accurate.

Motivation & Objective

• To address the limitations of existing image-based topic labeling methods that rely on expensive O(n²) re-ranking of pre-selected candidates.
• To develop a generic, scalable method that can evaluate the relevance of any arbitrary topic-image pair without requiring pre-filtering or iterative computation.
• To improve accuracy and efficiency in topic representation by jointly modeling textual and visual features using deep learning.
• To enable real-time, large-scale topic labeling using images as language-independent, intuitive labels.

Proposed method

• The model takes as input a concatenated vector of topic term embeddings (mean of 10 top terms), image caption embeddings, and a 1000D VGG-16 feature vector from the image.
• The input is passed through four fully connected hidden layers (256, 128, 64, 32 units) with ReLU activation and dropout regularization.
• The final output is a single real-valued score representing the predicted relevance of the image to the topic.
• The network is trained using mean absolute error loss and RMSProp optimization over 30 epochs with 5-fold cross-validation.
• The model is evaluated on a benchmark dataset of 300 topics with 20 candidate images each and human-annotated relevance scores (0–3).

Experimental results

Research questions

• RQ1 Can a deep neural network effectively predict the relevance of an image to an unseen topic without requiring pre-filtered candidate sets?
• RQ2 How does the joint modeling of textual and visual features improve topic labeling accuracy compared to methods using only text or image features?
• RQ3 Does the proposed method achieve better performance and efficiency than state-of-the-art approaches based on graph-based re-ranking?
• RQ4 To what extent do visual features alone outperform textual captions in the context of topic-image relevance prediction?

Key findings

• The proposed DNN model (Topic+Caption+VGG) achieved a Top-1 average rating of 2.12, significantly outperforming the next best method (Local PPR at 2.00) with p < 0.01.
• The model achieved nDCG-5 score of 0.81, the highest among all methods, indicating superior ranking performance for relevant image candidates.
• Using only visual features (DNN (Topic+VGG)) yielded a Top-1 rating of 2.04, outperforming the text-only model (1.94), suggesting images provide less noisy signals than captions.
• The model's O(n) inference complexity enables efficient evaluation over large-scale image repositories like Flickr, Getty, or ImageNet, unlike O(n²) graph-based methods.
• The DNN (Topic+Caption+VGG) model achieved statistically significant gains over WSABIE, LR, SVM, and both Global and Local PPR baselines on all metrics.