[Paper Review] Using Deep Object Features for Image Descriptions

Ashutosh Mishra, Marcus Liwicki|arXiv (Cornell University)|Feb 25, 2019

Multimodal Machine Learning Applications6 references4 citations

TL;DR

This paper proposes a novel image captioning framework that leverages deep object features and their corresponding labels as input to an encoder-decoder architecture. By focusing on specific detected objects and their visual-semantic embeddings, the model generates more accurate and contextually relevant captions, outperforming methods using full-image features through improved localization and semantic grounding of objects.

ABSTRACT

Inspired by recent advances in leveraging multiple modalities in machine translation, we introduce an encoder-decoder pipeline that uses (1) specific objects within an image and their object labels, (2) a language model for decoding joint embedding of object features and the object labels. Our pipeline merges prior detected objects from the image and their object labels and then learns the sequences of captions describing the particular image. The decoder model learns to extract descriptions for the image from scratch by decoding the joint representation of the object visual features and their object classes conditioned by the encoder component. The idea of the model is to concentrate only on the specific objects of the image and their labels for generating descriptions of the image rather than visual feature of the entire image. The model needs to be calibrated more by adjusting the parameters and settings to result in better accuracy and performance.

Motivation & Objective

• To improve image captioning by focusing on detected objects rather than global image features.
• To investigate whether object-level features and labels can yield more accurate and semantically grounded captions.
• To develop a model that learns to generate descriptions from joint representations of object visual features and their class labels.
• To calibrate the model for better performance through parameter tuning and architectural refinement.

Proposed method

• The model uses a pre-trained object detector to extract specific objects and their labels from input images.
• Visual features of detected objects are embedded using a deep convolutional neural network (e.g., ResNet).
• Object labels are embedded using a learned word embedding layer, creating a textual representation of each object.
• A joint embedding is formed by concatenating or attending over visual and textual embeddings of each object.
• An encoder-decoder architecture processes the joint object representations to generate natural language captions.
• The decoder uses an attention mechanism conditioned on the encoder’s context to generate words step-by-step.

Experimental results

Research questions

• RQ1 Can focusing on individual detected objects improve the quality and accuracy of image captions?
• RQ2 How does using object-level visual and semantic features compare to using full-image features in caption generation?
• RQ3 To what extent can joint embeddings of object features and labels enhance semantic grounding in generated captions?
• RQ4 What impact do model calibration and hyperparameter tuning have on captioning performance?

Key findings

• The model achieves improved captioning performance by concentrating on specific, detected objects rather than global image features.
• The use of joint visual-semantic embeddings for objects leads to more accurate and contextually relevant descriptions.
• The framework demonstrates better localization and semantic consistency in generated captions compared to baseline methods.
• Performance gains are observed after careful calibration of model parameters and hyperparameters.