[Paper Review] Dual-Path Convolutional Image-Text Embedding.

Zhedong Zheng, Liang Zheng+3 more|arXiv (Cornell University)|Nov 15, 2017

Multimodal Machine Learning ApplicationsComputer Science36 references47 citations

TL;DR

This paper proposes a dual-path convolutional network for joint image-text embedding, using end-to-end trainable CNNs with ReLU and batch normalization to jointly optimize visual and textual features. It introduces an instance loss with large margin optimization, achieving state-of-the-art performance on language person retrieval and competitive results on Flickr30k and MSCOCO.

ABSTRACT

This paper considers the task of matching images and sentences. The challenge consists in discriminatively embedding the two modalities onto a shared visual-textual space. Existing work in this field largely uses Recurrent Neural Networks (RNN) for text feature learning and employs off-the-shelf Convolutional Neural Networks (CNN) for image feature extraction. Our system, in comparison, differs in two key aspects. Firstly, we build a convolutional network amenable for fine-tuning the visual and textual representations, where the entire network only contains four components, i.e., convolution layer, pooling layer, rectified linear unit function (ReLU), and batch normalisation. End-to-end learning allows the system to directly learn from the data and fully utilise the supervisions. Secondly, we propose instance loss according to viewing each multimodal data pair as a class. This works with a large margin objective to learn the inter-modal correspondence between images and their textual descriptions. Experiments on two generic retrieval datasets (Flickr30k and MSCOCO) demonstrate that our method yields competitive accuracy compared to state-of-the-art methods. Moreover, in language person retrieval, we improve the state of the art by a large margin. Code is available at this https URL com/layumi/Image-Text-Embedding

Motivation & Objective

• To address the challenge of discriminative visual-textual embedding in a shared space for image-sentence matching.
• To overcome limitations of RNN-based text encoders and off-the-shelf CNNs in existing methods.
• To enable end-to-end learning of visual and textual representations using a lightweight, fully convolutional architecture.
• To improve inter-modal correspondence learning through a novel instance loss with large margin objective.

Proposed method

• The model uses a dual-path architecture with shared components: convolutional layers, ReLU activation, batch normalization, and pooling layers for both image and text streams.
• The entire network is end-to-end trainable, allowing direct optimization from raw data using full supervisory signals.
• An instance loss is introduced, treating each image-text pair as a unique class to enhance discriminative learning.
• The loss is combined with a large margin objective to strengthen inter-modal alignment and improve generalization.
• The network is trained end-to-end using backpropagation with supervised contrastive learning principles.

Experimental results

Research questions

• RQ1 Can a fully convolutional network effectively learn joint visual-textual representations without RNNs?
• RQ2 How does instance loss with large margin optimization improve cross-modal matching compared to standard contrastive loss?
• RQ3 Does end-to-end training of a lightweight CNN architecture outperform models using pre-trained RNNs and off-the-shelf CNNs?
• RQ4 To what extent does the method generalize across diverse retrieval tasks, including zero-shot and language person retrieval?

Key findings

• The method achieves competitive performance on the Flickr30k and MSCOCO retrieval benchmarks, matching state-of-the-art results.
• It significantly improves the state of the art on language person retrieval, demonstrating strong generalization to zero-shot settings.
• The end-to-end training scheme enables better feature alignment than models using pre-trained components.
• The instance loss with large margin objective enhances discriminative capacity, especially in fine-grained matching tasks.