[Paper Review] CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning

Yuxin Peng, Jinwei Qi+1 more|arXiv (Cornell University)|Oct 14, 2017

Advanced Image and Video Retrieval Techniques41 references65 citations

TL;DR

CM-GANs learns discriminative cross-modal common representations using cross-modal GANs with weight-sharing autoencoders and dual discriminators, achieving state-of-the-art cross-modal retrieval across multiple datasets.

ABSTRACT

It is known that the inconsistent distribution and representation of different modalities, such as image and text, cause the heterogeneity gap that makes it challenging to correlate such heterogeneous data. Generative adversarial networks (GANs) have shown its strong ability of modeling data distribution and learning discriminative representation, existing GANs-based works mainly focus on generative problem to generate new data. We have different goal, aim to correlate heterogeneous data, by utilizing the power of GANs to model cross-modal joint distribution. Thus, we propose Cross-modal GANs to learn discriminative common representation for bridging heterogeneity gap. The main contributions are: (1) Cross-modal GANs architecture is proposed to model joint distribution over data of different modalities. The inter-modality and intra-modality correlation can be explored simultaneously in generative and discriminative models. Both of them beat each other to promote cross-modal correlation learning. (2) Cross-modal convolutional autoencoders with weight-sharing constraint are proposed to form generative model. They can not only exploit cross-modal correlation for learning common representation, but also preserve reconstruction information for capturing semantic consistency within each modality. (3) Cross-modal adversarial mechanism is proposed, which utilizes two kinds of discriminative models to simultaneously conduct intra-modality and inter-modality discrimination. They can mutually boost to make common representation more discriminative by adversarial training process. To the best of our knowledge, our proposed CM-GANs approach is the first to utilize GANs to perform cross-modal common representation learning. Experiments are conducted to verify the performance of our proposed approach on cross-modal retrieval paradigm, compared with 10 methods on 3 cross-modal datasets.

Motivation & Objective

• Motivate bridging the heterogeneity gap between image and text modalities for cross-modal retrieval.
• Propose a cross-modal GAN framework to learn a discriminative common representation by modeling joint distributions.
• Preserve intra-modality semantic reconstruction while enforcing inter-modality correlation through adversarial training.
• Introduce weight-sharing cross-modal autoencoders to learn shared representations and maintain modality-specific information.

Proposed method

• Introduce cross-modal convolutional autoencoders (G_I and G_T) with shared final-layer weights to learn common representations (s_p^i, s_p^t) and reconstruct representations (r_p^i, r_p^t).
• Use two parallel GANs: an intra-modality discriminator (D_I, D_T) for distinguishing originals vs reconstructions, and an inter-modality discriminator (D_Ci, D_Ct) for cross-modal common representations.
• Formulate two adversarial losses: L_GAN1 for intra-modality reconstruction and L_GAN2 for inter-modality correlation, combined in a minimax objective.
• Train with a cross-modal adversarial procedure that alternates updating discriminative models and generative models to maximize mutual boost in learning discriminative common representations.
• Leverage weight-sharing in the encoder’s final layers and a softmax constraint to enforce semantic alignment between modalities.

Experimental results

Research questions

• RQ1 Can a GAN-based architecture learn a discriminative common representation that correlates heterogeneous data from different modalities (image and text)?
• RQ2 Does cross-modal adversarial training with intra- and inter-modality discriminators improve cross-modal retrieval performance?
• RQ3 Do weight-sharing cross-modal autoencoders effectively preserve intra-modality semantics while enabling cross-modal correlation?

Key findings

• CM-GANs achieves best retrieval accuracy compared with 10 state-of-the-art cross-modal retrieval methods across three datasets.
• Demonstrates effectiveness on Wikipedia, Pascal Sentence, and the authors’ XMediaNet datasets for cross-modal retrieval tasks.
• Shows that cross-modal convolutional autoencoders with weight-sharing capture cross-modal correlations while preserving semantic consistency within each modality.
• Validates the proposed cross-modal adversarial mechanism as a means to boost discriminative common representation learning.