[Paper Review] Multimodal Differential Network for Visual Question Generation

Badri N. Patro, Sandeep Kumar+2 more|arXiv (Cornell University)|Aug 12, 2018

Multimodal Machine Learning Applications51 references4 citations

TL;DR

This paper proposes a Multimodal Differential Network (MDN) for Visual Question Generation (VQG) that leverages exemplars—supporting and opposing images, captions, and tags—to learn implicit, differentiable embeddings capturing subtle visual-linguistic context. The method significantly outperforms state-of-the-art models on standard benchmarks (BLEU, METEOR, ROUGE, CIDEr), with gains up to 34.4% on CIDEr, and generates human-like questions validated by user studies.

ABSTRACT

Generating natural questions from an image is a semantic task that requires using visual and language modality to learn multimodal representations. Images can have multiple visual and language contexts that are relevant for generating questions namely places, captions, and tags. In this paper, we propose the use of exemplars for obtaining the relevant context. We obtain this by using a Multimodal Differential Network to produce natural and engaging questions. The generated questions show a remarkable similarity to the natural questions as validated by a human study. Further, we observe that the proposed approach substantially improves over state-of-the-art benchmarks on the quantitative metrics (BLEU, METEOR, ROUGE, and CIDEr).

Motivation & Objective

• To address the challenge of generating natural, human-like questions from images by modeling dynamic, context-sensitive visual-linguistic representations.
• To improve visual question generation by incorporating multimodal exemplars (supporting and opposing images, captions, tags) instead of relying on static or explicit cues.
• To develop a differentiable framework that enables end-to-end training and backpropagation through context modeling for better optimization.
• To validate the superiority of implicit differential context over explicit keyword-based context in generating engaging and contextually relevant questions.

Proposed method

• Uses a Multimodal Differential Network (MDN) that learns embeddings by comparing a target image with supporting (relevant) and opposing (irrelevant) exemplars.
• Employs a contrastive learning mechanism where the network learns to differentiate between relevant and irrelevant visual-linguistic contexts using feature-level differences.
• Integrates multiple context types—image features, captions, and part-of-speech tags (nouns, verbs, WH-words)—into a unified embedding space.
• Applies Hadamard product, element-wise addition, or attention-based fusion to combine image and text features, with attention learned via a soft attention mechanism over convolutional features.
• Uses a 1D convolutional layer or concatenation to fuse multimodal embeddings before feeding them into a question decoder.
• Employs a standard encoder-decoder architecture with an image encoder (VGG-19) and a decoder with LSTM or similar RNN, conditioned on the learned differential context.

Experimental results

Research questions

• RQ1 Can implicit differential embeddings derived from exemplars improve the naturalness and relevance of generated questions compared to explicit keyword-based context?
• RQ2 How does incorporating multiple context types (captions, tags, locations) affect the quality of generated questions in visual question generation?
• RQ3 Does the use of supporting and opposing exemplars lead to better optimization and generalization in end-to-end training of VQG models?
• RQ4 To what extent does the proposed MDN outperform existing state-of-the-art models on standard evaluation metrics (BLEU, METEOR, ROUGE, CIDER) and human evaluation?
• RQ5 Which fusion strategy (Hadamard, addition, concatenation, attention) yields the most effective multimodal representation for question generation?

Key findings

• The proposed Multimodal Differential Network achieves a 34.4% improvement in CIDEr score over the basic image-only model when using three WH-words as context with the concatenation fusion method.
• Using three nouns as context increases BLEU score by 1.6% and METEOR by 2% compared to the basic image model, while three verbs yield a 1.3% BLEU and 2.1% METEOR gain.
• The concatenation-based fusion model outperforms Hadamard and addition methods across all metrics, especially on CIDER, indicating better capture of multimodal interactions.
• The model with K=4 exemplars (four supporting and four opposing images) achieves the highest performance, showing that richer contrastive context improves generalization.
• Human evaluation confirms that generated questions are indistinguishable from human-annotated questions in 80% of cases, demonstrating high naturalness and engagement.
• Statistical significance tests confirm that the improvements over baselines are robust and not due to random variation.