[Paper Review] Exploring Diverse Methods in Visual Question Answering

Panfeng Li, Qikai Yang+4 more|arXiv (Cornell University)|Apr 21, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

This paper investigates three advanced methods—Generative Adversarial Networks (GANs), autoencoders, and attention mechanisms with Multimodal Compact Bilinear pooling (MCB)—for improving Visual Question Answering (VQA) on a balanced VQA dataset. It finds that autoencoder-based models outperform GANs on complex questions, while MCB-enhanced attention mechanisms achieve the best overall performance, especially on reasoning-intensive tasks, despite higher computational cost.

ABSTRACT

This study explores innovative methods for improving Visual Question Answering (VQA) using Generative Adversarial Networks (GANs), autoencoders, and attention mechanisms. Leveraging a balanced VQA dataset, we investigate three distinct strategies. Firstly, GAN-based approaches aim to generate answer embeddings conditioned on image and question inputs, showing potential but struggling with more complex tasks. Secondly, autoencoder-based techniques focus on learning optimal embeddings for questions and images, achieving comparable results with GAN due to better ability on complex questions. Lastly, attention mechanisms, incorporating Multimodal Compact Bilinear pooling (MCB), address language priors and attention modeling, albeit with a complexity-performance trade-off. This study underscores the challenges and opportunities in VQA and suggests avenues for future research, including alternative GAN formulations and attentional mechanisms.

Motivation & Objective

• To evaluate the effectiveness of GANs, autoencoders, and attention mechanisms in improving VQA performance on a balanced dataset.
• To investigate whether GAN-based approaches can effectively learn answer distributions conditioned on image-question pairs.
• To assess whether autoencoder-based embedding learning provides more robust representations for complex VQA tasks.
• To examine the role of MCB-based attention mechanisms in mitigating language priors and improving multimodal reasoning.
• To identify performance trade-offs between model complexity and accuracy across different question types (e.g., yes/no, number, open-ended).

Proposed method

• Utilized a pre-trained ResNet to extract image features and an RNN to encode question embeddings, which were then combined via element-wise multiplication and non-linear transformations.
• Designed a conditional GAN framework where the generator produces answer embeddings conditioned on image-question feature pairs, with random noise injected in three configurations (N0, N1, N2).
• Implemented two generator architectures: a simple single-layer linear head and a deeper multi-layer network with ReLU activations, both outputting 1000-dim vectors representing answer likelihoods.
• Applied pretraining strategies—specifically, pretraining the generator while keeping the discriminator untrained—to stabilize GAN training and improve data distribution learning.
• Employed Multimodal Compact Bilinear (MCB) pooling to model cross-modal interactions between visual and textual features, enhancing attention modeling and reducing language bias.
• Trained models on the VQA 1.9 validation dataset using a balanced subset, comparing performance across question types (Yes/No, Number, Other) and ablation settings.

Figure 1 : High Level Architecture of GAN Based System

Experimental results

Research questions

• RQ1 Can GAN-based models effectively generate accurate answer embeddings conditioned on image and question inputs in VQA?
• RQ2 How does pretraining the generator versus the discriminator affect GAN training stability and VQA performance?
• RQ3 Do autoencoder-based methods for learning joint image-question embeddings outperform GAN-based approaches on complex VQA tasks?
• RQ4 To what extent do MCB-enhanced attention mechanisms reduce language priors and improve reasoning on open-ended and complex questions?
• RQ5 What is the trade-off between model complexity and performance across different question types in VQA systems?

Key findings

• Pretraining the generator while keeping the discriminator untrained yielded the best GAN performance, with a 21.25% accuracy on 'All' questions, significantly outperforming non-pretrained baselines.
• Autoencoder-based methods achieved slightly better overall performance than GANs, particularly excelling on complex questions due to more stable and meaningful embedding learning.
• The MCB-enhanced attention mechanism outperformed both GAN and autoencoder baselines on complex reasoning tasks, correctly answering 'What year is the car?' and 'What color is the fridge?' in qualitative examples.
• On Yes/No questions, the GAN-based model achieved 54.65% accuracy when the generator was pretrained, indicating strong performance on binary classification tasks.
• The GAN-based model struggled with open-ended and complex questions, such as misclassifying 'What color is the fridge?' as 'dinner', highlighting limitations in reasoning generalization.
• The attention mechanism with MCB showed superior qualitative results, correctly identifying the fridge color as 'silver' while the GAN model returned 'gray', a near-correct but incorrect answer.

Figure 2 : High Level Architecture of AutoEncoder Based System