[Paper Review] Pythia v0.1: the Winning Entry to the VQA Challenge 2018

Yu Jiang, Vivek Natarajan+4 more|arXiv (Cornell University)|Jul 26, 2018

Multimodal Machine Learning ApplicationsComputer Science16 references165 citations

TL;DR

Pythia v0.1 is a modular VQA framework that improves the up-down attention model through architecture tweaks, learning schedule, feature fine-tuning, data augmentation, and diverse ensembling, achieving state-of-the-art results on VQA v2.0.

ABSTRACT

This document describes Pythia v0.1, the winning entry from Facebook AI Research (FAIR)'s A-STAR team to the VQA Challenge 2018. Our starting point is a modular re-implementation of the bottom-up top-down (up-down) model. We demonstrate that by making subtle but important changes to the model architecture and the learning rate schedule, fine-tuning image features, and adding data augmentation, we can significantly improve the performance of the up-down model on VQA v2.0 dataset -- from 65.67% to 70.22%. Furthermore, by using a diverse ensemble of models trained with different features and on different datasets, we are able to significantly improve over the 'standard' way of ensembling (i.e. same model with different random seeds) by 1.31%. Overall, we achieve 72.27% on the test-std split of the VQA v2.0 dataset. Our code in its entirety (training, evaluation, data-augmentation, ensembling) and pre-trained models are publicly available at: https://github.com/facebookresearch/pythia

Motivation & Objective

• Motivate the development of a modular VQA research platform named Pythia.
• Show that targeted architectural and training changes improve VQA accuracy.
• Demonstrate that data augmentation and fine-tuned features boost performance.
• Explore the benefits of grid features and diverse ensembling beyond standard seeds.

Proposed method

• Re-implement the bottom-up top-down (up-down) attention model as a modular framework.
• Replace gated tanh with weight normalization and ReLU; use Hadamard product for fusion and a sigmoid classifier.
• Use 300D GloVe embeddings, GRU-based question encoding, and a question attention module.
• Apply Adamax with a warm-up learning schedule and stepwise LR decay to improve training.
• Fine-tune bottom-up features with Detectron FPN-based detectors and 2048D fc6/fc7 features.
• Augment data with Visual Genome and VisDial, mirror images with left-right token swaps; incorporate grid features and 100 bounding-box proposals.
• Construct two ensembles: (i) same-model seeds; (ii) diverse models trained with different features and data sources.

Experimental results

Research questions

• RQ1 Can modularizing VQA research into interchangeable components improve reuse and performance?
• RQ2 What is the impact of architectural tweaks (activation, fusion), learning rate schedules, and feature fine-tuning on VQA accuracy?
• RQ3 Do data augmentation and additional grid-based image features improve performance beyond bottom-up features alone?
• RQ4 Does diverse-model ensembling outperform ensembles built from identical architectures with different seeds?

Key findings

• Baseline up-down achieved 65.32% test-dev and 65.67% test-std.
• Adaptations to the architecture raised test-dev to 66.91% (no test-std reported).
• Learning schedule improvements raised test-dev to 68.05%.
• Fine-tuning bottom-up features raised test-dev to 68.49%.
• Data augmentation raised test-dev to 69.24%.
• Grid features raised test-dev to 69.81%.
• Using 100 object proposals raised test-dev to 70.01% and test-std to 70.24%.
• Ensembling 30 diverse models yielded 72.18% test-dev and 72.27% test-std (state-of-the-art).