[Paper Review] TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines

Jingxiang Lin, Unnat Jain+1 more|arXiv (Cornell University)|Oct 31, 2019

Multimodal Machine Learning Applications109 references4 citations

TL;DR

This paper proposes TAB-VCR, a simple yet effective visual commonsense reasoning model that improves performance by leveraging object attributes and enhanced text-to-image grounding. By replacing complex attention mechanisms with a lightweight architecture and integrating attribute-aware visual features and additional object detections, TAB-VCR achieves state-of-the-art results with 50% fewer parameters, outperforming prior work by 5.3%, 4.4%, and 6.5% on VCR question answering, justification, and holistic tasks respectively.

ABSTRACT

Reasoning is an important ability that we learn from a very early age. Yet, reasoning is extremely hard for algorithms. Despite impressive recent progress that has been reported on tasks that necessitate reasoning, such as visual question answering and visual dialog, models often exploit biases in datasets. To develop models with better reasoning abilities, recently, the new visual commonsense reasoning (VCR) task has been introduced. Not only do models have to answer questions, but also do they have to provide a reason for the given answer. The proposed baseline achieved compelling results, leveraging a meticulously designed model composed of LSTM modules and attention nets. Here we show that a much simpler model obtained by ablating and pruning the existing intricate baseline can perform better with half the number of trainable parameters. By associating visual features with attribute information and better text to image grounding, we obtain further improvements for our simpler & effective baseline, TAB-VCR. We show that this approach results in a 5.3%, 4.4% and 6.5% absolute improvement over the previous state-of-the-art on question answering, answer justification and holistic VCR.

Motivation & Objective

• To address the limitations of complex, bias-prone models in visual commonsense reasoning by developing a simpler, more robust baseline.
• To improve model performance by incorporating visual attributes and extending object grounding beyond the original VCR annotations.
• To demonstrate that a minimal architecture with enhanced visual-textual grounding can outperform intricate models with significantly fewer trainable parameters.
• To identify and correct missing object detections in the VCR dataset that hinder accurate reasoning.

Proposed method

• The model uses a pre-trained image CNN fine-tuned for attribute prediction to extract visual features, enhancing object representation beyond standard image classification.
• It introduces a novel grounding mechanism using part-of-speech tagging and Wu-Palmer similarity to link noun phrases in text to visual objects, including those not present in the original VCR annotations.
• The method extends object detection to include previously untagged nouns (e.g., 'cart', 'coats') by reapplying the CNN to detect and label these objects in the image.
• A simplified feed-forward network replaces the LSTM and attention modules of the prior SOTA, reducing model complexity while maintaining performance.
• The model performs joint reasoning over image features, predicted attributes, and extended object tags to generate answers and rationales.
• Ablation studies confirm the effectiveness of attribute learning and extended grounding, with performance gains even when reducing model size.

Experimental results

Research questions

• RQ1 Can a simpler model architecture outperform complex, attention-based models in visual commonsense reasoning when enhanced with attribute and grounding improvements?
• RQ2 How does incorporating visual attributes and detecting previously untagged objects affect reasoning performance on the VCR benchmark?
• RQ3 To what extent do data biases and incomplete object annotations in the VCR dataset limit model performance, and how can they be mitigated?
• RQ4 Does reducing model complexity through pruning and ablation improve generalization and robustness in visual reasoning tasks?

Key findings

• TAB-VCR achieves a 5.3% absolute improvement on the VCR question answering subtask compared to the prior state-of-the-art.
• The model improves by 4.4% on the answer justification subtask, demonstrating better reasoning capability.
• It records a 6.5% absolute gain on the holistic VCR task, which evaluates overall reasoning performance.
• Despite having less than half the number of trainable parameters (15M vs. 30M), TAB-VCR outperforms the previous SOTA model.
• The ablation study confirms that attribute learning and extended object grounding are key contributors to performance gains.
• The model performs better on examples with richer textual grounding, indicating that improved alignment between image and language boosts reasoning accuracy.