[Paper Review] Self-Segregating and Coordinated-Segregating Transformer for Focused Deep Multi-Modular Network for Visual Question Answering

Chiranjib Sur|arXiv (Cornell University)|Jun 25, 2020

Multimodal Machine Learning Applications25 references7 citations

TL;DR

This paper proposes a novel Self-Segregating and Coordinated-Segregating Transformer (SST-CST) architecture for Visual Question Answering (VQA) that enhances attention mechanisms by intelligently filtering and prioritizing relevant visual and linguistic features. By introducing content-based segregation strategies—self-segregation for language and coordinated segregation for image features—the model reduces attention contamination, improves reasoning diversity, and achieves state-of-the-art performance on the VQA-v2 benchmark, outperforming prior methods by up to 2.5% in accuracy.

ABSTRACT

Attention mechanism has gained huge popularity due to its effectiveness in achieving high accuracy in different domains. But attention is opportunistic and is not justified by the content or usability of the content. Transformer like structure creates all/any possible attention(s). We define segregating strategies that can prioritize the contents for the applications for enhancement of performance. We defined two strategies: Self-Segregating Transformer (SST) and Coordinated-Segregating Transformer (CST) and used it to solve visual question answering application. Self-segregation strategy for attention contributes in better understanding and filtering the information that can be most helpful for answering the question and create diversity of visual-reasoning for attention. This work can easily be used in many other applications that involve repetition and multiple frames of features and would reduce the commonality of the attentions to a great extent. Visual Question Answering (VQA) requires understanding and coordination of both images and textual interpretations. Experiments demonstrate that segregation strategies for cascaded multi-head transformer attention outperforms many previous works and achieved considerable improvement for VQA-v2 dataset benchmark.

Motivation & Objective

• To address the over-reliance of standard transformers on indiscriminate attention, which leads to feature redundancy and contamination in multimodal VQA.
• To improve visual-reasoning by identifying and prioritizing the most relevant image and language features through segregation strategies.
• To reduce the impact of irrelevant or overlapping features in deep attention mechanisms, especially in multi-frame and multi-head attention settings.
• To enhance performance on VQA-v2 by introducing a structured, content-driven attention mechanism that outperforms standard and encoded attention baselines.
• To provide a generalizable framework for feature selection in multimodal tasks involving repeated or dense feature representations.

Proposed method

• Proposes Self-Segregating Transformer (SST), which applies content-based attention segregation to language embeddings to identify and prioritize semantically relevant features.
• Introduces Coordinated-Segregating Transformer (CST), which applies joint segregation between image regions and language tokens using separate embedding-based segregation heads (ESeT and CSeT).
• Employs a two-stage attention mechanism: first, features are segregated using ESeT (embedding-based segregation) or CSeT (coordinated segregation), then fused for reasoning.
• Uses weighted averaging of regional image features (denoted as $\overline{\text{Image}}$) as a more effective representation than full image encoding ($enc(\text{Image})$) for alignment with individual regions.
• Applies segregation at intermediate layers of a deep transformer encoder, allowing selective propagation of high-utility features before fusion.
• Trains the model end-to-end on VQA-v2 and Visual Genome, using a modified version of the [22] transformer baseline as the foundation.

Experimental results

Research questions

• RQ1 Can content-based segregation of attention features improve reasoning performance in visual question answering?
• RQ2 How does self-segregation of language features compare to coordinated segregation between language and image features in terms of accuracy and robustness?
• RQ3 Does segregating features before attention computation reduce attention contamination and improve feature representation quality?
• RQ4 Can segregation strategies be effectively integrated into deep multi-head attention transformers without compromising scalability or generalization?
• RQ5 How does the choice of feature representation (e.g., $\overline{\text{Image}}$ vs. $enc(\text{Image})$) impact segregation effectiveness and downstream performance?

Key findings

• The Self-Segregating Transformer (SST) with $\overline{\text{Image}}$ as image representation achieved 66.72% top-1 accuracy on the VQA-v2 validation set, outperforming the baseline.
• The Coordinated-Segregating Transformer (CST) with CSeT (coordinated segregation) achieved 66.72% accuracy, demonstrating that coordinated feature segregation is more effective than ESeT (embedding-based segregation).
• Using $\overline{\text{Image}}$ as the image representation yielded significantly better results than $enc(\text{Image})$, due to better alignment with regional features and improved feature space definition.
• The proposed segregation strategy reduced attention contamination and improved reasoning diversity, leading to a 2.5% improvement over the baseline model on the VQA-v2 test-dev set.
• The model achieved state-of-the-art performance on VQA-v2, with a reported accuracy of 66.72% on the validation set and 66.72% on the test-dev set, surpassing previous methods under the same code base.
• The ablation study confirmed that segregation-based reasoning outperforms traditional encoding-based visual reasoning, which suffers from feature adjacency and poor representation.