[Paper Review] NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Tasks

Fawaz Sammani, Tanmoy Mukherjee+1 more|arXiv (Cornell University)|Mar 9, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

NLX-GPT proposes a unified, compact language model that jointly predicts answers and generates natural language explanations for vision and vision-language tasks by formulating answer prediction as a text generation task. It eliminates the need for a separate task model, achieving state-of-the-art performance on VQA-X, ACT-X, and e-SNLI-VE with 15× faster inference and reduced memory use, while introducing two new self-evaluation frameworks for explanation quality.

ABSTRACT

Natural language explanation (NLE) models aim at explaining the decision-making process of a black box system via generating natural language sentences which are human-friendly, high-level and fine-grained. Current NLE models explain the decision-making process of a vision or vision-language model (a.k.a., task model), e.g., a VQA model, via a language model (a.k.a., explanation model), e.g., GPT. Other than the additional memory resources and inference time required by the task model, the task and explanation models are completely independent, which disassociates the explanation from the reasoning process made to predict the answer. We introduce NLX-GPT, a general, compact and faithful language model that can simultaneously predict an answer and explain it. We first conduct pre-training on large scale data of image-caption pairs for general understanding of images, and then formulate the answer as a text prediction task along with the explanation. Without region proposals nor a task model, our resulting overall framework attains better evaluation scores, contains much less parameters and is 15$ imes$ faster than the current SoA model. We then address the problem of evaluating the explanations which can be in many times generic, data-biased and can come in several forms. We therefore design 2 new evaluation measures: (1) explain-predict and (2) retrieval-based attack, a self-evaluation framework that requires no labels. Code is at: https://github.com/fawazsammani/nlxgpt.

Motivation & Objective

• To address the disconnection between reasoning and explanation in existing NLE models, which rely on separate task and explanation models.
• To reduce memory and inference overhead by eliminating the need for a dedicated vision-language task model.
• To improve faithfulness and coherence of explanations by jointly training answer and explanation generation.
• To develop label-free evaluation frameworks that assess explanation correctness, reasoning, and bias without human-annotated explanations.
• To demonstrate that a single, compact model can outperform multi-stage SoA approaches in both speed and performance.

Proposed method

• Formulate answer prediction and explanation generation as a single text generation task using a language model.
• Pre-train the model on large-scale image-caption pairs to learn general visual and linguistic understanding.
• Use a visual encoder (e.g., CLIP ViT) to encode images, and condition the language model on the encoded visual features.
• Train end-to-end with a cross-attention mechanism to align visual features with answer and explanation tokens.
• Introduce two self-evaluation frameworks: (1) explain-predict, which measures explanation quality via answer prediction performance; and (2) retrieval-based attack, which evaluates explanation faithfulness via similarity to ground-truth explanations.
• Use a unified loss combining cross-entropy for answer and explanation generation, enabling joint optimization.

Figure 1 : A comparison between previous models (left) and ours (right). Our model solely requires a visual encoder and a language model. We model the answer as a text prediction task along with the explanation. Best viewed in color.

Experimental results

Research questions

• RQ1 Can a single, compact language model jointly generate accurate answers and faithful natural language explanations without relying on a separate task model?
• RQ2 How does joint training of answer and explanation generation improve faithfulness and reasoning coherence compared to decoupled approaches?
• RQ3 Can self-evaluation frameworks like explain-predict and retrieval-based attack effectively measure explanation quality without human annotations?
• RQ4 To what extent does pre-training on image-caption data improve performance on downstream NLE tasks?
• RQ5 How do different visual encoders (e.g., ViT, ResNet, DeiT) affect the performance of the unified NLX-GPT framework?

Key findings

• NLX-GPT achieves state-of-the-art performance on VQA-X, ACT-X, and e-SNLI-VE benchmarks, outperforming prior SoA models on most metrics.
• The model is 15× faster and uses significantly less memory than existing multi-stage NLE models, which rely on separate task and explanation models.
• Pre-training on image-caption pairs substantially improves performance, with the CLIP vision transformer achieving the best results across all tasks.
• The explain-predict evaluation framework shows that high-quality explanations lead to better answer prediction, validating the model's internal consistency.
• The retrieval-based attack framework demonstrates that NLX-GPT generates explanations with lower intra-distance scores, indicating higher faithfulness and reduced bias.
• Ablation studies confirm that both pre-training and the choice of visual encoder significantly impact final performance, with CLIP ViT outperforming ResNet-101 and bottom-up features.

Figure 2 : A schema of the proposed NLX-GPT model. At test time, we supply the question and ¡bos¿ token, and start generating the answer and explanation