[Paper Review] Deconfounded Image Captioning: A Causal Retrospect

Xu Yang, Hanwang Zhang+1 more|arXiv (Cornell University)|Mar 9, 2020

Multimodal Machine Learning ApplicationsComputer Science104 references38 citations

TL;DR

The paper analyzes dataset bias in image captioning through causal inference and proposes DICv1.0, a deconfounded captioning framework using backdoor and front-door adjustments, to improve CIDEr-D scores.

ABSTRACT

Dataset bias in vision-language tasks is becoming one of the main problems which hinders the progress of our community. Existing solutions lack a principled analysis about why modern image captioners easily collapse into dataset bias. In this paper, we present a novel perspective: Deconfounded Image Captioning (DIC), to find out the answer of this question, then retrospect modern neural image captioners, and finally propose a DIC framework: DICv1.0 to alleviate the negative effects brought by dataset bias. DIC is based on causal inference, whose two principles: the backdoor and front-door adjustments, help us review previous studies and design new effective models. In particular, we showcase that DICv1.0 can strengthen two prevailing captioning models and can achieve a single-model 131.1 CIDEr-D and 128.4 c40 CIDEr-D on Karpathy split and online split of the challenging MS COCO dataset, respectively. Interestingly, DICv1.0 is a natural derivation from our causal retrospect, which opens promising directions for image captioning.

Motivation & Objective

• Identify how dataset bias distorts image captioning learning via confounders in visual-language data.
• Develop principled deconfounding methods using causal inference (backdoor and front-door) to learn true image-to-caption causal effects.
• Propose DICv1.0 framework that strengthens existing captioners by mitigating bias.
• Retrospect major image captioning models from a causal viewpoint to inform model design.

Proposed method

• Model the bias as confounders D (and S) affecting image features X and captions L.
• Use backdoor adjustment to compute P(L|do(X)) by averaging over confounders: P(L|do(X)) = sum_d P(L|X,d) P(d).
• Use front-door adjustment to handle unobserved confounders via a mediator Z: P(L|do(X)) = sum_z P(z|X) sum_x P(L|z,x) P(x).
• Instantiate DICv1.0 by selecting a mediator Z as a commonsense structure from ConceptNet and a backdoor-deconfounding vocabulary S, enabling both adjustments.
• Apply DICv1.0 to Up-Down and AoANet captioners to boost CIDEr-D: Up-Down from 126.4 to 129.5; AoANet from 128.7 to 131.1 (MS COCO test, c40: 128.4).

Experimental results

Research questions

• RQ1 How does dataset bias distort learning of the true image-to-caption causal effect in image captioning?
• RQ2 Can causal adjustments (backdoor and front-door) be practically applied to deconfound modern caption models?
• RQ3 Does a DICv1.0 framework improve standard captioners on benchmark datasets?
• RQ4 What role do mediators like structure vocabularies or commonsense triplets play in deconfounded captioning?

Key findings

• DICv1.0 can deconfound image captioning by applying backdoor and front-door adjustments to compute the interventional distribution P(L|do(X)).
• Using DICv1.0 with Up-Down and AoANet improves CIDEr-D scores on MS COCO: 126.4 → 129.5 and 128.7 → 131.1 respectively; the AoANet result includes 128.4 CIDEr-c40 on the test server.
• The framework employs a mediator Z (commonsense structure) and a backdoor-deconfounded vocabulary S to mitigate bias from confounders D and S.
• The front-door approach allows deconfounding when backdoor adjustment is impractical due to complex, unobserved confounders.
• The work provides a causal retrospective of major captioning models, informing the design of causal-deconfounded captioners.