[Paper Review] Causal Intervention for Weakly-Supervised Semantic Segmentation

Dong Zhang, Hanwang Zhang+3 more|arXiv (Cornell University)|Sep 26, 2020

Domain Adaptation and Few-Shot Learning76 references70 citations

TL;DR

This paper introduces CONTA, a causal-intervention framework using backdoor adjustment to remove context confounding in weakly-supervised semantic segmentation, improving image-level classifiers and pseudo-masks to boost segmentation performance.

ABSTRACT

We present a causal inference framework to improve Weakly-Supervised Semantic Segmentation (WSSS). Specifically, we aim to generate better pixel-level pseudo-masks by using only image-level labels -- the most crucial step in WSSS. We attribute the cause of the ambiguous boundaries of pseudo-masks to the confounding context, e.g., the correct image-level classification of "horse" and "person" may be not only due to the recognition of each instance, but also their co-occurrence context, making the model inspection (e.g., CAM) hard to distinguish between the boundaries. Inspired by this, we propose a structural causal model to analyze the causalities among images, contexts, and class labels. Based on it, we develop a new method: Context Adjustment (CONTA), to remove the confounding bias in image-level classification and thus provide better pseudo-masks as ground-truth for the subsequent segmentation model. On PASCAL VOC 2012 and MS-COCO, we show that CONTA boosts various popular WSSS methods to new state-of-the-arts.

Motivation & Objective

• Motivate and formalize the misled learning caused by context confounding in WSSS under image-level supervision.
• Propose a causal framework (SCM) to model pixels, context, and labels and derive a method to obtain P(Y|do(X)).
• Develop CONTA, a practical backdoor-adjustment-based algorithm to generate better pseudo-masks from observational data.
• Demonstrate that CONTA improves seed areas, pseudo-masks, and segmentation masks across multiple baselines and datasets.

Proposed method

• Formulate a structural causal model linking pixel-level image X, context C, and image-level labels Y.
• Use backdoor adjustment P(Y|do(X)) = sum_c P(Y|X, M=f(X,c)) P(c) to remove confounding effects.
• Approximate the unobserved confounder set C with class-specific average masks, creating M_t as an iterative EM-like update.
• Iteratively perform: (i) train classifier with P(Y|do(X)); (ii) generate CAM seed areas and pseudo-masks; (iii) train segmentation model; (iv) update confounder masks M_t via equation M_{t+1} = sum_i alpha_i c_i P(c_i).
• research_questions_narrowed_to_2-5_queries_N/A

Experimental results

Research questions

• RQ1 Can context confounding in image-level classification be mitigated to produce more accurate pseudo-masks for WSSS?
• RQ2 Does applying backdoor-adjusted causality via CONTA improve seed areas, pseudo-masks, and final segmentation across baselines?
• RQ3 How many iterative rounds of CONTA yield optimal improvements without overfitting?
• RQ4 Which backbone blocks and confounder representations maximize gains from CONTA?
• RQ5 Is CONTA effective across datasets (PASCAL VOC 2012 and MS-COCO) and different WSSS models?

Key findings

• CONTA improves CAM seed areas, pseudo-masks, and segmentation masks over baselines on PASCAL VOC 2012.
• On VOC 2012, CONTA achieves state-of-the-art 66.1% mIoU on val and 66.7% on test for SEAM+CONTA (ResNet-38).
• CONTA provides average gains of 0.9% (CAM), 2.0% (pseudo-mask), and 2.0% (segmentation mask) when applied to multiple baselines.
• On VOC 2012, IRNet+CONTA reaches 65.3% val and 66.1% test mIoU; SEAM+CONTA reaches 66.1% val and 66.7% test.
• On MS-COCO, SEC+CONTA achieves 23.7% mIoU on the val set, surpassing prior best by 1.3%.