[Paper Review] Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations

Wolfgang Stammer, Patrick Schramowski+1 more|arXiv (Cornell University)|Nov 25, 2020

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper introduces Neuro-Symbolic Explanatory Interactive Learning (NeSy XIL), a method that revises neuro-symbolic concept learners by interacting with symbolic, compositional explanations at the object-level rather than relying on visual attention maps. It demonstrates on a novel confounded dataset (CLEVR-Hans) that semantic feedback—such as 'never focus on color'—effectively eliminates confounding factors like gray cubes, improving model robustness and accuracy beyond visual explanation-based methods.

ABSTRACT

Most explanation methods in deep learning map importance estimates for a model's prediction back to the original input space. These "visual" explanations are often insufficient, as the model's actual concept remains elusive. Moreover, without insights into the model's semantic concept, it is difficult -- if not impossible -- to intervene on the model's behavior via its explanations, called Explanatory Interactive Learning. Consequently, we propose to intervene on a Neuro-Symbolic scene representation, which allows one to revise the model on the semantic level, e.g. "never focus on the color to make your decision". We compiled a novel confounded visual scene data set, the CLEVR-Hans data set, capturing complex compositions of different objects. The results of our experiments on CLEVR-Hans demonstrate that our semantic explanations, i.e. compositional explanations at a per-object level, can identify confounders that are not identifiable using "visual" explanations only. More importantly, feedback on this semantic level makes it possible to revise the model from focusing on these factors.

Motivation & Objective

• To address the limitation of visual explanations in identifying confounding factors in deep learning models.
• To enable human users to correct models not just at the input-level, but at the semantic concept level.
• To develop a neuro-symbolic framework that supports interactive revision of model behavior using symbolic feedback.
• To create a new benchmark dataset (CLEVR-Hans) with non-separable confounders to test model robustness.
• To demonstrate that global feedback rules at the symbolic level can effectively correct Clever-Hans behavior in neuro-symbolic models.

Proposed method

• Proposes a neuro-symbolic concept learner combining Slot Attention and Set Transformer for end-to-end differentiable object-centric representation learning.
• Introduces a novel loss function to penalize model reliance on incorrect symbolic explanations, particularly confounding attributes like color.
• Employs symbolic annotations of incorrect explanations to guide optimization, enabling revision from 'right for the wrong reason' to 'right for the right reason'.
• Enables Explanatory Interactive Learning (XIL) at both visual and conceptual levels by exposing symbolic, per-object explanations.
• Supports global feedback rules (e.g., 'never focus on gray') that generalize across multiple instances, improving model generalization.
• Uses a symbolic abstraction layer to enable human-machine interaction on the level of concepts, not raw input features.

Figure 1 : Neuro-Symbolic explanations are needed to revise deep learning models from focusing on irrelevant features via global feedback rules.

Experimental results

Research questions

• RQ1 Can symbolic, compositional explanations identify confounders that are invisible to standard visual explanation methods?
• RQ2 Can global feedback rules at the symbolic level effectively correct model behavior without retraining from scratch?
• RQ3 Does revising a neuro-symbolic model using symbolic explanations lead to improved robustness and accuracy on confounded data?
• RQ4 Can a neuro-symbolic model be interactively revised to avoid relying on spurious correlations such as object color?
• RQ5 How does symbolic feedback compare to visual explanation-based feedback in correcting Clever-Hans behavior?

Key findings

• On the CLEVR-Hans3 dataset, NeSy XIL reduced the relative L1 error in symbolic explanations by 73% compared to the default NeSy model.
• For class 1, the test accuracy improved significantly when applying a global feedback rule to ignore the color gray, demonstrating effective correction of confounding behavior.
• When provided with ground-truth symbolic representations, the model achieved near-perfect test accuracy, confirming the effectiveness of the symbolic revision mechanism.
• The use of a symbolic loss and RRR (Reinforcement-based Regularization Rule) loss enabled robust optimization even with negative user feedback.
• The method successfully eliminated reliance on the confounding factor (gray cube) across all classes, proving that symbolic feedback enables generalization of corrections.
• The results show that symbolic explanations are essential for identifying and correcting confounders that visual explanations alone cannot detect.

Figure 2 : Visual Explanations for ColorMNIST. (Left) the general data distribution between train and test split. (Right) a typical visual explanation of a CNN. Notice digit pixels are considered as important for the wrong prediction.