[Paper Review] When a Relation Tells More Than a Concept: Exploring and Evaluating Classifier Decisions with CoReX

Bettina Finzel, Patrick Hilme+2 more|arXiv (Cornell University)|May 2, 2024

Statistical and Computational Modeling4 citations

TL;DR

This paper proposes CoReX, a novel explainable AI method for convolutional neural networks (CNNs) that explains model decisions using concept and relation-based reasoning, improving interpretability beyond pixel-level relevance. By combining concept relevance with logic rules derived from inductive logic programming, CoReX produces faithful, contrastive explanations that reveal how spatial relations between concepts (e.g., 'spout right of handle') drive classification, enabling detection of misclassifications and model evaluation in complex domains like biomedicine.

ABSTRACT

Explanations for Convolutional Neural Networks (CNNs) based on relevance of input pixels might be too unspecific to evaluate which and how input features impact model decisions. Especially in complex real-world domains like biology, the presence of specific concepts and of relations between concepts might be discriminating between classes. Pixel relevance is not expressive enough to convey this type of information. In consequence, model evaluation is limited and relevant aspects present in the data and influencing the model decisions might be overlooked. This work presents a novel method to explain and evaluate CNN models, which uses a concept- and relation-based explainer (CoReX). It explains the predictive behavior of a model on a set of images by masking (ir-)relevant concepts from the decision-making process and by constraining relations in a learned interpretable surrogate model. We test our approach with several image data sets and CNN architectures. Results show that CoReX explanations are faithful to the CNN model in terms of predictive outcomes. We further demonstrate through a human evaluation that CoReX is a suitable tool for generating combined explanations that help assessing the classification quality of CNNs. We further show that CoReX supports the identification and re-classification of incorrect or ambiguous classifications.

Motivation & Objective

• To address the limitation of pixel-level explanations in capturing high-level, relational features critical in complex image classification tasks.
• To improve model evaluation by identifying which concepts and relations influence predictions, especially in ambiguous or borderline cases.
• To support interactive model correction by enabling users to specify which concepts or relations should be suppressed or emphasized.
• To develop a method that combines concept activation relevance with logic-based reasoning for more interpretable and faithful explanations.
• To demonstrate the utility of contrastive explanations—comparing similar but differently classified images—using relational knowledge.

Proposed method

• CoReX identifies and localizes concepts in intermediate CNN features using concept relevance propagation, assigning positive, negative, or neutral scores based on contribution to classification.
• It extracts meaningful concepts by clustering high-relevance pixel groups and labeling them with natural language terms (e.g., 'handle', 'spout').
• Concepts are linked via spatial and structural relations (e.g., 'spout right of handle') using inductive logic programming (ILP) to learn interpretable logic rules.
• The method constructs a surrogate model based on learned ILP theories, which constrains predictions by masking irrelevant concepts and enforcing relational constraints.
• Explanations are generated by contrasting a sample with a counterfactual example from the contrastive class, highlighting distinguishing relations.
• The approach supports interactive feedback: users can modify rules to correct misclassifications, enabling model regularization through human-in-the-loop refinement.

Figure 1 : Explaining why a sample image belongs to the target class “teapot” by contrasting it with a sample from the contrastive class “vase” based on identified concepts (handle, spout) and their relations (“spout right of handle”)

Experimental results

Research questions

• RQ1 Can concept- and relation-based explanations reveal discriminative features that pixel-level explanations miss, especially in complex image classification tasks?
• RQ2 How faithfully do CoReX explanations reflect the actual decision-making process of a trained CNN?
• RQ3 Can CoReX identify and explain misclassifications by highlighting incorrect or ambiguous concept relations?
• RQ4 To what extent can users leverage CoReX to correct model behavior through interactive rule modification?
• RQ5 Does the inclusion of relational knowledge improve the interpretability and evaluability of CNN decisions in real-world domains like biomedicine or fine-grained visual categorization?

Key findings

• CoReX explanations are highly faithful to the underlying CNN model, as validated through predictive consistency across multiple datasets and architectures.
• The method successfully identifies discriminative relations—such as 'spout right of handle'—that distinguish classes like 'teapot' and 'vase', which pixel-level methods fail to express.
• In the Adience dataset, CoReX uncovered a representative cluster defined by the relation 'eyes above smiling mouth', demonstrating its ability to capture subtle facial structure patterns.
• For the Picasso dataset, CoReX detected that misclassified male faces were excluded from rule coverage due to background artifacts, revealing model sensitivity to spurious features.
• The approach enabled interactive correction of misclassifications by modifying rules, showing potential for human-in-the-loop model refinement.
• An empty cluster in the Adience evaluation revealed a misclassified male sample not covered by any rule, highlighting the method’s ability to flag problematic cases for human review.

Figure 2 : Overview of our CoReX approach for explaining and evaluating CNN image classifications with concept- and relation-based explanations and constraints (concept masking and relational constraints)