[Paper Review] PyTorch, Explain! A Python library for Logic Explained Networks

Pietro Barbiero, Gabriele Ciravegna+2 more|arXiv (Cornell University)|May 25, 2021

Explainable Artificial Intelligence (XAI)19 references4 citations

TL;DR

This paper introduces PyTorch, Explain! — a Python library that enables deep neural networks to generate interpretable, first-order logic (FOL) explanations for their predictions. By integrating entropy-based learning objectives and symbolic concept inputs, the framework produces human-readable, logically structured explanations in disjunctive normal form, achieving high fidelity and simplifiability while maintaining state-of-the-art classification accuracy.

ABSTRACT

"PyTorch, Explain!" is a Python module integrating a variety of state-of-the-art approaches to provide logic explanations from neural networks. This package focuses on bringing these methods to non-specialists. It has minimal dependencies and it is distributed under the Apache 2.0 licence allowing both academic and commercial use. Source code and documentation can be downloaded from the github repository: https://github.com/pietrobarbiero/pytorch_explain.

Motivation & Objective

• To bridge the gap between black-box deep learning models and human-understandable explanations in safety-critical domains such as medicine and law.
• To develop a Python library that enables non-specialists to generate, evaluate, and simplify logic-based explanations from neural networks.
• To provide a formal, unambiguous, and quantitatively measurable explanation framework using first-order logic (FOL) for improved trust and interpretability.
• To support the integration of logic explanations into existing deep learning pipelines with minimal dependencies and broad accessibility.
• To enable the extraction of compact, modular, and simplifiable logic formulas from trained models for real-world deployment.

Proposed method

• The framework uses a neural network architecture with a symbolic input layer representing human-understandable concepts (e.g., 'has_hands', 'has_head').
• It applies an entropy-based learning objective, entropy_logic_loss, to encourage the model to produce predictions that align with first-order logic formulas.
• The model is trained using cross-entropy loss combined with a regularization term that promotes logical simplicity and fidelity.
• After training, the explain_class function extracts a logic formula in disjunctive normal form (DNF) that describes the decision boundary for a given class.
• The library supports quantitative evaluation using metrics like accuracy, fidelity, and complexity to validate the quality of the explanation.
• It enables simplification of logic formulas using techniques such as Quine–McCluskey, reducing redundancy and improving interpretability.

Experimental results

Research questions

• RQ1 Can deep neural networks be trained to produce first-order logic explanations that are both accurate and interpretable for non-expert users?
• RQ2 How can logic-based explanations be systematically extracted and evaluated for correctness and completeness in neural network predictions?
• RQ3 To what extent can logic explanations be simplified while preserving their predictive fidelity and semantic clarity?
• RQ4 Can a unified, lightweight Python library integrate state-of-the-art logic explanation techniques for broad adoption by researchers and practitioners?
• RQ5 How do logic-based explanations compare in performance and interpretability to standard black-box explanation methods in real-world classification tasks?

Key findings

• The library successfully extracts first-order logic explanations in disjunctive normal form (DNF) that are both semantically meaningful and quantitatively verifiable.
• The framework achieves high explanation accuracy, with test_explanation metrics confirming strong fidelity between predicted and actual class memberships.
• Logic formulas can be automatically simplified using algorithms like Quine–McCluskey, reducing complexity without sacrificing predictive performance.
• The integration of entropy-based regularization enables the model to learn decision rules that are both accurate and logically coherent.
• The library demonstrates that symbolic concept inputs and logic-based outputs can be seamlessly integrated into PyTorch workflows with minimal dependencies.
• The approach enables the creation of modular explanations that can be combined across different inputs, enhancing generalization and interpretability.