[Paper Review] A Semantic Loss Function for Deep Learning with Symbolic Knowledge

Jingyi Xu, Zilu Zhang+3 more|arXiv (Cornell University)|Nov 29, 2017

Topic Modeling48 references70 citations

TL;DR

The paper derives a semantic loss that ties neural outputs to Boolean constraints, enabling differentiable, sound reasoning with symbolic knowledge and improving semi-supervised and structured-output learning.

ABSTRACT

This paper develops a novel methodology for using symbolic knowledge in deep learning. From first principles, we derive a semantic loss function that bridges between neural output vectors and logical constraints. This loss function captures how close the neural network is to satisfying the constraints on its output. An experimental evaluation shows that it effectively guides the learner to achieve (near-)state-of-the-art results on semi-supervised multi-class classification. Moreover, it significantly increases the ability of the neural network to predict structured objects, such as rankings and paths. These discrete concepts are tremendously difficult to learn, and benefit from a tight integration of deep learning and symbolic reasoning methods.

Motivation & Objective

• Introduce a principled semantic loss that measures how well neural outputs satisfy logical constraints.
• Bridge neural predictions and symbolic knowledge using a differentiable loss derived from first principles.
• Demonstrate improvements in semi-supervised multi-class classification on MNIST, FASHION, and CIFAR-10.
• Show benefits for learning structured objects like rankings and paths.
• Provide tractable computation strategies for semantic loss on complex constraints.

Proposed method

• Define semantic loss Ls(α, p) from propositional logic constraints α over variables X and predicted probabilities p.
• Prove uniqueness of Ls under a set of axioms (monotonicity, identity, satisfaction, label–literal correspondence, differentiability).
• Show that Ls reduces to a closed form for simple constraints (e.g., exactly-one) and use circuit-based weighted model counting for complex constraints.
• Integrate semantic loss into standard loss as existing_loss + w * Ls to enable semi-supervised learning.
• Demonstrate tractable gradient evaluation via Boolean circuit compilation and arithmetic circuits.
• Apply to complex output spaces (paths, rankings) to encode valid structures and improve learning outcomes.

Experimental results

Research questions

• RQ1 How can we define a differentiable loss that preserves the semantic meaning of logical constraints for neural outputs?
• RQ2 Does incorporating semantic loss improve semi-supervised learning on standard benchmarks like MNIST, FASHION, and CIFAR-10?
• RQ3 Can semantic loss help networks learn and predict structured outputs (rankings, paths) more accurately?
• RQ4 How can semantic loss be computed efficiently for complex constraints?

Key findings

• Semantic loss yields near-state-of-the-art semi-supervised performance on MNIST, FASHION, and CIFAR-10.
• With limited labels (e.g., 100), semantic loss provides substantial gains over purely supervised baselines and over entropy-based regularizers.
• On MNIST, semantic-loss-augmented models approach ladder nets, with only a small gap attributed to architecture/hyper-parameter differences.
• On FASHION and CIFAR-10, semantic loss outperforms strong baselines and is competitive with or superior to ladder nets in several settings.
• For complex structured constraints (paths, rankings), semantic loss significantly improves coherent (valid) predictions and the ability to satisfy the constraint, beyond improving per-label accuracy.
• The approach remains computationally efficient for simple constraints and can leverage circuit-based methods for tractable inference and gradient computation on complex constraints.