[Paper Review] Inductive Learning of Answer Set Programs from Noisy Examples

Mark Law, Alessandra Russo+1 more|arXiv (Cornell University)|Aug 25, 2018

Logic, Reasoning, and Knowledge22 references19 citations

TL;DR

This paper introduces ILP_LOAS^noise, a noise-tolerant framework for learning Answer Set Programs (ASPs) from noisy examples, extending the ILASP3 system to handle imperfect data. It achieves high accuracy (over 90%) even with 20% noisy examples, outperforming other ILP systems by guaranteeing optimal solutions via a cost function without prior noise level knowledge.

ABSTRACT

In recent years, non-monotonic Inductive Logic Programming has received growing interest. Specifically, several new learning frameworks and algorithms have been introduced for learning under the answer set semantics, allowing the learning of common-sense knowledge involving defaults and exceptions, which are essential aspects of human reasoning. In this paper, we present a noise-tolerant generalisation of the learning from answer sets framework. We evaluate our ILASP3 system, both on synthetic and on real datasets, represented in the new framework. In particular, we show that on many of the datasets ILASP3 achieves a higher accuracy than other ILP systems that have previously been applied to the datasets, including a recently proposed differentiable learning framework.

Motivation & Objective

• To address the challenge of learning interpretable, human-understandable knowledge from noisy real-world data in cognitive systems.
• To extend the ILASP framework to handle noisy examples while preserving the ability to learn non-monotonic knowledge with defaults, exceptions, and preferences.
• To develop a system that guarantees optimal solutions without requiring prior knowledge of noise levels in the data.
• To evaluate the performance of ILASP3 on both synthetic and real-world datasets under varying noise conditions.
• To demonstrate that optimal hypothesis search leads to higher accuracy than non-optimizing ILP systems.

Proposed method

• Proposes ILP_LOAS^noise, a generalization of the Learning from Answer Sets (LAS) framework to handle noisy examples by incorporating a cost function that balances coverage and hypothesis complexity.
• Uses a cost function combining example coverage and hypothesis length to guide the search for optimal hypotheses, minimizing the cost without requiring a predefined noise threshold.
• Employs a hypothesis space generation strategy that computes all possible hypotheses upfront, enabling exhaustive search for optimal solutions.
• Applies the ILASP3 system to learn ASP programs from datasets with controlled noise levels, including synthetic (Hamiltonian, Journey preferences) and real-world (event theories, sentence chunking, preference learning) datasets.
• Introduces a noise-tolerant learning strategy where hypotheses are evaluated based on their ability to cover the majority of examples while minimizing complexity, even when some examples are mislabeled.
• Relies on answer set semantics to represent and learn non-monotonic knowledge, including negation as failure, enabling modeling of defaults and exceptions.

Experimental results

Research questions

• RQ1 Can a noise-tolerant ILP system learn high-accuracy ASP programs from datasets with up to 20% mislabeled examples?
• RQ2 Does guaranteeing optimal solution search via a cost function improve accuracy compared to non-optimizing ILP systems on noisy data?
• RQ3 How does ILASP3 perform on real-world datasets compared to existing ILP systems, including differentiable learning frameworks?
• RQ4 Can the system learn complex knowledge involving defaults, exceptions, and preferences from noisy examples without prior noise level estimation?
• RQ5 What is the impact of hypothesis space size on scalability, and how can future systems improve efficiency?

Key findings

• ILASP3 achieved over 90% accuracy on synthetic Hamiltonian and Journey preferences datasets even with 20% of examples labeled incorrectly.
• On real-world datasets such as event theories, sentence chunking, and preference learning, ILASP3 outperformed other ILP systems, including a recently proposed differentiable learning framework.
• The system consistently achieved higher accuracy than non-optimizing ILP systems because it guarantees finding optimal solutions based on the cost function.
• ILASP3 does not require prior knowledge of noise levels, unlike systems such as HYPER/N, which need the proportion of noise as input.
• Despite improvements, scalability remains limited due to the full hypothesis space computation at the start, especially for large or complex problems.
• The results suggest that cost function-based optimization leads to better generalization and higher accuracy, even when optimal hypotheses are not guaranteed to generalize perfectly to unseen data.