[Paper Review] Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey

Chao Yu, Ye, Shicheng+1 more|arXiv (Cornell University)|Apr 24, 2023

Data Stream Mining Techniques6 citations

TL;DR

This survey explores the integration of Knowledge Representation and Reasoning (KRR) into Reinforcement Learning (RL) to enhance sample efficiency, generalization, safety, and interpretability. By leveraging formal logic to represent high-level domain knowledge—such as facts, relations, and rules—it enables more abstract, transparent, and generalizable learning, addressing key limitations of traditional RL in complex, real-world environments.

ABSTRACT

Reinforcement Learning (RL) has achieved tremendous development in recent years, but still faces significant obstacles in addressing complex real-life problems due to the issues of poor system generalization, low sample efficiency as well as safety and interpretability concerns. The core reason underlying such dilemmas can be attributed to the fact that most of the work has focused on the computational aspect of value functions or policies using a representational model to describe atomic components of rewards, states and actions etc, thus neglecting the rich high-level declarative domain knowledge of facts, relations and rules that can be either provided a priori or acquired through reasoning over time. Recently, there has been a rapidly growing interest in the use of Knowledge Representation and Reasoning (KRR) methods, usually using logical languages, to enable more abstract representation and efficient learning in RL. In this survey, we provide a preliminary overview on these endeavors that leverage the strengths of KRR to help solving various problems in RL, and discuss the challenging open problems and possible directions for future work in this area.

Motivation & Objective

• Address the limitations of traditional RL in sample efficiency, generalization, and interpretability when applied to complex real-world problems.
• Investigate how formal knowledge representation (e.g., logical rules, facts, relations) can be integrated into RL to improve learning performance.
• Explore the role of KRR in enabling transfer learning, rule-based reasoning, and explainable policy learning in RL.
• Identify open challenges in theoretical analysis, scalability, and multi-agent applications of KRR-enhanced RL.
• Provide a structured overview of current methods and future research directions in the KRR-RL integration space.

Proposed method

• Use formal logic languages (e.g., first-order logic, answer set programming, temporal logic) to represent high-level domain knowledge such as objects, relations, rules, and constraints.
• Integrate symbolic knowledge into RL by encoding prior knowledge into reward functions, state representations, or policy structures using declarative logic.
• Enable task generalization through modular and compositional reasoning over known primitives (e.g., combining 'deliver coffee' and 'deliver mail' into a new task).
• Support interpretability and safety by replacing black-box policies with symbolic, human-readable rule-based policies.
• Apply KRR techniques such as automated reasoning, belief revision, and model checking to validate and verify agent behaviors.
• Combine function approximation with symbolic abstraction to scale KRR-RL to high-dimensional environments, while managing the trade-off between expressiveness and computational cost.

Experimental results

Research questions

• RQ1 How can KRR methods improve sample efficiency in reinforcement learning by incorporating prior knowledge?
• RQ2 In what ways does symbolic knowledge enhance policy generalization across tasks with different but related structures?
• RQ3 How does the use of formal logic improve the interpretability and safety of RL policies in critical domains?
• RQ4 What are the theoretical challenges in analyzing the convergence and sample complexity of KRR-integrated RL systems?
• RQ5 How can KRR-RL be scaled to complex, multi-agent environments with strategic interactions and partial observability?

Key findings

• Incorporating KRR into RL significantly improves sample efficiency by reducing the number of environment interactions needed for learning, especially in sparse-reward environments.
• KRR enables better generalization across tasks by allowing compositional reasoning over abstract, reusable knowledge components, such as combining existing task modules.
• Symbolic policies derived from logical rules enhance interpretability and trustworthiness, making RL more suitable for safety-critical applications like healthcare and autonomous driving.
• Current KRR-RL methods show promise in domains like OfficeWorld, Minecraft, and robotic control, but scalability to large-scale, real-world problems remains a key challenge.
• Theoretical analysis of KRR-RL systems is still in its infancy, with limited work on convergence, sample complexity, and stability under dynamic abstraction levels.
• A major open challenge lies in efficiently reasoning over strategic agent interactions in multi-agent RL, especially when integrating formalisms like Alternating-time Temporal Logic (ATL) for ability modeling.