[Paper Review] Relational Abstractions for Generalized Reinforcement Learning on Symbolic Problems

Rushang Karia, Siddharth Srivastava|arXiv (Cornell University)|Apr 27, 2022

Neural Networks and Applications4 citations

TL;DR

This paper proposes a relational deep reinforcement learning method that learns a generalizable, relational Q-function using automatically generated features from state trajectories. By leveraging deep learning and relational abstractions, the approach enables zero-shot transfer to larger problem instances with different object counts and names, reducing sample complexity by orders of magnitude compared to standard RL.

ABSTRACT

Reinforcement learning in problems with symbolic state spaces is challenging due to the need for reasoning over long horizons. This paper presents a new approach that utilizes relational abstractions in conjunction with deep learning to learn a generalizable Q-function for such problems. The learned Q-function can be efficiently transferred to related problems that have different object names and object quantities, and thus, entirely different state spaces. We show that the learned generalized Q-function can be utilized for zero-shot transfer to related problems without an explicit, hand-coded curriculum. Empirical evaluations on a range of problems show that our method facilitates efficient zero-shot transfer of learned knowledge to much larger problem instances containing many objects.

Motivation & Objective

• To address the challenge of scaling reinforcement learning to symbolic, factored state spaces that grow exponentially with object count.
• To enable efficient transfer of policies across problem instances with different object names and quantities without hand-coded curricula.
• To learn a generalizable Q-function that captures relational structure and enables zero-shot transfer to larger instances.
• To reduce sample complexity in stochastic, symbolic environments by leveraging automatically derived features from state trajectories.

Proposed method

• The method uses deep learning to approximate a relational Q-function that generalizes across problem instances with varying object counts and names.
• It automatically generates a feature list from state trajectory sequences, avoiding the need for hand-coded features or analytical models.
• The Q-function is trained using a generalized reinforcement learning (GRL) framework that supports inter-instance transfer via leapfrogging over object counts.
• The approach employs multilayer perceptrons (MLPs) to process relational features, enabling efficient inference and policy generalization.
• A leapfrogging strategy is used to train on smaller problem instances and transfer knowledge to larger ones, accelerating learning without explicit curriculum design.
• The method relies on Description Logic (DL)-based features derived from state transitions to capture relational structure and dynamics.

Experimental results

Research questions

• RQ1 Can a deep Q-function be learned in a relational, symbolic MDP setting without access to analytical action models?
• RQ2 Can automatically generated relational features from state trajectories support effective zero-shot transfer to larger problem instances?
• RQ3 Does the proposed GRL framework reduce sample complexity in stochastic, symbolic environments compared to standard RL?
• RQ4 To what extent can the learned Q-function generalize across changes in object names and quantities?
• RQ5 How effective is the leapfrogging strategy in enabling transfer without hand-coded curricula?

Key findings

• The proposed method reduces sampling requirements on larger problem instances by several orders of magnitude compared to standard Q-learning.
• The generalized Q-function enables zero-shot transfer to problem instances with different object counts and names, even when state spaces are entirely different.
• The approach outperforms existing transfer methods in symbolic, stochastic domains, particularly on harder and more complex tasks.
• Automatically generated features from state trajectories are sufficient to achieve strong performance, reducing or eliminating the need for analytical models or hand-coded features.
• The leapfrogging strategy enables effective curriculum learning without requiring domain-specific knowledge or hand-coded difficulty assessments.
• The method shows robust performance across multiple symbolic domains, including the Sysadmin(n) problem, where it generalizes a simple policy to large-scale instances.