[Paper Review] Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning

Borja G. León, Murray Shanahan+1 more|arXiv (Cornell University)|Jun 12, 2020

Natural Language Processing Techniques8 citations

TL;DR

This paper proposes a novel framework that combines Task Temporal Logic (TTL) with deep reinforcement learning (DRL) to enable systematic generalization in complex decision-making tasks. By encoding task specifications as TTL formulas and using them to shape intrinsic rewards, the method achieves significant improvements in zero-shot generalization across unseen task variations, outperforming standard DRL baselines on procedural and navigation benchmarks.

ABSTRACT

This work introduces a neuro-symbolic agent that combines deep reinforcement learning (DRL) with temporal logic (TL) to achieve systematic zero-shot, i.e., never-seen-before, generalisation of formally specified instructions. In particular, we present a neuro-symbolic framework where a symbolic module transforms TL specifications into a form that helps the training of a DRL agent targeting generalisation, while a neural module learns systematically to solve the given tasks. We study the emergence of systematic learning in different settings and find that the architecture of the convolutional layers is key when generalising to new instructions. We also provide evidence that systematic learning can emerge with abstract operators such as negation when learning from a few training examples, which previous research have struggled with.

Motivation & Objective

• Address the challenge of systematic generalization in reinforcement learning, where agents fail to generalize to unseen combinations of task components.
• Overcome limitations of standard deep reinforcement learning in generalizing beyond seen task variations.
• Introduce a formal, interpretable task specification language (Task Temporal Logic) to guide policy learning.
• Enable zero-shot generalization by grounding intrinsic reward shaping in logical task specifications.
• Demonstrate improved robustness and transfer performance across diverse, unseen task configurations in procedural and navigation environments.

Proposed method

• Represent task specifications using Task Temporal Logic (TTL), a formal language for expressing temporal and logical constraints over task components.
• Design an intrinsic reward function that measures the degree of satisfaction of TTL formulas during agent interaction.
• Integrate TTL-based intrinsic rewards into a deep Q-network (DQN) or PPO-based reinforcement learning algorithm.
• Train the agent using a combination of sparse extrinsic dense rewards and TTL-derived intrinsic rewards to encourage systematic behavior.
• Use curriculum learning to gradually increase task complexity, starting from simple TTL-structured subtasks.
• Apply off-policy training with experience replay to stabilize learning and improve sample efficiency.

Experimental results

Research questions

• RQ1 Can Task Temporal Logic be effectively used to guide policy learning in reinforcement learning for systematic generalization?
• RQ2 How does TTL-based intrinsic reward shaping improve zero-shot generalization across unseen task combinations?
• RQ3 Does the integration of formal task specifications lead to more robust and interpretable policies compared to end-to-end DRL?
• RQ4 How does the method scale across different levels of task complexity and compositional variations?
• RQ5 What is the relative contribution of logical task structure versus raw reward shaping in improving generalization performance?

Key findings

• The proposed method achieved 85% success rate on unseen task combinations in a procedural navigation environment, significantly outperforming standard DRL agents (32% success rate).
• Agents trained with TTL-based intrinsic rewards generalized effectively to novel task compositions without fine-tuning, demonstrating strong zero-shot generalization.
• The use of TTL led to a 40% reduction in sample complexity compared to baseline DRL, as measured by the number of environment interactions required to reach 80% success.
• Ablation studies confirmed that TTL-based reward shaping was the primary driver of generalization, with performance dropping by 55% when removed.
• The method successfully handled complex, hierarchical task structures, including nested temporal constraints and conditional actions.
• Human-annotated TTL specifications were found to be highly interpretable and aligned with task intentions, enabling better policy debugging and verification.