[Paper Review] Reinforcement Learning With Temporal Logic Rewards

Xiao Li, Cristian-Ioan Vasile+1 more|arXiv (Cornell University)|Dec 11, 2016

Formal Methods in Verification15 references4 citations

TL;DR

This paper proposes using Truncated Linear Temporal Logic (TLTL) with robustness degree as a reward function in reinforcement learning to enable faster and more reliable learning of complex, temporally structured robotic tasks. By encoding domain knowledge and constraints formally, the method outperforms heuristic rewards in both convergence speed and policy quality, demonstrated in simulation and on a Baxter robot for a toast-placing task with 100% success rate.

ABSTRACT

Reinforcement learning (RL) depends critically on the choice of reward functions used to capture the de- sired behavior and constraints of a robot. Usually, these are handcrafted by a expert designer and represent heuristics for relatively simple tasks. Real world applications typically involve more complex tasks with rich temporal and logical structure. In this paper we take advantage of the expressive power of temporal logic (TL) to specify complex rules the robot should follow, and incorporate domain knowledge into learning. We propose Truncated Linear Temporal Logic (TLTL) as specifications language, that is arguably well suited for the robotics applications, together with quantitative semantics, i.e., robustness degree. We propose a RL approach to learn tasks expressed as TLTL formulae that uses their associated robustness degree as reward functions, instead of the manually crafted heuristics trying to capture the same specifications. We show in simulated trials that learning is faster and policies obtained using the proposed approach outperform the ones learned using heuristic rewards in terms of the robustness degree, i.e., how well the tasks are satisfied. Furthermore, we demonstrate the proposed RL approach in a toast-placing task learned by a Baxter robot.

Motivation & Objective

• To address the challenge of designing effective reward functions for complex robotic tasks with rich temporal and logical structure.
• To incorporate domain knowledge and formal constraints into reinforcement learning via a specification language that captures task intentions and safety requirements.
• To improve learning efficiency and policy quality by replacing heuristic rewards with robustness degrees derived from temporal logic specifications.
• To demonstrate the feasibility and superiority of TLTL-based rewards in both simulated environments and real-world robotic manipulation tasks.

Proposed method

• Proposes Truncated Linear Temporal Logic (TLTL) as a specification language tailored for robotic tasks with finite-time trajectories and temporal dependencies.
• Defines a quantitative semantics called robustness degree to map TLTL formulae into real-valued reward functions.
• Uses the robustness degree as the primary reward signal in policy search reinforcement learning, replacing handcrafted heuristic rewards.
• Applies episode-based REPS and step-based policy search algorithms to train policies using TLTL rewards in both simulation and real-robot experiments.
• Employs a hierarchical decomposition of task specifications into sub-formulae, with robustness computed via nested min/max operations to identify active constraints at each time step.
• Normalizes robustness values manually to prevent imbalance in learning focus across different subtasks.

Experimental results

Research questions

• RQ1 Can temporal logic specifications with quantitative semantics improve the sample efficiency of reinforcement learning in complex robotic tasks?
• RQ2 How does learning with TLTL-based robustness rewards compare to learning with heuristic reward functions in terms of convergence speed and final policy quality?
• RQ3 Can a non-hierarchical reinforcement learning algorithm successfully learn a temporally structured task when guided by TLTL rewards?
• RQ4 To what extent does the robustness degree serve as a meaningful proxy for task satisfaction in real-world robotic manipulation?

Key findings

• Policies trained with TLTL-based rewards achieved 100% success rate in the Baxter robot’s toast-placing task, while policies trained with heuristic rewards failed to learn the correct gripper timing policy.
• Learning with TLTL rewards converged significantly faster and produced higher-quality policies than with heuristic rewards in both simulated and real-world environments.
• The robustness degree reward function effectively focused learning on active constraints at each time step, improving overall learning progress by prioritizing bottleneck subtasks.
• The TLTL-based approach outperformed both discrete and continuous heuristic reward functions even when using a less efficient episode-based RL algorithm.
• Manual normalization of robustness values was necessary to prevent imbalance in learning across different subtasks, indicating a need for adaptive normalization in future work.