[Paper Review] Guided Policy Search with Delayed Senor Measurements.

Connor Schenck, Dieter Fox|arXiv (Cornell University)|Sep 10, 2016

Reinforcement Learning in Robotics23 references3 citations

TL;DR

This paper extends guided policy search to non-Markovian environments with delayed sensor measurements by incorporating state augmentation, enabling effective learning for tasks like precise liquid pouring. The method maintains sample efficiency and achieves high precision despite sensor delays, demonstrating robust performance in real-world robotic control.

ABSTRACT

Guided policy search is a method for reinforcement learning that trains a general policy for accomplishing a given task by guiding the learning of the policy with multiple guiding distributions. Guided policy search relies on learning an underlying dynamical model of the environment and then, at each iteration of the algorithm, using that model to gradually improve the policy. This model, though, often makes the assumption that the environment dynamics are markovian, e.g., depend only on the current state and control signal. In this paper we apply guided policy search to a problem with non-markovian dynamics. Specifically, we apply it to the problem of pouring a precise amount of liquid from a cup into a bowl, where many of the sensor measurements experience non-trivial amounts of delay. We show that, with relatively simple state augmentation, guided policy search can be extended to non-markovian dynamical systems, where the non-markovianess is caused by delayed sensor readings.

Motivation & Objective

• Address the challenge of applying guided policy search to non-Markovian dynamical systems where sensor measurements are delayed.
• Enable sample-efficient reinforcement learning in robotic control tasks with delayed sensory feedback.
• Maintain policy generalization and performance despite environmental non-Markovianity caused by sensor delays.
• Demonstrate the effectiveness of state augmentation in transforming non-Markovian problems into Markovian ones for policy learning.

Proposed method

• Augment the state space with delayed sensor measurements to restore Markovian properties in the system dynamics.
• Use a learned dynamical model to simulate environment transitions during policy optimization, as in standard guided policy search.
• Apply guided policy search with the augmented state space to iteratively improve the policy using model-based rollouts.
• Integrate delayed sensor readings into the state representation to ensure the policy can make decisions based on complete historical observations.
• Train the policy using demonstrations and model-based rollouts, with the augmented state enabling accurate policy updates despite delayed feedback.
• Leverage the structure of the task (e.g., liquid pouring) to define meaningful state augmentation that captures relevant delayed information.

Experimental results

Research questions

• RQ1 Can guided policy search be effectively applied to non-Markovian systems with significant sensor measurement delays?
• RQ2 How does state augmentation with delayed measurements improve policy learning in such environments?
• RQ3 Does the proposed method maintain sample efficiency and performance in real-world robotic control tasks with delayed feedback?
• RQ4 To what extent does the augmented state space restore the Markov property necessary for guided policy search?
• RQ5 Can the method generalize to complex manipulation tasks like precise liquid pouring under sensor delays?

Key findings

• The proposed method successfully extends guided policy search to non-Markovian systems with delayed sensor measurements through state augmentation.
• State augmentation enables the policy to achieve high precision in liquid-pouring tasks despite non-trivial sensor delays.
• The method maintains sample efficiency by leveraging model-based rollouts and guided learning, even with delayed observations.
• The approach demonstrates robust performance in real-world robotic experiments, showing practical viability for delayed-sensor environments.
• The results indicate that non-Markovian dynamics caused by sensor delays can be effectively mitigated using structured state augmentation.