[Paper Review] Metric State Space Reinforcement Learning for a Vision-Capable Mobile Robot

Viktor Zhumatiy, Faustino Gomez+2 more|ArXiv.org|Mar 7, 2006

Reinforcement Learning in Robotics21 references3 citations

TL;DR

This paper proposes Piecewise Continuous Nearest Sequence Memory (PC-NSM), a metric-based instance learning algorithm that enables vision-capable mobile robots to learn control policies directly from real-world sensory data without prior environment modeling or state space discretization. By using a continuous metric over state-action trajectories and k-nearest neighbors with variable radius, the method achieves data-efficient learning in piecewise-continuous perceptual spaces while handling partial observability, successfully demonstrating autonomous navigation and target acquisition on a real robot.

ABSTRACT

We address the problem of autonomously learning controllers for vision-capable mobile robots. We extend McCallum's (1995) Nearest-Sequence Memory algorithm to allow for general metrics over state-action trajectories. We demonstrate the feasibility of our approach by successfully running our algorithm on a real mobile robot. The algorithm is novel and unique in that it (a) explores the environment and learns directly on a mobile robot without using a hand-made computer model as an intermediate step, (b) does not require manual discretization of the sensor input space, (c) works in piecewise continuous perceptual spaces, and (d) copes with partial observability. Together this allows learning from much less experience compared to previous methods.

Motivation & Objective

• To enable direct, real-world reinforcement learning in vision-capable mobile robots without relying on hand-crafted environment models.
• To address the challenge of piecewise-continuous perceptual spaces common in visual robot environments, where discontinuities arise from object boundaries or occlusions.
• To reduce data requirements by avoiding manual discretization of the sensor input space and enabling efficient generalization using a metric-based nearest-neighbor approach.
• To handle partial observability in real-time robot control by incorporating temporal state history through a continuous metric on observation sequences.
• To develop a method that is robust to environmental changes, such as repositioning obstacles or targets, without retraining.

Proposed method

• The algorithm extends McCallum’s Nearest Sequence Memory (NSM) to piecewise-continuous perceptual spaces by using a continuous metric over state-action trajectories instead of discrete state grids.
• It employs a k-nearest neighbors approach with variable radius to generalize from stored experiences, allowing higher resolution near state space boundaries where discontinuities occur.
• A discounted metric is applied to the history of observations, enabling the agent to remember past visual states and recover from temporary loss of target visibility.
• The method stores all collected experience and uses the nearest neighbors in the metric space to estimate Q-values via a modified Q-learning update rule.
• The algorithm operates directly on raw visual input without requiring preprocessing or manual discretization of the perceptual space.
• It uses a continuous action space implicitly by selecting actions based on the nearest neighbors in the metric space, with future extensions considering explicit action metrics.

Experimental results

Research questions

• RQ1 Can a reinforcement learning algorithm learn effective navigation policies directly from real-world visual input without prior environment modeling?
• RQ2 How can a learning algorithm efficiently generalize in piecewise-continuous perceptual spaces where discontinuities exist due to object boundaries or occlusions?
• RQ3 Can a metric-based nearest-neighbor approach outperform fixed-grid or function approximation methods in data efficiency and robustness for real robot control?
• RQ4 How can partial observability in vision-based robot control be mitigated using temporal memory and a continuous metric over observation sequences?
• RQ5 To what extent can the learned policy generalize across changes in obstacle or target positions without retraining?

Key findings

• The PC-NSM algorithm successfully enabled a real mobile robot to learn to navigate and locate a target using only visual input, without any pre-built environment model.
• The robot achieved stable performance after approximately 70 minutes of training, accumulating rewards more rapidly over time, with the fourth trial completed with minimal deliberation.
• The algorithm demonstrated robustness to environmental changes: repositioning the target or obstacles did not degrade performance, as the policy was based on perception rather than fixed coordinates.
• The use of a discounted metric allowed the robot to recover from temporary loss of target visibility by leveraging recent visual history, reducing the risk of getting stuck.
• The method outperformed previous approaches in data efficiency by avoiding manual discretization and enabling adaptive generalization near discontinuities.
• The robot successfully navigated complex scenarios, including narrow passages and specularly reflecting wall gaps, with only minor delays due to perceptual ambiguities.