[Paper Review] Learning Policies with External Memory

Leonid Peshkin, Nicolas Meuleau+1 more|ArXiv.org|Mar 2, 2001

Reinforcement Learning in Robotics16 references92 citations

TL;DR

This paper proposes a stigmergic reinforcement learning approach using external memory to enable agents to learn optimal policies in partially observable Markov decision processes (POMDPs). By augmenting observations with memory bits and using algorithms like vaps and SARSA(λ), the method enables memory-efficient credit assignment, with vaps outperforming SARSA(λ) on complex non-Markovian tasks such as multi-location load-unload problems.

ABSTRACT

In order for an agent to perform well in partially observable domains, it is usually necessary for actions to depend on the history of observations. In this paper, we explore a {\it stigmergic} approach, in which the agent's actions include the ability to set and clear bits in an external memory, and the external memory is included as part of the input to the agent. In this case, we need to learn a reactive policy in a highly non-Markovian domain. We explore two algorithms: SARSA(λ), which has had empirical success in partially observable domains, and VAPS, a new algorithm due to Baird and Moore, with convergence guarantees in partially observable domains. We compare the performance of these two algorithms on benchmark problems.

Motivation & Objective

• To address the challenge of learning effective policies in partially observable environments where standard Markov assumptions fail.
• To investigate whether external memory can enable reactive agents to simulate optimal behavior in non-Markovian domains.
• To compare the performance of vaps and SARSA(λ) in learning policies with external memory on benchmark POMDP problems.
• To evaluate the impact of credit assignment mechanisms on learning stability and convergence in highly non-Markovian settings.

Proposed method

• Augments the agent's observation space with a set of external memory bits to encode historical information.
• Uses a reactive policy that maps observations and memory states to actions, including memory-setting and clearing actions.
• Employs SARSA(λ) with eligibility traces and Boltzmann exploration for policy learning, with learning rate and temperature decay strategies.
• Applies vaps (Value-Adjusted Policy Search), a convergence-guaranteed algorithm for POMDPs, using true Boltzmann exploration and credit assignment based on policy error.
• Represents Q-functions in tabular form with weights for each observation-action pair.
• Uses a modified exploration strategy with decaying temperature and fixed learning rate schedule to stabilize learning.

Experimental results

Research questions

• RQ1 Can external memory enable a reactive agent to learn optimal policies in non-Markovian domains?
• RQ2 How do vaps and SARSA(λ) compare in learning performance on benchmark POMDP problems with increasing non-Markovian complexity?
• RQ3 Does improved credit assignment in vaps lead to more stable learning and faster convergence than SARSA(λ) in highly non-Markovian environments?
• RQ4 What hyperparameter settings (e.g., β, λ, learning rate, temperature decay) are most effective for each algorithm in non-Markovian tasks?
• RQ5 Can vaps consistently converge to near-optimal policies where SARSA(λ) fails, particularly in multi-location load-unload problems?

Key findings

• On the one-loading-location load-unload problem, both vaps and SARSA(λ) converged to the optimal policy within fewer than 100 runs, achieving a trial length of 9 steps.
• In the two-loading-location problem, vaps consistently converged to a near-optimal policy, while SARSA(λ) failed to stabilize, often diverging after reaching near-optimal performance.
• The modified vaps with true Boltzmann exploration outperformed the original vaps formulation reported by Baird and Moore in terms of learning stability and convergence speed.
• SARSA(λ) with λ=1 and ε=0 showed better performance than other λ values, but remained unstable due to uniform punishment of all state-action pairs during exploration.
• vaps required β=1 for convergence, indicating that Bellman error is ineffective as a learning signal in highly non-Markovian domains.
• A base learning rate of α₀=0.5, along with temperature decay from c_max=1.0 to c_min=0.2, yielded robust performance across both algorithms and problem types.