[Paper Review] Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards

Xingyu Lu, Stas Tiomkin+1 more|arXiv (Cornell University)|Dec 21, 2019

Reinforcement Learning in Robotics32 references4 citations

TL;DR

This paper proposes using predictive coding to learn unsupervised representations that shape rewards in deep reinforcement learning with sparse rewards. By maximizing mutual information between consecutive states via Contrastive Predictive Coding, the method generates noise-resistant, dynamics-aware reward signals that match the sample efficiency of hand-crafted rewards across robotic manipulation, navigation, and locomotion tasks.

ABSTRACT

While recent progress in deep reinforcement learning has enabled robots to learn complex behaviors, tasks with long horizons and sparse rewards remain an ongoing challenge. In this work, we propose an effective reward shaping method through predictive coding to tackle sparse reward problems. By learning predictive representations offline and using these representations for reward shaping, we gain access to reward signals that understand the structure and dynamics of the environment. In particular, our method achieves better learning by providing reward signals that 1) understand environment dynamics 2) emphasize on features most useful for learning 3) resist noise in learned representations through reward accumulation. We demonstrate the usefulness of this approach in different domains ranging from robotic manipulation to navigation, and we show that reward signals produced through predictive coding are as effective for learning as hand-crafted rewards.

Motivation & Objective

• To address the challenge of sample inefficiency in deep reinforcement learning with sparse rewards.
• To develop a reward shaping method that leverages unsupervised representation learning without requiring domain-specific reward engineering.
• To create reward signals that are robust to noise in learned representations and emphasize environment dynamics.
• To demonstrate that predictive coding-based reward shaping performs as well as hand-crafted rewards in complex control tasks.

Proposed method

• The method uses Contrastive Predictive Coding (CPC) to learn representations that maximize mutual information between consecutive states in trajectories.
• Predictive features are extracted from offline trajectories collected via random or exploratory policies.
• These features are used to shape rewards in the representation space, providing dense feedback to the agent while training on raw observations.
• The reward shaping is applied as a dense, sparse-reward-agnostic signal, decoupling representation learning from policy training.
• The approach is compatible with standard deep RL algorithms like PPO, operating in a two-stage pipeline: offline pretraining of encoder, then online RL with shaped rewards.
• Noise robustness is enhanced by avoiding direct use of embeddings as features and instead using them solely for reward shaping, reducing sensitivity to imperfect representations.

Experimental results

Research questions

• RQ1 Can unsupervised predictive coding representations provide effective reward signals in sparse-reward reinforcement learning?
• RQ2 How does predictive coding-based reward shaping compare to hand-crafted rewards in terms of sample efficiency and final performance?
• RQ3 Can predictive coding representations generalize across different textures and environmental variations without explicit supervision?
• RQ4 Does using predictive features for reward shaping outperform using them as state features in policy training?
• RQ5 How does the method perform in environments with long horizons and limited exploration?

Key findings

• The proposed method achieved performance on par with hand-crafted rewards in robotic manipulation, navigation, and locomotion tasks, demonstrating equivalent sample efficiency.
• Using predictive coding features for reward shaping significantly outperformed using them as state features, especially in noisy or undertrained representation regimes.
• The method showed robustness to background texture variations, with embeddings clustering by physical state (e.g., arm angle) rather than visual texture, indicating effective abstraction of dynamics.
• Reward shaping via predictive coding enabled stable learning even with limited exploration, outperforming standard sparse-reward training in environments like Reacher and Pendulum.
• The approach reduced sensitivity to noisy representations by decoupling representation learning from policy training, leveraging reward accumulation across trajectories.
• The method was effective across diverse environments, including those with high-dimensional observations like pixel inputs, without architectural or algorithmic modifications.