[Paper Review] Predictive Coding for Locally-Linear Control

Rui Shu, Tung Nguyen+6 more|arXiv (Cornell University)|Mar 2, 2020

Advanced Control Systems Optimization33 references4 citations

TL;DR

This paper proposes a decoder-free Learning Controllable Embedding (LCE) framework that replaces explicit next-observation prediction with predictive coding to learn a controllable latent space for locally-linear control. The method, called PC3, achieves superior control performance with improved sample efficiency and faster training by minimizing predictive suboptimality through an information-theoretic objective.

ABSTRACT

High-dimensional observations and unknown dynamics are major challenges when applying optimal control to many real-world decision making tasks. The Learning Controllable Embedding (LCE) framework addresses these challenges by embedding the observations into a lower dimensional latent space, estimating the latent dynamics, and then performing control directly in the latent space. To ensure the learned latent dynamics are predictive of next-observations, all existing LCE approaches decode back into the observation space and explicitly perform next-observation prediction---a challenging high-dimensional task that furthermore introduces a large number of nuisance parameters (i.e., the decoder) which are discarded during control. In this paper, we propose a novel information-theoretic LCE approach and show theoretically that explicit next-observation prediction can be replaced with predictive coding. We then use predictive coding to develop a decoder-free LCE model whose latent dynamics are amenable to locally-linear control. Extensive experiments on benchmark tasks show that our model reliably learns a controllable latent space that leads to superior performance when compared with state-of-the-art LCE baselines.

Motivation & Objective

• Address the challenge of controlling high-dimensional, unknown dynamical systems using raw sensory inputs.
• Overcome the limitations of existing LCE methods that rely on explicit next-observation prediction via a high-capacity decoder.
• Develop a parameter-efficient, decoder-free LCE model that ensures predictive suboptimality is minimized for effective latent-space planning.
• Enable reliable and efficient locally-linear control by learning a controllable latent dynamics model through predictive coding.
• Demonstrate that predictive coding is a theoretically sound and empirically superior alternative to next-observation prediction in LCE frameworks.

Proposed method

• Formulate the LCE problem through an information-theoretic objective that minimizes predictive suboptimality in the latent space.
• Replace explicit next-observation prediction with predictive coding, which learns to reconstruct future latent states from context using contrastive prediction.
• Introduce the Predictive Coding-Consistency-Curvature (PC3) objective that jointly optimizes representation quality, dynamics consistency, and curvature regularization.
• Use a contrastive predictive coding loss to ensure the latent dynamics model is predictive and suitable for multi-step planning.
• Train the encoder and latent dynamics model end-to-end without a decoder, reducing model complexity and parameter count.
• Apply locally-linear control methods (e.g., LQR) directly in the latent space after training, leveraging the learned predictive dynamics.

Experimental results

Research questions

• RQ1 Can predictive coding serve as a viable and theoretically justified alternative to explicit next-observation prediction in LCE frameworks?
• RQ2 Does a decoder-free LCE model trained via predictive coding achieve better control performance and sample efficiency than baselines relying on next-observation prediction?
• RQ3 How does predictive coding impact the interpretability and controllability of the learned latent space compared to baselines?
• RQ4 What is the contribution of each component in the PC3 objective (predictive coding, consistency, curvature) to final control performance?
• RQ5 Can predictive coding enable effective and efficient locally-linear control in high-dimensional observation spaces without requiring a decoder?

Key findings

• The PC3 model significantly outperforms state-of-the-art LCE baselines, including PCC and SOLAR, across multiple benchmark control tasks, with a mean performance improvement of 11.3% on average.
• PC3 achieves 1.85× faster training than PCC and 52.8× faster than SOLAR, demonstrating substantial computational efficiency gains.
• The latent representations learned by PC3 are more interpretable and semantically meaningful, especially in tasks like Pendulum, where PCC's next-observation prediction leads to overly conservative representations.
• Ablation studies show that predictive coding alone is insufficient for good planning performance; the addition of consistency and curvature regularization is essential for stable and accurate latent dynamics.
• PC3 enables effective reuse of the learned latent dynamics across multiple control tasks without retraining, unlike task-specific methods like SOLAR.
• The model achieves high success rates on challenging tasks such as Swing Up and 3-Link, where PCC and SOLAR frequently fail due to poor representation quality or high computational cost.