[Paper Review] Robust Locally-Linear Controllable Embedding

Ershad Banijamali, Rui Shu+3 more|arXiv (Cornell University)|Oct 15, 2017

Model Reduction and Neural Networks12 references25 citations

TL;DR

This paper proposes Robust Locally-Linear Controllable Embedding (RCE), a novel model that directly estimates the predictive conditional density $ p(\mathbf{x}_{t+1}|\mathbf{x}_t) $ using a bottlenecked generative model with structured dynamics for robust locally-linear control. Unlike E2C, RCE uses a variational inference scheme that conditions on future observations, reducing approximation error and significantly improving performance under noisy dynamics.

ABSTRACT

Embed-to-control (E2C) is a model for solving high-dimensional optimal control problems by combining variational auto-encoders with locally-optimal controllers. However, the E2C model suffers from two major drawbacks: 1) its objective function does not correspond to the likelihood of the data sequence and 2) the variational encoder used for embedding typically has large variational approximation error, especially when there is noise in the system dynamics. In this paper, we present a new model for learning robust locally-linear controllable embedding (RCE). Our model directly estimates the predictive conditional density of the future observation given the current one, while introducing the bottleneck between the current and future observations. Although the bottleneck provides a natural embedding candidate for control, our RCE model introduces additional specific structures in the generative graphical model so that the model dynamics can be robustly linearized. We also propose a principled variational approximation of the embedding posterior that takes the future observation into account, and thus, makes the variational approximation more robust against the noise. Experimental results show that RCE outperforms the E2C model, and does so significantly when the underlying dynamics is noisy.

Motivation & Objective

• Address the statistical shortcomings of E2C, which lacks a likelihood-based objective and uses a non-robust variational approximation.
• Develop a principled method to learn low-dimensional embeddings that support robust locally-linear control in high-dimensional observation spaces.
• Introduce a generative model that explicitly models the linearization point as a random variable to enable structured, locally-linear dynamics.
• Design a variational inference framework that conditions on future observations to reduce posterior approximation error.
• Ensure the model is robust to noise in system dynamics while maintaining compatibility with existing locally-optimal control algorithms like iLQG.

Proposed method

• Model the predictive conditional density $ p(\mathbf{x}_{t+1}|\mathbf{x}_t) $ using a bottlenecked graphical model with a latent variable $ \mathbf{z}_t $, inspired by BCDE.
• Treat the local linearization point as a random variable in the generative model to enforce structured, locally-linear dynamics.
• Construct a variational posterior $ q(\mathbf{z}_t|\mathbf{x}_t, \mathbf{x}_{t+1}) $ that explicitly conditions on the future observation $ \mathbf{x}_{t+1} $, improving approximation accuracy.
• Optimize a variational lower bound on the data likelihood that accounts for the full sequence, not just pairwise marginals.
• Decouple the generative model from the recognition model, enabling modular training and better generalization.
• Use a factorized recognition model that leverages determinism in transition dynamics to improve inference efficiency.

Experimental results

Research questions

• RQ1 Can a model that directly estimates the predictive conditional density $ p(\mathbf{x}_{t+1}|\mathbf{x}_t) $ achieve better control performance than E2C in high-dimensional, noisy environments?
• RQ2 Does conditioning the variational posterior on future observations reduce the variational approximation error and improve robustness to system noise?
• RQ3 Can structured modeling of the linearization point in the generative model enable more accurate and stable locally-linear control?
• RQ4 How does the proposed RCE framework compare to E2C in terms of reconstruction, prediction, and planning performance across multiple control benchmarks?
• RQ5 Is the separation between generative model and recognition model beneficial for training stability and performance in complex control tasks?

Key findings

• RCE significantly outperforms E2C in planning loss across all benchmarks, especially under noisy dynamics: 61.1±16.2 vs. 97.1±34.1 in the inverted pendulum task.
• In the cart-pole balancing task, RCE achieves a 90% success rate under noise, compared to E2C’s 60%, with lower prediction and planning losses.
• On the three-link robot arm, RCE maintains 90% success rate in noiseless conditions and 80% under noise, while E2C drops to 65%.
• RCE reduces reconstruction and prediction losses by up to 30% compared to E2C in high-dimensional visual control tasks.
• The model’s performance gap widens under noise, demonstrating that the future-conditional variational inference in RCE effectively mitigates noise-induced approximation errors.
• RCE achieves better generalization and robustness due to the clean separation of generative modeling and amortized inference, enabling stable training and improved control policy learning.