[Paper Review] DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs

Yunbo Wang, Bo Liu+5 more|arXiv (Cornell University)|Sep 28, 2019

Adversarial Robustness in Machine Learning25 references6 citations

TL;DR

DualSMC proposes a differentiable, particle-based framework for continuous POMDPs that jointly optimizes state filtering and uncertainty-aware planning using two Sequential Monte Carlo processes: one for belief estimation over states and another for planning over future trajectories. It achieves state-of-the-art performance in complex, partially observed environments with visual inputs, including 3D navigation and robotic control, while maintaining interpretability through particle-based belief representations and adversarial training for improved filtering.

ABSTRACT

A major difficulty of solving continuous POMDPs is to infer the multi-modal distribution of the unobserved true states and to make the planning algorithm dependent on the perceived uncertainty. We cast POMDP filtering and planning problems as two closely related Sequential Monte Carlo (SMC) processes, one over the real states and the other over the future optimal trajectories, and combine the merits of these two parts in a new model named the DualSMC network. In particular, we first introduce an adversarial particle filter that leverages the adversarial relationship between its internal components. Based on the filtering results, we then propose a planning algorithm that extends the previous SMC planning approach [Piche et al., 2018] to continuous POMDPs with an uncertainty-dependent policy. Crucially, not only can DualSMC handle complex observations such as image input but also it remains highly interpretable. It is shown to be effective in three continuous POMDP domains: the floor positioning domain, the 3D light-dark navigation domain, and a modified Reacher domain.

Motivation & Objective

• To address the challenge of planning under partial observability in continuous POMDPs where state uncertainty significantly impacts decision quality.
• To develop a differentiable, interpretable framework that maintains belief over unobserved states using particles, enabling robust inference and planning.
• To integrate filtering and planning through shared dynamics and belief propagation, improving sample efficiency and policy performance.
• To enable end-to-end training with complex observations such as images while preserving interpretability through particle-based representations.
• To benchmark and demonstrate superior performance on challenging continuous POMDP environments, including visual navigation and robotic control tasks.

Proposed method

• DualSMC employs two coupled Sequential Monte Carlo (SMC) processes: a state estimation filter using particles in the real state space and a planning SMC that samples in the future trajectory space.
• It introduces an adversarial particle filter that enhances filtering accuracy by leveraging a discriminator to refine particle weights based on their consistency with observations.
• The planner uses a differentiable Q-function and policy network to compute trajectory weights based on predicted rewards and state-action transitions, with uncertainty-aware action selection.
• The top state particles from the filter are used as initial belief for the planner, and the model performs model predictive control by re-planning at each step with only the first action executed.
• The transition model is shared between filtering and planning, and the entire system is trained end-to-end using supervised signals for the filter and reinforcement learning for the planner.
• The framework uses a belief propagation mechanism that ties the filter and planner via particle weights and state transitions, enabling uncertainty-aware decision-making.

Experimental results

Research questions

• RQ1 Can a differentiable, particle-based filtering method improve state estimation in continuous POMDPs with complex observations like images?
• RQ2 How can planning be made uncertainty-aware in continuous POMDPs by leveraging particle-based belief representations?
• RQ3 Can adversarial training improve the robustness and accuracy of differentiable particle filters in partially observed environments?
• RQ4 To what extent can a shared dynamics model improve sample efficiency and performance in joint filtering and planning?
• RQ5 How does the DualSMC framework compare to existing deep reinforcement learning and POMDP methods in terms of performance and interpretability on continuous control tasks?

Key findings

• DualSMC achieves state-of-the-art performance across three continuous POMDP benchmarks: floor positioning, 3D light-dark navigation, and a modified Reacher environment.
• The model outperforms existing methods in the 3D light-dark navigation task, which involves rich visual observations and long-horizon partial observability.
• The adversarial particle filter significantly improves belief accuracy compared to standard differentiable particle filters, especially in high-uncertainty scenarios.
• The integration of top state particles as initial belief for planning leads to faster convergence and more robust action selection.
• Model predictive control with re-planning at each step ensures adaptability and stability, even under model inaccuracies.
• The framework maintains high interpretability through particle-based belief states, enabling failure analysis and model debugging.