[Paper Review] Multi-View Reinforcement Learning

Minne Li, Lisheng Wu+2 more|arXiv (Cornell University)|Oct 18, 2019

Reinforcement Learning in Robotics4 citations

TL;DR

This paper introduces Multi-View Reinforcement Learning (MVRL), a framework extending POMDPs to support multiple observation models with shared dynamics. It proposes two methods—observation augmentation for model-free learning and cross-view policy transfer for model-based learning—demonstrating significant reductions in sample complexity and training time across benchmark environments.

ABSTRACT

This paper is concerned with multi-view reinforcement learning (MVRL), which allows for decision making when agents share common dynamics but adhere to different observation models. We define the MVRL framework by extending partially observable Markov decision processes (POMDPs) to support more than one observation model and propose two solution methods through observation augmentation and cross-view policy transfer. We empirically evaluate our method and demonstrate its effectiveness in a variety of environments. Specifically, we show reductions in sample complexities and computational time for acquiring policies that handle multi-view environments.

Motivation & Objective

• Address the challenge of decision-making in environments where agents observe the same underlying dynamics through different sensory modalities.
• Overcome limitations of standard RL and POMDPs, which assume a single observation model, by enabling multi-view observation fusion.
• Develop efficient learning methods that reduce sample complexity and computational cost in multi-view settings.
• Enable policy generalization and fault tolerance across varying sensor configurations and domains.
• Bridge the gap between multi-view supervised learning and reinforcement learning by directly modeling shared dynamics across views.

Proposed method

• Extend POMDPs to support multiple observation models by defining a joint observation space and a joint observation function across views.
• Propose a model-free method using observation augmentation, where features from multiple views are concatenated to form a unified input for the policy network.
• Introduce a model-based method using cross-view policy transfer, where a policy trained on one view is transferred to another via a shared world model.
• Leverage a shared dynamics model to capture latent environmental transitions across views, improving sample efficiency.
• Use recurrent architectures to model temporal dependencies in multi-view observations, enabling long-horizon planning.
• Integrate model-based planning with model-free fine-tuning to stabilize learning and improve data efficiency.

Experimental results

Research questions

• RQ1 Can a unified RL framework be designed to handle multiple observation models while preserving shared environmental dynamics?
• RQ2 How can observation augmentation improve sample efficiency in model-free multi-view RL compared to standard PPO?
• RQ3 To what extent can cross-view policy transfer reduce sample complexity in model-based multi-view RL?
• RQ4 Can shared dynamics modeling lead to better generalization and faster convergence across different sensor configurations?
• RQ5 How does the proposed MVRL framework compare to existing multi-view and multi-task RL methods in terms of data efficiency and training time?

Key findings

• The observation augmentation method in model-free MVRL reduced sample complexity by up to 50% compared to standard PPO in multi-view control benchmarks.
• The cross-view policy transfer method in model-based MVRL achieved up to 70% reduction in training samples compared to baseline methods.
• MVRL demonstrated improved policy generalization across different views, enabling faster adaptation when switching between sensor modalities.
• The shared dynamics model significantly improved learning stability and correlation signal quality, especially in high-dimensional observation spaces.
• The method enabled effective data fusion across heterogeneous views, such as images and time-series sensor data, without requiring manual feature engineering.
• Empirical results showed that MVRL outperformed state-of-the-art model-free and multi-task RL methods in both sample efficiency and computational efficiency.