[Paper Review] Domain-Robust Visual Imitation Learning with Mutual Information Constraints

Edoardo Cetin, Oya Çeliktutan|arXiv (Cornell University)|Mar 8, 2021

Human Pose and Action Recognition36 references4 citations

TL;DR

This paper proposes DisentanGAIL, a domain-robust visual imitation learning framework that uses adversarial training with a latent representation bottleneck regularized by mutual information constraints to disentangle task completion signals from domain-specific variations. The method enables effective imitation from high-dimensional observations of a different expert agent, achieving strong performance across diverse control tasks under significant domain shifts in appearance and embodiment.

ABSTRACT

Human beings are able to understand objectives and learn by simply observing others perform a task. Imitation learning methods aim to replicate such capabilities, however, they generally depend on access to a full set of optimal states and actions taken with the agent's actuators and from the agent's point of view. In this paper, we introduce a new algorithm - called Disentangling Generative Adversarial Imitation Learning (DisentanGAIL) - with the purpose of bypassing such constraints. Our algorithm enables autonomous agents to learn directly from high dimensional observations of an expert performing a task, by making use of adversarial learning with a latent representation inside the discriminator network. Such latent representation is regularized through mutual information constraints to incentivize learning only features that encode information about the completion levels of the task being demonstrated. This allows to obtain a shared feature space to successfully perform imitation while disregarding the differences between the expert's and the agent's domains. Empirically, our algorithm is able to efficiently imitate in a diverse range of control problems including balancing, manipulation and locomotive tasks, while being robust to various domain differences in terms of both environment appearance and agent embodiment.

Motivation & Objective

• Address the challenge of visual imitation learning when expert demonstrations are observed from a different agent’s perspective, rather than from the agent’s own point of view.
• Overcome the limitations of agent-centric imitation, which require kinesthetic or teleoperated demonstrations and lead to unnatural user-agent interactions.
• Enable effective imitation in high-dimensional control tasks despite domain shifts in environment appearance and agent embodiment.
• Develop a method that learns a domain-invariant representation of task completion levels without relying on hand-engineered features or strong assumptions about task structure.
• Ensure robustness and scalability to complex, real-world imitation scenarios by enforcing mutual information constraints in a generative adversarial framework.

Proposed method

• Introduce a novel discriminator with a latent bottleneck that encodes only task-completion-related information, regularized via mutual information constraints.
• Enforce two mutual information constraints: one between the latent representation and the origin of observations (agent vs. expert), and another to prevent domain information from being encoded in the latent space.
• Implement a double statistics network (2St) to stabilize training and prevent domain information disguising during mutual information estimation.
• Apply spectral normalization to the discriminator’s layers to improve training stability and prevent mode collapse.
• Use off-policy reinforcement learning to train the agent using the reward signal derived from the adversarial discriminator.
• Design the model to disentangle goal-completion signals from irrelevant domain-specific features such as background appearance or agent morphology.

Experimental results

Research questions

• RQ1 Can mutual information constraints effectively disentangle task completion signals from domain-specific variations in visual observations?
• RQ2 How does the proposed method perform under significant domain shifts in both environment appearance and agent embodiment?
• RQ3 What is the impact of domain information disguising on mutual information estimation, and how can it be mitigated during training?
• RQ4 Does the proposed architecture scale to high-dimensional control tasks such as locomotion and manipulation?
• RQ5 How do architectural components like spectral normalization and the double statistics network contribute to training stability and performance?

Key findings

• DisentanGAIL achieves state-of-the-art performance on a diverse set of control tasks, including Reacher, Inverted Pendulum, Hopper, and 7DOF-Pusher, with mean returns of 0.973±0.074 and 1.021±0.023 in the Reacher and Inverted Pendulum tasks, respectively.
• The method demonstrates robustness to domain shifts: performance in alternative target environments with altered backgrounds (e.g., darker floor in Hopper, green table in 7DOF-Pusher) remains strong, with a return of 0.709±0.078 in Hopper and 0.835±0.039 in 7DOF-Pusher, compared to 0.749±0.026 and 0.901±0.044 in original environments.
• Ablation studies show that both spectral normalization and the double statistics network contribute positively to performance, with the latter having a slightly greater impact, especially in the Inverted Pendulum task.
• Removing spectral normalization leads to faster initial convergence but reduced training stability, indicating a trade-off between speed and robustness.
• The double statistics network effectively prevents domain information disguising, which otherwise degrades mutual information estimation and harms policy performance.
• The model successfully disentangles task-relevant features from irrelevant domain-specific cues, such as floor tile appearance, enabling effective imitation even when visual backgrounds are significantly different.