[Paper Review] Imitation Learning from Pixel-Level Demonstrations by HashReward

Xin-Qiang Cai, Yao-Xiang Ding+2 more|arXiv (Cornell University)|Sep 9, 2019

Multimodal Machine Learning Applications39 references4 citations

TL;DR

This paper proposes HashReward, a novel imitation learning method that uses supervised hashing to balance dimensionality reduction and discriminative training in high-dimensional pixel environments. By jointly optimizing embedding learning and reward generation, HashReward enables effective adversarial imitation learning, significantly outperforming state-of-the-art methods on Atari and MuJoCo benchmarks with large performance gaps.

ABSTRACT

One of the key issues for imitation learning lies in making policy learned from limited samples to generalize well in the whole state-action space. This problem is much more severe in high-dimensional state environments, such as game playing with raw pixel inputs. Under this situation, even state-of-the-art adversary-based imitation learning algorithms fail. Through empirical studies, we find that the main cause lies in the failure of training a powerful discriminator to generate meaningful rewards in high-dimensional environments. Although it seems that dimensionality reduction can help, a straightforward application of off-the-shelf methods cannot achieve good performance. In this work, we show in theory that the balance between dimensionality reduction and discriminative training is essential for effective learning. To achieve this target, we propose HashReward, which utilizes the idea of supervised hashing to realize such an ideal balance. Experimental results show that HashReward could outperform state-of-the-art methods for a large gap under the challenging high-dimensional environments.

Motivation & Objective

• To address the failure of adversary-based imitation learning in high-dimensional state spaces, particularly with raw pixel inputs.
• To identify the root cause of performance degradation in existing methods as an imbalance between dimensionality reduction and discriminative training.
• To design a unified framework that preserves discriminative information during dimensionality reduction while enabling effective reward signal generation.
• To improve generalization and sample efficiency in imitation learning under limited expert demonstrations.
• To demonstrate that supervised hashing can effectively balance the discrimination-rewarding trade-off in adversarial imitation learning.

Proposed method

• Proposes HashReward, a joint optimization framework that integrates supervised hashing with adversarial imitation learning.
• Uses a Siamese network architecture to learn 64-bit hash codes from expert and expert-like policy trajectories.
• Introduces a contrastive loss with hard negative mining to preserve discriminative structure in the embedding space.
• Employs a discriminator that receives hash codes as input, enabling it to generate meaningful reward signals based on semantic similarity.
• Trains the policy and discriminator end-to-end using a GAIL-like objective, with the discriminator's loss incorporating both classification and contrastive supervision.
• Applies a temperature-scaled contrastive loss to stabilize training and improve generalization of the learned embeddings.

Experimental results

Research questions

• RQ1 Why do state-of-the-art adversary-based imitation learning methods fail in high-dimensional pixel environments?
• RQ2 What causes the degradation in performance when using unsupervised dimensionality reduction in imitation learning?
• RQ3 How can dimensionality reduction and discriminative training be balanced to preserve meaningful reward signals?
• RQ4 Can supervised hashing effectively preserve discriminative information while reducing dimensionality for imitation learning?
• RQ5 Does a unified training procedure for hashing and discrimination lead to better policy generalization than isolated pre-training?

Key findings

• HashReward significantly outperforms GAIL, VAIL, and GAIL-UH on Atari games, achieving higher final returns and faster convergence.
• The pseudo-reward curve of HashReward closely matches the true reward, indicating stable and meaningful reward signals, unlike GAIL-UH and VAIL, which show poor or biased reward signals.
• t-SNE visualization confirms that HashReward embeddings form globally clustered structures with increasing overlap between expert and policy samples over time, indicating improved policy imitation.
• 64-bit embedding analysis shows that policy-generated codes become increasingly similar to expert codes as training progresses, verifying that HashReward learns a meaningful, semantically structured embedding space.
• The discriminator in HashReward avoids over-discrimination, maintaining a balanced trade-off between discrimination and reward generation, unlike GAIL and GAIL-AE, which suffer from overfitting to discrimination.
• Experiments confirm that unsupervised hashing (e.g., GAIL-UH) fails to preserve discriminative information, leading to ineffective reward signals and poor policy performance.