[Paper Review] Robust Maximum Entropy Behavior Cloning

Mostafa Hussein, Brendan Crowe+2 more|arXiv (Cornell University)|Jan 4, 2021

Adversarial Robustness in Machine Learning26 references4 citations

TL;DR

This paper proposes Robust Maximum Entropy Behavior Cloning (RM-ENT), a novel imitation learning framework that autonomously detects and excludes adversarial demonstrations by leveraging entropy-based weighting in a maximum entropy policy learning setup. The method achieves superior robustness and sample efficiency compared to standard behavior cloning and IRL, especially under adversarial data contamination, without requiring a simulator or additional samples.

ABSTRACT

Imitation learning (IL) algorithms use expert demonstrations to learn a specific task. Most of the existing approaches assume that all expert demonstrations are reliable and trustworthy, but what if there exist some adversarial demonstrations among the given data-set? This may result in poor decision-making performance. We propose a novel general frame-work to directly generate a policy from demonstrations that autonomously detect the adversarial demonstrations and exclude them from the data set. At the same time, it's sample, time-efficient, and does not require a simulator. To model such adversarial demonstration we propose a min-max problem that leverages the entropy of the model to assign weights for each demonstration. This allows us to learn the behavior using only the correct demonstrations or a mixture of correct demonstrations.

Motivation & Objective

• To address the critical limitation in imitation learning where adversarial or incorrect demonstrations degrade policy performance.
• To develop a method that automatically identifies and excludes adversarial demonstrations without prior labeling or simulator access.
• To maintain high performance and sample efficiency in the presence of noisy or adversarial data by leveraging maximum entropy principles.
• To enable robust policy learning using only expert demonstrations, even when some are suboptimal or adversarial.
• To outperform standard behavior cloning and inverse reinforcement learning in both accuracy and robustness under data contamination.

Proposed method

• The framework formulates a min-max optimization problem that maximizes policy entropy while constraining feature expectation matching between expert and learned policies.
• It assigns adaptive weights to each demonstration based on their contribution to policy entropy, effectively identifying and downweighting adversarial or random demonstrations.
• The method uses Lagrangian relaxation to solve the constrained optimization problem, enabling end-to-end training via gradient-based methods.
• A neural network is used to parameterize the policy, and weights are learned jointly with the policy to distinguish between reliable and adversarial demonstrations.
• The approach does not require a simulator or extra rollouts, making it time- and sample-efficient.
• The entropy of the policy distribution is used as a proxy for demonstration quality: higher entropy indicates more randomness or adversarial behavior.

Experimental results

Research questions

• RQ1 Can a behavior cloning framework automatically detect and exclude adversarial demonstrations without prior labeling or simulator access?
• RQ2 How does entropy-based weighting improve robustness in imitation learning under data contamination?
• RQ3 To what extent does the proposed method outperform standard behavior cloning and IRL in the presence of adversarial or random demonstrations?
• RQ4 Can the method maintain high performance even when the majority of demonstrations are adversarial?
• RQ5 How does the method’s sample and time efficiency compare to existing IRL and BC approaches?

Key findings

• In a 5×5 grid world, RM-ENT correctly assigned weights of 0.5 to correct demonstrations and 0.0 to adversarial ones, achieving 83% accuracy in mixed sets.
• With random demonstrations, the framework achieved 92% accuracy, demonstrating its ability to detect random or unstructured behavior based on entropy.
• On Mountain-Car and Acrobot tasks, RM-ENT maintained stable performance even when adversarial demonstrations were introduced, while BC and IRL accuracy dropped significantly.
• The method achieved faster convergence than competing IRL and BC baselines, as shown in training time comparisons on OpenAI Gym environments.
• When adversarial demonstrations outnumbered correct ones, all methods converged to random-like policies, but RM-ENT still outperformed others in early stages due to effective detection.
• The framework successfully excluded adversarial demonstrations even when only two correct demonstrations were available, proving its robustness to low signal-to-noise ratios.