[Paper Review] Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning

Yan Zha, Lin Guan+1 more|arXiv (Cornell University)|Oct 11, 2021

Explainable Artificial Intelligence (XAI)35 references4 citations

TL;DR

This paper proposes SERLfD, a self-explanation guided reinforcement learning framework that improves learning from ambiguous human demonstrations by having agents infer task-relevant object relations (e.g., spatial or color-based predicates) as self-explanations. By integrating a self-explanation predictor (SE-Net) into GAN-IRL, the method enhances policy learning stability and performance, especially in tasks with ambiguous or noisy demonstrations, outperforming standard RLfD baselines.

ABSTRACT

Our work aims at efficiently leveraging ambiguous demonstrations for the training of a reinforcement learning (RL) agent. An ambiguous demonstration can usually be interpreted in multiple ways, which severely hinders the RL-Agent from learning stably and efficiently. Since an optimal demonstration may also suffer from being ambiguous, previous works that combine RL and learning from demonstration (RLfD works) may not work well. Inspired by how humans handle such situations, we propose to use self-explanation (an agent generates explanations for itself) to recognize valuable high-level relational features as an interpretation of why a successful trajectory is successful. This way, the agent can provide some guidance for its RL learning. Our main contribution is to propose the Self-Explanation for RL from Demonstrations (SERLfD) framework, which can overcome the limitations of traditional RLfD works. Our experimental results show that an RLfD model can be improved by using our SERLfD framework in terms of training stability and performance.

Motivation & Objective

• To address the challenge of training reinforcement learning agents from ambiguous human demonstrations, where multiple interpretations hinder stable and efficient learning.
• To overcome limitations of existing RLfD methods that fail when demonstrations are noisy or lack clear task-relevant signals.
• To simulate human-like introspection by enabling agents to generate self-explanations—identifying relevant object relations—as a mechanism to disambiguate demonstrations.
• To improve sample efficiency and policy performance in complex, real-world robotic tasks involving both explicit motor actions and implicit relational knowledge.
• To develop a framework that leverages background knowledge of domain predicates to guide the disambiguation of demonstrations without requiring explicit labels.

Proposed method

• Introduce a self-explanation guided RL framework (SERLfD) that combines Generative Adversarial Inverse Reinforcement Learning (GAN-IRL) with a non-linear self-explanation predictor (SE-Net) to infer task-relevant predicate utilities from raw observations.
• Train the SE-Net to predict utility weights for symbolic predicates (e.g., 'block_at_yellow', 'ring_at_blue') based on demonstration trajectories, enabling the agent to identify which relations are most relevant to task success.
• Augment the RL agent’s state or reward function with predicted self-explanations to guide policy learning, improving generalization and reducing sensitivity to noisy or ambiguous demonstrations.
• Use a discriminator in GAN-IRL that distinguishes between successful and failed trajectories, with the SE-Net providing interpretive signals to improve discrimination and policy optimization.
• Leverage a shared vocabulary of domain predicates and classifiers to ground symbolic relations in visual observations, enabling symbolic reasoning over raw sensory input.
• Train the SE-Net end-to-end with the RL policy using adversarial imitation learning, where the agent learns to generate explanations that align with successful demonstrations.

Experimental results

Research questions

• RQ1 Can self-explanation mechanisms improve the stability and performance of RL agents learning from ambiguous human demonstrations?
• RQ2 How can an agent infer task-relevant object relations (e.g., spatial or color-based predicates) from non-expert, unstructured demonstrations without explicit labels?
• RQ3 To what extent does incorporating self-explanations into the reward or state space improve sample efficiency and generalization in RLfD?
• RQ4 How do self-explanations help disambiguate conflicting interpretations of the same demonstration, such as assigning objects to target regions based on color or location?
• RQ5 Can the SE-Net effectively identify and prioritize causally important relations (e.g., 'block_at_yellow' over 'block_at_L1') in complex robotic tasks?

Key findings

• The SERLfD framework significantly improves training stability and final performance compared to standard RLfD baselines, especially in environments with ambiguous demonstrations.
• In the Robot-Push domain, SACfD+SE and TD3fD+SE models learned to prioritize task-relevant predicates such as 'block_at_yellow' and 'ring_at_blue', aligning with human intent even when demonstrations were ambiguous.
• In the Robot-Remove-and-Push domain, the models correctly identified high-utility predicates like 'is_cube_pushedTo_blue' and 'is_cylinder_pushedTo_yellow', which guided effective behavior in complex manipulation tasks.
• In the discrete Pacman domain, RL agents enhanced with SE-Nets outperformed baseline methods, demonstrating improved decision-making in timing pellet consumption for optimal ghost capture.
• The self-explanation predictor (SE-Net) successfully learned to assign higher utility weights to relevant predicates over time, with predictions shifting from less relevant to more task-critical relations as the task progressed.
• Visualizations of self-explanations confirmed that the model dynamically adjusted its interpretation of task-relevant relations, such as favoring 'ring_at_L1' over 'ring_at_blue' in later steps when only one target remained.