[Paper Review] Inferring Personalized Bayesian Embeddings for Learning from Heterogeneous Demonstration

Rohan Paleja, Matthew Gombolay|arXiv (Cornell University)|Mar 14, 2019

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes a Bayesian LfD framework that infers personalized embeddings for heterogeneous human demonstrators using Bayesian neural networks (BNNs), enabling accurate, individualized behavior prediction. By incorporating counterfactual reasoning via pairwise action comparisons, the method improves data efficiency and outperforms state-of-the-art baselines on synthetic and real-world StarCraft II datasets.

ABSTRACT

For assistive robots and virtual agents to achieve ubiquity, machines will need to anticipate the needs of their human counterparts. The field of Learning from Demonstration (LfD) has sought to enable machines to infer predictive models of human behavior for autonomous robot control. However, humans exhibit heterogeneity in decision-making, which traditional LfD approaches fail to capture. To overcome this challenge, we propose a Bayesian LfD framework to infer an integrated representation of all human task demonstrators by inferring human-specific embeddings, thereby distilling their unique characteristics. We validate our approach is able to outperform state-of-the-art techniques on both synthetic and real-world data sets.

Motivation & Objective

• Address the challenge of human behavioral heterogeneity in Learning from Demonstration (LfD), where traditional methods assume homogeneity or use clustering that reduces data efficiency.
• Overcome limitations of prior clustering-based LfD approaches that sacrifice model capacity and fail to handle mixed or intra-cluster variability.
• Develop a Bayesian neural network framework that infers person-specific embeddings while leveraging shared structure across demonstrators.
• Improve data efficiency in low-data regimes by introducing counterfactual reasoning through pairwise comparisons of actions taken vs. not taken.
• Propose a hybrid training strategy (Hybrid-Bayes Shift) to ensure robust performance during early inference before Bayesian embeddings converge.

Proposed method

• Employ Bayesian neural networks (BNNs) to learn latent embeddings that represent individual human decision-making characteristics, enabling personalized behavior modeling.
• Integrate counterfactual reasoning by comparing the action a human took against all other possible actions in each state, effectively augmenting the training signal.
• Formulate the learning objective to maximize the likelihood of observed actions while penalizing unobserved actions, using a probabilistic ranking loss.
• Use a hybrid training strategy (Hybrid-Bayes Shift) that combines a non-Bayesian initial model with gradual Bayesian fine-tuning to stabilize early predictions.
• Train a BNN to infer embeddings end-to-end via backpropagation through the Bayesian layers, allowing uncertainty-aware adaptation to individual demonstrators.
• Apply the framework to both synthetic jobshop scheduling and real human gameplay in StarCraft II, using action descriptors and state features as input.

Experimental results

Research questions

• RQ1 Can Bayesian neural networks be effectively used to infer personalized embeddings that capture individual decision-making patterns in heterogeneous LfD?
• RQ2 To what extent does counterfactual reasoning via pairwise action comparisons improve data efficiency and model performance in low-data LfD settings?
• RQ3 Can a hybrid training strategy (Hybrid-Bayes Shift) achieve robust early performance while allowing Bayesian embeddings to converge over time?
• RQ4 Are Bayesian embeddings transferable across different neural network architectures, such as from an action-prediction network to a value-prediction network?
• RQ5 Does the proposed method outperform state-of-the-art clustering-based LfD approaches in terms of prediction accuracy and generalization?

Key findings

• The proposed Bayesian LfD framework with personalized embeddings significantly outperforms prior state-of-the-art methods, including k-means clustering-based baselines, on both synthetic and real-world datasets.
• Counterfactual reasoning via pairwise comparisons led to a multi-fold improvement in performance, especially in low-data regimes, by effectively increasing the effective training signal.
• The Hybrid-Bayes Shift algorithm achieved strong early performance by leveraging a non-Bayesian model during initial training, before transitioning to full Bayesian inference.
• Empirical results on StarCraft II demonstrated that the model could accurately predict human actions with high precision, even when training data was limited.
• The study found that Bayesian embeddings were not reliably transferable across different network architectures (e.g., from action to value network), suggesting that network-invariant embeddings are not guaranteed without explicit design.
• The results indicate that BNNs are well-suited for LfD in heterogeneous settings, offering uncertainty estimates and improved adaptation to individual human behaviors.