[Paper Review] Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning

Mark Edmonds, Xiaojian Ma+4 more|arXiv (Cornell University)|Nov 25, 2019

Reinforcement Learning in Robotics29 references4 citations

TL;DR

This paper proposes a theory-based causal transfer framework that integrates instance-level associative learning with abstract-level causal structure learning to enable robust knowledge transfer in dynamic environments. By using Bayesian inference to learn causal models from interactions, the agent achieves human-like transfer performance in the OpenLock 'escape room' environment, outperforming model-free RL agents that fail to generalize across trials.

ABSTRACT

Learning transferable knowledge across similar but different settings is a fundamental component of generalized intelligence. In this paper, we approach the transfer learning challenge from a causal theory perspective. Our agent is endowed with two basic yet general theories for transfer learning: (i) a task shares a common abstract structure that is invariant across domains, and (ii) the behavior of specific features of the environment remain constant across domains. We adopt a Bayesian perspective of causal theory induction and use these theories to transfer knowledge between environments. Given these general theories, the goal is to train an agent by interactively exploring the problem space to (i) discover, form, and transfer useful abstract and structural knowledge, and (ii) induce useful knowledge from the instance-level attributes observed in the environment. A hierarchy of Bayesian structures is used to model abstract-level structural causal knowledge, and an instance-level associative learning scheme learns which specific objects can be used to induce state changes through interaction. This model-learning scheme is then integrated with a model-based planner to achieve a task in the OpenLock environment, a virtual ``escape room'' with a complex hierarchy that requires agents to reason about an abstract, generalized causal structure. We compare performances against a set of predominate model-free reinforcement learning(RL) algorithms. RL agents showed poor ability transferring learned knowledge across different trials. Whereas the proposed model revealed similar performance trends as human learners, and more importantly, demonstrated transfer behavior across trials and learning situations.

Motivation & Objective

• Address the challenge of transferring causal knowledge across environments with different surface features but shared underlying mechanics.
• Overcome the limitations of model-free reinforcement learning, which fails to generalize due to reliance on state-action associations without structural abstraction.
• Develop a hybrid learning system that combines bottom-up associative learning with top-down causal structure induction to form transferable world models.
• Enable agents to generalize across novel but structurally similar environments by learning invariant causal structures and reusable causal hypotheses.
• Demonstrate that integrating causal theory induction with model-based planning leads to human-like transfer behavior, even in complex hierarchical tasks.

Proposed method

• Employ a Bayesian framework to induce causal theories from interactions, treating causal hypotheses as probabilistic beliefs updated via evidence.
• Use a hierarchy of Bayesian structures to represent abstract-level causal relationships, enabling generalization across domains with shared mechanics.
• Implement an instance-level associative learning scheme that maps specific object interactions (e.g., pushing/pulling levers) to observable state changes.
• Integrate the learned causal model with a model-based planner to guide action selection in the OpenLock virtual environment.
• Leverage active exploration to test and refine causal hypotheses, updating belief distributions over the causal hypothesis space based on outcomes.
• Formalize the learning process as a belief update mechanism over a structured hypothesis space, where each hypothesis encodes a potential causal mechanism.

Experimental results

Research questions

• RQ1 Can a hybrid learning system combining instance-level associative learning and abstract-level causal structure learning enable effective knowledge transfer across similar but different environments?
• RQ2 How does the integration of causal theory induction with model-based planning improve generalization compared to model-free RL in hierarchical tasks?
• RQ3 Why do human learners and the proposed model show transferable behavior across trials, while model-free RL agents fail to generalize?
• RQ4 What role does causal abstraction play in reducing failure paths and improving planning efficiency in complex environments?
• RQ5 How can hypothesis space enumeration be made scalable while preserving the ability to discover correct causal structures through active exploration?

Key findings

• The proposed model demonstrated consistent performance trends across trials, mirroring human learning behavior, while model-free RL agents failed to transfer knowledge between trials.
• In the CC (consistent configuration) condition, the agent achieved reliable success by identifying invariant causal structures, showing robustness to surface-level changes.
• In the CE (variable environment) condition, the agent successfully navigated more complex failure paths due to its ability to reason about abstract causal structure, unlike model-free RL agents.
• The model outperformed baseline RL algorithms in both task completion rate and sample efficiency, particularly in environments with structural invariance but perceptual variation.
• The agent’s ability to generalize was attributed to its dual learning mechanism: learning specific instance-level interactions and abstract causal structures simultaneously.
• The study revealed an unexpected asymmetry in difficulty: the CE condition had more failure paths than CC, which the model handled effectively through causal abstraction and hypothesis testing.