[Paper Review] Building Machines that Learn and Think for Themselves: Commentary on Lake et al., Behavioral and Brain Sciences, 2017

Matthew Botvinick, David G. T. Barrett+17 more|arXiv (Cornell University)|Nov 22, 2017

Reinforcement Learning in Robotics23 references5 citations

TL;DR

This commentary advocates for developing autonomous AI agents that learn and construct their own internal models with minimal human engineering, emphasizing model-based reasoning and self-supervised learning. It highlights progress in meta-learning, world model learning, and hierarchical reinforcement learning, arguing that autonomy is essential for scaling to real-world complexity beyond pre-defined formal models.

ABSTRACT

We agree with Lake and colleagues on their list of key ingredients for building humanlike intelligence, including the idea that model-based reasoning is essential. However, we favor an approach that centers on one additional ingredient: autonomy. In particular, we aim toward agents that can both build and exploit their own internal models, with minimal human hand-engineering. We believe an approach centered on autonomous learning has the greatest chance of success as we scale toward real-world complexity, tackling domains for which ready-made formal models are not available. Here we survey several important examples of the progress that has been made toward building autonomous agents with humanlike abilities, and highlight some outstanding challenges.

Motivation & Objective

• To advance artificial intelligence toward human-level learning and reasoning by emphasizing autonomy in model-building.
• To reduce reliance on hand-engineered models by enabling agents to learn and exploit internal representations independently.
• To address real-world complexity where formal models are unavailable or impractical to design.
• To survey progress in autonomous learning systems, including meta-learning, world model learning, and hierarchical reinforcement learning.
• To identify key challenges in scaling autonomous agents toward robust, generalizable intelligence.

Proposed method

• Leveraging meta-learning (few-shot learning) to enable rapid adaptation to new tasks with minimal data.
• Employing world model learning through predictive world models that simulate future states and outcomes.
• Using hierarchical reinforcement learning to decompose complex tasks into manageable subgoals and abstract policies.
• Integrating self-supervised learning to extract structured representations from unstructured sensory data.
• Designing agents that can both generate and refine internal models through interaction and experience.
• Applying differentiable neural networks and differentiable world models to enable end-to-end training of autonomous reasoning systems.

Experimental results

Research questions

• RQ1 How can AI agents autonomously construct and refine internal models without extensive human-provided supervision?
• RQ2 What mechanisms enable agents to generalize from few examples to new, unseen tasks?
• RQ3 How can hierarchical and compositional reasoning be learned end-to-end in complex environments?
• RQ4 What role does predictive world modeling play in enabling autonomous, goal-directed behavior?
• RQ5 What are the key limitations in scaling autonomous learning to real-world domains with high complexity and uncertainty?

Key findings

• Meta-learning approaches enable agents to generalize from few examples, demonstrating human-like few-shot learning capabilities.
• World model learning allows agents to simulate future states and plan effectively, even in environments with sparse rewards.
• Hierarchical reinforcement learning enables agents to decompose complex tasks into subgoals, improving sample efficiency and scalability.
• Self-supervised world models trained on raw sensory data can capture structured, disentangled representations useful for reasoning and planning.
• Autonomous agents trained with end-to-end differentiable models show improved robustness and adaptability in complex, dynamic environments.
• Despite progress, significant challenges remain in achieving reliable, generalizable reasoning in real-world domains lacking formal models.