[Paper Review] Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machines

Sreejan Kumar, Carlos G. Correa+8 more|arXiv (Cornell University)|May 23, 2022

Adversarial Robustness in Machine Learning7 citations

TL;DR

This paper proposes co-training meta-reinforcement learning agents using natural language descriptions and program abstractions—particularly those with learned primitives—to instill human-like inductive biases. By leveraging human-generated language and program induction with library learning, the method improves generalization on human-adept tasks while reducing performance on human-poor tasks, demonstrating that abstraction through compression in these representations drives more human-like behavior.

ABSTRACT

Strong inductive biases give humans the ability to quickly learn to perform a variety of tasks. Although meta-learning is a method to endow neural networks with useful inductive biases, agents trained by meta-learning may sometimes acquire very different strategies from humans. We show that co-training these agents on predicting representations from natural language task descriptions and programs induced to generate such tasks guides them toward more human-like inductive biases. Human-generated language descriptions and program induction models that add new learned primitives both contain abstract concepts that can compress description length. Co-training on these representations result in more human-like behavior in downstream meta-reinforcement learning agents than less abstract controls (synthetic language descriptions, program induction without learned primitives), suggesting that the abstraction supported by these representations is key.

Motivation & Objective

• To investigate whether natural language and program abstractions can serve as repositories of human inductive biases for training artificial agents.
• To determine if co-training with these representations leads to more human-like behavior in meta-reinforcement learning agents.
• To examine whether abstraction—particularly through compression in language and program representations—drives improved generalization and human-like inductive biases.
• To compare the effectiveness of human-generated vs. synthetic language and program abstractions with and without learned primitives in shaping agent behavior.
• To explore the feasibility of scaling such representations using active learning and data augmentation for larger environments.

Proposed method

• Co-train meta-reinforcement learning agents using auxiliary tasks that predict representations derived from natural language task descriptions and program abstractions.
• Use a domain-specific language (DSL) for program generation on 2D binary grid environments, including both base primitives and learned library abstractions.
• Apply library learning to automatically discover and incorporate higher-order programmatic primitives that compress description length, mimicking human abstraction.
• Train agents to predict both human-generated language descriptions and programs with learned primitives as auxiliary supervision during meta-RL training.
• Compare performance across control conditions: synthetic language, base DSL programs, and programs with learned libraries, using human performance as a benchmark.
• Employ a task distribution from Kumar et al. [10] designed to distinguish subtle differences in inductive biases, enabling fine-grained analysis of agent behavior.

Experimental results

Research questions

• RQ1 Can co-training with human-generated natural language descriptions improve meta-RL agents' alignment with human inductive biases?
• RQ2 Does using program abstractions with learned primitives—especially those that compress description length—lead to more human-like behavior in agents?
• RQ3 How do different forms of representation (synthetic vs. human language, base DSL vs. library-extended programs) affect the emergence of human-like inductive biases?
• RQ4 Is there a measurable correspondence between the abstraction level in human language and program representations, suggesting shared cognitive compression mechanisms?
• RQ5 To what extent does the abstraction level in language and program representations drive improved generalization and human-like behavior in downstream meta-RL tasks?

Key findings

• Agents co-trained on human-generated language and programs with learned primitives showed significantly more human-like behavior than those trained on synthetic language or base DSL programs.
• The use of program abstractions with learned primitives led to a measurable decrease in performance on task distributions where humans perform poorly, indicating that the agent adopted a more human-like inductive bias.
• A strong correspondence was observed between human-generated language descriptions and programs synthesized using library learning, suggesting that both use higher-order abstractions to compress information in a similar way.
• Co-training with representations that support abstraction through compression—especially in language and program abstractions—was found to be a key driver of human-like behavior in downstream meta-RL agents.
• Agents trained with human language and library-extended programs outperformed control conditions on task distributions where humans are adept, demonstrating improved generalization.
• The results suggest that abstraction via compression in natural language and program representations is a critical mechanism for instilling human-like inductive biases in artificial agents.