[Paper Review] OMNI: Open-endedness via Models of human Notions of Interestingness

Jenny Zhang, Joel Lehman+2 more|arXiv (Cornell University)|Jun 2, 2023

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

OMNI proposes using foundation models as a model of human notions of interestingness (MoI) to guide open-ended learning by prioritizing tasks that are both learnable and genuinely interesting. By leveraging pre-trained FMs to assess novelty and value, OMNI outperforms uniform sampling and learning-progress-only baselines in discovering diverse, high-impact behaviors across environments like Crafter and BabyAI.

ABSTRACT

Open-ended algorithms aim to learn new, interesting behaviors forever. That requires a vast environment search space, but there are thus infinitely many possible tasks. Even after filtering for tasks the current agent can learn (i.e., learning progress), countless learnable yet uninteresting tasks remain (e.g., minor variations of previously learned tasks). An Achilles Heel of open-endedness research is the inability to quantify (and thus prioritize) tasks that are not just learnable, but also $ extit{interesting}$ (e.g., worthwhile and novel). We propose solving this problem by $ extit{Open-endedness via Models of human Notions of Interestingness}$ (OMNI). The insight is that we can utilize foundation models (FMs) as a model of interestingness (MoI), because they $ extit{already}$ internalize human concepts of interestingness from training on vast amounts of human-generated data, where humans naturally write about what they find interesting or boring. We show that FM-based MoIs improve open-ended learning by focusing on tasks that are both learnable $ extit{and interesting}$, outperforming baselines based on uniform task sampling or learning progress alone. This approach has the potential to dramatically advance the ability to intelligently select which tasks to focus on next (i.e., auto-curricula), and could be seen as AI selecting its own next task to learn, facilitating self-improving AI and AI-Generating Algorithms. Project website at https://www.jennyzhangzt.com/omni/

Motivation & Objective

• To address the critical gap in open-ended AI research: the inability to quantify and prioritize truly interesting tasks beyond mere learnability.
• To overcome the limitations of hand-crafted interestingness metrics, which often fail to capture human intuition about novelty and value.
• To enable AI agents to autonomously generate and focus on a self-curated, diverse curriculum of tasks that are both learnable and worth pursuing.
• To explore whether foundation models—trained on vast human-generated data—can internalize and generalize human-like judgments of interestingness for use in AI self-improvement.

Proposed method

• OMNI uses foundation models (FMs) as a model of interestingness (MoI), leveraging their pre-trained understanding of human concepts like novelty, value, and originality from massive text corpora.
• The MoI evaluates candidate tasks by prompting FMs to judge whether a task is learnable and interesting, using few-shot examples to guide reasoning.
• Tasks are ranked based on the FM’s output, with higher scores indicating greater expected learning progress and perceived interestingness.
• The agent samples tasks from the ranked list, ensuring exploration of diverse, non-redundant behaviors rather than minor variations of known tasks.
• The method integrates both learning progress (via policy performance) and FM-based interestingness into a unified scoring function for task selection.
• OMNI is evaluated using both sentence-level models (e.g., GPT-3.5) and sentence embedding models (e.g., BGE), with ablation studies comparing their performance.

Experimental results

Research questions

• RQ1 Can foundation models effectively serve as a proxy for human notions of interestingness in open-ended learning?
• RQ2 How does FM-based task selection compare to uniform sampling and learning-progress-only baselines in promoting diverse and high-impact skill acquisition?
• RQ3 To what extent can FMs distinguish between learnable yet uninteresting tasks (e.g., minor variations) and genuinely novel, valuable tasks?
• RQ4 Can the model of interestingness be improved by integrating multiple aspects of interestingness (e.g., novelty, value, complexity) into a single FM prompt?
• RQ5 What are the safety implications of allowing AI to autonomously select its next learning task based on human-like interestingness judgments?

Key findings

• OMNI significantly outperforms uniform sampling and learning-progress-only baselines in average task success rate across all training steps in the Crafter environment.
• OMNI achieves higher success rates on tasks considered boring (e.g., repetitive or compound tasks) compared to OMNI-embed, indicating better discrimination of true interestingness.
• The method successfully identifies and prioritizes non-redundant, high-impact tasks, such as learning to collect both wood and stone only after mastering each individually, avoiding overestimation of compound tasks.
• OMNI-embed, a simpler sentence-embedding baseline, performs comparably to OMNI in some settings but fails to distinguish subtle differences in task novelty, especially in compound task scenarios.
• GPT-4 demonstrates strong capability in integrating multiple aspects of interestingness (e.g., novelty, value, learnability) when prompted with few-shot examples, suggesting potential for scalable, adaptive MoI design.
• The results suggest that foundation models can internalize and generalize human-like judgments of interestingness, enabling AI to autonomously curate its own curriculum of learning tasks.