[Paper Review] The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputs

Lance Ying, Katherine M. Collins+6 more|arXiv (Cornell University)|Jun 25, 2023

Topic Modeling4 citations

TL;DR

This paper introduces the Neuro-Symbolic Inverse Planning Engine (NIPE), a hybrid model that uses a large language model (LLM) to translate linguistic descriptions into symbolic program semantics, which condition a Bayesian probabilistic generative model for goal inference. NIPE outperforms LLMs alone in predicting human judgments on complex social reasoning tasks involving rational agent planning, particularly in spatial and rule-based obstacle course scenarios.

ABSTRACT

Human beings are social creatures. We routinely reason about other agents, and a crucial component of this social reasoning is inferring people's goals as we learn about their actions. In many settings, we can perform intuitive but reliable goal inference from language descriptions of agents, actions, and the background environments. In this paper, we study this process of language driving and influencing social reasoning in a probabilistic goal inference domain. We propose a neuro-symbolic model that carries out goal inference from linguistic inputs of agent scenarios. The "neuro" part is a large language model (LLM) that translates language descriptions to code representations, and the "symbolic" part is a Bayesian inverse planning engine. To test our model, we design and run a human experiment on a linguistic goal inference task. Our model closely matches human response patterns and better predicts human judgements than using an LLM alone.

Motivation & Objective

• To model human-like social reasoning about agents' goals from linguistic inputs, particularly in complex, dynamic environments.
• To address the limitations of purely neural LLMs in capturing rational, goal-directed planning and efficient path inference.
• To integrate symbolic Bayesian inverse planning with neural language models for more structured, interpretable, and accurate social inference.
• To test whether combining neural language understanding with symbolic probabilistic reasoning improves prediction of human intuitive judgments.
• To develop a scalable, linguistically grounded framework for modeling theory of mind in AI agents using natural language descriptions.

Proposed method

• Use a large language model (LLM) to parse natural language descriptions of environments, agent actions, and world dynamics into executable symbolic program representations.
• Translate linguistic inputs into structured symbolic semantics that condition a probabilistic generative model over possible world states and agent behaviors.
• Condition the generative model on the symbolic semantics to define a prior over possible environments and agent plans.
• Apply Bayesian inverse planning to infer the most likely goal of an agent by reasoning backward from observed actions and world constraints.
• Employ a nested planner within the inverse planning process to simulate rational, goal-directed behavior under efficiency and constraint assumptions.
• Train and evaluate the model on a custom linguistic goal inference task inspired by a gameshow obstacle course with locked doors and multiple trophies.

Figure 1: Overview of our model, the Neuro-Symbolic Inverse Planning Engine (NIPE), which reasons about linguistic goal inference scenarios. ( Left, green ) We design and measure human judgments on a linguistic goal inference domain (inspired by Zhi-Xuan et al. ( 2020 ) , which asks about the final

Experimental results

Research questions

• RQ1 Can a neuro-symbolic model better predict human intuitive goal inferences than LLMs alone in complex social reasoning scenarios?
• RQ2 How well do LLMs alone handle counterintuitive rational planning, such as ignoring closer targets when pursuing a more distant goal?
• RQ3 To what extent does symbolic Bayesian inverse planning improve reasoning about agent efficiency and goal-directedness in spatial tasks?
• RQ4 How do linguistic inputs about world dynamics, actions, and configurations jointly influence goal inference in a probabilistic framework?
• RQ5 Can a hybrid neuro-symbolic system produce more coherent, structured, and human-aligned social inferences than end-to-end neural models?

Key findings

• NIPE closely matches human response patterns in a linguistic goal inference task involving a complex obstacle course with locked doors and multiple trophies.
• NIPE significantly outperforms LLM baselines (including GPT-4) in predicting human judgments, especially on complex variants like the Spatial and Color-Different domains.
• On the Spatial variant, where efficient path planning is required, LLMs incorrectly predict agents are targeting closer trophies (e.g., silver), while humans and NIPE correctly infer the agent is aiming for a more distant target (e.g., bronze), demonstrating NIPE’s ability to model rational planning.
• The LLM baseline shows near-zero correlation with human judgments on the Color-Different variant, indicating it fails to consistently apply abstract rules about world dynamics.
• NIPE achieves strong correlation with human judgments across all four task variants, indicating robustness to linguistic and structural complexity.
• Qualitative analysis reveals that NIPE correctly infers goals based on rationality and efficiency, while LLMs often rely on surface-level cues like proximity or perceived value (e.g., gold vs. silver trophies).

Figure 2: Correlation between NIPE and baseline LLMs vs. human judgments, aggregated across the goal-inference stimuli. Our model ( Right ) produces posterior probabilities over goals that correlate strongly overall with human judgments; GPT-4 correlates moderately overall ( Middle ) and GPT-3.5 ( L