[Paper Review] LaMPP: Language Models as Probabilistic Priors for Perception and Action

Belinda Z. Li, William Chen+2 more|arXiv (Cornell University)|Feb 3, 2023

Natural Language Processing Techniques4 citations

TL;DR

LaMPP introduces a framework that uses pre-trained language models (LMs) as probabilistic priors in structured graphical models to improve perception and control tasks under uncertainty. By integrating LMs' world knowledge into prior distributions over labels and actions, LaMPP enhances zero-shot, out-of-distribution, and in-distribution generalization in semantic segmentation, navigation, and action recognition, with measurable gains on rare and novel inputs.

ABSTRACT

Language models trained on large text corpora encode rich distributional information about real-world environments and action sequences. This information plays a crucial role in current approaches to language processing tasks like question answering and instruction generation. We describe how to leverage language models for *non-linguistic* perception and control tasks. Our approach casts labeling and decision-making as inference in probabilistic graphical models in which language models parameterize prior distributions over labels, decisions and parameters, making it possible to integrate uncertain observations and incomplete background knowledge in a principled way. Applied to semantic segmentation, household navigation, and activity recognition tasks, this approach improves predictions on rare, out-of-distribution, and structurally novel inputs.

Motivation & Objective

• To address the lack of robust common-sense priors in perception and control systems, especially under data sparsity or distributional shift.
• To leverage the rich, world-embedded knowledge encoded in large language models (LMs) for non-linguistic tasks such as image segmentation and action recognition.
• To enable principled integration of uncertain linguistic priors with task-specific, non-linguistic observations using probabilistic graphical models.
• To improve model generalization on rare, out-of-distribution, and structurally novel inputs by combining LM priors with domain-specific likelihood models.

Proposed method

• Formalize perception and decision-making as Bayesian inference in a probabilistic graphical model, where the label space Y is governed by a prior P(Y, Y') parameterized by a language model.
• Use the language model to define P(Y) — a distribution over structured labels (e.g., object co-occurrences in scenes) — based on natural language statistics.
• Combine the LM prior with a task-specific observation model P(X|Y), such as a vision model for images or a video encoder for action sequences.
• Perform approximate inference via sampling or optimization to compute P(Y|X) ∝ P(Y)P(X|Y), enabling uncertainty-aware predictions.
• Apply the framework to three tasks: semantic segmentation, household navigation, and video action recognition, using LMs to guide label and action sequence inference.
• Calibrate LM priors through prompt engineering and evaluation on in-domain and out-of-domain examples to improve alignment with real-world distributions.

Experimental results

Research questions

• RQ1 Can language models serve as effective, general-purpose probabilistic priors for perception and control tasks beyond natural language understanding?
• RQ2 How does integrating LM-derived priors improve performance on rare, out-of-distribution, and structurally novel inputs in vision and robotics tasks?
• RQ3 To what extent can language models’ world knowledge be effectively composed with task-specific, non-linguistic observations in a principled probabilistic framework?
• RQ4 What are the limitations of current LMs in providing calibrated priors for complex decision-making, especially in sequential action prediction?

Key findings

• LaMPP improves zero-shot and out-of-distribution generalization in semantic segmentation, with a 3.5% absolute gain in mIoU on rare object categories.
• In video action recognition, LaMPP improves average step recall by 8.2% on held-out transitions in the OOD setting, indicating strong generalization to unseen action sequences.
• The method achieves consistent improvements across all three tasks—semantic segmentation, navigation, and action recognition—demonstrating broad applicability of LM priors.
• LaMPP shows measurable gains even within the training distribution, suggesting that LM priors help correct biases and improve robustness in standard settings.
• The improvement is most pronounced for rare or structurally novel inputs, where prior knowledge from LMs compensates for data scarcity.
• Despite gains, LM priors for action sequences remain less reliable than for object co-occurrences, due to poor calibration and prompt-order bias in LMs.