[Paper Review] Sasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models

Evan King, Haoxiang Yu+2 more|arXiv (Cornell University)|May 16, 2023

AI in Service InteractionsComputer Science49 references7 citations

TL;DR

The paper investigates using large language models to interpret under-specified smart-home goals, identifies failure modes, and presents Sasha, an iterative-reasoning smart-home assistant evaluated in real-user studies.

ABSTRACT

Smart home assistants function best when user commands are direct and well-specified (e.g., "turn on the kitchen light"), or when a hard-coded routine specifies the response. In more natural communication, however, human speech is unconstrained, often describing goals (e.g., "make it cozy in here" or "help me save energy") rather than indicating specific target devices and actions to take on those devices. Current systems fail to understand these under-specified commands since they cannot reason about devices and settings as they relate to human situations. We introduce large language models (LLMs) to this problem space, exploring their use for controlling devices and creating automation routines in response to under-specified user commands in smart homes. We empirically study the baseline quality and failure modes of LLM-created action plans with a survey of age-diverse users. We find that LLMs can reason creatively to achieve challenging goals, but they experience patterns of failure that diminish their usefulness. We address these gaps with Sasha, a smarter smart home assistant. Sasha responds to loosely-constrained commands like "make it cozy" or "help me sleep better" by executing plans to achieve user goals, e.g., setting a mood with available devices, or devising automation routines. We implement and evaluate Sasha in a hands-on user study, showing the capabilities and limitations of LLM-driven smart homes when faced with unconstrained user-generated scenarios.

Motivation & Objective

• Explore how LLMs can support loosely-constrained user goals in smart homes.
• Identify practical challenges and failure modes when using LLMs for smart-home control.
• Develop Sasha with iterative reasoning to improve action plans and reduce mis-targeting.
• Evaluate LLM-driven smart-home control through a hands-on user study and real-world deployment.

Proposed method

• Prototype system uses zero-shot prompts with a JSON home template to generate executable action plans from natural-language commands.
• Empirical study with 20 participants analyzes 600 labels and free-form rationales to identify failure modes and satisfaction levels.
• Sasha introduces iterative reasoning to guide the LLM toward high-quality plans and reduce hallucinations.
• Implementation of Sasha in a test home with unconstrained user commands to assess capabilities and limitations in realistic scenarios.
• Use of prompt engineering and JSON-based planning, plus post-processing to ensure valid action plans.

Experimental results

Research questions

• RQ1 RQ1: What unique capabilities are unlocked when LLMs are used for smart home control?
• RQ2 RQ2: What practical challenges will LLM-based systems present?
• RQ3 RQ3: What system design choices can address these practical challenges?
• RQ4 RQ4: How well can this new form of smart home support user goals in unconstrained scenarios?

Key findings

• LLMs can flexibly generate creative action plans in response to under-specified commands.
• LLMs exhibit failure patterns that reduce usefulness and user satisfaction.
• Sasha's iterative reasoning reduces false positives and erroneous device targeting.
• The system can realize immediate goals and higher-level persistent goals through automation in a test home.
• A real-world user study demonstrates the capabilities and remaining limitations of LLM-driven smart homes under unconstrained scenarios.