[Paper Review] Building Proactive Voice Assistants: When and How (not) to Interact

Ondrej Miksik, I. Munasinghe+18 more|arXiv (Cornell University)|May 4, 2020

AI in Service Interactions38 references4 citations

TL;DR

This paper proposes a framework for proactive voice assistants that initiate interactions based on contextual awareness, personalization, and spatial AI to deliver timely, non-intrusive information. The authors validate their design through a prototype and user study, finding that privacy is a critical concern, and users value proactive assistance only when it is unobtrusive and contextually appropriate.

ABSTRACT

Voice assistants have recently achieved remarkable commercial success. However, the current generation of these devices is typically capable of only reactive interactions. In other words, interactions have to be initiated by the user, which somewhat limits their usability and user experience. We propose, that the next generation of such devices should be able to proactively provide the right information in the right way at the right time, without being prompted by the user. However, achieving this is not straightforward, since there is the danger it could interrupt what the user is doing too much, resulting in it being distracting or even annoying. Furthermore, it could unwittingly, reveal sensitive/private information to third parties. In this report, we discuss the challenges of developing proactively initiated interactions, and suggest a framework for when it is appropriate for the device to intervene. To validate our design assumptions, we describe firstly, how we built a functioning prototype and secondly, a user study that was conducted to assess users' reactions and reflections when in the presence of a proactive voice assistant. This pre-print summarises the state, ideas and progress towards a proactive device as of autumn 2018.

Motivation & Objective

• To address the limitations of reactive voice assistants that only respond to user-initiated commands.
• To design a system that proactively delivers relevant information at the right time without disrupting ongoing activities.
• To investigate how privacy and user comfort can be maintained in proactive interaction systems.
• To validate the feasibility and user acceptance of proactive voice assistants through a prototype and user study.
• To establish design principles for when and how voice assistants should initiate interactions autonomously.

Proposed method

• Developed a custom hardware platform with multi-modal sensors (audio, visual, environmental) for real-time context perception.
• Implemented a Spatial AI model to fuse sensory inputs and high-level semantic understanding of the environment.
• Designed a decision-making module that evaluates user state (e.g., alone, busy, engaged) and urgency of information.
• Integrated personal data sources (calendar, email, digital behavior) to identify relevant proactive events.
• Enabled dynamic adaptation of interaction timing and mode based on real-time context and user preferences.
• Conducted a live-lab user study with participants interacting with the prototype to assess reactions and privacy concerns.

Experimental results

Research questions

• RQ1 When is it appropriate for a voice assistant to initiate a proactive interaction without user prompting?
• RQ2 How can a voice assistant determine the right moment to deliver information without causing distraction or annoyance?
• RQ3 What role does personalization and context awareness play in making proactive interactions feel natural and useful?
• RQ4 How do users perceive privacy risks in proactive systems that use cameras, microphones, and behavioral data?
• RQ5 What design principles ensure that proactive interactions enhance user experience rather than degrade it?

Key findings

• Users strongly value privacy, and even in a controlled lab setting, privacy concerns emerged as a central challenge for proactive systems.
• Participants did not raise privacy issues during the study, but this does not guarantee acceptance in real home environments.
• Proactive interactions were perceived as beneficial only when they were timely, contextually relevant, and non-intrusive.
• The system’s ability to detect user state (e.g., alone, busy) significantly improved the appropriateness of intervention timing.
• Users preferred proactive updates when they were personalized and derived from trusted sources like their own calendar or email.
• There is a clear trade-off between functionality and privacy: proactive features require persistent access to sensitive data, demanding strong transparency and configurability.