[Paper Review] Different Types of Voice User Interface Failures May Cause Different Degrees of Frustration

Shiyoh Goetsu, Tetsuya Sakai|arXiv (Cornell University)|Feb 10, 2020

AI in Service Interactions9 references4 citations

TL;DR

This study investigates how different types of voice user interface (VUI) failures impact user frustration, using a controlled experiment with a custom Alexa-based calendar app. It finds that users are most frustrated by 'Reason Unknown' failures (often caused by intent misclassification), less so by 'Speech Misrecognition,' and least by 'Utterance Pattern Match' issues, suggesting that preventing intent misclassification is key to reducing frustration.

ABSTRACT

We report on an investigation into how different types of failures in a voice user interface (VUI) affects user frustration. To this end, we conducted a pilot user study ($n=10$) and a main user study ($n=30$), both with a simple voice-operated calendar application that we built using the Alexa Skills Kit. In our pilot study, we identified three major failure types as perceived by the users, namely, Reason Unknown, Speech Misrecognition, and Utterance Pattern Match Failure, along with more fine-grained failure types from the developer's viewpoint such as Intent Pattern Match Failure and Intent Misclassification. Then, in our main study, we set up three user tasks that were designed to each induce a specific failure type, and collected user frustration ratings for each task. Our main findings are: (a)Users may be relatively tolerant to user-perceived Speech Misrecognition, and not so to user-perceived Reason Unknown and Utterance Mattern Match Failures; (b)Regarding the relationship between developer-perceived and user-perceived failure types, 68.8\% of developer-perceived Intent Misclassification instances caused user-perceived Reason Unkown failures. From (a) and (b), a practical design implication would be to try to prevent Intent Misclassification from happening by carefully crafting the utterance patterns for each intent.

Motivation & Objective

• To understand how different VUI failure types affect user frustration from the user’s perspective.
• To compare developer-perceived failure types with user-perceived failure types in a controlled VUI environment.
• To identify which failure types lead to the highest levels of user frustration in voice interactions.
• To derive practical design implications for reducing frustration in VUI applications based on failure perception and impact.

Proposed method

• Conducted a pilot study with n=10 participants to identify major failure types from both developer and user perspectives.
• Built a custom voice-operated calendar application using the Alexa Skills Kit to simulate controlled failure scenarios.
• Designed three distinct user tasks (T1, T2, T3) to induce specific developer-perceived failure types: D1 (Intent Pattern Match Failure), D4 (Intent Misclassification), and others.
• Collected user frustration ratings for each task to assess the emotional impact of different failure types.
• Mapped developer-perceived failures (D1–D7) to user-perceived failures (U1–U3) to analyze perception gaps.
• Used statistical analysis to compare mean frustration scores across tasks and identify significant differences.

Experimental results

Research questions

• RQ1 How do different types of VUI failures, as perceived by users, affect their levels of frustration?
• RQ2 To what extent do developer-perceived failure types (e.g., Intent Misclassification) align with user-perceived failure types (e.g., Reason Unknown)?
• RQ3 Which failure type leads to the highest user frustration in voice interaction scenarios?
• RQ4 Can specific design strategies, such as refining utterance patterns, reduce the occurrence of high-frustration failure types?

Key findings

• Users showed significantly lower frustration with 'Speech Misrecognition' (U2), which constituted 46.2% of failures in T1, compared to other failure types.
• Users reported the highest frustration with 'Reason Unknown' failures (U1), which made up 68.3% of failures in T3, and were statistically significantly more frustrating than other types.
• 68.8% of developer-perceived 'Intent Misclassification' (D4) failures were perceived by users as 'Reason Unknown' (U1), indicating a strong perception gap.
• 'Utterance Pattern Match Failure' (U3), induced by D1 (Intent Pattern Match Failure), was perceived correctly by 48.0% of participants, showing partial alignment between developer and user views.
• The mean frustration score for T3 (U1 failures) was significantly higher than for T1 (U2) and T2 (U3), confirming that lack of explanation increases frustration.
• The study concludes that preventing 'Intent Misclassification' through careful utterance pattern design is a key strategy to reduce user frustration in VUIs.