[论文解读] The Partner Modelling Questionnaire: A validated self-report measure of perceptions toward machines as dialogue partners

Philip R. Doyle, Iona Gessinger+8 more|arXiv (Cornell University)|Aug 14, 2023

Speech and dialogue systems被引用 10

一句话总结

本论文开发并验证了 Partner Modelling Questionnaire (PMQ)，这是一个18项自我报告量表，用以衡量用户将机器视为对话伙伴的感知，涵盖三项研究。

ABSTRACT

Recent work has looked to understand user perceptions of speech agent capabilities as dialogue partners (termed partner models), and how this affects user interaction. Yet, currently partner model effects are inferred from language production as no metrics are available to quantify these subjective perceptions more directly. Through three studies, we develop and validate the Partner Modelling Questionnaire (PMQ): an 18-item self-report semantic differential scale designed to reliably measure people's partner models of non-embodied speech interfaces. Through principal component analysis and confirmatory factor analysis, we show that the PMQ scale consists of three factors: communicative competence and dependability, human-likeness in communication, and communicative flexibility. Our studies show that the measure consistently demonstrates good internal reliability, strong test-retest reliability over 12 and 4-week intervals, and predictable convergent/divergent validity. Based on our findings we discuss the multidimensional nature of partner models, whilst identifying key future research avenues that the development of the PMQ facilitates. Notably, this includes the need to identify the activation, sensitivity, and dynamism of partner models in speech interface interaction.

研究动机与目标

• 为非具身言语接口开发一个可靠的自我报告伙伴模型测量量表。
• 识别PMQ的因子结构并精简条目以增强鲁棒性。
• 评估与相关量表的聚合效度和辩别效度。
• 在12周和4周的时间间隔内评估时-测重测信度。
• 展示伙伴模型的多维性质并为未来研究提供指南。

提出的方法

• 利用对照格技术和主观量表回顾来生成大量条目。
• 使用探索性和确认性因子分析来识别并验证三因子结构（沟通能力与可靠性；沟通中的人性化；沟通的灵活性）。
• 通过CFA比较23项模型与18项模型以选择稳健版本。
• 使用SASI和I-D-AQ量表评估聚合效度和辨别效度。
• 使用ICC与皮尔逊相关在12周和4周时间间隔内检验时-测重测信度。

实验结果

研究问题

• RQ1 PMQ的潜在因子结构是什么？
• RQ2 18项PMQ版本是否在独立样本中具有鲁棒性和效度？
• RQ3 PMQ是否与相关主观量表显示聚合效度和辨别效度？
• RQ4 PMQ在不同间隔下的时-测重测信度是多少？

主要发现

• PMQ包含三个因子：沟通能力与可靠性；沟通中的人性化；沟通的灵活性。
• 18项PMQ显示出良好的内部一致性，并且相较23项版本具有稳健的CFA拟合。
• PMQ与SASI显示聚合效度，与I-D-AQ显示辨别效度。
• PMQ在12周和4周时显示出强大的时-测重测信度。
• 18项模型提供了对语音接口伙伴模型的简明而稳健的测量。