[Paper Review] A Conceptual Framework for Implicit Evaluation of Conversational Search Interfaces

Abhishek Kaushik, Gareth J. F. Jones|arXiv (Cornell University)|Apr 8, 2021

Speech and dialogue systems37 references4 citations

TL;DR

This paper proposes a multidimensional conceptual framework for implicit evaluation of conversational search (CS) interfaces, integrating dimensions from interactive information retrieval and conversational systems—such as usability, cognitive load, knowledge gain, and user experience. The framework uses post- and pre-search questionnaires with Likert-scale scoring and statistical analysis to assess system performance and user learning, enabling both quantitative and qualitative evaluation of CS effectiveness beyond simple task completion.

ABSTRACT

Conversational search (CS) has recently become a significant focus of the information retrieval (IR) research community. Multiple studies have been conducted which explore the concept of conversational search. Understanding and advancing research in CS requires careful and detailed evaluation. Existing CS studies have been limited to evaluation based on simple user feedback on task completion. We propose a CS evaluation framework which includes multiple dimensions: search experience, knowledge gain, software usability, cognitive load and user experience, based on studies of conversational systems and IR. We introduce these evaluation criteria and propose their use in a framework for the evaluation of CS systems.

Motivation & Objective

• To address the limitation of existing conversational search (CS) evaluations that rely only on task completion and user feedback.
• To develop a comprehensive evaluation framework that captures deeper aspects of user interaction and learning in CS.
• To integrate dimensions from interactive information retrieval and conversational systems into a unified evaluation model for CS.
• To enable implicit assessment of user experience, cognitive load, and knowledge gain during CS interactions.
• To provide a practical, empirically grounded tool for evaluating and improving CS systems in real-world scenarios.

Proposed method

• The framework uses a two-part questionnaire: one for post-search evaluation (exploration) and one for pre- and post-search knowledge assessment (contentment).
• Each questionnaire item uses a 7-point Likert scale (0–7), with mean scores analyzed for statistical significance using dependent or independent t-tests.
• Qualitative analysis assigns color-coded annotations (red, yellow, green) to mean scores based on thresholds: <2 (red), 2–4 (yellow), >4 (green) to indicate negative, neutral, or positive evaluations.
• Two independent analysts annotate responses with a Kappa coefficient of approximately 0.85 to ensure reliability.
• Knowledge gain is measured by comparing pre- and post-search summaries using three parameters: Dqual (quality), Dintrp (interpretation), and Dcrit (critical thinking), with knowledge increase defined as Dqual > 1.5, Dintrp > 1, and Dcrit > 0.
• The framework is validated through statistical testing and inter-rater reliability checks to identify interface weaknesses in specific dimensions such as usability or cognitive load.

Experimental results

Research questions

• RQ1 How can conversational search systems be evaluated beyond simple task completion and user feedback?
• RQ2 What multidimensional factors influence the effectiveness and user experience of conversational search interfaces?
• RQ3 To what extent does conversational search contribute to user knowledge gain and cognitive development?
• RQ4 How can implicit evaluation metrics be systematically collected and analyzed to assess CS performance?
• RQ5 In what ways can usability, cognitive load, and user experience be quantitatively and qualitatively assessed in CS interactions?

Key findings

• The proposed framework enables a more comprehensive evaluation of conversational search systems by incorporating usability, cognitive load, and knowledge gain beyond traditional performance metrics.
• Statistical analysis using dependent and independent t-tests allows for valid comparison between conventional and conversational search systems.
• The use of color-coded annotations (red, yellow, green) based on mean Likert scores provides a clear visual indicator of system strengths and weaknesses in specific dimensions.
• Inter-rater reliability was confirmed with a Kappa coefficient of approximately 0.85, indicating strong consistency in qualitative annotation of user responses.
• Knowledge gain was quantitatively assessed using a three-parameter model (Dqual, Dintrp, Dcrit), with a threshold-based definition for significant learning (Dqual > 1.5, Dintrp > 1, Dcrit > 0).
• The framework successfully identifies specific interface components needing improvement, such as software usability or cognitive load, based on dimension-specific analysis.