[Paper Review] Thinking Assistants: LLM-Based Conversational Assistants that Help Users Think By Asking rather than Answering

Soya Park, Subramonyam, Hari+1 more|arXiv (Cornell University)|Dec 10, 2023

AI in Service Interactions4 citations

TL;DR

This paper introduces *Thinking Assistants*, LLM-based conversational agents that promote deep reflection in users by asking probing questions rather than providing direct answers. The system, *\Sys*, supports prospective graduate students in refining research interests through personalized, mentor-like dialogue, resulting in 65% user satisfaction and twice as long engagement when discussing personal research compared to querying about professors.

ABSTRACT

Many AI systems focus solely on providing solutions or explaining outcomes. However, complex tasks like research and strategic thinking often benefit from a more comprehensive approach to augmenting the thinking process rather than passively getting information. We introduce the concept of "Thinking Assistant", a new genre of assistants that help users improve decision-making with a combination of asking reflection questions based on expert knowledge. Through our lab study (N=80), these Large Language Model (LLM) based Thinking Assistants were better able to guide users to make important decisions, compared with conversational agents that only asked questions, provided advice, or neither. Based on the results, we develop a Thinking Assistant in academic career development, determining research trajectory or developing one's unique research identity, which requires deliberation, reflection and experts' advice accordingly. In a longitudinal deployment with 223 conversations, participants responded positively to approximately 65% of the responses. Our work proposes directions for developing more effective LLM agents. Rather than adhering to the prevailing authoritative approach of generating definitive answers, LLM agents aimed at assisting with cognitive enhancement should prioritize fostering reflection. They should initially provide responses designed to prompt thoughtful consideration through inquiring, followed by offering advice only after gaining a deeper understanding of the user's context and needs.

Motivation & Objective

• To address the challenge of self-discovery in high-stakes decisions like graduate school applications, where mentorship is often inaccessible or intimidating.
• To reduce user frustration in decision-making by shifting from information delivery to reflective dialogue, especially for uncertain or identity-forming choices.
• To design a conversational assistant that fosters commitment to research interests through active engagement and wise feedback, mimicking mentorship.
• To evaluate whether a thinking assistant model improves user engagement and satisfaction compared to traditional Q&A-based chatbots in academic decision-making contexts.
• To explore the tension between reflective dialogue and user expectations for definitive answers, particularly regarding application success likelihood.

Proposed method

• The system, *\Sys*, uses GPT-4 fine-tuned on curated data from participating HCI professors, including research evolution, mentoring styles, and key qualities.
• It operates in two modes: 'probing' mode, which asks reflective questions to deepen user thinking, and 'answering' mode, which provides factual information about professors.
• The assistant uses a dynamic response strategy that ends most turns with a follow-up question to sustain dialogue and encourage reflection.
• A secondary safety bot validates factual claims, especially regarding publications and program details, to prevent misinformation.
• The system was iteratively designed and validated with participating professors to ensure accurate representation of their expertise and mentoring approaches.
• User conversations were logged and analyzed across 223 interactions to assess engagement, satisfaction, and response patterns.

Figure 1. \Sys is a thinking assistant that helps prospective graduate students reflect on research interests. Based on the user’s question, the assistant chooses between a ‘probing’ mode, understanding user interests and prompting reflection, and an ‘answering’ mode that offers with relevant inform

Experimental results

Research questions

• RQ1 How does a thinking assistant that emphasizes questioning over answering affect user engagement and satisfaction in graduate school application contexts?
• RQ2 To what extent do users prefer reflective dialogue over direct answers when making high-stakes decisions like graduate school applications?
• RQ3 How does personal disclosure of research interests influence the quality and depth of interaction with a thinking assistant?
• RQ4 What are the limitations of a thinking assistant when users expect definitive answers, such as 'yes' or 'no' on admission chances?
• RQ5 How can a thinking assistant balance reflective scaffolding with factual information retrieval without undermining user trust?

Key findings

• Users reported a 65% satisfaction rate with responses, with significantly higher engagement when discussing their own research interests rather than querying about professors.
• Conversations involving personal research disclosure averaged six messages per interaction, indicating deeper engagement, while information-only queries were shorter and less satisfying.
• Users who sought definitive answers—such as 'Do I have a chance?'—found the reflective mode unsatisfactory, highlighting a mismatch between user expectations and the assistant’s design.
• Even basic factual retrieval, such as paper recommendations, was perceived as helpful, despite the availability of public information, suggesting value in curated, accessible responses.
• The system’s safety bot successfully corrected factual inaccuracies, demonstrating the importance of validation layers in LLM-based assistants.
• The dual-mode design (probing vs. answering) enabled flexibility, but users struggled to adapt when the assistant prioritized reflection over information delivery.

Figure 2. \Sys interface. Left: Before a user initiates conversation, it shows description of the chatbot. In the interface, we suggest users starting by sharing about their research interest. Right: The chatbot primarily focuses on asking thought-provoking questions and probing users’ research inte