[Paper Review] LLM-based Smart Reply (LSR): Enhancing Collaborative Performance with ChatGPT-mediated Smart Reply System

Ashish Bastola, Hao Wang+6 more|arXiv (Cornell University)|Jun 21, 2023

Artificial Intelligence in Healthcare and Education8 citations

TL;DR

The paper presents an LLM-based Smart Reply (LSR) system that uses ChatGPT to generate context-aware responses in Slack, reducing cognitive workload and increasing work performance and productivity in simulated collaborative tasks.

ABSTRACT

Interactive user interfaces have increasingly explored AI's role in enhancing communication efficiency and productivity in collaborative tasks. The emergence of Large Language Models (LLMs) such as ChatGPT has revolutionized conversational agents, employing advanced deep learning techniques to generate context-aware, coherent, and personalized responses. Consequently, LLM-based AI assistants provide a more natural and efficient user experience across various scenarios. In this paper, we study how LLM models can be used to improve work efficiency in collaborative workplaces. Specifically, we present an LLM-based Smart Reply (LSR) system utilizing the ChatGPT to generate personalized responses in professional collaborative scenarios while adapting to context and communication style based on prior responses. Our two-step process involves generating a preliminary response type (e.g., Agree, Disagree) to provide a generalized direction for message generation, thus reducing response drafting time. We conducted an experiment where participants completed simulated work tasks involving a Dual N-back test and subtask scheduling through Google Calendar while interacting with co-workers. Our findings indicate that the proposed LSR reduces overall workload, as measured by the NASA TLX, and improves work performance and productivity in the N-back task. We also provide qualitative analysis based on participants' experiences, as well as design considerations to provide future directions for improving such implementations.

Motivation & Objective

• Evaluate whether an LLM-based smart reply system can reduce cognitive workload in collaborative work tasks.
• Assess the impact of LSR on work performance and productivity in a simulated workplace setting.
• Explore user experience factors, trust, privacy, and design considerations for AI-assisted workplace communication.
• Provide design recommendations and future directions for AI-driven collaboration tools.

Proposed method

• Three-part system: N-back task as simulated work, Google Calendar for subtasks, and Slack integrated with LSR.
• Two-step LSR workflow: generate a preliminary response type (e.g., Agree/Disagree) to guide message generation, then present three AI-generated response options.
• Use ChatGPT (GPT-3.5-turbo) to generate replies based on the last ten messages; user selects a button to send the AI-generated reply.
• Mixed-methods evaluation with quantitative measures (N-back accuracy, messages per minute, NASA TLX) and qualitative data (surveys and semi-structured interviews).
• Participants interacted with coworkers of varying personas (Jeff, Tony, Janine) in online simulations, with and without LSR.

Experimental results

Research questions

• RQ1 RQ1: Does the LSR impact work performance, productivity, and workload in a collaborative workplace?
• RQ2 RQ2: What are the key factors that impact the user experience during collaborative works?

Key findings

• LSR significantly improved work performance: mean N-back accuracy rose from 73.79% (SD 14.93) to 79.37% (SD 8.96); difference 5.58% (p = 0.025).
• LSR increased productivity: average messages per minute rose by 40.36% (p = 3.74e-06).
• NASA TLX results showed reduced mental and temporal demand with LSR; Mental Demand (t = 2.7102, p = 0.0154), Temporal Demand (t = 3.6794, p = 0.0020) improved, while Performance showed a marginal change (t = -2.5156, p = 0.0229).
• Participants reported smoother workflow, faster return to tasks, and perceived politeness/quality in AI-generated replies, with some concerns about context accuracy and control.
• Design considerations include interface trade-offs between control and automation, trust/privacy concerns, and the need for editable final messages to improve usability.