[Paper Review] Impact of Guidance and Interaction Strategies for LLM Use on Learner Performance and Perception

Harsh Kumar, Ilya Musabirov+6 more|arXiv (Cornell University)|Oct 13, 2023

Technology Adoption and User Behaviour88 references10 citations

TL;DR

The paper investigates four pedagogically informed guidance strategies (list of suggestions, example-based instruction, metacognitive questioning, and solve-then-refine) across two learner approaches (LLM-first vs. self-first) and two study settings to assess effects on learner performance, confidence, and trust in LLMs.

ABSTRACT

Personalized chatbot-based teaching assistants can be crucial in addressing increasing classroom sizes, especially where direct teacher presence is limited. Large language models (LLMs) offer a promising avenue, with increasing research exploring their educational utility. However, the challenge lies not only in establishing the efficacy of LLMs but also in discerning the nuances of interaction between learners and these models, which impact learners' engagement and results. We conducted a formative study in an undergraduate computer science classroom (N=145) and a controlled experiment on Prolific (N=356) to explore the impact of four pedagogically informed guidance strategies on the learners' performance, confidence and trust in LLMs. Direct LLM answers marginally improved performance, while refining student solutions fostered trust. Structured guidance reduced random queries as well as instances of students copy-pasting assignment questions to the LLM. Our work highlights the role that teachers can play in shaping LLM-supported learning environments.

Motivation & Objective

• Assess how common guidance strategies affect learners’ use of LLMs in educational tasks.
• Examine how guidance interacts with learner approach (LLM-first vs. self-first) to shape outcomes.
• Evaluate effects on performance, confidence, trust, and perceived helpfulness of LLMs over different interaction stages.
• Explore how guidance strategies influence student engagement and quality of interaction with LLM tutors.

Proposed method

• Conduct two studies: a formative field study in an undergraduate CS classroom (N=145) and a controlled Prolific study (N=356).
• Manipulate four guidance strategies (G1: List of Suggestions, G2: Example-based instruction, G3: Metacognitive questioning, G4: Solve then refine) in factorial designs.
• Use two learner approaches: LLM-First Approach (LFA) and Self-First Approach (SFA) influenced by guidance.
• Measure performance on assignments and final exams, plus perceptions of LLMs (confidence in responses, helpfulness, willingness to interact again, error tolerance) and self-perception (confidence in answers).
• In Study 2, compare prompted vs. unprompted LLMs and analyze pre-, post-, and post-task confidence and trust trajectories.

Figure 1 . Different forms of guidance strategies.

Experimental results

Research questions

• RQ1 RQ1: How do guidance types influence learners’ initial conversation patterns with an LLM tutor?
• RQ2 RQ2: What is the impact of guidance strategies on learners’ task performance and learning outcomes with LLMs?
• RQ3 RQ3: How do guidance strategies affect learners’ confidence in using LLMs for problem-solving across interaction stages?
• RQ4 RQ4: How do guidance strategies influence learners’ trust in LLM responses during different interaction stages?
• RQ5 RQ5: In what ways do guidance methods shape learners’ self-confidence in problem-solving across interaction stages?

Key findings

• Structured guidance reduces random queries and decreases verbatim copying of assignment questions to the LLM; it also promotes deeper engagement.
• Direct LLM answers marginally improve performance; refining student solutions with the LLM fosters trust.
• Example-based instruction unexpectedly increases unrelated queries or attempts to break the chatbot.
• Metacognitive questioning increases rephrasing of questions but may increase copying in some cases.
• In the Prolific study, there were no statistically significant differences in task performance across guidance types or LLM types (prompted vs unprompted); confidence in using LLMs increases initially but drops after attempting real tasks, indicating a potentially negative interaction experience.

Figure 2 . Schematic of the formative study in a CS classroom. 2 ( List of Suggestions : present vs. absent) x 2 ( Example-based Instruction : present vs. absent) x 2 ( Metacognitive-questioning based Instruction : present vs. absent) x 2 ( Solve, then refine with LLM : present vs. absent) between s