[Paper Review] Integrating A.I. in Higher Education: Protocol for a Pilot Study with 'SAMCares: An Adaptive Learning Hub'

Syed Hasib Akhter Faruqui, Nazia Tasnim+3 more|arXiv (Cornell University)|May 1, 2024

Innovative Teaching and Learning Methods6 citations

TL;DR

This paper protocol describes a year-long randomized controlled trial to evaluate SAMCares, an LLM-based adaptive learning hub using Retriever-Augmented Generation (RAG) to support SHSU students with context-aware AI tutoring. It outlines study design, data collection, and open-source release plans.

ABSTRACT

Learning never ends, and there is no age limit to grow yourself. However, the educational landscape may face challenges in effectively catering to students' inclusion and diverse learning needs. These students should have access to state-of-the-art methods for lecture delivery, online resources, and technology needs. However, with all the diverse learning sources, it becomes harder for students to comprehend a large amount of knowledge in a short period of time. Traditional assistive technologies and learning aids often lack the dynamic adaptability required for individualized education plans. Large Language Models (LLM) have been used in language translation, text summarization, and content generation applications. With rapid growth in AI over the past years, AI-powered chatbots and virtual assistants have been developed. This research aims to bridge this gap by introducing an innovative study buddy we will be calling the 'SAMCares'. The system leverages a Large Language Model (LLM) (in our case, LLaMa-2 70B as the base model) and Retriever-Augmented Generation (RAG) to offer real-time, context-aware, and adaptive educational support. The context of the model will be limited to the knowledge base of Sam Houston State University (SHSU) course notes. The LLM component enables a chat-like environment to interact with it to meet the unique learning requirements of each student. For this, we will build a custom web-based GUI. At the same time, RAG enhances real-time information retrieval and text generation, in turn providing more accurate and context-specific assistance. An option to upload additional study materials in the web GUI is added in case additional knowledge support is required. The system's efficacy will be evaluated through controlled trials and iterative feedback mechanisms.

Motivation & Objective

• Investigate whether GenAI tools can enhance learning experiences and academic performance in higher education.
• Assess knowledge gains, student satisfaction, and cognitive load when using SAMCares versus traditional materials.
• Develop and evaluate an adaptive, context-aware AI tutoring system grounded in SHSU course notes.

Proposed method

• Use LLaMa-2 70B as the base model with a fixed SHSU course notes knowledgebase.
• Implement a Retriever-Augmented Generation (RAG) system to provide context-aware responses.
• Provide a web-based GUI with options to upload additional study materials and secure SHSU credentialed access.
• Conduct a year-long stratified randomized controlled trial with 150 freshmen from SHSU Engineering Technology.
• Collect quantitative data (exam scores, topic assessments) and qualitative data (pre-/post-surveys, interviews, eye-tracking).
• Perform power analysis to justify sample size (80% power, alpha 0.05) and analyze learning gains and satisfaction as primary/secondary outcomes.

Figure 1 : Schematic Representation of Randomization and Data Collection Process for SAMCares Tool Evaluation

Experimental results

Research questions

• RQ1 Does access to SAMCares improve test scores and topic assessments compared with traditional materials?
• RQ2 What are the effects of SAMCares on student satisfaction and perceived cognitive load during learning?
• RQ3 How do eye-tracking metrics (engagement and fatigue) differ between SAMCares users and non-users?
• RQ4 What qualitative themes emerge regarding usability, perceived usefulness, and equity in AI-assisted learning?

Key findings

• No empirical results are reported in this protocol paper; the study aims to determine whether SAMCares improves learning outcomes and satisfaction.
• The central hypothesis is that SAMCares integration will yield greater knowledge gains, higher student satisfaction, and lower cognitive load than traditional methods.
• The protocol emphasizes rigorous data collection (exams, surveys, eye-tracking, interviews) and plans for de-identified data sharing and open-source release of SAMCares.
• The study acknowledges limitations such as single-site scope, potential misuse, infrastructure needs, and accessibility considerations for students with disabilities.

Figure 2 : Retriever-Augmented Generation (RAG) system for SAMCares : Process flowchart for generating context-aware responses using vector embeddings and LLAMA 2 70B model.