[Paper Review] Large language model-powered chatbots for internationalizing student support in higher education

Achraf Hsain, Hamza El Housni|arXiv (Cornell University)|Mar 16, 2024

AI in Service Interactions4 citations

TL;DR

This study designs and evaluates a GPT-3.5 and GPT-4 Turbo-powered chatbot for higher education student support, leveraging Python, LangChain, and Chroma Vector Store to deliver real-time, accurate, and context-aware responses. The chatbot significantly outperforms traditional methods in user preference, response quality, and error rate, demonstrating strong potential for enhancing internationalization and digital transformation in higher education.

ABSTRACT

This research explores the integration of chatbot technology powered by GPT-3.5 and GPT-4 Turbo into higher education to enhance internationalization and leverage digital transformation. It delves into the design, implementation, and application of Large Language Models (LLMs) for improving student engagement, information access, and support. Utilizing technologies like Python 3, GPT API, LangChain, and Chroma Vector Store, the research emphasizes creating a high-quality, timely, and relevant transcript dataset for chatbot testing. Findings indicate the chatbot's efficacy in providing comprehensive responses, its preference over traditional methods by users, and a low error rate. Highlighting the chatbot's real-time engagement, memory capabilities, and critical data access, the study demonstrates its potential to elevate accessibility, efficiency, and satisfaction. Concluding, the research suggests the chatbot significantly aids higher education internationalization, proposing further investigation into digital technology's role in educational enhancement and strategy development.

Motivation & Objective

• To address the growing need for scalable, multilingual student support in internationalizing higher education institutions.
• To reduce reliance on traditional, time-consuming support channels by deploying AI-driven chatbots.
• To improve student engagement and satisfaction through real-time, context-aware, and accurate information retrieval.
• To evaluate the effectiveness of LLM-powered chatbots in handling diverse student inquiries with low error rates.
• To support institutional digital transformation by integrating advanced LLM technologies into student services.

Proposed method

• Developed a chatbot using GPT-3.5 and GPT-4 Turbo via the OpenAI API for natural language understanding and response generation.
• Implemented LangChain to orchestrate retrieval-augmented generation (RAG) workflows, enhancing response relevance.
• Built a Chroma Vector Store to index and retrieve high-quality, institution-specific transcripts for accurate context grounding.
• Constructed a curated, timely, and relevant transcript dataset to train and test the chatbot’s information retrieval capabilities.
• Integrated memory and context retention features to enable multi-turn, coherent conversations.
• Deployed the system using Python 3, enabling real-time interaction and dynamic response generation.

Experimental results

Research questions

• RQ1 How effective is an LLM-powered chatbot in delivering accurate and comprehensive responses to student inquiries in higher education?
• RQ2 To what extent does the chatbot improve user satisfaction compared to traditional student support methods?
• RQ3 What is the error rate of the chatbot in handling diverse, real-world student queries in an academic context?
• RQ4 How does the integration of retrieval-augmented generation (RAG) enhance response quality and relevance?
• RQ5 In what ways can such a chatbot support institutional internationalization and digital transformation efforts?

Key findings

• The chatbot demonstrated high efficacy in providing comprehensive and contextually relevant responses to student inquiries.
• Users preferred the chatbot over traditional support methods, indicating improved user experience and satisfaction.
• The system achieved a low error rate, confirming reliability in handling complex and varied queries.
• Real-time engagement and memory capabilities enabled sustained, coherent, and personalized interactions.
• The integration of Chroma Vector Store and RAG architecture significantly enhanced information retrieval accuracy.
• The chatbot proved effective in improving accessibility and efficiency of student support services in higher education.