[论文解读] The Utility of Large Language Models and Generative AI for Education Research

Andrew Katz, Umair Shakir+1 more|arXiv (Cornell University)|May 29, 2023

Topic Modeling被引用 9

一句话总结

本文展示了一种工作流，使用文本嵌入、聚类和生成式语言模型（例如 GPT-3.5）对超过 1,000 份工程教育论文进行主题分析，展示了可扩展定性分析的归纳与演绎标注方法。

ABSTRACT

The use of natural language processing (NLP) techniques in engineering education can provide valuable insights into the underlying processes involved in generating text. While accessing these insights can be labor-intensive if done manually, recent advances in NLP and large language models have made it a realistic option for individuals. This study explores and evaluates a combination of clustering, summarization, and prompting techniques to analyze over 1,000 student essays in which students discussed their career interests. The specific assignment prompted students to define and explain their career goals as engineers. Using text embedding representations of student responses, we clustered the responses together to identify thematically similar statements from students. The clustered responses were then summarized to quickly identify career interest themes. We also used a set of a priori codes about career satisfaction and sectors to demonstrate an alternative approach to using these generative text models to analyze student writing. The results of this study demonstrate the feasibility and usefulness of NLP techniques in engineering education research. By automating the initial analysis of student essays, researchers and educators can more efficiently and accurately identify key themes and patterns in student writing. The methods presented in this paper have broader applications for engineering education and research purposes beyond analyzing student essays. By explaining these methods to the engineering education community, readers can utilize them in their own contexts.

研究动机与目标

• Demonstrate how NLP and large language models can be used to analyze unstructured text data in engineering education at scale.
• Showcase an inductive, data-driven approach to theme identification via embeddings and clustering.
• Demonstrate a deductive labeling approach using pre-defined O*NET SOC and career-satisfaction labels.
• Evaluate the usefulness and feasibility of automating initial qualitative analysis to inform teaching and program development.

提出的方法

• Convert student essays to sentence-level text for high-resolution analysis.
• Embed sentences with a pre-trained transformer encoder (Sentence Transformers mpnet) to obtain semantic vectors.
• Cluster sentence embeddings with agglomerative clustering to identify thematically similar statements.
• Use GPT-3.5 to generate 15-word summaries for each cluster to distill core themes.
• Cluster the 15-word summaries to consolidate major themes.
• Prompt GPT-3.5 to generate 3–5 word labels for each final theme cluster to form an initial codebook.

实验结果

研究问题

• RQ1 Can embeddings and clustering identify semantically similar statements to reveal underlying themes in student career-interest essays?
• RQ2 Can a generative language model generate meaningful, concise theme labels and classifications for unlabeled text?
• RQ3 How effective are inductive vs. deductive labeling approaches (with O*NET SOC and career-satisfaction codes) in annotating engineering education writing?
• RQ4 Can the model self-assess labeling accuracy to reduce human curation effort?

主要发现

• NLP techniques can feasibly identify key career-interest themes from over 1,000 student essays.
• A multi-step inductive workflow yields thematically labeled codes that capture major patterns in student writing.
• Deductive labeling using O*NET SOC codes and career-satisfaction factors can apply predefined labels to student text via prompting.
• The model can perform accuracy checks by rating label applicability, correlating with human judgments in tested examples.
• Automation reduces initial qualitative analysis time while preserving themes relevant to teaching strategies and program planning.