[论文解读] Towards an optimised evaluation of teachers' discourse: The case of engaging messages

Samuel Falcón, Jaime León|arXiv (Cornell University)|Dec 18, 2024

Discourse Analysis in Language Studies被引用 5

一句话总结

本文提出了一种通过在真实课堂转录数据上训练两个大型语言模型来识别和分类具有吸引力的教师话语的方法，以优化对教师参与性话语的评估，达到较高的灵敏性和特异性，并分析不同年级和年份的消息类型的变化。

ABSTRACT

Evaluating teachers' skills is crucial for enhancing education quality and student outcomes. Teacher discourse, significantly influencing student performance, is a key component. However, coding this discourse can be laborious. This study addresses this issue by introducing a new methodology for optimising the assessment of teacher discourse. The research consisted of two studies, both within the framework of engaging messages used by secondary education teachers. The first study involved training two large language models on real-world examples from audio-recorded lessons over two academic years to identify and classify the engaging messages from the lessons' transcripts. This resulted in sensitivities of 84.31% and 91.11%, and specificities of 97.69% and 86.36% in identification and classification, respectively. The second study applied these models to transcripts of audio-recorded lessons from a third academic year to examine the frequency and distribution of message types by educational level and moment of the academic year. Results showed teachers predominantly use messages emphasising engagement benefits, linked to improved outcomes, while one-third highlighted non-engagement disadvantages, associated with increased anxiety. The use of engaging messages declined in Grade 12 and towards the academic year's end. These findings suggest potential interventions to optimise engaging message use, enhancing teaching quality and student outcomes.

研究动机与目标

• 提高评估教师话语效率，减少编码劳动。
• 开发一种从课堂转录中识别并分类参与性信息的方法学。
• 在跨多个学年的真实音频记录课堂数据上验证该方法。

提出的方法

• 在两学年的真实课堂转录数据上训练两作用于识别参与性信息的两个大型语言模型。
• 通过报告的灵敏度和特异性来衡量模型在识别与分类方面的表现。
• 将训练好的模型应用于第三个学年的转录数据，以分析按教育水平和学年时间划分的消息类型的频率与分布。

实验结果

研究问题

• RQ1 大型语言模型能否从课堂转录中准确识别教师话语中的参与性信息？
• RQ2 在识别和分类参与性信息方面，模型的灵敏度和特异性是多少？
• RQ3 参与性信息的类型在教育水平和学年中的时点上有何变化？

主要发现

• 第一项研究在识别方面达到84.31%的灵敏度、在分类方面达到91.11%的灵敏度，特异性分别为97.69%和86.36%。
• 关注参与度的好处并与改善结果相关的参与信息普遍存在。
• 约三分之一的信息强调与增加焦虑相关的非参与性的不利因素。
• 在第三年的数据中，12年级的参与性信息下降并且在学年末出现下降趋势。