[論文レビュー] ChatGPT is Good but Bing Chat is Better for Vietnamese Students

Xuan-Quy Dao, Ngoc-Bich Le|arXiv (Cornell University)|Jul 17, 2023

Online Learning and Analytics被引用数 10

ひとこと要約

本論文は Vietnamese National High School Graduation Examination (VNHSGE) におけるChatGPTとBingChatを比較し、文学を除く大半の科目でBingChatが概して優れていると述べ、文学科ではChatGPTが優位であることを発見した。BingChatの優位性はGPT-4、ベトナムでの利用可能性、およびハイパーリンク/引用機能に起因すると説明している。

ABSTRACT

This study examines the efficacy of two SOTA large language models (LLMs), namely ChatGPT and Microsoft Bing Chat (BingChat), in catering to the needs of Vietnamese students. Although ChatGPT exhibits proficiency in multiple disciplines, Bing Chat emerges as the more advantageous option. We conduct a comparative analysis of their academic achievements in various disciplines, encompassing mathematics, literature, English language, physics, chemistry, biology, history, geography, and civic education. The results of our study suggest that BingChat demonstrates superior performance compared to ChatGPT across a wide range of subjects, with the exception of literature, where ChatGPT exhibits better performance. Additionally, BingChat utilizes the more advanced GPT-4 technology in contrast to ChatGPT, which is built upon GPT-3.5. This allows BingChat to improve to comprehension, reasoning and generation of creative and informative text. Moreover, the fact that BingChat is accessible in Vietnam and its integration of hyperlinks and citations within responses serve to reinforce its superiority. In our analysis, it is evident that while ChatGPT exhibits praiseworthy qualities, BingChat presents a more apdated solutions for Vietnamese students.

研究の動機と目的

• Assess the efficacy of two state-of-the-art LLMs (ChatGPT and BingChat) in supporting Vietnamese students across multiple subjects via the VNHSGE dataset.
• Compare performance of the two models against Vietnamese student performance on the VNHSGE.
• Evaluate whether BingChat can substitute ChatGPT for education in Vietnam.

提案手法

• Evaluate ChatGPT and BingChat on a VNHSGE-based evaluation set covering mathematics, literature, English, physics, chemistry, biology, history, geography, and civic education.
• Note that BingChat uses GPT-4 while ChatGPT uses GPT-3.5, affecting comprehension, reasoning, and generative capabilities.
• Analyze results against human Vietnamese student performance and discuss accessibility and citation features (hyperlinks).

実験結果

リサーチクエスチョン

• RQ1 RS1: What is BingChat’s performance on the VNHSGE examination?
• RQ2 RS2: How does BingChat’s ability compare to ChatGPT’s ability on the VNHSGE examination?
• RQ3 RS3: How does BingChat’s ability compare to Vietnamese students’ ability on the VNHSGE examination?
• RQ4 RS4: Can BingChat replace ChatGPT in education in Vietnam?

主な発見

• BingChat generally outperforms ChatGPT across most subjects on the VNHSGE dataset.
• ChatGPT performs better than BingChat only in literature.
• BingChat’s advantages are linked to GPT-4, availability in Vietnam, and provision of hyperlinks for sources.
• In subjects requiring heavy computation (mathematics, physics, chemistry), both models show limited performance relative to human students, though BingChat is competitive in some cases.
• BingChat’s ability to access online information and cite sources is highlighted as a practical educational advantage.