[Paper Review] Seeing ChatGPT Through Students' Eyes: An Analysis of TikTok Data

Anna‐Carolina Haensch, Sarah Ball+2 more|arXiv (Cornell University)|Mar 9, 2023

Artificial Intelligence in Healthcare and Education14 citations

TL;DR

The study analyzes the 100 most-liked English TikTok videos with #chatgpt (Feb 2023) to characterize themes, engagement, and educational implications from a student perspective.

ABSTRACT

Advanced large language models like ChatGPT have gained considerable attention recently, including among students. However, while the debate on ChatGPT in academia is making waves, more understanding is needed among lecturers and teachers on how students use and perceive ChatGPT. To address this gap, we analyzed the content on ChatGPT available on TikTok in February 2023. TikTok is a rapidly growing social media platform popular among individuals under 30. Specifically, we analyzed the content of the 100 most popular videos in English tagged with #chatgpt, which collectively garnered over 250 million views. Most of the videos we studied promoted the use of ChatGPT for tasks like writing essays or code. In addition, many videos discussed AI detectors, with a focus on how other tools can help to transform ChatGPT output to fool these detectors. This also mirrors the discussion among educators on how to treat ChatGPT as lecturers and teachers in teaching and grading. What is, however, missing from the analyzed clips on TikTok are videos that discuss ChatGPT producing content that is nonsensical or unfaithful to the training data.

Motivation & Objective

• Understand how students learn about ChatGPT from TikTok and what themes dominate the discourse.
• Assess engagement patterns (likes, shares, views) across promotional and critical content related to ChatGPT.
• Identify gaps in student knowledge and implications for teaching and academic integrity.

Proposed method

• Collect the 100 most-liked English-language TikTok videos with the hashtag #chatgpt collected on a single day (Feb 7, 2023).
• Record metrics: cumulative views, likes, shares, and creator IDs for each video.
• Develop and apply a manual content-classification scheme with iterative coder agreement (two authors; random sample checked for reliability).
• Use Cohen’s kappa to assess inter-rater reliability (reported as 0.896).
• Classify videos into major categories (Promotional, Critical, Business with ChatGPT, The future of society, AI tools list, Entertainment) with subcategories for the first two categories.

Figure 1: Play, like, and share count for all the selected videos.

Experimental results

Research questions

• RQ1 What themes and topics dominate the most-liked TikTok videos about ChatGPT among English-language content?
• RQ2 How do engagement metrics (likes, shares, views) compare between promotional and critical ChatGPT videos on TikTok?
• RQ3 To what extent is content related to academic pursuits present in these top videos, and how does this relate to educational practices?

Key findings

• 53 of the 100 videos are promotional and present ChatGPT positively.
• Average promotional video: 213,081 likes, 9,013 shares, 2,133,447 views.
• Average critical video: 312,417 likes, 7,935 shares, 2,828,875 views.
• Total promotional: 11,293,300 likes, 477,670 shares, 113,072,700 views.
• Total critical: 3,749,000 likes, 95,224 shares, 33,946,500 views.
• A Welch two-sample t-test showed no significant difference in means between promotional and critical counts, but promotional content accumulated far more engagement in total.

Figure 2: Play, like and share count for the selected videos, grouped by category.