[Paper Review] Quantitative Analysis of AI-Generated Texts in Academic Research: A Study of AI Presence in Arxiv Submissions using AI Detection Tool

Arslan Akram|arXiv (Cornell University)|Feb 9, 2024

Artificial Intelligence in Healthcare and Education6 citations

TL;DR

The paper builds a dataset from physics, mathematics, and computer science arXiv submissions and evaluates an AI-detection method, reporting 98% accuracy on the detector.

ABSTRACT

Many people are interested in ChatGPT since it has become a prominent AIGC model that provides high-quality responses in various contexts, such as software development and maintenance. Misuse of ChatGPT might cause significant issues, particularly in public safety and education, despite its immense potential. The majority of researchers choose to publish their work on Arxiv. The effectiveness and originality of future work depend on the ability to detect AI components in such contributions. To address this need, this study will analyze a method that can see purposely manufactured content that academic organizations use to post on Arxiv. For this study, a dataset was created using physics, mathematics, and computer science articles. Using the newly built dataset, the following step is to put originality.ai through its paces. The statistical analysis shows that Originality.ai is very accurate, with a rate of 98%.

Motivation & Objective

• Motivate the need to detect AI-generated components in scholarly submissions to ensure originality and integrity.
• Create a dataset of arXiv articles across physics, mathematics, and computer science to test detection methods.
• Evaluate the effectiveness of an AI-detection approach on the constructed dataset.
• Assess the accuracy and reliability of the detection tool in identifying AI-generated content.

Proposed method

• Construct a dataset of arXiv papers from physics, mathematics, and computer science.
• Apply a newly built AI-detection method to the dataset via an HTTP URL pipeline.
• Perform statistical analysis to assess the detector's accuracy.
• Report accuracy metrics and qualitative observations from the results.

Experimental results

Research questions

• RQ1 Can the proposed AI-detection method reliably identify AI-generated content in arXiv submissions?
• RQ2 How does the detector perform across disciplines (physics, mathematics, computer science)?
• RQ3 What is the overall accuracy of the detection tool on the constructed dataset?
• RQ4 What are the limitations and potential sources of error in the detection approach?

Key findings

• The study uses a dataset drawn from physics, mathematics, and computer science articles.
• The AI-detection pipeline was tested on arXiv submissions through an HTTP URL.
• Statistical analysis indicates the detector achieves high accuracy on the dataset.
• The reported accuracy of the method is 98%.
• The paper includes 6 figures and 1 table to illustrate results.