Testing of Detection Tools for AI-Generated Text

by   Debora Weber-Wulff, et al.

Recent advances in generative pre-trained transformer large language models have emphasised the potential risks of unfair use of artificial intelligence (AI) generated content in an academic environment and intensified efforts in searching for solutions to detect such content. The paper examines the general functionality of detection tools for artificial intelligence generated text and evaluates them based on accuracy and error type analysis. Specifically, the study seeks to answer research questions about whether existing detection tools can reliably differentiate between human-written text and ChatGPT-generated text, and whether machine translation and content obfuscation techniques affect the detection of AI-generated text. The research covers 12 publicly available tools and two commercial systems (Turnitin and PlagiarismCheck) that are widely used in the academic setting. The researchers conclude that the available detection tools are neither accurate nor reliable and have a main bias towards classifying the output as human-written rather than detecting AI-generated text. Furthermore, content obfuscation techniques significantly worsen the performance of tools. The study makes several significant contributions. First, it summarises up-to-date similar scientific and non-scientific efforts in the field. Second, it presents the result of one of the most comprehensive tests conducted so far, based on a rigorous research methodology, an original document set, and a broad coverage of tools. Third, it discusses the implications and drawbacks of using detection tools for AI-generated text in academic settings.


page 21

page 23

page 25

page 27

page 38

page 39

page 40

page 42


Game of Tones: Faculty detection of GPT-4 generated content in university assessments

This study explores the robustness of university assessments against the...

Detection of Fake Generated Scientific Abstracts

The widespread adoption of Large Language Models and publicly available ...

Playing with Words: Comparing the Vocabulary and Lexical Richness of ChatGPT and Humans

The introduction of Artificial Intelligence (AI) generative language mod...

Towards Understanding the Interplay of Generative Artificial Intelligence and the Internet

The rapid adoption of generative Artificial Intelligence (AI) tools that...

Will ChatGPT get you caught? Rethinking of Plagiarism Detection

The rise of Artificial Intelligence (AI) technology and its impact on ed...

Can AI Put Gamma-Ray Astrophysicists Out of a Job?

In what will likely be a litany of generative-model-themed arXiv submiss...

Towards an Understanding and Explanation for Mixed-Initiative Artificial Scientific Text Detection

Large language models (LLMs) have gained popularity in various fields fo...

Please sign up or login with your details

Forgot password? Click here to reset