Counter Turing Test (CT2): AI-Generated Text Detection is Not as Easy as You May Think – Introducing AIDetectabilityIndex

By Megha Chakraborty et al

Read the original document by opening this link in a new tab.

Table of Contents

Abstract
1 Proposed AI-Generated Text Detection Techniques (AGTD) – A Review
1.1 Watermarking
1.2 Perplexity Estimation
1.3 Burstiness Estimation
1.4 Negative Log-Curvature (NLC)
2 Design Choices for CT2 and ADI Study
2.1 LLMs: Rationale and Coverage
2.2 Datasets: Generation and Statistics
3 De-Watermarking: Discovering its Ease and Efficiency

Summary

With the rise of prolific ChatGPT, the risk and consequences of AI-generated text has increased alarmingly. This paper introduces the Counter Turing Test (CT2), a benchmark consisting of techniques aiming to offer a comprehensive evaluation of the robustness of existing AGTD techniques. Our empirical findings unequivocally highlight the fragility of the proposed AGTD methods under scrutiny. The paper also introduces the AI Detectability Index (ADI) as a measure for LLMs to infer whether their generations are detectable as AI-generated or not. The study discusses de-watermarking experiments and design choices for the CT2 and ADI study.