The Fingerprint of ChatGPT: DNA-GPT is a GPT-Generated Text Detection Method Using Divergent N-Gram Analysis

ChatGPT has become an essential part of our daily lives at this point. Most of us use it daily to solve mundane tasks or get guidance on how to tackle complex problems, get recommendations about decisions, etc. More importantly, AI-assisted writing has become the norm for the majority, and we even started to see the effects already as companies started to replace their copywriters with ChatGPT.

While GPT models have proved to be useful assistants, they have also introduced challenges, such as the proliferation of fake news and technology-aided plagiarism. Instances of AI-generated scientific abstracts deceiving scientists have led to a loss of trust in scientific knowledge. Therefore, it looks like detecting AI-generated text will become crucial as we progress further. However, it is not straightforward as it poses fundamental difficulties, and the progress in detection methods lags behind the rapid advancement of AI itself. 

Existing methods, such as perturbation-based approaches or rank/entropy-based methods, often fail when the token probability is not provided, as in the case of ChatGPT. Additionally, the lack of transparency in the development of powerful language models poses an additional challenge. To effectively detect GPT-generated text and match the advancements of LLMs, there is a pressing demand for a robust detection methodology that is explainable and capable of adapting to continuous updates and improvements.

So, at this point, the need for a robust AI-generated text detection method is increasing. But, we know that LLMs advance faster than the detection methods. So, how can we come up with a method that can keep up with the advancement in LLMs? Time to meet DNA-GPT.

DNA-GPT addresses two scenarios: white-box detection, where access to the model output token probability is available, and black-box detection, where such access is unavailable. By considering both cases, DNA-GPT aims to provide comprehensive solutions. 

DNA-GPT builds upon the observation that LLMs tend to decode repetitive n-grams from previous generations, while the human-written text is less likely to be decoded. The theoretical analysis focuses on the possibility of AI-generated text in terms of true positive rate (TPR) and false positive rate (FPR), which adds an orthogonal perspective to the current debate on detectability.

The assumption is that each AI model possesses its distinctive DNA, which can manifest either in its tendency to generate comparable n-grams or in the shape of its probability curve. Then, the detection task is defined as a binary classification task, where given a text sequence S and a specific language model LM like GPT-4, the goal is to classify whether S is generated by the LM or written by humans.

DNA-GPT is a zero-shot detection algorithm for texts generated by GPT models, catering to both black-box and white-box scenarios. The effectiveness of the algorithms is validated using the five most advanced LLMs on five datasets.  Moreover, the robustness of the algorithm is tested against non-English text and revised text attacks. Additionally, the detection method provides the capability for model sourcing, enabling the identification of the specific language model used for text generation. Finally, DNA-GPT includes provisions for providing explainable evidence for detection decisions.

Check Out The Paper and Github. Don’t forget to join our 23k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we missed anything, feel free to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club