AI Detectors

A contact had sent him an interview request that he received from a Michael Chen, a reporter at a news site.  Johnston determined that the interview request was AI-written, that reporter Michael Chen was an AI-chatbot, that the news site was all AI-generated, and that it was funded by OpenAI to disseminate pro-AI “news.”

The post is worth reading, showing as it does how to get under the hood to see how these AI schemes operate.  What got my attention was Johnston’s first step:  “When I ran the message through Pangram, an AI content detector that credibly claims a near-zero false-positive rate, the email came back as “fully AI-generated.”

This introduced me to the world of AI detectors.  Check out Pangram.  It claims to be 99% accurate and offers special programs for teachers, universities, law firms, recruiters, publishers, and other entities that need to weed out AI plagiarism and fakery.  You can try it for free, entering a text at the website.  You can sign up for a free account, which allows you four uses per day, or get an individual account for $20 per month, or a professional account for $65 per month, allowing for more extensive, unlimited uses.  I would think a school for find that useful for all of their teachers.

It turns out, there are lots of AI detectors available.  Here is a review of over 30 of them by Anangsha Alammyan, with the top 6 in her judgement.  She usefully says which of these are best for various purposes.  I list them as a public service, but you’ll want to read her descriptions and explanations for each:

1. QuillBot: Best Overall

2. Winston AI: Most Reliable for Education & SEO

3. PangramLabs: Best for Writers & Publishers

4. Quetext: Best AI Detector with a Suite of AI Tools

5. Originality AI: Best for Students and Researchers

6. Humalingo: Most Consistent AI Detector Against Humanized Text

But a big caveat:  AI detectors are not always accurate!  The Wikipedia article on Artificial Intelligence Content Detection makes that clear.  There are two potential problems:  “False negatives” take place when the software says the text is not AI-generated, when it really was.  “False positives” take place when the software says the text is AI-generated, when it was actually written by a real person.  False negatives let AI plagiarism get by, but False positives are even more problematic because they can lead to false accusations.

Furthermore, the AI industry has developed “humanizing” technology to make AI-generated slop sound more like a human being.  Putting in a few grammatical mistakes, some colloquialisms, some inconsistency are “human” touches, so that a reader will assume that this is a communication from an actual person and helping the text get through the AI detectors.

Humalingo, #6 on Alammyan’s list, can be used both to “humanize” an AI text and to detect a humanized text! One use of these AI detectors, I gather, is to run your own original writing through them, just to make sure it doesn’t sound like AI, so as to get flagged as a false positive!  Thus, the grammar-check proofreading site Grammarly has added an AI-sounding check as a feature.

My impression, though, is that AI detection is getting better at the same time that AI is getting better.  It’s like an arms race:  One country develops a new means of attack.  The adversary country develops a new means of defending against it.  So the other country develops counter-measures to evade the defense.  So the adversary develops a defense against the counter-measures.  And so on, infinitum, giving us the history of military technology.  So for AI detectors, it’s important to stay up to date, which is why the subscription model makes sense.

If I were still in the classroom as an English professor, I think I might specify among the  criteria for my writing assignments that they must successfully go through an AI detector, thus proving that they were not AI-generated or that they do not have the characteristics of AI-generated writing.  If the assignment is flagged, that would show that the writing is not authentically original, either because it was AI-generated or because it is derivative in the same way that Large Language Models (by definition) are derivative.

Then again, I mustn’t surrender my vocation to technology in the name of countering technology.  I am a human teacher trying to reach human students.  What I need to assess is my students’ thinking, creativity, and insights.  I need to identify that distinctly human element in their work and bring that out and encourage them to cultivate it further, so that they distinguish themselves from machines.  When machines try to be like humans, it is especially important that humans try not to be like machines.