Hot take from a phd candidate who has spent way too many hours thinking about this:
everyone is focused on catching ai-generated content. the whole conversation is about “how do we detect it” and “what tools work.” but i think the bigger problem right now is false positives
heres what ive seen in my department alone this semester:
- 3 grad students accused of using ai, all cleared after review
- 1 professor’s published paper flagged by a new tool the journal adopted (published in 2019, before chatgpt existed)
- an international student whose second-language writing patterns triggered false positives repeatedly
the human cost of false positives is enormous. accusation alone can damage academic careers, create anxiety, and disproportionately affect non-native english speakers
i feel like the detection industry has a incentive to overreport rather than underreport. a tool that says “probably fine” doesnt sell as well as one that says “67% ai detected!”
am i off base here?
You’re not off base at all. I’ve raised this exact concern with my department. The disproportionate impact on non-native English speakers is particularly troubling and has been documented in several studies. Students writing in a second language often produce text that pattern-matches to AI output because they use simpler structures and common phrases.
The incentive problem you identified is real. Detection tools are selling fear, and a tool with a high false positive rate actually seems more thorough to buyers who equate “more flags” with “better detection.” It’s perverse.
Seeing this at the high school level too. the most vulnerable students - ESL students, students with learning differences who use writing aids, students who’ve been taught formulaic essay structures - are the ones getting flagged most often
i stopped using detection tools entirely after one of my best ESL students came to me in tears. now i focus on knowing my students writing and having conversations about their work. its more work but its actually accurate
The empirical data supports your observation. Liang et al. (2023) found that GPT detectors disproportionately classify non-native English writing as AI-generated, with false positive rates up to 61.3% for TOEFL essays. That’s worse than random.
The commercial incentive structure you describe is well-documented in other detection domains. Drug testing companies historically faced the same criticism: high sensitivity sells better than high specificity even when false positives cause more harm than false negatives.
@JonahHex99 that story about your ESL student breaks my heart. and you’re right - the students who need the most support are getting punished by these tools
@Marc_Delrieu 61.3% false positive rate on TOEFL essays is staggering. thats literally worse than a coin flip. how is any institution making decisions based on tools with that kind of performance
Not a teacher or researcher but as a student this whole thread validates what ive been feeling. the anxiety of submitting work knowing a broken tool might flag you is genuinely affecting how i write. and not in a good way. im making my writing worse on purpose to avoid detection which is the exact opposite of what education should do