Detecting LLM-Generated Web Fiction with "Classical" Machine Learning (AIGC Text Detection)简体中文 / [English]<br>Home<br>About<br>Writing<br>Category<br>Friends<br>Search

Previous post Next post Back to top Share post

1. TL;DR & Demo<br>2. Background (aka Useless Rambling)<br>3. Research Attempt – No Luck<br>4. A (Somewhat) Successful Attempt – scikit-learn SVM4.1. Data Generation<br>4.2. Training<br>4.3. JS Implementation for Web Demo<br>4.4. Testing Performance<br>4.5. Attack and Defense – Bypassing Detection(?)4.5.1. Classic Translation Method<br>4.5.2. LLM Prompt Method

5. Epilogue

This article is currently an experimental machine translation and may contain errors. If anything is unclear, please refer to the original Chinese version. I am continuously working to improve the translation.

TL;DR & Demo<br>As of early 2026, mainstream LLM-generated text exhibits strong statistical patterns that can be effectively distinguished from human-written content using traditional machine learning models. I suspect this is how many so-called “AI plagiarism checkers” actually work under the hood.<br>Online Demo: https://lyc8503.github.io/AITextDetector/<br>The model used in this demo is not trained on general-purpose data , nor has it undergone rigorous optimization or iteration. Its single-sentence detection accuracy is approximately 85% on the test set. Please read through this article before use to understand potential limitations.

The core code (drafts) and trained model files are available on GitHub: lyc8503/AITextDetector<br>Background (aka Useless Rambling)<br>Back when I was still writing my thesis at school half a year ago, rumors were already spreading about checking papers for AIGC (AI-generated content). I tested several platforms—CNKI, Wanfang, and some third-party AIGC detection services—and found they could indeed distinguish between my hand-written text and LLM-generated text with decent accuracy.<br>That sparked my curiosity about how AIGC detection actually works(and how to bypass it).<br>But I was juggling too many things at the time—obsessed with radio, Minecraft, Touhou—and after a few failed attempts, I shelved the idea.<br>Eventually, I faked my way through the thesis, and life moved on. But recently, while browsing Lofter, I stumbled upon entire tags flooded with low-quality, wildly out-of-character AI-generated fanfics.<br>How can I tell they’re AI at a glance? Well, some folks(or gals) don’t even bother cleaning up Markdown formatting or AI-generated section headers before posting—and then they slap half the article behind a paywall 😓<br>Most AI-generated texts, however, are harder to spot—they’re buried among diverse writing styles, varied prompts, and not immediately obvious. By the time you realize something feels off, it’s too late. Some texts are borderline impossible to prove as AI, leaving me paranoid. After swallowing a few too many AI-generated turds, I’d finally had enough. Lofter browsing stops here—time to open VS Code!<br>Yes, that’s how I ended up reviving my weekend project idea: building an AI-generated text detector…<br>Research Attempt – No Luck<br>The internet is now almost entirely polluted with ads when searching for AIGC detection. Every result is just another essay-AI-rewriting service. Back then, I dug through the noise and found something called text perplexity.<br>The idea is simple: use an existing LLM to estimate the probability of each word appearing in a given sentence. If nearly every word ranks high in the LLM’s predictions (Top-N), the sentence is likely AI-generated. Conversely, if many words are unexpected, it’s more likely human-written.<br>Sounds promising, right? I spent some time trying this method, but results were disappointing—plenty of false positives and false negatives, and no reasonable threshold could be set. Plus, there are practical issues: high inference cost, poor cross-model generalization, difficulty deploying large models locally, and closed-weight models being hard to integrate. Overall, this approach isn’t elegant or reliable.<br>Failed attempt—got tricked by a bunch of soft-ad "tutorials"<br>A (Somewhat) Successful Attempt – scikit-learn SVM<br>Since online resources were useless, back to old-school alchemy.<br>Scikit-learn, activate! Following its Roadmap, we can directly pick Linear SVC and Naive Bayes as good starting points for our classification task.<br>(Whisper: this also matched my gut feeling—LLMs have detectable word-choice patterns; even a Naive Bayes classifier should pick them up. I just didn’t expect the signal to be this strong.)<br>Data Generation<br>Old-school alchemy traditional classifiers need labeled data—so we need human-written texts and confirmed LLM-generated texts for training.<br>My approach: I pulled data I’d scraped in 2023 from a certain Ford-like and River-like platform, filtering for articles published between 2010–2022 (pre-ChatGPT). I only filtered out extremely low-engagement or very short pieces, then randomly sampled nearly 10,000 multi-thousand-character texts as human-written samples.<br>Then, I used an...