Thread

Show HN: Training a model to identify AI web content from structure alone

74 points · 27 comments · jochenmadler

Loading the complete thread in the background. This saved snapshot is available now. Refresh

jochenmadler · 1790082211 · focus · HN ↗
Hey HN! We’re Vincent and Jochen from Sitefire (<a href="https://sitefire.ai">https://sitefire.ai). We have been working together for years, with backgrounds in RL/optimization at Stanford and software engineering from Technical University Munich (TUM).
With Sitefire (YC W26), we help marketing teams get recommended by AI Search (ChatGPT, Google AI Overviews, AI Mode, Claude, etc.). Our software monitors prompts, sees which web pages get cited, and uses these insights to help marketing teams take action, e.g. create YouTube videos or write the right blog posts.
This means we have a commercial stake in AI-generated web content. And for now, high-information, AI-generated content works great to get cited and recommended in AI Search.
But after talking to hundreds of marketing teams, it became clear that everyone despises AI-generated content (“AI slop”). And yet, everyone still wants to leverage AI to create content. So we asked ourselves: what characterizes AI slop? Can we train a model to identify it from human-generated web pages?
Researchers from the University of Maryland and Google DeepMind already asked this question for fiction. Their paper StoryScope (Russell et al., 2026) showed that you can tell AI-written stories from human ones by their structure alone, without looking at the words.
We ported their pipeline to commercial web pages. Using the Wayback Machine, we collected 2,250 blog posts from 268 B2B company websites that were written before ChatGPT existed. For each blog post, five AI models (GPT-5.4, Claude Sonnet 4.6, Gemini 3 Flash, DeepSeek V3.2, Kimi K2.5) wrote their own version.
Instead of looking at the words, we looked at how each post is built. We had an AI model answer 214 questions about every post, e.g. how hard it pushes its own product, whether it backs up its claims with sources, or whether it quotes a named expert. Then we trained a classifier on these answers.
On blog posts it had never seen before, our classifier told AI-generated and human posts apart with 98% accuracy, getting only 19 of 1,740 wrong. Why does it work so well? Because all five AI models write in a similar shape. Mapping every AI model’s values for these features, we see they cluster together, while the human values sit apart and spread out much more. Of the 1% most unique blog posts in our data set, 149 are human, only 4 are AI.
So what characterizes AI slop? It tells you the same thing three times. The title already promises what you'll get ("How to Cut Onboarding Time in Half"), the intro lays out what's coming, and the ending says it all again. 77% of the AI posts end by repeating their main point, compared to only 12% of the human posts. We call it the tidy, self-announcing blog post.
Still, each AI model has its own accent. We trained a second classifier to tell which of the five AI models wrote a post, or whether a human did. It picks the right author 79% of the time, where random guessing (1 in 6) would get 17%. Almost all of its mistakes are mix-ups between the AI models, not between human and AI. The cool thing about structural features is that you can't simply reword your way out of it. We had each AI model rewrite its own posts until, on average, 73% of their original 13-word sequences were gone, and the AI slop classifier still worked just as well.
We're building this into Sitefire: our agents get a structural understanding of text, so the posts they write go deeper and vary the way human writing does. There's a lot we haven't tested yet, like the myriad of humanizer tools, human rewriting, restructuring a post, or prompting an AI model to explicitly avoid these habits. And our human posts are mostly from 2020 to 2022, while the AI posts were generated in August 2026. Structure can't really tell when a human post was written, but it's still not a same-year comparison.
We published the study with all the figures on arXiv: <a href="https://arxiv.org/abs/2609.15369" rel="nofollow">https://arxiv.org/abs/2609.15369. The code is on GitHub: <a href="https://github.com/pulse-energy-eu/slopshape" rel="nofollow">https://github.com/pulse-energy-eu/slopshape
We're pretty sure your own blog isn't AI slop, is it? We built a checker that runs one of your posts through the ten features from the paper, so you can see for yourself (the full report asks for a work email): <a href="https://sitefire.ai/slop-checker">https://sitefire.ai/slop-checker.
Think you can tell AI slop from human writing? We also made a little game to see if you can keep up with our model, which gets all five rounds right: <a href="https://sitefire.ai/spot-the-slop">https://sitefire.ai/spot-the-slop.
asdff · 1790101547 · focus · HN ↗
The idea is interesting but in looking at methods and github I feel the tooling leaves me wanting. I mean you are trusting LLMs here to establish, vet, and detect your various thresholds that were then used to train the classifier. I'd rather see this sort of thing done deterministically with actual code vs lossy human english prompts and a dependency on token spend to a single third party (who will probably pull the underlying model used in what a few short years probably) to replicate the results or try and use different training data.
1. jochenmadler · 1790178850 · focus · HN ↗
  [dead]
pooploop64 · 1790101778 · focus · HN ↗
Is the goal of this to help AI pick corn kernels out of it's own shit for the purposes of slightly raising the bar on how sloppy the slop is? Or are you trying to trick AIs into eating a higher amount of their own shit than they already are? This feels like a factory built specifically to manufacture pollution.
1. tomhow · 1790209774 · focus · HN ↗
  We've banned this account.
blackboxdev · 1790102287 · focus · HN ↗
[dead]
1. dang · 1790109291 · focus · HN ↗
  Can you please not post AI-generated or AI-edited comments to HN? It's not allowed here - see <a href="https://news.ycombinator.com/newsguidelines.html#generated">https://news.ycombinator.com/newsguidelines.html#generated and <a href="https://news.ycombinator.com/item?id=47340079">https://news.ycombinator.com/item?id=47340079.
  Of course, it's impossible to know for sure what was LLM processed or not, but some of your posts (like this one) have been getting classified that way.
  1. jumploops · 1790111115 · focus · HN ↗
    Funnily enough, that comment, when passed to the OP's slop detector[0], returns "80% human"
    It's very clearly AI-generated, and thus a bit ironic (:
    [0]<a href="https://sitefire.ai/slop-checker/r/UkRAuqW1y-1T2qhbRQfLt2Jq7no">https://sitefire.ai/slop-checker/r/UkRAuqW1y-1T2qhbRQfLt2Jq7...
    pooploop64 · 1790112592 · focus · HN ↗
    Maybe they ran it through OP's de-sloppifier to see how well it works. And look how well it did! The slop detector doesn't suspect a thing! I can already smell the value this thing is adding to the world.
c7b · 1790107200 · focus · HN ↗
Would be cool to compare your model to what you would have gotten with Jev. It's wild that it seems possible that a universal classifier could compete with a specifically-trained one (but would be curious).
bryanrasmussen · 1790111872 · focus · HN ↗
The idea is that AI slop is non-creative boring crap, to really determine if you are able to identify AI slop then it should be determined if you misidentify human slop as AI slop.
Idea 1: Identify the worst most boring human marketing, organizational, bureaucratic texts from the a time before AI was writing it, anonymize this text to make sure there is no reference to current events that can be used to determine that it is not AI. And then see if the AI will say hey, that is not AI slop.
Idea 2: Have people parody AI slop. Can it determine the parody is still not AI slop?
1. beepbooptheory · 1790113677 · focus · HN ↗
  [delayed]
  1. bryanrasmussen · 1790141341 · focus · HN ↗
    I don't think I said anything about word level?
    The 12% of human posts that do what AI do in repeating things, are they human slop?
    I personally don't think it is possible to identify human and AI, it is however probably more possible to identify unoriginal and boring quality writing and art.
    This was perhaps not clear from my first post, as I tend to imply points rather than tediously stating them.
    beepbooptheory · 1790157570 · focus · HN ↗
    [delayed]
    bryanrasmussen · 1790181159 · focus · HN ↗
    see I think it's like finding a paper that does certain comparative work between different kinds of apple pie that says it can determine if apple pie was made by a German, and I said for me to really trust your claims of being able to tell if the pie was made by a German these are the ways I would expect you to test on non-German people who are widely agreed to cook like German people.
    Because if you are not testing on people that are supposed to be the most German-like in their cooking then your ability to find the German cooking in a city renowned for its Thai food is not that impressive.
    Then in response to your first question I posit that actually it is not possible to identify if pie was made by Germans but probably relatively easy to identify if it was made by people who cook in a German manner. And maybe that is actually more beneficial.
    I realize that from communicating with people over the years that things which seem crystal clear to me may seem opaque to others, but I think your analogy is somewhat unfairly structured.
    Note: apologies to German cooks and their cooking, although I personally only like currywurst. But I needed something to make the analogy more like what I felt had been communicated, and you were pseudo-randomly picked.
    beepbooptheory · 1790220476 · focus · HN ↗
    [delayed]
2. qurren · 1790123403 · focus · HN ↗
  Personally I don't actually care if it's human slop or AI slop. If it's slop it's slop and I don't want to read it.
  The amount of slop on the internet was on the rise well before AI, actually.
3. jochenmadler · 1790179058 · focus · HN ↗
  [dead]
4. bryanrasmussen · 1790181725 · focus · HN ↗
  somebody named JochenMadler says below that this was essentially the study (for some reason their comment is dead, not sure why)
  I agree it is somewhat close to the study, but we are not sure, because it is not known how much is really boring dull marketing copy. In choosing blog posts from pre-AI times I suppose you might have difficulty finding the worst examples, and might accidentally get higher quality work.
  >Using the Wayback Machine, we collected 2,250 blog posts from 268 B2B company websites that were written before ChatGPT existed
  Not sure what metric was used to determine these 2250 blog posts? But there are certainly a lot of ways they can select higher quality posts by accident.
  1. bryanrasmussen · 1790185797 · focus · HN ↗
    On getting domains that are Human slop.
    You anonymized domains of pre-AI sources, are there any domains that had an excessive number of "telling you the same thing three times" or other AI tells among them?
    Of the percentage that was misidentified, do they come from any sources in particular?
    If you get a lot of content from these sources and run against the model do they perform worse? If they do how do they perform with word choice detectors? I would expect that structural slop is related to word choice slop among humans.
    Anyway these are things I would be interested in as being the point where AI slop rubs up against the human slop which it learned from.
5. bryanrasmussen · 1790182740 · focus · HN ↗
  On the point of Idea 1, I missed the point where they try to make sure the text is not in the training set, which is good, but the anonymization it mentions is I think only website anonymization, the anonymization is I think more like stuff like
  "President Bush in the state of The Union last month said"
  is a statement that should only have been written in a factual document during the Pre-AI era, enough of those and the AI might turn that into math that says Reference to X as being current means NOT AI where X is a range of things that nowadays can only be referred to as the past, except in fiction.
BobbyTables2 · 1790134228 · focus · HN ↗
Some people are just very robotic.
Today I spent 20 minutes in a call to customer support. The representative spoke so robotic-ally (both cadence and word choice), I was extremely confident they were an AI for the first 15 minutes. Was quite certain the automated AI had transferred me to a more advanced one…
Turns out they were just likely in a low cost Latin American country and trying really hard. I felt bad one could be reduced to such…
1. userbinator · 1790137844 · focus · HN ↗
  Probably more like "some jobs are just very robotic". CSRs aren't paid to think for themselves, but rather to follow a script. I suspect that human would behave a lot more like a human outside of work.
  1. BoxOfRain · 1790161183 · focus · HN ↗
    Yeah I worked in an inbound call centre one summer as a student, and I disliked it for that reason. You'd get angry customers yelling at you going 'why can't you do anything other than your script' and you're just sacked if you do anything else. You develop a skin as thick as a hippo and just plough on with it, since you're not actually empowered to do much that would help the customer.
    People were all very human the second the phone was put down, though everyone smoked like chimney myself included. You kind of have to if you're being screamed at all day by random members of the public. Also to this day I am extremely polite and patient with people in inbound call centres! I suspect the world would have infinitely better manners if everyone was required to do a year in some customer service role.
2. FL410 · 1790138402 · focus · HN ↗
  Oh it gets worse. Apparently Comcast is now using a voice-changing AI - it's a real (probably Latin or Asian) person getting their voice replaced on-the-fly to sound "American."
  1. BobbyTables2 · 1790207017 · focus · HN ↗
    It might have been that!
    The cadence was extremely robotic and consistent. Only thing that makes me think it wasn’t AI was the very end when they repeated a confirmation code and didn’t use the NATO names for the letters the second time around.
  2. ranger_danger · 1790326222 · focus · HN ↗
    I know it's more trendy to bash them for this... but I honestly struggle a lot to understand call center people with thick accents from other countries, and I often find myself wondering "am I racist for wanting to speak to a native right now" just because I cannot make out what they are saying.
DylanMerigaud · 1790144764 · focus · HN ↗
Structure alone can differentiate AI content, interesting approach.
1. jminnl · 1791021229 · focus · HN ↗
  Except - detail - it doesn't work...
evantbyrne · 1790169867 · focus · HN ↗
Immediately saw false positives on content written before ChatGPT. Not difficult to see how the methodology is wrong when it considers restating the thesis in the conclusion to be signal.
1. jochenmadler · 1790178319 · focus · HN ↗
  Can you elaborate on the false positives?
  Re methodology: Restating the thesis is one signal of many. 77% of the AI posts close that way, but so do 12% of the human ones. The classifier in the paper never decides on one feature, but their combination.
  1. evantbyrne · 1790210282 · focus · HN ↗
    Human writing was flagged as AI generated by the tool with dubious explanations given. Other human written posts were marked down substantially. I mentioned one feature, which seems to be double counted to mark down essays a whopping 20%, but none of the features listed on the results page gave me confidence in the classifier. Are you sure this is a slop detector and not just an essay grader with reverse scoring?
sofiarossi98 · 1790242560 · focus · HN ↗
[dead]
Brenndalampe1 · 1790251772 · focus · HN ↗
[dead]
eshaanpm · 1790260890 · focus · HN ↗
[dead]
katyarailabs · 1790316652 · focus · HN ↗
The "tells you the same thing three times" finding rings true. That's the structure every content-marketing course taught long before LLMs, and the models seem to have learned the most average version of it.
What I like about structural features ial, not just a detector. If you fine-tune a model to write for your brand, "did the new version get less sloppy than the one we're running?" is a measurable question with this. Disclosure: I work on Ookami (github.co open-source layer for serving andfine-tuning models. It only puts a fine-tuned model live if it beats the current one on evaluators you plug in. Since slopshape is open sourceof those evaluators: fine-tune on yourbest human posts, and the gate rejects any version whose posts score more AI-shaped than the live one. I haven't tried it yet, but it's the kinde for. One question for you: were the 19 misclns writing to SEO templates (listicles,"ultimate guides")? If so, the classifier might be detecting "written to a template" more than "written by a model", which is arguably the more us
nazbeisenovna · 1790622858 · focus · HN ↗
[dead]
jminnl · 1791021073 · focus · HN ↗
So, on my 100% hand written blog it comes up as 50% 'likely human written with AI assistance'. Back to the drawing board I guess.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.