Thread

Stay discoverable in search while disallowing AI training

87 points · 50 comments · djfergus

1vuio0pswjnm7 · 1789532445 · focus · HN ↗
"Cloudflare classifies bots by behavior, and a single bot can exhibit more than one behavior."
Is that really true
CF classifies anyone not using a popular browser with Javascript enabled as a "bot"
CF fingerprints www users
As an example, look at CF's Permissions-Policy HTTP response header on a site with CF "bot protection", i.e., the "checking your browser" CAPTCHA nonsense (challenges.cloudflare.com). Then look at IA's Permissions-Policy response header. One CDN is advertiser-focused, the other is user-focused
IA = Internet Archive
1. devmor · 1789533488 · focus · HN ↗
  As far as I can tell, after months of fighting being DDoSed by Anthropic and OpenAI across 50+ sites - Cloudflare also allows what it considers "good bots" through all of your bot blocking rules, with no option to turn this off unless you pay them money.
  1. RobotToaster · 1789546108 · focus · HN ↗
    The "good bots" also just happen to be from companies that pay cloudflare a lot of money, I imagine.
    actionfromafar · 1789550138 · focus · HN ↗
    Goodness Tokens
  2. basilikum · 1789552654 · focus · HN ↗
    How did you dertmine it was those two companies? Also did you disallow them in robots.txt?
    devmor · 1789571194 · focus · HN ↗
    No, I definitely spent months fighting off bots across multiple hosts and never set up a robots.txt file anywhere nor did I look at the analytics dashboard on cloudflare.
    basilikum · 1789584592 · focus · HN ↗
    I'm asking sincerely, how do you get to the conclusion that it's those two?
    devmor · 1789591695 · focus · HN ↗
    To that point, Cloudflare quite literally tells you the source of the bot traffic on your sites that use it as a WAF! And what percentage they are allowing through, despite you setting rules specifically to block them by name.
    It’s baffling.
2. userbinator · 1789534544 · focus · HN ↗
  CF classifies anyone not using a popular browser with Javascript enabled as a "bot"
  This is my biggest complaint about CF. They are implicitly supporting user-agent discrimination in favour of Big Browser, instead of discriminating on actual behaviour.
  ...and of course there are already companies running tons of VMs with "officially sanctioned" browser + OS stacks, that can get past all these "protections", for a fee.
  "AI bots" is the newest boogeyman they came up with to take away freedom.
3. irregularbowels · 1789576629 · focus · HN ↗
  [dead]
4. [deleted] · 1789590239 · focus · HN ↗
  [deleted]
5. 1vuio0pswjnm7 · 1789698916 · focus · HN ↗
  11,9,10

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.