Thread

1. United857 · 1790012362 · focus · HN ↗

   There's a difference between automated industrial scale scraping and well-behaved agents acting on the behalf of individuals. Right now there isn't a robust, standard way to distinguish between them, so sites just block known datacenter IPs and throw out the baby with the bathwater.

   That's a main advantage of running your own local claw setup using your residential connection -- difficult/impossible to block.

   That said, eventually a site blocking all agents would be like blocking all search engines, something that hurts more than it helps as agentic interactions become "the norm". WebMCP or similar support will likely be a basic expectation at some point.

   1. dgellow · 1790013840 · focus · HN ↗

      > Right now there isn't a robust, standard way to distinguish between them

      There is, it’s called an API

   2. stephen_cagle · 1790013877 · focus · HN ↗

      It's worse than that, they will block all agents and specifically make exceptions for maybe 2 search engines per country. Thereby strengthening existing players and weakening all incumbants.

      Cloudflare for instance does allow indexing by Google and Bing I think by default, but does challenge other bots. Double check me I am speaking from memory.

   3. oblio · 1790014085 · focus · HN ↗

      > That's a main advantage of running your own local claw setup using your residential connection -- difficult/impossible to block.

      <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;waf&#x2F;features&#x2F;bot-control&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;waf&#x2F;features&#x2F;bot-control&#x2F;

Unofficial Hacker News client; not affiliated with Y Combinator.