I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).
Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.
It's sad, but this train has left the station a quarter of a century ago imo. Many people have since become billionaires scraping the web without anyone agreeing (Google, Yahoo, and now possibly Anthropic and OpenAI).
There is a huge difference between a clearly identifiable, robots.txt-respecting, largely symbiotic search engine spider versus this new AI-driven casual sociopathy of firing shotguns of deliberately masked bots at sites in ways which benefit no one except for, maybe, the bot-owner.
I agree with you, though, that the future has almost certainly already been decided, and see the eventual outcome of all this selfishness to be mandatory device attestation to access most services on the internet—which will of course never be allowed on any open platform. AI engineers who believed in freedom to compute (or just individual freedom more generally) have committed perhaps the greatest self-own in the history of our species to date.
dvt · · focus · HN ↗
Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.
frabcus · · focus · HN ↗
dvt · · focus · HN ↗
csnover · · focus · HN ↗
I agree with you, though, that the future has almost certainly already been decided, and see the eventual outcome of all this selfishness to be mandatory device attestation to access most services on the internet—which will of course never be allowed on any open platform. AI engineers who believed in freedom to compute (or just individual freedom more generally) have committed perhaps the greatest self-own in the history of our species to date.