> Here’s what’s going on. The Internet Archive’s Wayback Machine has been hit by waves of high-volume automated traffic, and we’ve put protections in place to keep the service running.
I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior.
In addition to the load it puts on this vital non-profit piece of Internet infrastructure, we've also already seen some sites opt out of the Wayback Machine to prevent their content from being scraped via this alternative route.
Unfortunately I sometimes have to browbeat Claude into acting like an agent of the user is supposed to. Usually it works, though last time it refused to recognize my moral argument (on the grounds that it's not bound to my interests exclusively and needs to protect the interests of its maker too).
Do you think there should be a way for a site to tell an agent it isn't allowed access? I'm not sure where I land on this exactly tbh, so no judgment cast.
Has it been settled whether robots.txt applies to user-driven chat sessions and if things like the crawl delay should be applied to say an end-user, an ip address, a harness provider, etc? My understanding is robots.txt is more for training exclusions, but less so for agent work.
AI bros think they should be exempt from robots.txt. Administrators of big services beg to differ. No solid consensus has arisen. I bet it's gonna take a lawsuit or two to see how it shakes out.
If a new directive was introduced that allows for an explicit setting in robots.txt, do you think the bros would follow it anyway? Something like `ALLOW AGENTS` or `DISALLOW AGENTS`
I wouldn't want them to. The whole point of using agents to do stuff on the web for me, is for them to do the stuff on the web for me.
This is the reverse of "do not track" case. It'll not be effective because every service will set it to DISALLOW by default anyway, because it costs them nothing, and for most services, it actually is what they want anyway - most of businesses on the web are making money on wasting people's time, and for that, they need to force themselves on people; end-user automation defeats that, so they actively fight it (and complain a lot).
wget ignores robots.txt outside of recursive mode. I think it's correct to do so, and I think an AI loading a handful of pages in response to a command should be about the same.
robots.txt applies (or should, in my opinion) to anything that automatically follows a link. Basically any software that is not a human-controlled web browser or single-shot curl command. Everything else: robot.
robots.txt was only intended to help search index crawlers not get stuck in endless crawl loops for badly designed websites.
What you suggest is explicitly not a purpose of robots.txt per RFC9309[1]:
"These rules are not a form of access authorization."
HTTP 429 and HTTP 403 are what servers are meant to return to clients to slow them down or tell them to stop doing something without having first gained authorisation.
simonw · · focus · HN ↗
I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior.
In addition to the load it puts on this vital non-profit piece of Internet infrastructure, we've also already seen some sites opt out of the Wayback Machine to prevent their content from being scraped via this alternative route.
bradly · · focus · HN ↗
> Hacker News and the Rails forum are blocking the text fetcher, so I'm using the browser workflow to inspect the pages directly
TeMPOraL · · focus · HN ↗
Unfortunately I sometimes have to browbeat Claude into acting like an agent of the user is supposed to. Usually it works, though last time it refused to recognize my moral argument (on the grounds that it's not bound to my interests exclusively and needs to protect the interests of its maker too).
bradly · · focus · HN ↗
aaron_m04 · · focus · HN ↗
bradly · · focus · HN ↗
xena · · focus · HN ↗
ghaff · · focus · HN ↗
recursive · · focus · HN ↗
TeMPOraL · · focus · HN ↗
I wouldn't want them to. The whole point of using agents to do stuff on the web for me, is for them to do the stuff on the web for me.
This is the reverse of "do not track" case. It'll not be effective because every service will set it to DISALLOW by default anyway, because it costs them nothing, and for most services, it actually is what they want anyway - most of businesses on the web are making money on wasting people's time, and for that, they need to force themselves on people; end-user automation defeats that, so they actively fight it (and complain a lot).
xena · · focus · HN ↗
Dylan16807 · · focus · HN ↗
bityard · · focus · HN ↗
dhx · · focus · HN ↗
What you suggest is explicitly not a purpose of robots.txt per RFC9309[1]:
"These rules are not a form of access authorization."
HTTP 429 and HTTP 403 are what servers are meant to return to clients to slow them down or tell them to stop doing something without having first gained authorisation.
[1] <a href="https://datatracker.ietf.org/doc/html/rfc9309#section-1" rel="nofollow">https://datatracker.ietf.org/doc/html/rfc9309#section-1
mitxela · · focus · HN ↗