> Here’s what’s going on. The Internet Archive’s Wayback Machine has been hit by waves of high-volume automated traffic, and we’ve put protections in place to keep the service running.
I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior.
In addition to the load it puts on this vital non-profit piece of Internet infrastructure, we've also already seen some sites opt out of the Wayback Machine to prevent their content from being scraped via this alternative route.
It is. They will most likely eventually need to move to a walled model for Wayback due to scraper aggressiveness (like Reddit deprecating anonymous old.reddit.com), or behind Cloudflare for aggressive bot and scraping protection. Hard to defend against abuse of a public resource when its intent is public access with as little restriction as possible.
As someone who operates a large non-profit public data driven website, I have some VERY strong feelings about scrapers. We looked into various commercial solutions (Datadome, HUMAN) and based on our traffic estimates from logs we'd be looking at at least 250k/yr for bot mitigation. Anubis is offering a temporary reprieve, but after reading the recent kernel.org article [0] it's increasingly clear that this is a temporary bandaid.
The cheapest solution is to require a login and rate limit by API key. I also have strong feelings about the tragedy of the commons.
simonw · · focus · HN ↗
I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior.
In addition to the load it puts on this vital non-profit piece of Internet infrastructure, we've also already seen some sites opt out of the Wayback Machine to prevent their content from being scraped via this alternative route.
toomuchtodo · · focus · HN ↗
<a href="https://en.wikipedia.org/wiki/Tragedy_of_the_commons" rel="nofollow">https://en.wikipedia.org/wiki/Tragedy_of_the_commons
(no affiliation)
ronsor · · focus · HN ↗
On the other hand, the Internet Archive is a non-profit offering a free public resource.
toomuchtodo · · focus · HN ↗
itintheory · · focus · HN ↗
The cheapest solution is to require a login and rate limit by API key. I also have strong feelings about the tragedy of the commons.
[0] <a href="https://people.kernel.org/monsieuricon/creepy-crawlies" rel="nofollow">https://people.kernel.org/monsieuricon/creepy-crawlies
toomuchtodo · · focus · HN ↗
<a href="https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=author%3Adang%20%E2%80%9Cstrong%20feelings%E2%80%9D&sort=byDate&type=comment" rel="nofollow">https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...