> Here’s what’s going on. The Internet Archive’s Wayback Machine has been hit by waves of high-volume automated traffic, and we’ve put protections in place to keep the service running.
I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior.
In addition to the load it puts on this vital non-profit piece of Internet infrastructure, we've also already seen some sites opt out of the Wayback Machine to prevent their content from being scraped via this alternative route.
We used to always "scrape" the wayback machine for any sort of news article we actually paid to consume. I was absolutely shocked by major news sites making very important edits to an article without any sort of editorial notice!
Sadly this sort of thing is probably not really possible anymore, but I can't really blame anyone for making this sort of decision. I can't imagine how much more traffic they get now vs 2021 when we were doing this.
But anyway, no, I wouldn't keep finding reasons. I donate to them every year already. Somebody asked if I would be willing to pay and my answer was "yes, but".
It would need to be improved because certain aspects of it suck right now, not only the error this post is about. They only need go as far as their forums and github repos to see the community feedback.
Not at all. I donate to Internet Archive, but we're talking here about paying for unobstructed access to but one part of their service: Wayback Machine. Two different things.
100% of the people who write "I would pay, if..." or "I would pay, but..." are people who are never going to pay even a dime. You might be the exception, and sorry for bunching you up with them. You have paid already by donation.
I think that we should all pay when asked for things which we find useful, even if they aren't perfect. If nobody else is offering anything, then we have to take what's being offered. When there's a market, more providers will begin offering their versions.
I was thinking the same thing... paid access for high volume users or scrapers could actually help fund the non-profit. Maybe let website owners decide which scrapers are allowed to use their content, or allow them to get paid for use of it. If news and other sites were getting paid, maybe they could go back to optimizing for good content instead of clicks.
I think that would get into murky water really quickly with the rights holders (/content creators) not exactly being thrilled the Wayback Machine is essentially monetizing their IP behind their back.
Interesting, in all the years I have never noticed that IP has 2 meanings (well probably more...) Yeah, I am an engineer and usually try to avoid the legal BS. Although I hate that AI has made stealing legal if you are big enough.
It wouldn't be behind their back, like I said, "Maybe let website owners decide which scrapers are allowed to use their content, or allow them to get paid for use of it."
They already provide this service at <a href="https://archive-it.org/archive-it/" rel="nofollow">https://archive-it.org/archive-it/ though for some reason they don't seem to publicize it. Some info at <a href="https://help.archive.org/help/archive-it-information/" rel="nofollow">https://help.archive.org/help/archive-it-information/ as well.
I mean we already paid for the article from the source itself. I guess I'd expect a better "diff" source from them, but if they dont even update the article itself, i guess i wouldn't expect a paid service to have those updates either?
ah, i think i misunderstood your original post. if you mean the wayback-machine/arkive, then I suspect it'd be hard to justify? You are essentially paying a third-party source to validate that diffs didn't go through on the source material.
with llms, at some point it probably becomes easier to use your paid api connection to manage your own cached version yourself?
simonw · · focus · HN ↗
I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior.
In addition to the load it puts on this vital non-profit piece of Internet infrastructure, we've also already seen some sites opt out of the Wayback Machine to prevent their content from being scraped via this alternative route.
pantsforbirds · · focus · HN ↗
Sadly this sort of thing is probably not really possible anymore, but I can't really blame anyone for making this sort of decision. I can't imagine how much more traffic they get now vs 2021 when we were doing this.
subarctic · · focus · HN ↗
msephton · · focus · HN ↗
carlosjobim · · focus · HN ↗
msephton · · focus · HN ↗
But anyway, no, I wouldn't keep finding reasons. I donate to them every year already. Somebody asked if I would be willing to pay and my answer was "yes, but".
It would need to be improved because certain aspects of it suck right now, not only the error this post is about. They only need go as far as their forums and github repos to see the community feedback.
jakderrida · · focus · HN ↗
Their reply is 100% based on the content of your post.
Dylan16807 · · focus · HN ↗
carlosjobim · · focus · HN ↗
msephton · · focus · HN ↗
carlosjobim · · focus · HN ↗
I think that we should all pay when asked for things which we find useful, even if they aren't perfect. If nobody else is offering anything, then we have to take what's being offered. When there's a market, more providers will begin offering their versions.
bonestamp2 · · focus · HN ↗
DaSHacka · · focus · HN ↗
g-b-r · · focus · HN ↗
usr1106 · · focus · HN ↗
usr1106 · · focus · HN ↗
bonestamp2 · · focus · HN ↗
mitxela · · focus · HN ↗
progval · · focus · HN ↗
bee_rider · · focus · HN ↗
cloakley · · focus · HN ↗
pantsforbirds · · focus · HN ↗
pantsforbirds · · focus · HN ↗
with llms, at some point it probably becomes easier to use your paid api connection to manage your own cached version yourself?