Hacker News

I run some small websites, including a tiny forum that’s been a goldmine for scrapers. I had to significantly tweak some firewall rules and configuration after scrapers behind residential proxies suddenly accounted for over 99% of requests.

However, I also relaxed rules for automated traffic that was well-behaved, and I went out of my way to ensure that the Wayback Machine was able to hit everything. I should kick a small donation their way. They provide an incredibly valuable service and I love the benefit that I get from them just for personal side projects.

This is absolutely something that's happening. There are even paid scraper API's that offer "Wayback Machine fallback" as a feature.

We used to always "scrape" the wayback machine for any sort of news article we actually paid to consume. I was absolutely shocked by major news sites making very important edits to an article without any sort of editorial notice!

Sadly this sort of thing is probably not really possible anymore, but I can't really blame anyone for making this sort of decision. I can't imagine how much more traffic they get now vs 2021 when we were doing this.

Just yesterday from my one of my sessions with Sol:

> Hacker News and the Rails forum are blocking the text fetcher, so I'm using the browser workflow to inspect the pages directly

I wonder if the entire internet is going to slowly move behind logins and allow lists for specific trusted crawlers at some point.

Open access doesn't seem sustainable.

But I might just grumpy about spending another hour this week adjusting rules to prevent bots.

I've personally been using the Wayback Machine more often because I increasingly find myself being blocked from websites who are trying to keep out scrapers even though I'm just a regular person with JS disabled (along with a bunch of other stuff)

Nearly every time a link is posted to HN to a site behind some form of wall, a high voted comment on the post will be a link to an archive site bypassing the owners wall. Bot owners are not the only ones routinely circumventing the choices of content owners.

Do they offer bulk torrent downloads as an alternative?

You might be right but -- why would they have watied until these recent weeks?

What sites would they be targeting? Generic "just give me anything"? Whenever I check regular sites on IA, the coverage is spotty -- they'll have the homepage and a few important pages, but it quickly fizzles out.

Very understandable, you can't store all 15000 pages of any random website and update them etc etc, but that makes them pretty useless for indirect scraping because you usually don't want a tiny taste, you want everything.

My use of wayback has skyrocketed recently due to anti-bot measures.

I often cannot get past captchas, and archive.org is one of the fallbacks I try.

However, archive.is, etc are more reliable.

I wish the internet archive acted more like a library system, where multiple organizations could mirror the content.

They are a big single point of failure, and I’m shocked Trump/SCOTUS haven’t intentionally burnt the archives down yet.

> I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior.

Appalling, yes. But also expected. I'm surprised they haven't been the target of scrapers for years. But sites putting their content behind login walls and other anti-bot mechanisms has certainly exacerbated this. But again, this isn't at all a surprising progression.

> we've also already seen some sites opt out of the Wayback Machine to prevent their content from being scraped via this alternative route.

To be fair, another big motivation was likely users on sites like HN using archive.org (and similar sites) to get around their paywalls. In fact, I'd be surprised if this wasn't a big motivator.

Again, it sucks, but it's not at all surprising to see it progress like this. I wouldn't be surprised to see similar blocks on other archive sites eventually.

Perhaps it would be sensible for the wayback machine to not show paywalled articles for the first, say, 3 months.

It is. They will most likely eventually need to move to a walled model for Wayback due to scraper aggressiveness (like Reddit deprecating anonymous old.reddit.com), or behind Cloudflare for aggressive bot and scraping protection. Hard to defend against abuse of a public resource when its intent is public access with as little restriction as possible.

https://en.wikipedia.org/wiki/Tragedy_of_the_commons

(no affiliation)

Nah, there's actual value in hitting historical versions and with agents the gap between "how long has this product been offered by this company" and "I should go to wayback machine and do a binary search to find the earliest snapshot that contains this product offering " has closed.

I say name and shame!

I donate to them every year b/c I fully agree they’re doing a thankless critical job very well.

I've been donating $5 to them monthly for I don't know how long. I've only recently bumped it up to $25. They're the heroes of the internet age

fully expect them and wikipedia to get assaulted by AI companies to monopolize data source access in the near future.

The future is bleak :\

I assume that's because the IP range of your company network overlaps with a range used by some scrapers, and if it doesn't happen on your phone even in the corporate network, then IA probably checks some extra signals like the user agent in addition to the IP

Your workplace is probably redirecting traffic through a datacenter IP range. Especially if they have their own datacenters like google, microsoft, oracle, amazon, etc.

Try making a vpn via digital ocean for example and you'll see similar patterns.

My buddy said he could not access it even from a residential IP, it was blacklisted for some reason.

Could be due to some scrapers from either your work ISP block, or the larger block which lends IPs to multiple workplaces.

They seem to be aggressively blocking IPv6 source IPs. I ran into this problem over the past month traveling. I got nothing but 429 errors until I switched on my VPN (which is IPv4 only) and magically the Wayback machine worked again. The lack of transparency on the part of IA is very frustrating.

I'm a little more skeptical that this is "AI is big so it is worse" issue. Yes, AI is big in scale, but this has been the case for almost every popular free service. They either start:

- charging (news / journalist services)

- gate-keeping (X forcing log-ins)

- enshittifying (lots of ads and degraded service)

The fact that the way back machine is incredibly useful but most people didn't know about it or use it very much doesn't change the fact that it has basically become very popular... only with LLM agents rather than humans. Ads alone aren't enough to support human traffic for many sites with human traffic.

Torches and pitchforks?

It's not. There is no evidence, just propaganda.

People will just bring their bots over and you'll be back to square one. Bots scraping existed before LLM companies decided to go nutso on the internet.

Gatekeeping information is not the solution

the root of the root of all evil: NAT

How exactly does IPv6 "stop crawlers".

If anything, it will make it harder to block due to the vastness of the IPv6 address space.

If you get something, it isn't a donation.

> Shouldn't the solution be to gate bulk access for automated services for a price?

Fine in theory but determined scrapers will use residential proxies in bulk.

Presumedly they collect that, but the vast majority of blocks are correct and not errors. This info allows them to look up the user’s request to label it as legitimate.

Plenty of threads on HN about this, Anubis does not work.

They get marked as inaccessible, but still exist in IA data

See also: https://wiki.archiveteam.org/index.php/List_of_websites_excl...

Note that Archive Team is separate from the Internet Archive.

Yes. It's impossible to tell which because the split is residential proxies, dataset curators, and AI companies all being separate actors. However I fucking guarantee you it's out there and people are too cowardly to be honest about it so they don't get sued out of existence.

Oh yeah, absolutely.

Not while the new dukes of the internet wielding massive armies of hijacked smart TVs have a better time browsing the web than I have; as a mere peasant with just a few IP addresses. There would be no reason to sign up and pay up for bulk access, unless open access is shut down.

- Find some new way for Cloudflare to acquire paying customers. If their business did not depend on the status quo, they would be exceptionally well positioned to roll out the technical & organizational frameworks that that make massive botnets a thing of the past.

Yeah it’s kinda crazy to me that what was once a back alley python script is now accepted as ‘fine, free for all’. The new era of bros really are smth else.

As a public resource the hope is for Wayback to be free to access. I imagine putting up a paywall would be their last resort.

S3 price gouges on bandwidth.

I think that'd raise serious copyright concerns, if the Wayback machine started selling other people's intellectual property.

> It serves nobody except CloudFlare and hardware companies when one side set up blockers and the other side spend money putting VPN SDKs in consumer TVs

The problem with this perspective is that it ignores the victimization which is happening to all sorts of sites right now.

On one hand, you have content owners/suppliers which are trying to place restrictions on how much free bulk use is allowed.

When scrapers go to exotic lengths to evade the blocks, eg by using thousands of ephemeral IP addresses to collect an entire corpus, saying stuff like that makes it sound like it's all a wash.

"Oh, what a silly situation... How did we ever end up like this? It's not good for anyone ..."

No, there is a victim trying to defend themselves from rampant theft of resources, and a corporate asshole which doesn't care about the effects of their actions.

Sites would just block the Internet Archive crawler as well.

Micropayments would solve so many Internet problems. It's not too late to adopt.

Content creators could charge by page instead of depending on malware/ad/surveillance revenue. Spam is cut if there's a charge per mail. Scraping abuse goes away, along with a bunch of DDOS garbage.

The impact is a few cents per page or mail, negligible for a human. But if you're consuming a trillion pages per day, you'd reconsider.

My guess? Because even with a paid endpoint, the type of unscrupulous yahoo that is DDOSing IA today would probably still abuse the free endpoints because they can. The revenue that might come from a paid endpoint could help to scale up, but with how slow IA usually seems, I suspect there is an upper limit to how much traffic they can serve without a LOT more revenue.

This is a major "this is why we can't have nice things" situation in my opinion. IA is one of the most valuable gems of the Internet. The only thing that even comes close to preserving our shared history. The damage being caused (both by the effective DDOSing and by the knock-on impact that abuse has in encouraging publishers to remove their content from the archive) is incredibly serious.

> Why not just offer a paid endpoint for the crawlers?

Because then you're definitely violating US copyright law. There are four prongs of fair use analysis, and one of them is the "nature of the use." In this case, you'd be turning into a commercial use.

It’s one thing to archive other companies content, it’s another to sell the access to it

> Asking users to email them with details of their OS, browser, IP address is just crazy.

It's surely to serve as data to help tell humans apart from bots.

> Changes made by IA shouldn't become my responsibility.

They're a free service. It's ultimately not their responsibility to service you either.

OTOH I do appreciate their openness. It beats those times when I try visiting a site, only to get a cryptic 403 error or similar and no suggestion the site would like to hear from me.