Hacker News

Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users.

> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.

There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.

Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds.

I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...

PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".

> The anubis author has stated they recognize it's an arms race, but PoW scales.

The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.

The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.

And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.

It’s a neat concept, but the answer and future to my eyes look bleak.

I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.

Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.

At least anubis works for me. (I run umatrix)

Unfortunately whatever HN is using routinely blocks my login with "Sorry."

some websites just always give me 403.

Well, we don't use a captcha either. If it were a choice between a captcha and a proof of work system, we'd have to reevaluate things. Luckily, for now, we're able to get away with a much lighter touch.

> but PoW scales

Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.

> becoming much more obvious and easy to block, or they have to use massive amounts of compute.

If you believe this, please contact me: I think compute is free[1] and can probably help you out.

[1]: https://news.ycombinator.com/item?id=30175269

Or you can go full Reddit and just block anything that seems even remotely suspicious.

Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too.

Using VPN? Blocked.

Your iPhone is too old, blocked.

Your screen brightness too low? Believe or not, blocked.

> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.

You can't do that any more. Too many ISPs, especially mobile carriers, don't hand out anything resembling a fixed IP address any more. It's CGNAT and constantly changing IP addresses alllll the time now.

PoW can theoretically scale effectively infinite because it can mine cryptocurrency. Millions of compromised IoT devices hitting your server? Now you have enough money for a faster server.

It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?

Edit: the article says millions of times per hour? (!?)

The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?

Ironically in early 2023 a lot of websites went out of their way to block Common Crawl. Unsurprisingly that shifted scraping toward individual actors whereas the previous solution in research was to download CC dumps and process them.

I'm sure there are those who would participate, either because they want their data to be captured by AI labs or as a form of compromise.

That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).

I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.

You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.

Feels like it would be a good time for freenet and the like to catch on.

The Bright Data “free” VPN they’re talking about requires the user to go through steps to enable it.

These aren’t as simple as downloading a free game and then the phone is compromised as long as it’s installed.

The users who install these things don’t care about permissions prompts. They’ll follow instructions to tap any prompt the instructions ask. They want the free thing and don’t care what they have to do to get it.

GrapheneOS allows you to deny network access per app pretty trivially. Google Play services make it a bit more difficult because the app might marshall the network request through that; I'm not sure how to verify that behavior when it happens.

A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.

> I think that nobody would care if I use wget or curl for few pages

If only you were the only one doing it...

How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?

just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.

Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.

Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!

I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).

I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.

Proof of work, even "custom", where the user does not need a particular interaction with the page, does not work. The scrapers are running headless Chrome and solving the work. They do not care, they do not pay the bill, the compromised system's owner pays the bill.

I have such system for the registration form on one of my website to prevent the double validation of emails to be used to spam emails of victims. The PoW challenge prevents less than 10% of the bots.

First of all I am very critical of the $100 per consumer device per month figure.

Also all major browsers block crypto miners on webpages now (for good reasons) so it may prove difficult to allow "good" mining scripts while still blocking "bad" ones.

I don't think this is a practical solution

Better skip the PoW part as it'll be wildly inefficient for most of the work done.

Instead, exchange web traffic for actual $. Say, some kind of tokens that are easily turned back into hard cash through a 3rd party.

Requesting a 100KB file? Okay, that'll be a $0.00002 token, please! (visitor's user agent provides it in a manner transparent to regular web users). Requesting a 3MB image? Okay, that'll be a $0.0005 token, please!

Result: niche websites earn hard cash. It doesn't matter much if you're hammered as long as the hammering comes with a corresponding flow of tokens (read: $). No token(s)? No service.

Regular web users would pay for those tokens through their normal internet service fees, and otherwise not be bothered. Massive scrapers would have to pay somehow for the tokens to be served web data at all.

In effect: put the bulk of public web sites behind a paywall. But with the bar low enough & in a manner that it's transparent for regular web users. Clicked "reload" by accident? Oops, internet service bill got upped by 2 micro-$.

Proof of work captchas are widely deployed, especially on more niche sites. Kiwiflare is one (used for a harassment forum)

Proof of work doesn't help if the abuser is massively decentralized and is using other peoples moneys.

They discuss why this might be a bad idea in the article.

And it's made necessary because another group of people thought that selling IP blocking services would be a good idea. One party sells walls, another party sells ladders.

Well, one party gives away free walls if you agree to fill your castle with surveillance cameras you don't control.

I think they also have to operate in countries that don't mind shady things like this.

This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.

TIL:

> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.

WTF. That's just botnets.

Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...

Thank god for residential proxies.

Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet

Opnsense has a traffic capture feature in the interface diagnostics menu, if you want to spot check what servers the devices are currently talking to.

Should be pretty obvious: client devices and internal services will have no traffic >95% of the time, just NTP for timekeeping, DHCP lease renewal, and associated ARP (running total: two dozen packets if you monitor them for a full 24h), then any system updaters (readily identifiable by the initial DNS requests), and finally of course you'll see the traffic of the service that the device hosts, if any, which can be easily dismissed by not looking at incoming connections (scraping uses outgoing connections)

> Tracking which devices still send lots of data when no one else is using the network?

That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).

[0]: https://github.com/jow-/nlbwmon

[1]: https://www.maxchernoff.ca/files/grafana-network-bandwidth.p...

[2]: https://docs.mossdef.org/adblock-fast/

I think all of these mitigations are unfortunate. They hurt one of the things that makes the web cool: it's a stable, stateless, idempotent way to access data.

This makes it a prime target for aggressive scraping by LLM companies, but it also makes it accessible and fast, and a prime target for benign use (like archive.org or "read later" services).

For my own sites, I'll eat the cost of the crawlers (mitigated by making the sites as efficient as possible) and keep them available to everyone.

> I'm guessing there's some egregious regulation that makes micropayments impossible.

More likely there isn't any kind of universal standard that's easy to implement for browser makers, has low overhead, and preserves internet users' anonymity as much as possible.

The currently existing friction of using micropayments is the problem here, I suspect.

Well the argument appears to be, people put them in their apps instead of ads. (Or more likely on top of ads.) The argument is money.

The users presumably don't know about this, or you know, they clicked, "I agree."

Nearly Half of LG Smart TV Apps Contain Residential Proxies

https://news.ycombinator.com/item?id=48635954

Of course the issue with blocking residential IPs is that then they would be prevented from doing normal things on the internet.

At which point, millions of people will be forced to complain to their local representatives and... hey presto? :)

Why would anyone who doesn't have a use for a residential proxy have an argument for residential proxies?

I use them to scrape closed sites to make the information more open. For example YouTube.

Blocking in ipv6 works roughly the same way as in ipv4, just that the scale is different. Instead of blocking something like a company's /24 or an ISP's /16 when they don't respond to abuse messages, you block the company's /48 or the ISP's /32. It'll vary per organisation how large a range they got exactly but you can see that in WHOIS. End users are no longer at a /32 (v4) but at /64 (v6), or some prosumers might have a /29 (v4) and /56 (v6). Same concept, just a different prefix length

> do you want random IoT devices talking to your web server?

Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing

So far I’ve only encountered one site that blocked automated access to its web feed. I assume that was just an oversight.

> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.

Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.

I don't think people sit around going "Grrrr who can I ban next?". Instead this stuff gets noticed because you see the webserver at 99% CPU utilization for 2 days straight, check the logs, and see you are somehow getting crawled by half the IPs in New York City.

If it weren't a real problem, these types of articles and services wouldn't exist.

Why the air quotes? Evading a ban and using (potentially ill gotten) residential ips to circumvent that refusal of service, is a bad actor.

Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.

I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.

"He who has the gold makes the rules" is older than the pyramids.

> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.

What exactly should be illegal here? Scraping websites? AI agents? Not following robots.txt?

I ran a gitweb server which was battered by bots so I eventually had to take it down. Gitweb! You can just connect using the git protocol and download everything vastly more efficiently!

In other words, they don't care at all. For them, residential bandwidth is completely free.

Right. This whole controversy makes no sense as a pure scraping thing. It seems more like someone is trying to take the web offline.

Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.

That does not explain why so many residential VPNs operate with so many IPs in countries where there are no social media bans. Here in France:

- i know many people who buy shady IPTV boxes from stores/markets for like 50€/year

- i know some people who use "smart lightbulbs" and other nonsense

- almost everyone i know plays free smartphone games, which as LWN reminded, may contain a shady SDK

> The moment it does you'll be able to tell because Bitcoin will crash to 0.

The "workaround" for PoW is running the PoW computation on hardware that's better suited for the task. Bitcoin mining has been using ASIC for many years now.

Let's say a legitimate user is willing to wait for one minute on a budget phone. Then your PoW is limited to what that phone can compute in one minute. But on the attacker's specialized hardware this computation only costs fractions of a penny, so they are barely hindered by it.

The SHA256 based PoW scheme has a very heavy ASIC advantage. People have tried to design PoW scheme that minimize the custom hardware advantage, but I'm not sure if they managed to close the gap far enough to make PoW feasible for this application.

Residential proxy users don't have the ability to run compute on their proxies.

I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.

We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.

https://news.ycombinator.com/newsguidelines.html

Edit: If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.

People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue. The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.

> Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.

If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network.

It doesn't seem like monopoly abuse to exclude such malware from application stores, just like key loggers or apps intercepting other apps network traffic without the user being aware of it (say the banking app's network traffic and password entry).

Noone complained we can't discriminate DC IPs, though to be fair some (imo bad) operators did just that. This is not even about preventing bots, which has perfectly legitimate usecases (eg. Internet Archive).

This is about filtering out bad bots/actors who have no respect for your resources and will drain all of it causing bad experience for everyone. But because they know they don't respect robots.txt or even simple rate-limiting, they have to employ so-called residential VPNs. They're residential in that they route through real user connections, and so you can't block the IP/subnet without dropping a certain amount of legitimate human-driven traffic.

Personal example: some time ago, i had to disable a wordpress plugin on a site that was causing 100% CPU usage on the whole box (hosting dozens of wordpress instances). That plugin was a simple calendar, but a bot was repeatedly scraping non-existent (or rather, "no event planned for this day") pages for every date in the calendar that you can represent in the DB timestamp, clearing the cache as it went to try and find new events for 1000 years ago. Whoever operates this IP space doesn't matter to me, i'd just like to block them because they don't respect robot.txt… but i can't because they use a "residential proxy" and will change IP address every hour or so.

This is definitely an issue, where data centre addresses are, by default, second class citizens or even persona non grata. The problem is that this reputation has been, unfortunately, well-earned.

There's no easy answer here. The ephemerality and pseudonymity of VPS address usage screams untrustworthy, and the only way to reign that in is better identification of who is using the VPS/address or significantly more restrictive rules applied to data/port usage. And I'm not sure if I like the general direction that points towards - away from the "open internet".