Hacker News

Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.

The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".

(a) Prohibition.—A person may not—

(1) deploy, direct, or cause to be deployed, a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service; or

(2) intentionally misrepresent, conceal, or obscure the nature of the bot to appear as a human user for use in connection with a generative AI model or service.

(6) STEALTH BOT.—The term “stealth bot” means a bot that accesses, retrieves, scans, indexes, scrapes, or otherwise interacts with a website, digital platform, or online service without prior disclosure of its identity and purpose, in particular by—

(A) failing to identify itself including through a valid and accurate user-agent string; or

(B) failing to disclose the specific nature and purpose of the bot, including any proposed use and purpose of page content (including text and data mining, search indexing, inferencing, artificial intelligence development, support, or operations (such as training, fine tuning, retrieval augmented generation)) or other automated processing at the time access is requested and in a format that the website operator can access.

A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.

You are entitled to your own emails, posts, etc but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.

It's a human level distinction that doesn't have technical distinction—the law handles these kinds of things just fine. The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.

The world you envision where we decide that the law can't assess intent is certainly an interesting one but I don't think it would be the most functional.

A browser does not send 10k requests per minute to open a webpage. Even in the most and modern bloated webpages. And even if it did it wouldn't be to a single website.

A user operating a scraper or a crawler IS operating a scraper or a crawler.

The legal system isn't stupid - it doesn't think that a human using a web browser is a robot.

I mean it is stupid but not in that way.

What good is blocking the ones in your jurisdiction when the internet is global and then they just operate from somewhere else?

> Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.

How is geofencing supposed to help? The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country and you ultimately end up blocking primarily real humans who don't all know how to change their apparent geolocation.

> A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question.

There is a major way to distinguish them. If you pass a law requiring them to do something, journalists from major publications have legal departments who then order them to follow the law (and then can't do their journalism), whereas the bad actor is either not interested in following laws or is not in your jurisdiction to be subject to them to begin with.

> A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country.

Eh?

a) Window shopping for the next time the out-of-country visitor is in the country.

b) An out-of-country visitor can contract someone who's in the country to purchase things from the shop and ship those things to them.

c) Queries and requests from out-of-country visitors can cause the shop to notice that setting things up so that they can ship internationally actually would be a money-making thing for them to do.

I do agree that the obscenely antisocial bots run for the benefit of the LLM Manufacturing Industry make it very hard for small-time operators to keep their websites up and running at any reasonable cost. Hopefully the major LLM Manufacturers will fail to secure enough funding over the next few months, their VC-backed primary customers will then have their money faucets shut off, and the attack bots will reduce down to a dull roar because folks have stopped paying "infinity" dollars for them.

I think it’s far more likely the internet reverts to its 1998 state: hobby sites which don’t expect payment. Will the quality of some sites go down? Sure, but there are so many people on the internet with niche interests that I think we still end up with plenty of content. Reviews, music, discussion, news, etc. If anything this change represents less centralisation. Maybe it’ll be more confusing to know where to find certain discussion or reviews on a particular product, but this is why search engines were invented. We kind of forgot that.

So like peer to peer? What incentive is there to keep seeding?

Sure, but would that be one that contains services and apps ordinary people use everyday, or would they all have jumped ship like Netflix did? That version would still be an open web, just a very different one.

> trawlerbots use smart TV residential proxies and will copy someone else’s UA.

What bugs me the most are the ones that cycle through a list of LLM user agents.. What are you actually? You can't be Baidu and Bing at the same time.

> My experience with a crawler is that people don’t actually provide a robots.txt far more often.

I think many don't see the point of robots.txt as it will just be ignored by most bots anyway, so instead concentrate on more complicated measures.

Those would fall under the "stealth" bots subject to a fine.

The whole point is that if you can prove a company is doing this kind of shit, they should be on the hook for it. Of course, I'm not expecting any company in China to actually pay the fines, but this opens up some options such as seizing foreign assets.

As others point out, the anime girl must exist because otherwise the sites can’t afford to serve you, amidst a new pattern of DDoS that is attributed to AI but I see as just DDoS becoming more prevalent (remember how sites used to occasionally be taken down by DDoS? Before Cloudflare (and Anubis) they’d just have to stay offline until the DDoSers got bored of wasting money for no good reason. Today’s “scrapers” apparently have new reasons that make them never bored of wasting money, but cannot be good enough to justify the aggressiveness that makes them DDoSers).

Hence why my solution is, ironically, micropayments. That would be so cheap that any human could even access thousands of webpages per day for pennies (or however expensive sending a signal across wires is…it can’t be much), including shared dumps of scraped data from billions of pages.

When I opened the article above I was asked to subscribe to a newsletter. I dont know whether to laugh or cry.

And Anubis: it is uncomfortably close to erotica, in a way that makes me panic when it pops up while I'm at the office. I wouldn't care if it was a picture of a puppy or something, but not an anime catgirl.

You seem quite incensed by the anime girl of Anubis. Unfortunately for you it is here to stay for a while, along with cloudflair's implementation and others, as they do reportedly make a difference to the sites using them as shields. Until something better is found, the other option is that the resources otherwise serving more bot traffic than they can afford simply go dark which would be worse.

Equating bot prevention measures with “please give us permission for 1,000s of companies to stalk you around the Internet, no? Here are some dark patterns to try make you accidentally agree” is rather unfair IMO. They are an inconvenience but for the protection of the sites rather than for the betterment-of-adtech-via-conning-you-the-viewer-with-malicious-compliance.

> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit

They could, but given these methods seem to work currently most (almost all?) are not. They will start doing so if the amount of resources protected in this way increases to the point where it is worth it, or course, that is why memory-hard processes are being worked on which will make that route less practical.

> cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits

I can't disagree there, though that would be quite an inefficient process and requires a bunch of money up front from who-ever champions the legal challenge, so I doubt it would actually be at all practical.

What's the alternative? Invisible algorithmic fuzzy recognition on the server side that just randomly bans you if you look like a bad bot, with no recourse? At least if you solve a captcha and dismiss the cookie banner you can access the website.

> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.

Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.

> Another web is possible but web populists don't really believe it.

How? Sharing knowledge with the world used to be an obviously good thing. Now it's a way to help tech companies replace people with machines. What alternative web could change this state of things?

>You can already buy shares in a nations economy - it is called owning currency.

An interesting point! Yes, that is definitely one way that currency could be looked at...

>The big difference from stocks is that most countries issue new shares (dollars, etc) to keep the price stable (indirectly via careful control of interest rates).

Agreed -- but if we were really to analyze things, scrutinize the way things really work at the highest degree of resolution, then we would have to observe that:

a) There are multiple countries/currencies in the world.

b) That each country's/currency's inflation rate is different than all of the other countries/currencies.

c) If each country's/currency's inflation rate is different, this means that each countries's issuance of new shares (and control of interest rates) -- their financial policies, are, by comparison, either more effective, on par, or less effective than those of other countries used for comparison...

Phrased another way:

All countries do not control the rate of inflation of their currencies equally.

Phrased another way:

Some nations are better at controlling their currency's rate of inflation than other nations.

True or not true?

>I am simplifying terribly

As am I! :-)

>but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.

Again, an interesting point!

And yes, I could agree with that!