Hacker News

LLM bots make only a part of traffic and really easy to set up (and I think this will eventually throttled by OAI or Anthropic themselves).

But there are bad actors - bots that try every possible urls and traversals and cross scripting and so on… I see it in our access logs. This cannot be solved otherwise than some greylisting/blacklisting on ISP level. E.g. this is so huge that we have started to block all Chinese inbound traffic. I am sorry for that because Deepseek and other good actors are probably blocked as well but IP rotation made it impossible to filter bad ones only… (and I suppose that DS uses EU or US addresses as well so they shouldn’t be blocked in full).

I get tens of thousands of IPs making 1 or 2 requests against pages on The Consensus (job search, code viewing, etc.). They're not super expensive pages. But I don't want to deal with the bot traffic (mostly because it messes up view counts I use to sell sponsorships). But then if I require an account (even a free one), valid readership is also going to drop with the new friction.

I could build something more complex to discover the patterns, or use Turnstile maybe. I'm not really sure at the moment.

Can't speak for everyone. I'm very gently scraping a lot of libraries for public domain periodical scans. The scans often exist nowhere else, or not in the same quality. Sometimes I find a certain library has perhaps the only known remaining intact copy of a periodical or a specific page.

The problem with the libraries is that all these scans are behind walls, so while the texts are public domain nobody can easily access them. I'm trying to liberate them and re-upload them outside the walls.

Until recently a lot of libraries would allow you to remotely sign up for a library card, but in the last 12 months almost all of them have reverted to having to bring ID in person to the library to validate your account, which puts the brakes on things.

We're currently in the middle of an AI goldrush. There are probably hundreds of companies trying to do model pretraining. Anthropic and OpenAI and so on probably care about optics and implementing their crawlers properly. But there will be many firms few of us have heard of. I would guess some of them are Chinese, for example.

It's 11 THOUSAND searches ON JUST THAT ONE TOPIC in one day when they are used to seeing less than 11 THOUSAND searches in one day TOTAL. None of these systems were built for this type of traffic. The total traffic from these crawlers is many times more than just 11,000. It's more like 11,000 searches on 100,000 topics.

The problem is unintentional DOS due to traffic can constitute a felony under the CFAA; the case in question being Pulte Homes, Inc. v. Laborers' International Union 2011.

After Pulte fired an employee represented by the union, LIUNA urged members to call and send email to the company, expressing their opinions. As a result of the increased traffic, the company's email system crashed. The Sixth Circuit ruled that the LIUNA's instruction to call and email "intentionally caused damage," reversing the lower court's decision.

Now thankfully we have Sandvig v. Barr, 2020 where the DC Federal Court ruled that the CFAA does not criminalize the violation of a website's terms of service, but imo all bets are off in terms of interpretation of the law in the wild west of AI.

Agreed, this just reads like the universities have no idea how to actually serve data effectively. Just bad architecture and lazy or incompetent administrators.

It was 11,000 searches on just one topic (Finnish music). Not sure how many topics.