Hacker News

Thank you! I think that's a great idea, and will definitely look into implementing this.

I would just like to add some cautionary anec-data: there are widespread cases in certain jurisdictions where rightsholders are known to seed the same torrents themselves, just to turn around and send love letters to leechers that connect to them. A good example is Germany with movies and TV shows.

Now, I don't know if, say, Wolters Kluver would/does the same thing, and what the realistic risk of an individual receiving such a letter is, but I think it makes it worthwhile to go over the actual law in your jurisdiction before diving head first on things like this.

I'm not saying it's wrong to seed these things, I'm just saying it might be a good idea to weigh the risks if you don't have a cool 500€ in cash to part ways with.

Levin uses that feature exactly! It is not unique in finding what torrents to seed; It's unique in that it dynamically uses the available diskspace (removing / adding data when needed / possible), and automatically turning off when not plugged-in / on wifi connection.

that feature has a "max terabytes" field. phones typically do not have terabytes of storage, and even if they did, people may not want to seed that much

DMCA letter sounds like small potatoes when we talk about letting random people write stuff to your disk space and using your bandwidth.

I'm in Italy. Most people I know have been pirating movies, series and games [1] for 20+ years, via torrents and eMule (yes, eMule is still big in Italy), and nobody ever received any letters.

But there's a big exception: as soon as you start pirating soccer, they're going to come after you.

[1] I've personally stopped pirating games a long time ago, because it's just easier and safer to buy them on Steam or GOG. Gaben was 100% right when he said "Piracy is almost always a service problem".

In Germany you can expect to get a letter from some law firm, confirmed by some judge that orders you to pay 100s or 1000s of euros if you don't use a vpn

They will attempt to download DMCA files from you as often as possible and then calculate the amount of times times price of the product to come up with a fictional damages amount

US colocated seedbox with ~10k film and tv torrents seeding at any given time, the last letter I got was ~2014 IIRC, before that it was several a year. I never responded to any of them.

I don't think I'm especially good at covering my tracks, so either they've abandoned individual enforcement in favor of going after distributors or they no longer bother with non-residential IPs.

Happens every day in the US. Mostly video and music (MPA/RIAA). There's also been some effort put into extorting ISPs for the activities of their customers, but the effectiveness of that is still being determined as cases work their way through the court system. We should have a better idea this summer after the supreme court decides on the $1 billion in damages one ISP was ordered to pay to a bunch of RIAA labels.

It will be a lot more profitable to sue ISPs than it is to try to sue poor parents and grandparents for what children do online.

In France, for movies/music you get 2 warning letters, then a scary one that says you can now get to court possibly.

Didn't really hear about people getting fines for this, but the law exists.

I've heard Finland sends out letters, same with Japan. Are there actual consequences, or can they just be ignored?

Norway I haven't heard of anyone getting anything in the past decade. The ISPs supposedly get letters from lawyers but just toss them, since the intersection of the burden of proof and our privacy laws make it such that nothing can really be done.

I think there was some ISP that gave out names and IP addresses to one of the firms years ago, but nothing happened and the police said "we have better things to do".

I find it absurd that with all of the dhit going on in the world right now that any legal resources are being spent on copyright enforcement.

I haven't heard about someone ever getting a letter for seeding books, but maybe I'm lucky. In any case, I'll add a notice to the README, thank you for the suggestion.

I wonder if he uses spaces or tabs in his source code.

You configure Levin to "always leave 2GB available". Levin checks the available diskspace using a simple statvfs call, deducts 2GB, and sees that as its budget. It then checks your diskspace every minute (more or less, depending on the device) to see if anything changes. If more free space is suddenly available, it will download more content. If there's less than 2GB available, it will immediately start deleting its own files until 2GB are free.

You can check the content of the torrents, just like any torrent. The client isn't a "one shot" LLM produce, I've been spending quite some time on it. What actual concerns do you have?

Just like you can read source code written by humans (and should if you take this stance) you can also read source code generated by LLMs. Then, when you find something unsavory and feel that your sentiment is warranted, make a contribution.

Who do you think Anna is

What do you have in mind?

I don't recall there being movies on Napster.

It's related in the sense that it works in the background, using the spare resources you have. Whether you see the thing it does as a good thing or theft is really up to you. I guess some people had their own reasons for not supporting the SETI@home objectives either. In any case, I'm perfectly happy with an analogy like "it's like going to the library, making a copy of all the books and making the copies available for everyone for free".

What did they steal?

Yes. It's a basic scraper that fetches the document, parses it for URLs using regex, then fetches all those, repeat forever.

I've done honeypot tests with links in html comments, links in javascript comments, routes that only appear in robots.txt, etc. All of them get hit.

I assume this might be changing. Anecdotally, from what I've read here, I think we're starting to see headless browsers driven by LLMs for the purposes of scraping (to get around some of the content blocks we're seeing). Perhaps this is a solution to a problem that won't work now, but in the future, maybe.

Absolutely.

I assume that there are data brokers, or AI companies themselves, that are constantly scraping the entire internet through non-AI crawlers and then processing data in some way to use it in the learning process. But even through this process, there are no significant requests for LLMs.txt to consider that someone actually uses it.

I think it depends. LLMs now can look up things on the fly to bypass the whole "this model was last updated in December 2025" issue of having dated information. I've literally told Claude before to look up something after it accused me of making up fake news.

This is a file for a LLM, not a scraper, so anti-scraping mitigations seem sort of beside the point.

claude --plan "let's develop a plan to detect and mitigate tarpits"

Ten minutes later, the ball is back in your court.

And to try to get them execute bb(5) ;)

Thanks for the clarification.

>for use in LLMs such as Claude (1)

From your website, it seems to me that LLMs.txt is addressed to all LLMs such as Claude, not just 'individual client agents' . Claude never touched LLMs.txt on my servers, hence the confusion.

1. https://llmstxt.org

As a project that started with a lot of idealism about how software _should_ be built, I would totally expect Bun to have an llms.txt file even if Claude wasn't using it. It's a project that is motivated in part by leading by example.

I also noticed this LLMs.txt at bun.sh, so for me it looks like some sort of advertising.

Did they do that before they were bought by Anthropic? Perhaps it's just part of a CI process that nobody's going to take an axe to without good reason.

Actually, I noticed an interesting behaviour in LLMs.

We had made a docs website generator (1) that works with HTML (2) FRAMESET and tried to parse it with Claude.

Result: Claude doesn't see the content that comes from FRAMESET pages, as it doesn't parse FRAMEs. So I assume what they're using is more or less a parser based on whole-page rendering and not on source reading (including comments).

Perhaps, this is an option to avoid LLM crawlers: use FRAMEs!

1. https://github.com/tirrenotechnologies/hellodocs

2. https://www.tirreno.com/hellodocs/

It would be foolish to use the LLM directly without a wrapper that detects prompt injection attempts.

Good question, at least OAI-SearchBot is hitting robots.txt.

I assume the real issue is that what overloads the servers like security bots, SEO crawlers, and data companies — are the ones that don't respect robots.txt in full, but they wouldn't respect LLMs.txt either.

In tirreno (our product), we catch every resource request on the server side, including LLMs.txt and agents.md, to get the IP that requested it and the UA.

What I've seen from ASNs is that visits are coming from GOOGLE-CLOUD-PLATFORM (not from Google itself), and OVH. Based on UA, users are: WebPageTest, BuiltWith, and zero LLMs based on both ASN and UA.

1. https://github.com/tirrenotechnologies/tirreno

This is only done at the DNS level, so using a different DNS (such as Quad9) solves that issue. For background info, I can recommend [1, 2].

[1]: https://www.youtube.com/watch?v=Uxmu25mUZgg [2]: https://cuiiliste.de/

Yay, MITM in the wild :)

I got it on my phone, but not with my local ISP.

In other news, Project Gutenberg not completely censored in Germany. Well done, Germany. https://cand.pglaf.org/germany/index.html

And the works that previously had lead to Project Gutenberg being unavailable from Germany IP addresses will go into public domain in 2027.

I can access the site just fine from Germany. Tried Vodafone and Congstar but I don't use their DNS servers.

its possible your browser used DoH. Some have started shipping it by default to encrypt DNS traffic (and use their own resolvers of course). Or maybe your ISP doesn't care

Which ISP please?

Just checked and it's blocked for me if I turn off my VPN - am on VirginMedia.

I'm on Vodafone in Spain and I see

> Error code: PR_CONNECT_RESET_ERROR

If I try the http version, I get redirected to https://bloqueadaseccionsegunda.cultura.gob.es/ (which also fails with PR_CONNECT_RESET_ERROR).

If it wasn't enough that half the internet gets unusable whenever there is football on TV (which is fucking stupid), now we're also getting rid of free (text!) information it seems.

For Virgin Media, redirects to https://assets.virginmedia.com/site-blocked.html

> Virgin Media has received an order from the High Court requiring us to prevent access to this site.

Appears that UK EE has it blocked too. Tried this morning waiting for the train in to work.

https://en.wikipedia.org/wiki/Anna%27s_Archive#United_Kingdo...

>In December 2024, the UK Publishers Association won an order from the High Court of Justice requiring major ISPs to block Anna's Archive and other copyright-infringing sites, extending a list of sites blocked since 2015 under section 97A of the Copyright, Designs and Patents Act

Others have already posted, but the biggest domestic British ISPs block a variety of things, like SciHub, Libgen, Pirate Bay, or Anna's Archive. Coverage varies a lot though, so I assume ISPs have some discretion and enforcement is patchy.

Also in the UK and can also see it fine.

I wonder if it's blocked simply by DNS manipulation and therefore only people using the ISP DNS have issues.

In the UK I'm currently getting:

Hmmm… can't reach this page

Check if there is a typo in annas-archive.li.

DNS_PROBE_FINISHED_NXDOMAIN

I am in the UK and I can't see it unless I use a VPN. I get

This site can’t provide a secure connection annas-archive.li sent an invalid response. ERR_SSL_PROTOCOL_ERROR

Neither was the data LLMs were trained on.

At least this isn't saddled with a profit motive and the destruction of the consumer computing market.

Data doesn't belong to anyone, data is free :) zero-copy cost, delivery at speed of light.

It is. They gathered it. They stored it. They served it. That's how data should work and eventually will.

I’m not completely sure there was an API from the start. I’ve thought the only way is to get a DB dump (which sounds pretty reasonable to me).

Hah! I learned of Anna's a few months ago. I posted a slightly snarky comment on the lack of an explainer and got downvoted to oblivion

https://news.ycombinator.com/item?id=46169388

  >>  You know, it wouldn't kill them to add some fucking details to the main page rather than making you dig for it. The TL;DR:

  WTF is a Anna's Archive: Hi, I’m Anna. I created Anna’s Archive, the world’s largest shadow library. This is my personal blog, in which I and my teammates write about piracy, digital preservation, and more.

  WTF this post is about: Exclusive access for LLM companies to largest Chinese non-fiction book collection in the world.

That is just not true. These AI scrapers are hammering all types of sites and causing their bills to explode.

https://www.pcmag.com/news/wikipedia-faces-flood-of-ai-bots-...

The nature of archives is that they are constantly updated.

That's a good point I suppose.

I guess I'm just kind of sad. LLMS appropriately sourcing material could have been such a boom for artists in a way. I guess I feel like it was a missed opportunity for some mutual benefit.

Would have been a really interesting at least.

They did already? OK. I somehow missed that.

Thing is, when they're pirating books, they're flagrantly violating ip laws in ways which big tech companies do themselves. When they're pirating music, they're flagrantly violating IP laws on a type of IP the big tech companies are directly selling. They're making a lot of new enemies.

Book publishers have less money than record labels, so less lawyers too

they have individual IPFS links but they don't work 100% of the times

Thanks. I also enabled DNS-over-HTTPS for good measure.

https://wiki.archlinux.org/title/XDG_Base_Directory

https://specifications.freedesktop.org/basedir/latest

originally published as a standard in 2003, apparently.

HTTP equivalent:

https://www.rfc-editor.org/rfc/rfc8615

https://en.wikipedia.org/wiki/Well-known_URI

Big corps are bad, human culture is great. Thats the red thread here.

They only derived payment because other humans find value in the robots output. In the end it’s still benefiting humans.

Well, I can go along with your analogy, and say that yeah, I'd be annoyed at the owner of the lemonade stand. Those marathon runners are trampling all my vegetables, and you're just trying to make a quick buck selling lemonade? People (me included) are annoyed at LLM creators scraping the web and gobbling up all copyrighted material, but it's mis-calibrated to get annoyed at Anna's Archive performing some sort of digital selling of stolen goods?

Well yes, it would pretty clearly be classed as "prompt injection" given that it's trying to get the LLM to give them money or "persuade" a human to give them money. Of course the fault lies mainly with whoever deployed the LLM in the first place, but I still think it's misguided to try to convince LLM "agents" to make financial transactions in order to benefit yourself. It'd be much more ethical to just block them.