Hacker News

Can you translate the American for me? “Wide variance”, does that they’re being assholes?

The second half of this post chronicles the author's Herculean efforts to make everything work well in the cases you describe.

> One of the big things that blocked this shipping for so long was not having an escape hatch of some kind to allow clients that disable WebAssembly by policy to get through the gate. [...] This combination of factors means that there would need to be another implementation of the proof of work code in JavaScript that would actually execute the number crunching.

> because I dislike things running in the background without my knowledge

Point of order: WASM doesn't let things run in the background. To do that you need WebWorkers (a javascript feature, not WASM-specific).

It's amusing (although disheartening to me as someone who builds online games) that wasm is now being treated like a flash plugin by sysadmins. You're not entirely wrong, but it's just funny to me that we took a full ten years to get back to roughly the same place. Now with Ruffle running Flash in wasm, no less! We build shit that doesn't work in the DOM, search engines hate it, walled gardens try to block it, admins fear it, they shut it down and the whole thing starts over.

I don't mean to be rude, but please search "wasm2js" in the article.

I really don't see why you'd disable WebAssembly and allow JavaScript instead. Expecting developers to make a fallback for a baseline-supported feature for any reason but backwards compatibility with runtimes that do not so support WASM is extremely silly.

"I disable things and expect others to do things for me, in their own time, without being paid, to bring solutions to my paranoia"

You do see how selfish it is, right? While the author did introduce such a solution, it's such a massive waste of time.

In the non-webdev realm, it's gobsmacking that you can put "FROM debian/eol:buzz" in a Dockerfile, and it will do what it says...

> Weirdly, the response from the rustc side (IIRC we also talked with one of the devs of wasm component on discord a bit) has contained a lot of (paraphrasing) "in retrospect we think wasm32-unknown-unknown target was designed badly, so we'd prefer if you either dealt with it or switched to another wasm target <with its different set of downsides> than us fixing it". (I acknowledge that the way std was implemented on that target was unusual and "not clean", but it works _really well_ for a lot of people.)

That's about right. Specifically, wasm32-unknown-unknown is serving multiple targets, which aren't very compatible, and it implements std by panicking in all the bits that can't work.

One target is "you truly have absolutely nothing, and don't really have much of `std` at all". Another target is "you're in a browser, and can do anything a typical browser can provide" (ideally targeting the future where wasm has full browser APIs without going through javascript). Another target is "hosted environment that isn't a browser, and which capabilities you have depends on the environment". Those are three very different things.

I would argue that we need at least two new targets, possibly three:

- `wasm32-none` or similar (by analogy with `x86_64-unknown-none`), a target where you don't have anything except for what you bring with you. The equivalent of writing kernel code.

- `wasm32-wasi-hosted` (or pick another name), where you the hoster of the wasm can supply whatever capabilities you have available.

- `wasm32-wasi-browser`.

It's possible that `none` could be a special case of `hosted`, where the services you provide are "none".

I'm working on this with a private prototype. I'm probably going to lean towards using a Service Worker (https://developer.mozilla.org/en-US/docs/Web/API/Service_Wor...) to renew challenges at a reduced difficulty. Stay tuned!

Or something that shadier websites could calculate ahead of time in your browser, then spend on your behalf.

I feel like Anubis is ironically speedrunning a lot of discoveries the crypto folks have already made several years ago...

Privacy Pass does something similar to what you describe: https://developers.cloudflare.com/waf/tools/privacy-pass/

I wonder if you could convince the Anubis author to implement x402 payments to bypass the PoW

Then we could implement an exchange, so that if you generate too much Anubis POW, you could exchange it with others.

We could link it to a site, you generate for HN, I for Reddit, but it so happens that you visited Reddit more and I HN, so we depleted our Anubis POW, so we could exchange some Reddit Anubis with some HN Anubis.

Oops, there's your proof I didn't write this with AI! Fixing, sorry.

Scrapers scrape all sorts of nonsense, notably every possible git diff from git hosts. They clearly aren't concerned with rationality.

the answer to this is in the post: where wasm is not available, he transpiles the wasm binary back to Javascript

In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.

One of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.

I'm not a fan of Anubis for various reasons but the idea that bot traffic in only harmful with dynamic pages must die. CPU (yes, even to serve static pages) is not free, bandwidth is definitely not free. There's an idea that serving a static page to a bot has a marginal cost tending to zero, but it's never really zero and serving them by millions definitely has a cost.

Also, while some pages may look like static HTML pages, they may be generated on the fly by an expensive and/or slow backend, which adds to the cost. I happen to maintain servers for academics and some content management systems are slow and have an expensive CPU cost. While it's OK for the low number of humans interested in the subjects they deal with, it's definitely not fit for massive bot scrapping. And before you ask, no, it's not always practical to have cache upfront or to pre-generate all pages.

Others have answered your question. A lot of the sites listed here https://anubis.techaro.lol/docs/user/known-instances/ have dynamic content; Git web interfaces in particular (Codeberg, the Linux kernel, FFMPEG, and more are on the list) are vulnerable to poorly or maliciously configured scrapers.

Shitty scrapers hitting images over and over again is also a problem. Not just /pic/1.webp - /pic/2.webp, but in practice I've seen three AI scrapers that would request things like /pic/1.webp?version=webp&foo=bar and other random query parameters that none of my domains even use. It's just stuck brute-forcing files or something, sucking up loads of bandwidth downloading the same image for no reason.

It also depends on if you want your content to be scraped. Any bot ignoring robots.txt should be considered potentially malicious and worth considering such mechanisms for.

The anubis docs website is used as an example of how anubis works.

Also, if you are not using a big CDN (e.g. Cloudflare), you still need to pay for the traffic (even if you don't pay as much for the CPU). This is doubly true for smaller CDNs like Bunny.

some people don't want to be scrapped at all, and most bots ignore `robots.txt`

no, absolutely not, any malicious or dishonest bot traffic is harmful, generating unnecessary transfer and literally stealing data

It's in the range of 32-40Ki before compression. After compression it gets to be completely negligible.

The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.

Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.

The assumption is that there will always be some population distribution in terms of effort expended as well as in terms of problems caused. At present it's the lowest effort actors that are causing the highest number of problems. So an 80% (or even just a 20%) solution largely solves the practical problem faced by site operators.

If against all odds it turns out that skiddies (or agents or AI billionaires or what have you) are willing to burn inordinate amounts of RAM and CPU just to scrape autogenerated cgit pages that are of approximately zero value to them to begin with then I guess we'll just have to cross that bridge when we come to it.

afaik argon2 should make the GPU less helpful

> at least in crypto there is a token you can sell

That would defeat the purpose. The goal is to make scraping costly, not profitable.

Well, there was CoinHive which did this exact thing 6-7 years ago, but that system got abused a bit much

PoW was originally anti-spam technology.

Refresh, the Anubis docs site regenerates its cookie secret on every deploy as part of the "holy shit did I break everything?" verification step.

The goal isn't to eliminate scrapers, its to prevent a distributed scraping network from requesting 10000 pages a second each from 10000 different websites.

Yes, I also strongly suspect that this is only going to move more parts of scrapers onto consumer devices. The egress proxies are already there, why not use a little bit of the compute as well?