Hacker News

I think Hotz basically created super specific software for the gpus that throws away anything that doesn't contribute to inference (not turing complete, for example).

> I'm also confused why this is 12U. My whole rig is 4u.

I imagine that's because they are buying a single SKU for the shell/case. I imagine their answer to your question would be: In order to keep prices low and quality high, we don't offer any customization to the server dimensions

It will work fine but it’s not necessarily insane performance. I can run a q4 of gpt-oss-120b on my Epyc Milan box that has similar specs and get something like 30-50 Tok/sec by splitting it across RAM and GPU.

The thing that’s less useful is the 64G VRAM/128G System RAM config, even the large MoE models only need 20B for the router, the rest of the VRAM is essentially wasted (Mixing experts between VRAM and/System RAM has basically no performance benefit).

This does not match my experience with 120B~ models. I run Qwen3.5 122b A10B on about 80GB of vRAM just fine.

Was that cheaper than a Blackwell 6000?

But yeah, 4x Blackwell 6000s are ~32-36k, not sure where the other $30k is going.

What models are you testing? A 120b model with hybrid attention should fit within 80gb of VRAM fine at a 4-bit quant. Also, 4-bit quants that are done well are generally fine. They certainly don’t make the model unusable.

> There's no way the red v2 is doing anything with a 120b parameter model.

I don't see the 120B claim on the page itself. Unless the page has been edited, I think it's something the submitter added.

I agree, though. The only way you're running 120B models on that device is either extreme quantization or by offloading layers to the CPU. Neither will be a good experience.

These aren't a good value buy unless you compare them to fully supported offerings from the big players.

It's going to be hard to target a market where most people know they can put together the exact same system for thousands of dollars less and have it assembled in an afternoon. RTX 6000 96GB cards are in stock at Newegg for $9000 right now which leaves almost $30,000 for the rest of the system. Even with today's RAM prices it's not hard to do better than that CPU and 256GB of RAM when you have a $30,000 budget.

> And there's no room for kv, so you'll OOM around 4k of context.

Can't you offload KV to system RAM, or even storage? It would make it possible to run with longer contexts, even with some overhead. AIUI, local AI frameworks include support for caching some of the KV in VRAM, using a LRU policy, so the overhead would be tolerable.

Honestly two rtx 8000s would probably have a better return on investment than the red v2. I have an eight gpu server, five rtx 8000, three rtx 6000 ada. For basic inference, the 8000s aren't bad at all. I'm sure the green with four rtx pro 6000s are dramatically faster, but there's a $25k markup I don't honestly understand.

I find that the most respected writing about AI has very few signs of being written by AI. I'm guessing that's because people in the space are very sensitive to the signs and signal vs. noise.

Good? That's what I want out of all websites. I don't want to read what an AI believes is the best thing for a website, I want to know the honest truth.

I don't view this as irony. This seems like good sense in understanding when AI usage will make things better and when it will not.

I am a little surprised that they openly solicit code contributions with "Invest with your PRs" but don't have any statement on AI contributions.

Maybe the volume for them is ok that well-intentioned but poor quality PRs can be politely(or otherwise, culture depending) disregarded and the method of generation is not important.

If you’re spending $65,000 on this thing, needing two circuits seems like a minor problem

"locally-trained models for everyone"

Wouldn't there be a massive duplication of effort in that case? It'll be interesting to see how the costs play out. There are security benefits to think about as well in keeping things local-first.

If I'm spending at least 12k USD on the machine then doing some electrical works to accommodate it is not a big deal.

When you’re dealing with this kind of power it’s easier just to colocate where you’ll typically get two separate feeds of 208v

A typical U.S. 240V circuit is actually just two 120V circuits. Fairly trivial to rewire for that.

3200W at ~240V is ~15A, that's just a regular household socket, at least in Europe. I imagine 240V sockets in the US are at least 15A.

No need for separate circuits, just use a double adapter.

Why is hn so obsessed Scott whether something is _written_ by ai or not? Who cares? Judge content, not form.

Oh wait, I get it, it's bike shedding.

Big companies are pushing cloud really hard, and yea the hardware prices too is a problem. People still buy Google cloud and OneDrive when they could literally pickup an old computer from trash and Frankenstein it into a NAS server.

Sometimes you can compete with the big boys simply because they built their infra 5+ years ago and it’s not economically viable for them to upgrade yet, because it’s a multi-billion dollar process for them. They can run a deficit to run you out of the business, but if you’re taking less than 0.01% of their business, I doubt they’d give a crap.

> The exabox is interesting.

Can it run Crysis?

Would a hedge fund that does not want to trust to a public AI cloud just buy chassis, mobos, GPUs, etc, and build an equivalent themselves? I suspect they value their time differently.

They’re kickstarting a TINY device that is pocketable and aimed at consumers. I’ve backed it (full disclosure).

i think the real gap isnt at the high end tho. theres a whole segment of people who just want to run a 7-8b model locally for personal use without dealing with cloud APIs or sending their data somewhere. you dont need 4 GPUs for that, a jetson or even a mini pc with decent RAM handles it fine. the $12k+ market feels like it's chasing a different customer than the one who actually cares about offline/private AI

M3 max tflops is tiny compared to the 12k box. It's not even comparable.

it's for fools. i bought 160gb of vram for $1000 last year. 96gb of p40 VRAM can be had for under $1000. And it will run gpt-oss-120b Q8 at probably 30tk/sec

I haven’t seen tinygrad used for any mainstream production project or thing of value, yet.

Besides a lot of self congratulatory pats on the back for how elegant it is. Honestly, when I read it, it looked confusing as all the other ML libraries. Not actually simple like Karpathy’s stuff.

All that to say, I do really want it to succeed. They should probably hire some practical engineers and not just guys and gals congratulating themselves how elegant and awesome they are.

Your framing of this section is misleading. On the site it's preceded by a FAQ-style 'question':

> Can you fill out this supplier onboarding form?

That's very important context, as anyone who has been asked to fill out a supplier onboarding form (hi) will attest.

> arrogant to the point of being almost hostile

First encounter with geohot eh?

He's not actually selling the exabox yet. It sounds like he put up a hypothetical config to see if anyone is interested.

There isn't a $10MM device right now, just $64M and under. I doubt the order process will remain the same in 12 months when the $10MM device becomes available

The specs for the “exabox” scream “this is a joke” to me.

> 20,000 lbs

> concrete slab

Huge-scale IT systems are typically delivered in one or more 42/44u cabinets, and are designed to be installed on raised floors.

> arrogant to the point of being almost hostile.

The YouTube rap video of geohotz telling Sony lawyers suing him to blow him is still up.

His style of dealing with corporate matters is certainly unconventional

I imagine that the FAQ might get updated when there’s actually a $10M machine for sale

That third party link is from a different company (Tiiny with an extra i)

Now I'm wondering if the HN title was submitted by some AI bot that couldn't tell the difference.

Kind of yes, except there is no block reward.

AMD now has 32 GiB Radeon AI Pro 9700. 4 of these (just under 2k each) would put you at 128 GiB VRAM

I was more worried by the 600kW power requirement... that's 200 houses at full load (3kw) in southern europe... which likely means 400 houses at half load.

the town near my hometown has 650 – 800 houses (according to chatgpt).

crazy.

I mean the difference in performance is quite big too. However, the 10,000,000 is a little bit too much (imo).

For reference, 12k gets you at least 4 Strix Halo boxes each running GPT-OSS-120B at ~50tok/s.

Look for llmfit on github. This will help with that analysis. I've found it reasonably accurate. If you have Ollama already installed, it can download the relevant models directly.

It's pretty close to what people have been frankenbuilding on r/LocaLLaMa... It's nice to have a prebuild option.

If you wanted a box built by geohot, most recently known for signing on to Elons Twitter and then bailing, it's for you

Most privacy talk folds on contact with a quote. Latency and convenience beat philosophy fast once someone wants a dashboard next week, and a lot of "data sensitivity" talk is just the corporate version of buying "organic" food until the price tag shows up.

If private inference is actually non-negotiable, then sure, put GPUs in your colo and enjoy the infra pain, vendor weirdness, and the meeting where finance learns what those power numbers meant.

Depends. If token speed isn't a big deal, then I think strix halo boxes are the meta right now, or Mac studios. If you need speed, I think most people wind up with something like a gaming PC with a couple 3090 or 4090s in it. Depending on the kinds of models you run (sparse moe or other), one or the other may work better.

At $7.2k + tax:

* RAM - $1500 - Crucial Pro 128GB Kit (2x64GB) DDR5 RAM, 5600MHz CP2K64G56C46U5, up to 4 sticks for 128GB or 256GB, Amazon

* GPU - $4700 - RTX Pro 5000 48GB, Microcenter

* CPU/Mobo bundle - $1100 - AMD Ryzen 7 9800X3D, MSI X870E-P Pro, ditch the 32GB RAM, Microcenter

* Case - $220, Hyte Y70, Microcenter

* Cooler - $155, Arctic Cooling Liquid Freezer III Pro, top-mount it, Microcenter

* PSU - $180, RM1000x, Microcenter

* SSD - $400 - Samsung 990 pRO 2TB gen 4 NVMe M.2

* Fans - $100 - 6x 120mm fans, 1x 140mm fan, of your choice

Look into models like Qwen 3.5

Sadly $5k is sort of a no-man's land between "can run decent small models" and "can run SOTA local models" ($10k and above). It's basically the difference between the 128GB and 512GB Mac Studio (at least, back when it was still available).

The DGX Spark is probably the best bang for your buck at $4k. It's slower than my 4090 but 128gb of GPU-usable memory is hard to find anywhere else at that price. It being an ARM processor does make it harder to install random AI projects off of GitHub because many niche Python packages don't provide ARM builds (Claude Code usually can figure out how to get things running). But all the popular local AI tools work fine out of the box and PyTorch works great.

Biggest Mac Studio you can get. The DGX Spark may be better for some workflows but since you're interested in price, the Mac will maintain it's value far longer than the Spark so you'll get more of your money out of it.

Fully aware of the DGX spark I've actually been looking into AMD Ryzen AI Max+ 395/392 machines. There's some interesting things here like https://www.bee-link.com/products/beelink-gtr9-pro-amd-ryzen... and https://www.amazon.com/GMKtec-5-1GHz-LPDDR5X-8000MHz-Display... ... haven't pulled the trigger yet but apparently inferencing on these chips are not trash.

Machines with the 4xx chips are coming next month so maybe wait a week or two.

It's soldered LPDDR5X with amd strix halo ... sglang and llama.cpp can do that pretty well these days. And it's, you know, half the price and you're not locked into the Nvidia ecosystem

> What’s the most effective ~$5k setup today?

Mac Studio or Mac Mini, depending on which gives you the highest amount of unified memory for ~$5k.

With $5k you have to make compromises. Which compromises you are willing to make depends on what you want to do - and so there will be different optimal setup.

DGX Spark is a fantastic option at this price point. You get 128GB VRAM which is extremely difficult to get at this price point. Also it’s a fairly fast GPU. And stupidly fast networking - 200gbps or 400gbps mellanox if you find coin for another one.

Look at the GPU and RAM spec; 120b seems workable.

Old news. ROCm works a lot better now than it did a year ago.

Looking at these prices I’m just thinking that as a user it makes no sense to buy this when you can just use the subsidised stuff from AI companies and then buy it a few years later at a tiny % of the cost.

If you're OK letting them train on, and maybe keep, your data, then it's hard to beat cloud prices vs. local.

I can't find sources but I think they are building it for Comma.ai (geohot's other company) so that Comma can scale up their training datacenter.

And... what about 20k lbs and 1360 cubic feet screams "tiny" :)

A non-trivial share of this market won’t show up in public data. That makes most estimates unreliable by default

VC funded startups

A company which doesn't want the big LLM providers to see it's prompts or data - military, health, finance, research

An NVL72 rack or Helios rack also "functions as a single GPU".

TPU pods

He builds a batch every few months.

In this case, they're taking wire transfers, so they're definitely building them once they get the cash.

Anyone who wants to run/train/finetune a local llm.

“Not your weights, not your brain.”

These things don’t have Flash Attention or either have a really hacked together version of it. Is it viable for a hobby? Sure. Is it viable for a serious workload with all the optimizations, CUDA, etc.. Not really.

I'd go with strix halo if you're looking at that old of tech.

the latest AMD GPUs are RX 9070 XT w/32GB each

They answered your question with a pretty specific uptime target. Calling it a dodge and then moving the goalposts with a new question as your follow up doesn’t speak to you acting in good faith.

Can't see replies, what did they say?

That thing is NOT related to tinybox or tinygrad in any way. It is basically copyright infringement. Unless you’re astroturfing here I suggest you get your money back.

Yeah, and labeling AMD "Driver Quality" as "Good" (for comparison, they label nvidia's driver quality as "Great").

It’s not for people to buy. It’s for companies to buy. Compare to salary, and it’s cheap.

Wire transfer has no comission or extra costs associated to it, so I find it very honest.

man, cmon. a little more effort.

He was always defending democracy and freedom before, and that was his argument for the local AI thing? What changed?

He had a video on Youtube where he proudly gloated about how he voted for Trump in not one but two elections, how happy he is that he can now openly talk about it, how its a fresh start for US, how catastrophic Harris would have been.

Did he take down the video because of embarrassment or did he fear negative impact on his sales?

Damn, that's a take.

For those unaware, Mencius Moldbug is the pen name of Curtis Yarvin, thought leader for the Silicon Valley branch of right-wing technofascist weirdos which includes Peter Thiel and apparently half of a16z.

Geohot has always been an arrogant cunt who thinks he's better than everyone else. That blog post is totally on brand.

Geohotz's politics are fairly straightforward once you understand his background. Geohotz is the prodigy child who, at the age of ~16 accomplished amazing technical feats on his own.

And his politics are a derivative of Great Man Theory, and his positions on things like democracy follow from that. This idea, and those espoused by some of the VC/tech elite like Peter Theil are that singular hardworking genius individuals can change the world on their own, and everyone who not in this top 0.1% are borderline NPCs.

They do this both because of their genius/hardwork, and also because they are willing to break the rules that are set forth by this bottom 99.9%.

I'm starting to call this ideology Authoritarian techno-Libertarianism. Its a delibriately oxymoronic name that I use, because these "Great Men" are definitely trying to change the world. IE, they are trying to impose their goals and values on the world without getting the buyin of other people.

Thats the "authoritarian" part. And then the "libertarian" part is that they are going about this imposition of their will on the world by doing it all themselves, through their own hard work.

Think "Person invents a world changing technology, that some people thing is bad, and just releases it open source for anyone to use". AI models are a great example, in fact. Once that technology is out there the genie cannot be put back into the bottle and a ton of people are going to lose their jobs, ect.

A distain for democracy follows directly from things like this. You dont wait for people to vote to allow you to change the world by inventing something new. You just do and watch the results.

Nvidia has $4K DGX Spark, $120K DGX Station, $500K DGX, and $7M NVL72.

> See our bounty page to judge if you might be a good fit. Bounties pay you while judging that fit.

Literally the line above that