Hacker News

> but are they functionally useful or helpful to anyone currently?

I've been building a SaaS that deals with data that can't be distributed to third parties. Some of the useful AI stuff I can add is not time sensitive and can run overnight. Things like this allow me to use higher quality models without selling my house for GPUs.

> but are they functionally useful or helpful to anyone currently?

Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do.

MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro.

The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).

I don't need any of these to help with coding. skill issue.

If I could justify wear and tear and electricity, I was willing to do something like this for batch processing. The batches would be a bunch of prompts whose outputs I'd look at the next day. Maybe common operations, like QA or refactoring, on whatever software I wrote.

If so, I could use a larger model than I have real-time hardware for. The largest, well-trained models can often get the output mostly right in one try. I also would be using AI's as a supplement to, not replacement for, my own brain. So, issues with the outputs wouldn't be a problem because I'm just keeping what's helpful.

If I still need to re-generate it all, it might still save money over time by avoiding cloud costs. Also, hardware that's already paid for is a sunk cost that doesn't inflate over time. Glitches in loading or destroying VM's might blow up into a big bill.

Running 292 tps on K3 would make your gpu a money printer.

Assuming 30% gpu power utilization because of all the loading and unloading 29.2 kJ per token

It is a bit more than four tokens per millifortnight.

So, instead of having one or two maintained projects that work super well, now we have hundreds or thousands of people half-assing it locally each time they need it? Doesn't sound efficient.

This is complete dependency on LLM tools all the way from development to usage, and is risky as well as prone to failure

Or Deepseek with a fiver.

Opus 4.6 was already enough to tackle these projects vibe coding.

these things can be done with local models; so don't say 500$ credits, say "accelerated climate change disasters by $500/year"

"This one is made with premium vibes"

Not an expert in this field, but the "expert" is consisting of multiple layers. To keep it small in terms of memory print, this project streams each layer (dividing even further).

It also requires extra space because of decomposition of the layers. Normally the file format optimized for compute intense workloads. But here the bottleneck is the memory capacity.

Also guessing that you need to be able to hold at least 3-layers at once in the memory, given M x N = R operation, M is the previous layer, N is next, and R is the result. on the next "layer", the R (result) becomes M, gets computed against the next layer, N, yielding the further result R'. And so on, until all layers are processed.

I assume it's horribly slow, but can be put in a non-intrusive background task...

Obviously needs downloading eventually :).

It seems like this tool saves on both disk space and RAM, then. Classic trade off: speed vs space.