IBM CEO says open weight models are now good and cheap on-prem (theregister.com)
184 points by Betelbuddy 6 hours ago
jacquesm 4 hours ago
It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.
gorgoiler 2 hours ago
Gaming used to be done in arcades for 25c a play*.
Over time NeoGeo realized they could sell you the hardware to play at home (“on-prem”) at arcade quality albeit at a price out of reach to most people. Alongside them, Nintendo and Atari gave you 80% of the fun at 20% of the cost. History will view games arcades as a flash in the pan compared to gaming at home.
Namco, Capcom, and Konami went into selling software instead of hardware but it feels hard to imagine buying a home copy of Fable** for $X,000 any time soon.
⸻
* That’s for video games. In fairground arcades you’d literally pay for tokens then use tokens for play. The analogy almost writes itself.
** The model, not the video game.
asdf88990 an hour ago
If you think the idea of buying Fable like model is hard, you should really try some of the latest open models like Qwen 3.8. They’re getting there very fast.
jacquesm 35 minutes ago
otterley 4 hours ago
How much did you shell out for this setup? What’s the TTFT and the output tokens/sec rate? How much electricity does it consume?
jacquesm 4 hours ago
Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
glassresearch 3 hours ago
gr_norm 3 hours ago
selcuka 3 hours ago
bitexploder 3 hours ago
I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
jacquesm 3 hours ago
jamesfinlayson an hour ago
At work two different departments have it on their radar to do - apparently there are GPUs waiting to be used for something like it.
jacquesm an hour ago
Smart move to get them bought ahead of time.
tannertech an hour ago
8 gpus is cheap now?
jacquesm 31 minutes ago
Well... that's a tricky one. I bought a lot of these second hand. I can sell them for more than they originally cost new, so indeed, it is not 'cheap'.
Then again, I run a business and for the business it is absolutely manageable, it is roughly the price of a single job. Privately this would be a different matter but I assume that if you would do this privately you'd simply get a single (but powerful) GPU second hand and offload to RAM. Not as fast but with llama.cpp, 'ninfer', DwarfStar and various other clones of those two you can get some pretty decent performance out of single GPUs.
I've been buying powerful hardware since the 90's, starting with SGI and then ever larger Linux boxes. I make my living with them so it's never been a huge decision to invest some of the proceeds back into the tools. But on an absolute scale, yes, it is a lot of money.
sjdh76 an hour ago
Depends on context of what people build.
I mean IBM already has a track record of building gigantic amounts of garbage and over charging large corps and govts for it coasting on their brand name and networks to get deals. So using some cheap ass model fits their MO. Having used their shit mountain called websphere, what an IBM CEO says is not something I take too seriously.
jacquesm 27 minutes ago
IBM has two faces. Their consultancy arm is the exact opposite of what I would call a good company, even so, they make a good bit of money. The hardware side of the business is still quite impressive, yes, it is too expensive but their power rack boxes are effectively a little mainframe (check the IO stats on those things) and the performance is - I have to admit - impressive. That said I would never buy it because I can get much better value elsewhere. Even so I think they could do this if they really wanted to.
dansquizsoft 4 hours ago
> Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct)
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
sp527 4 hours ago
You need to reread what was written. No one is proposing to take away your right to leak proprietary edge to the rapacious frontier labs that need to eat most of the economy in order to survive.
Frannky 2 hours ago
You can try open models via OpenRouter and omp without having to buy or rent any GPU. I routinely try open models to see where they are. GLM 5.3 with DSFV4.1 as advisor via omp feels Opus 4.6 level, i.e. totally able to ship stuff.
I do use Anthropic and OpenAI mostly because it costs me 1/40 and 1/50 of the api prices, which is way cheaper than Chinese subscriptions, but I’m ready to jump ship when they pull the plug. Fortunately they have nothing that forces me to keep paying them, and I don’t pay the crazy enterprise subscription costs, just the personal account prices.
I also die to be able to run the OSS setup locally to stop being spied on all the time.
For context, I averaged 10 hours a day prompting LLMs from when ChatGPT came out. At the start it was just function generation, then single file modules, then modules, and now I reason, design, implement and test with them.
An interesting thing is that with omp you can plan with a smart model and then implement with a dumber one. So for example, plan with Kimi 3.1 via OpenRouter and then implement with DSFV4 with zero data retention active, sorted by cheapest, and minimal tps accepted.
I get the people who are pissed about their cushy jobs evaporating, but if you know how to use these technologies it’s very freeing because it’s like having a company of people working for you, except they are not human so they’re not wasting their life working for you instead of using it for themselves.
You can make them do pretty much everything that is fuzzy-verifiable, you don’t have to worry about their feelings or that they won’t do what you asked them to do or stop, and you can activate them when you want at scale in parallel or not use them at all for days.
For me it’s a dream honestly. I never felt so much empowered in my life while being enabled to unlock my creativity with almost no boundaries.
deno an hour ago
In OMP just setting the "task" role (which controls the default subagent model choice) to some cheap Flash model like Deepseek or Gemini goes a long way towards making better use of your quota.
Sol and Opus both are quite eager to use subagents and since the subagent tasks are often well specified and then verified by the main model there's hardly any advantage to using anything better for the subagent anyway, so IMO it's a no-brainer.
For me switching the main model to Kimi K3.1 Pro would put me towards 100% open, but I don't know if it uses subagents effectively. Any other open models I noticed just don't utilize them very much.
jains99 5 hours ago
But hardware to run them is conveniently something IBM would be happy to sell you.
jacquesm 4 hours ago
If IBM would ship a piece of affordable hardware that can run large LLMs efficiently I'd be lining up.
solyin 4 hours ago
"Affordable" and "IBM" don't belong in the same sentence together, but they're happy to sell you their inference-capable hardware if you've got the wallet for it.
jacquesm 4 hours ago
mbreese 2 hours ago
I didn't think that IBM had gear that mortals would want to run. If you're not running a mainframe, what do they have to sell you? They still sell Power servers and it looks like they have an AI card that can be used on both. But, if we're talking about the IBM CEO talking about on-premises, what are they really talking about hardware-wise? It's not something they would sell, is it?
This is more of a statement for their consulting customers, right?
numpad0 4 hours ago
Yes but unironically? IBM should be what NVIDIA would be competing with their NVL72 and DGX products. It doesn't make sense that IBM is just watching NVIDIA take its lunch.
jacquesm 3 hours ago
I think they will need a far more aggressive management before such a thing could happen. But I can dream about it.
verdverm 3 hours ago
> It doesn't make sense that IBM is just watching NVIDIA take its lunch.
If you have ever worked at IBM, it makes total sense. Completely dysfunctional middle management and on up.
anukin 4 hours ago
Not till Jensen buys his next leather jacket.
hypfer 2 hours ago
Of course IBM - the guys that sell you mainframes - would be saying this.
This is a win though. But not for the reasons people might think it is.
It is a win because it's a massive player now countering the new (temporary) behemoths. With all the leverage, budget and connections.
And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people. Meaning that they're going to be far closer to the baseline of sane humans just from averaging alone.
ggeorgovassilis 2 hours ago
The guy that sells mainframes also gives away the Granite models for free. I've found them to be quite capable in content retrieval and long-running workflows, even (or especially?) the small ones.
djelu 4 hours ago
Considering the disclosed numbers for DeepInfer and the usefulness/popularity of DeepSeek 4.1 Flash, I expect it is inevitable that more companies are going to consider the self-hosting option seriously, and perhaps play with the idea of offering inference as a vendor.
trollbridge 4 hours ago
While he thinks most businesses can build useful AI infrastructure in just two or three racks, he feels many will struggle to get it running.
“The question becomes: how do you do that?” he said. “The question that the enterprise struggles with is: ‘Where are my kernel engineers? Where are the people who worry about the underlying network? Who can make sure that the model is clustered on GPUs that are close to each other in network proximity? Who can make sure that you’re doing the right caching of results so you get an incredible response?’”
liorn 3 hours ago
It's not a one time thing though - onprem inference requires continuous monitoring and tuning to get the most out of your GPUs. Looking at 25-50% performance benefits. Today, it does require dedicated Inference Engineers...
Buttons840 4 hours ago
That's based on the idea that humans can learn and adapt and solve problems, and that's not what companies want to think about when they're seeking a profitable transaction such as hiring.
LoganDark 4 hours ago
Do not do that if you can't prompt it correctly. It is a potential disaster waiting to happen. LLMs are likely to pull things off the shelf without real defense in depth and that's a problem. Maybe slightly less if it's only in an internal network, but still.
matt3210 4 hours ago
RIP frontier AI, the top customers can easily self host
IncreasePosts 4 hours ago
What happens when their competitors decide to pay for a more intelligent frontier model?
selcuka 3 hours ago
What happens when your competitors invest in better and more expensive tools or materials in general? They (potentially) make better products, but those products also cost more. It's turtles all the way down.
gscott 2 hours ago
You buy the competitor and kill their culture from the inside, ruin them and take a tax loss on the company.
verdverm 3 hours ago
depends on the economics and much more, the best Ai cannot do a lot of things, 90% of human tasks by some reports
we technologists tend to extend what we see and think to other realms where it doesn't make sense
collinmcnulty 4 hours ago
Can we please stop giving weight to CEOs talking their book? Like of course IBM wants you to think this, it would be great for them if it’s true. And maybe it is true! But IBM’s CEO saying it’s true tells me just as much as if a frontier lab said it’s not: nothing.
nvme0n1p1 4 hours ago
"CEO Said A Thing!" Journalism
lordgrenville 2 hours ago
Great term. Plus the related "Politician Said a Thing!" journalism
verdverm 3 hours ago
thank you, I was trying to re-find this the other day
epsteingpt 4 hours ago
But you know - it would be funny if somehow Open Weights are the thing that completely save IBM
SoftTalker 4 hours ago
Linux already served as a life-preserver for them also.
MattDamonSpace 4 hours ago
It’s a conversation point off which a discussion can be had
gjvc 4 hours ago
the only credible "conversation" is "well he would say that, wouldn't he?"
gjvc 4 hours ago
"we are delighted..." said a spokesman
SV_BubbleTime 4 hours ago
I’ve been in tech for a long time… I have no idea what IBM does. Now, I get that I’m not the customer, that their customer is “business” but not my business… no… bigger business. The biggest of businesses I assume. I think it must either be a club I’m not in, or IBM is three half racks in a trenchcoat submitting invoices for millions to companies where people just pay it because “it’s IBM, it must be important”.
andsoitis 33 minutes ago
> I have no idea what IBM does
For one thing, amongst many, IBM mainframes process 70% of global transactions on a value basis
jamesfinlayson 3 hours ago
Ironically one of the smaller places I've worked at tried to use an IBM product once - I'm still not clear on what it was meant to do or how it was better than any cheaper alternative, so I assume someone in leadership was fooled by a very impressive pitch.
Lots of work went into integration and it never went live.
CookieCrisp 4 hours ago
Good enough for the old world, but I'm not so sure it's good enough for what's coming.
tomrod 4 hours ago
What do you think is coming that isn't already in the old world?
Cerebras with Claude 8?
LoganDark 4 hours ago
OpenAI most likely pushed Cerebras into a contractual agreement not to do business with Anthropic through 2030. ("Most likely" because the names of the companies Cerebras is contractually obligated not to sell to are redacted in public filings)