Hacker News

The metal masked ROM is basically only 2 metal/contact layers. It's not a full new design and tapeout. You could roll a new set of parameters every ~2-3months. It's not an architectural change. See statements below.

https://www.eetimes.com/taalas-specializes-to-extremes-for-e...

https://www.turingpost.com/p/taalas

https://cambrian-ai.com/taalas-launches-hardcore-chip-with-i...

Part of the key is that by moving even from 6nm to 3-4nm one could embed a 20-30B model as part of a MoE (or only a subset of activated layers) on a single reticle die (note B300s are already multi-reticle), with a separate predictive/dispatch model controlling them each on a separate chip. This is without even stacking CiM ROM die. Moving the layer activations (and KV cache etc) between die requires relatively high speeds (and low latency), but distributed with multiple die in parallel might well be doable even with standard multilane PCIe. Of course KV cache prefill could also be handled by external GPUs. I'm sure AMD will make some reasonable choices.

> Maybe once LLM improvements asymptote further?

Maybe! But it also doesn't require the rate of improvement to slow down. As long as some current model is eventually "good enough" for general use, it could still be a market-killer at a very low marginal price thanks to ASIC. Even if slower, much more expensive models are 10x better, that doesn't actually diminish the utility of the ASIC model, as long as it's "good enough".

They could etch the model architecture, without the weights into the chip.

This way newly post-trained model can be loaded and served the same day.

How much of that 16mo is design versus just production? If there was a “plug and play” chip where you just BYO weights, how long would it take?

The bigger issue seems to be that these chips can’t hold that many weights at the moment.

(I’m curious if chips with large weights in them would be more tolerant or less to yield issues. If you flip a few bits in the weights, does it really matter at scale?)

tapeout could shrink but days per mask layer (DPML) does not have much margin..

I think Sol is already good enough though.

It may not matter. Think about why SOTA model companies are exploring chips. What do chips offer?

If SOTA models haven’t peaked, then the SOTA model companies would still be churning out better and better intelligence.

> Taalas needed a giant chip (6nm) for an 8B model.

You're phrasing it like it was kind of an inherent technical limitation with this kind of burning weights into silicon. Which is also not new, it goes back to the 1980s with fixed function digital signal processors and little linear regressions or hardware classifiers for industrial control systems, all are the same basic principle.

It's just usually not worth it to go super small process node, because most models people thought to turn into silicon were pretty small parameter sizes. We're talking 10-100 weight regression or at most 2-4k weight neural net, used in some instrument or factory equipment. You can do a decent MNIST OCR with a 4k weight neural net. For this, 180/130nm is fine.

Or you might think it's required with their special 4-bit as transistor thing (plausible). It's more that when you're experimenting and iterating, TSMC 6nm is their advertised path for rapid prototyping at cost for proof of concepts. And that's already in hot demand, while good luck if you're a startup trying to break in with 3/4nm as your first run.

Nope. But we are hitting some pretty impressive levels with 128B models.

The other thing is, a lot of the time, model performance is improved with more 'thinking' time.

The thinking time is just more tokens... but instead of say 1000 tokens or 10,000 tokens worth of thinking its 1,000,000... how does that improve model performance? Could a 128B model hit levels of GPT Sol?

Keep in mind that what previous work has done on a single chip with weights baked in was on a 8b parameter model. Sol is likely something in the 5T parameter range, perhaps higher. Serving the whole thing at BF16 is on the order of $3m in hardware just to serve it at all, and closer to $1-1.5m of hardware if it was being served as NVFP4. And power draw starting at high tens to low hundreds of kilowatts.

Let's say a magic set of chips comes along to host this. Maybe it's 2-3x more efficient in size and power. You're still talking a form factor that's a good chunk of a rack, draws tens of kilowatts, and could actually be sold at a similar if not higher price point because the OPEX is so much lower.

It may be useful but it's certainly uneconomic to spend >$1m to self host the model, plus ongoing power and maintenance costs, plus the cost to adapt whatever building you're in to be able to power it.

Man wouldn’t it be cool to be able to slot a massive ROM AI chip into the external AI drive of the pc…

Also right now Sol 5.6 Max is super slow but if it were way faster on a chip (like Taalas' Llama 8b demo) then it would be an extreme value multiplier. But the model is so large that "baking it onto a chip" doesn't seem straightforward.

That'd be ungodly expensive.

Maybe! (1) Would SOL level intelligence be useful 3 years from now? 5 years? (2) would dedicated chips be the most affordable way to run this model in 3-5 years?

I suspect the answer to both of these questions is yes right now, but I agree it’s borderline.

It depends when the good enough level hits. Pretty sure we are almost there for most common applications of AI.

There are other options. I worked for a startup called NVXL and we were programming DNNs into FPGAs using OpenCL, on custom boards we built to plug into NVME. It worked great, but it wasn't fast enough at the time to compete with Nvidia, or even Intel AVX512. Ultimately the company failed, but maybe some hybrid like that could work for LLMs? I haven't been up to date on how DNNs and LLMs look like under the hood these days, but there's got to be someone doing something similar.

Well we'll see those surplus chips being repurposed for toys then. Who wouldn't want a new Furby that can actually hold a conversation.

No, Cerebras holds the weights in SRAM - they are changeable, not baked in.

I would assume asic based llm would work really well. Why did it not go well?

Ternary?

EAX was very powerful in its heyday, but it has died because of a thousand cuts.

First we had to have the audio processor. Good EAX was available on top of the line cards, and they were not always cheap. Lower end chips got less features.

Then we had to have the speaker setup to have the greatest sound, or needed to get a real 5.1 headphones, which were bulky and never provided the same fidelity.

Then Microsoft changed the Windows driver model, cutting the driver's direct access to the card. All of the timing sensitive effects were gone in an instant. I remember installing the new drivers and getting literally nothing. Sound Blaster was the only card with an hardware mixer, and Microsoft didn't feel like enabling them. Mixing at the DirectX layer killed the cards.

Soundblaster's very closed stance didn't help them either. None of the cards after Audigy2 worked with Linux when I had my desktop system.

After my Audigy2ZS, I moved to Asus Xonar D2X. Its positional audio capabilities were nice, but I mostly bought it for its Linux support and sound quality, and that was top notch in that regards.

Then sound cards became commodity. Everybody stopped making good cards. Musicians moved to audio interfaces, audiophiles moved to DACs.

Just looked to the SoundBlaster website. Internal cards are very limited. One DAC, one DTS enabled 7.1 sound card for PC cinema systems, three game oriented lower end cards, nothing else.

Eeeeeh idk about the Sound Blaster comparison. Creative earned their place in the early-mid 90s solely because they were the one company making a sound card with drivers that actually worked properly.

It wasn't really the cool reverb effects or wave tables, though those were a nice bonus. It was just "I can tell my computer to make sound and it actually makes sound without days of troubleshooting."

Granted, similar things could be said about 3dfx. It's was a 3D card with drivers that actually worked.

And then there's the obvious "sound blasters and voodoos go in my computer, jalapeno goes in someone else's computer" thing.

On board got "good enough" and the separate cards died away.

In fairness on board (depending on the board but on the whole) is pretty good.

There is drastically more power and profit in the software ultimately.

Apple is in the software first, the hardware second. Everyone at Apple has been trained to understand this for decades, and Jobs pointed it out endlessly. Apple's real moat is software (services, iOS, experience, MacOS).

Windows, Office, Azure, et al. Microsoft accumulated approximately one zillion dollars in profit on the back of software. It's a vastly superior business to anything hardware has traditionally seen. Nvidia is the first true juggernaut hardware profit machine, and the AI boom in extended hardware (RAM, storage) will prove temporary (even if there is a feast during that time). Microsoft's advantage and moat was Windows-Office for decades. It was a far better business than Intel's chip biz.

Google is a software company first. Every aspect of what made them and maintains them is software first, hardware second. They're a $400 billion software company. Their ad machine is software. Search is software.

Facebook is software. Instagram is software. WhatsApp is software. A $200 billion software company. They're not selling hardware, they're selling ads via software, they're monetizing users that use their software.

AWS is at least half software as an entity in terms of complexity, competitive advantage, et al. That's a two trillion dollar business.

LLMs can run successfully with various hardware approaches. The software is the value at the end of this, regardless of the hardware under it. The sole exception so far that may be sustainable is Nvidia, and we'll see if the bottom falls out from under that margin monster (China, specialized AI chips, whatever it happens to be that cuts under them massively).

Hardware always gets its margin squeezed eventually because it's a manufactured good (with inventory, fabs, etc). Software is hyper margin by default, you have to layer a lot of garbage on top of it to kill the margin. Nvidia is 33 years old, they have had a rich business for three years, that's it.

The AI boom is the sole reason anything in hardware has looked great in the past 20 years. Check the margins & op income for the top 20 hardware companies, from TI to AMD to Intel to Nvidia to Micron to Sandisk to Samsung to TSMC to ASML, prior to the AI boom of the past couple years. It won't last indefinitely. And after the return to a more normal environment happens, the hyper margins in software will persist.

I bet cost is of no issue with the capx where it is at. It is almost certainly organizational. Meta throws money at every problem and it never seems to workout for them.

kind of a moot point if you can't get your brain to not do everything else. I think it's a fun comparison, even if it's not a 100% equivalence.

Right, I can do the talked about ~3 tok/sec output and drive a car, hold my bladder, and eat chips at the same time.

Take that, Jalapeno!

And the brain is literally only producing electrochemical signals.

I don’t see how tokens can’t produce speech or track metabolic needs. You can talk to chatgpt can’t you? Or do you mean literally talking? Because that’s not a brain function, that’s the mouth, vocal chords, and lungs.

I do believe that this is the trade off. We are more efficient but slower in terms of thinking (at the same level of intelligence). Some animals go much further in terms of that trade off, see https://en.wikipedia.org/wiki/Portia_(spider) for example.

What exactly are you questioning?

Careful, let's not put the whole matrix into stasis outside of business hours.

Productivity is not the only reason to let these meatbags burn oxygen.

Or maybe significantly lower.

Along with Leopold Aschenbrenner so maybe not so much.

shit posting

Bots do that because other platforms remove or hide posts with bad words

What credentials do you need to write a substack about an industry so it’s not grifting?

Say more about this please

Are they? https://jon4hotaisle.substack.com/p/influence-as-a-service-s...

This is exactly what I see happening now.

Codex keeps doing these usage resets. What do I do? Burn even more tokens than ever before. I know I'm not the only one.

If we are applying Jevons paradox to this then the unit being consumed is not tokens but the inputs for token production - power, capex, something else. To draw an analogy to the steam engine, coal:electricity::mechanical-work:tokens. Jevons paradox does not talk about mechanical work becoming cheaper in the short term setting up a sort of rubber band of demand creating spiking prices for mechanical work. Compared to the renaissance, mechanical work was much cheaper throughout the industrial revolution and remains cheaper to this day. We can still definitely say that the easier it is to produce tokens, the cheaper they will be.

the total cost spent on tokens may go up, but i just cant imagine per token costs going up

It is incredibly cheap now. What sectors are you thinking of?

That's when demand is higher than capacity. Now imagine places like Gigalab and Chinese labs are online and able to produce significant percentage of chips. That could cause real surge in prices.

I think you're reducing a very complex thing (the global economy) into a very simplistic model (Jevons' paradox) and thinking both are the same thing. This has no predictive power or rigor. You're just wishing things would happen as they did before, without considering that conditions and situations change significantly, and instead of Jevon's paradox, we look back at today 50 years from now and talk about Jensen's paradox.

This doesn't mean the concept is BS, but one single concept cannot explain away everything in such a system.

The demand for them is growing _per person_, not just across the wider economy, if tokens cost half as much but you want to use 3 times as much you're going to have to pay more.

Maybe 1000s of tokens per second unlocks realtime robotic decision making, and now every robot needs to continuously stream tokens to and from the cloud to operate. That could 1000x demand overnight, just to speculate :)

The price has been going down for ages, its not clear what you are pointing at

Nah, Jevon’s Paradox says that cheaper tokens will mean increased overall energy consumption.

If we can’t even build data centers, the least disruptive industrial use possible, there’s no hope to reindustrialize the US or anywhere outside of China.

>polluting on-site generators

how much pollution do you believe modern gas-turbine engines to produce?

>Not to mention the water use controversy.

what percentage of US water usage do you believe is by AI data centers?

OpenAI just dropped the price of Luna by 80% and Sol by 20-30%

We should be mindful of the context that many of these providers VERY likely have been selling their subscriptions at a substantial loss

So as much as i agree “more profits to stakeholders screw the customer”, i think its more of an emergency to get to profitability before the music stops.

Yes, I can bet on this happening. If anything, this is a net gain for consumers as it is a competitive market.

If the populist campaign is to Make Affordable DRAM Again, then it's not a terrible solution.

The current datacenter owners love a compute-bound world anyhow. A moratorium on new datacenters would increase their valuation, encourage efficiency and make computers cheap again. If Chinese labs can ship frontier models under 1T parameters, why not American labs too?

these GPUs make computation faster, I understand as of now maybe all the computation is used to generate yet another junk LinkedIn post or unnecessary RFC, but at some point this craze should settle and we will be left with powerful computation machines, which can be used for computing more useful things

10-90 is the fair split.

The same level of intelligence gets roughly 10x cheaper per year. So you might both be correct where a large part are commodity tasks but frontier is hard and valuable and not commodities.

> I don't believe models will be commodified because each model is unique with strengths and weaknesses.

They are all converging.

This depends heavily on what the use-case is. Yes, if it's a coder making software and having to read LLM output then writing style matters. If the LLM is used in an automated data processing pipeline with a capped level of complexity, entirely different aspects matter and LLMs become more interchangeable.

> Its not like Steel which is more or less the same no matter where you purchase it from.

I’m not an expert in metallurgy by any means, but this seems really off. There are many recipes for steel and varied processes that also impact the final product.

They evaporate the water which is what makes it cool so effeciently.

Yes they are for water cooling.

At the datacenter side, it depends on the method of cooling. You can chill the air or the chips directly (or both), doesn't matter, you still need to cool, and that still needs water. The question is, where is the water being used?

- If they use either evaporative cooling or a liquid-cooled heat exchanger, that uses tons of water consistently. This requires less energy (it's mostly passive) so you use more water.

- If they use closed-loop water cooling and/or heat pumps/electric chillers, that uses much less water - at the DC. But it does require more energy to circulate the water, run fans, etc. If you are using more energy, where is the energy coming from? It's coming from power plants, which require... you guessed it... more water (e.g. thermoelectric, hydroelectric, geothermal, concentrated solar). They need water in order to generate the power, and lots of it. Coal, natural gas, nuclear, and concentrated solar, all use steam to generate energy. Nuclear also uses water to cool the reactor. And water is used extensively to extract coal, oil, and natural gas. Geothermal uses water in the ground.

You can't not use a ton of water in one fashion or another. It just depends what method, and on what end the water is used. And the crazy thing is, most new datacenters are being built in places with extremely little water. Guess how that's gonna work out as the planet gets hotter?

I don't know why I got downvoted to hell for stating facts every datacenter architect knows. HN be HN'in.

> NVidia has enormous operating margins, so a competitive solution doesn't have to match or beat NVidia's scale efficiencies; it just has to beat delivered cost.

But then that means you have no actual moat against the behemot, right? Your competitor can move into the market as soon as they want to, at much better cost (so at slightly better price)... and Nvidia certainly can adapt much faster around hard hardware specs innovation than a new entrant ever could.

Nvidia buys the memory it uses on its GPUs, same as all other ASICs.

To give some context, Intel started making DRAM, I think they were actually the company that came up with modern memory techniques. They exited the market and pursued a more lucrative moat with CPUs.

Nvidia does not make RAM

That doesn't excuse them from wrecking the market for ordinary person.

NVIDIA produces memory?

RAM chips are not hard to produce compared to many other types of semiconductors; Intel started in the memory game and left because the margins weren't great and they were going to fold. The failure rates on these chips are actually very tolerable; you can have a very bad yield and still have a viable chip due to things like ECC.

Yes, it is difficult, but shafting working class is easy, therefor it is okay.

If the rich decided to buy all drinking water, you would probably be saying that's okay, making water is difficult, shortly before dying.

It is being used as part of a combined system. For example AWS is pushing for Trainium + WSE 3. The WSE 3 does the decode and the Trainium does the prefill.

Even in nvidia land rubin + LPU does a similar thing.

It has its downsides of course - if your traffic swings prefill heavy to decode heavy, you can't suddenly use your lpu for prefill. With GPUs they're totally interchangeable. Tradeoffs.

And yet, the top comment is about “hardcoding” weights into the silicon.

Man, if only someone made like, chips that could lots of different calculations all at the same time!