Hacker News

Chip CEO here. It really depends on what "design" or "production" means. Does "design" mean that the design was complete? Does "production" mean the beginning of production, i.e. tapeout? If measuring from RTL-freeze to tapeout, this is a fairly typical (even somewhat unimpressive) timeline (accounting for some unexpected issues) for a large, complex 3nm chip. If measuring from concept (no RTL at all, block diagram of architecture) to tapeout, this is an amazing timeline. The truth is probably somewhere in between. A more concrete statement would use actual technical milestones and gates.

The hardware description languages (HDL) used in chip development are like programming languages. The existing models understand them and can do a lot with them. You don’t need to have separate, specialty models designed for this work to use LLMs in chip design workflows.

Design verification also involves a lot of traditional programming which benefits from LLMs.

So it’s not meaningless at all. You could download some of the open source chip design software today and the LLMs could even help you get started on your own tiny chip if you are so interested.

Broadcom already has a ton of IP for AI SoCs. I'm guessing the hard parts of this inference chip was already designed by Broadcom and OpenAI simply told Broadcom what it wanted. It's likely very similar to Google's TPU.

  Early testing shows that the first-generation accelerator will deliver performance per watt substantially better than current state-of-the-art

Personally, I don't think OpenAI should have invested in this project. It's too early for them. They should have focused on models like Anthropic and win there. When they're profitable, they can take on these projects.

The risk here is very high for OpenAI because AI has a hard cap in energy. If you have a gigawatt, you should only install the best chips. If Nvidia's chips are better, then this is a wasted project and likely wasted billions.

[0]https://developer.nvidia.com/blog/scaling-token-factory-reve...

Right. There are two possible meanings and shades in-between:

1) OpenAI genuinely have AI technologies that can improve chip design (bold, unlikely claim, needs evidence)

2) OpenAI designed test/verification models and kernels that could be run on the simulated hardware to test its performance

As you and others have said, it's hard to trust when they are happy to write something that could easily only mean the latter but sounds like the former.

There is a lot of verilog out there, it's pretty feasible that they had AI assistance writing more to design their chip.

It doesn't have to be revolutionary, it could just be AI-assisted design and lined up well enough with their operations for a custom ASIC to be worth it.

Written with AI is the new written in Rust. Both are nonsensical statements and tell noting about the quality of the software.

Without context, both are warnings about the quality of the developers.

> the use of email, spam filters and spellchecker to accelerate parts of the design and optimization process

honestly you don't realise how much more efficient it is until you are stuck using the wrong flavour of outlook, the spam filter breaks or sloppy spelling, punctuation and grammar force you to clarify details needlessly.

I feel like "the use of OpenAI models to accelerate parts of the design and optimization process" just means that engineers were using ChatGPT to sanity check their designs and suggest potential optimizations, though that's just my take (and I'm quite cynical about AI marketing in general!)

I think this kind of "hard work" is a perfect fit for AI, and something where the complexity for a human is incorrectly extrapolated to LLMs.

Tirelessly wading through heaps of specifications and documentation with very clear goal definitions is hard for a human but easy for an AI. Meanwhile, taking UX and edge cases into account in a business application is easy for a human but hard for an AI.

My girlfriend works at Broadcom doing chip design, and based on what she's told me they JUST got claude code like 3 weeks ago, so I really doubt this means anything beyond them vibe coding some scripts or something...

VHDL, VLSI are well documented languages, with well build test and verification frameworks and harnesses. Even just by iteration you could get there if you have the money to pay for it.

NVIDIA already designs most of their chips using AI. Why would you assume it's meaningless marketing?

realistically, how hard are AI accelerators to design?

I would assume they've already made as big a deal of it as they can without outright lying too much. Read the rest of the press release.

FWIW, Google is now on their 8th generation TPU, having put out the last 4 generations on a 1-year cadence.

AlphaChip is what a chip design with AI is. I'm very suspicious that OpenAI has anything like this or they would be bragging about it.

https://deepmind.google/blog/how-alphachip-transformed-compu...

I just read a claim on Twitter that the reason these companies (Google and Amazon as well as OpenAI) are using Broadcom isn't just for design expertise, but because Broadcom have allocation agreements in place with TSMC and the memory manufacturers.

I recently put 2+2 together.

Broadcom has become wealthy by being Google's TPU hardware partner, including sharing their TSMC capacity with Google, and evidently now they are doing the same thing with OpenAI. What a brilliant way to take advantage of the AI gold rush!

I wish they weren't using their piles of money to extort money out of the software industry like they are with VMWare and Bitnami.

Their demo is almost unbelievably fast, but as I understand it, the limitation of Taalas's strategy is KV-cache. This grows with context length, so either needs to be stored in SRAM (small) or streamed in (slow). Even for a tiny model like the Llama 8B they have in their demo, the KV cache will be ~64kb per token at 8-bit quantization, so at a 1,000-token sequence length you are already at 64MB of SRAM for a single user. This is probably why their demo only lets you generate 1,000 tokens: they can't go beyond that without slowing down inference.

So I'm curious what their strategy is. It seems to me that the options are: 1. Target smaller usecases that can live with a tiny context window 2. Use huge amounts of SRAM (at which point they look like Groq or Cerebras) 3. Make it up with extreme KV-cache compression/quantization 4. Run linear-attention/sliding window attention models

Other commenters have mentioned robotics as a potential application, which sounds interesting.

> seems like soooo much efficiency waiting to be unlocked at the chip level

Well if you are exclusively using GPUs that are general purpose, of course you leave so much efficiency on the table. That’s why Google started making TPUs more than a decade ago. I remember that kerfuffle when Google fired Timnit Gebru when Gebru’s paper used GPUs to calculate the environment impact of LLMs while ignoring the efficiency of TPUs; this basically made Jeff Dean very angry due to that wide efficiency gap.

It'd be cool to see more of this type of thing, but I have to imagine the ability for it to be updated to a brand-new model as new models come out is limited. If that is the case, it's going to be an extremely hard sell.

In a chatbot, 17k tok/s is a neat but nearly useless showcase. In a coding agent it is a meaningful improvement. In robotics, it could be an absolute revolution.

8B models aren't useful in general, but for specific use cases they can provide an enourmous amount of intelligence - nVidia's Tesla/Waymo competitor is a 7B LLM with a 2B diffusion model, and running that at those speeds could be an order of magnitude cheaper than existing solutions.

Low latency is nice. But it would be more interesting if they could demonstrate the efficiency of energy consumption.

It seems technically interesting, but they seem very sparse on details. I don't know if I like the idea of a single unchanging model forever on a chip. How much more expensive would the silicon be if they used rewritable ROM for the weights? Such an arrangement would permit fine-tunes of the model it was designed for, which might minimize concerns about the model becoming outdated.

I think hardware like this is the future for LLM-providers once we reach a point where the models aren't advancing much any more. You could argue we're close now.

The hyperscalers like AWS will made great use of these to serve up models that will be relevant for several years. But right now, we're still seeing significant bumps in model quality every couple of months - especially with open-weight models like Deepseek/Kimi/GLM.

Until that point, though, I don't see how this is ever going to be cost effective vs general purpose hardware.

I also think we'll see miniature versions of this baked into mobile hardware for super fast and efficient on-device LLMs.

this appeared some time ago, https://taalas.com/, but I'm sure there's others thinking these same thoughts. this would be best for small models imo, nothing frontier because that changes too fast

By the way, you've seen Cerebras? It's not gone as far as what you described - loads of cores and RAM but you still load up the weights onto it as software and they need to be streamed into the chip for large models - but it is a whole wafer.

You don't need a single wafer, you can split the model into many smaller different chips and connect inputs/outputs.

Skip VHDL and directly go for GDSII / OASIS. Try to find similar vectors so you get re-usable blocks.

You can dynamically calibrate a chip by fine tuning output.

>> I wanna see an inference chip where the weights are part of the rom of the chip.

I've been wondering about that for a while now. For a lot of tasks putting weights in ROM is probably OK. OTOH:

>> There would be 1 multiplier per weight...

I'm not sure that is a good idea. Maybe if its quantized down to 2 bits... Otherwise maybe a small ROM near each multiplier (or row of them or whatever) so the multipliers could handle N distinct matrix operations without having to move the data from far away.

Another fun thought is to have a row of MAC units on DRAM so a DRAM row would be a vector. Row size might be 64Kbit or 8K weights if they're 8bit. This also keeps the weights and calcs on the same chip. I'm not sure this would put enough multipliers on one chip though. Systolic arrays can have tens or hundreds of thousands each doing one op per clock cycle.

This may be extreme, or, completely stupid, but, why are we not using genetics to "grow" chips in a chemical soup yet? Similar to Verilog/VHDL, don't we have some similar language to express circuits using gene sequences?

> "Downside is this chip would be huuuuge - a whole wafer."

Why don't we have chips like that? If a CPU the size of a postage stamp can do x amount of performance, imagine how much performance you could get if you used an entire wafer of chips running in parallel. Obviously there would be certain use cases, like you couldn't fit an entire wafer in a phone, but still

One token per clock cycle at 1B parameters would imply 2 ExaFLOPS, consuming about 10 KWs

I've also been thinking about this. Although the forward pass of a transformer model also involves some heavier operations like normalization, reciprocals, exponentiations or other non-linearities (GeLU, SiLU) which may (though typically don't) involve learned weights as operands.

Supposedly memristors would be ideal for this (and it would be reprogrammable), but then again, memristors seem to be the carbon nanotubes of the computing world.

> weights [as] part of the rom of the chip

Not really that: you are pointing to Compute-In-Memory (CIM) - techniques where the data (here, a multiplier value) is part of the processor (here, the multiplying circuit).

The problem of "fetch and process" is bypassed completely architecturally: the data is there where the processing happens - it's not moved, there is no latency.

firmware upgrade would mean flashing a huge BIN file.

How would the pipelining work when the next token depends on the last token?

“ Wafer level faults probably won't matter though - neural nets are resistant to a few missing or wrong weights.”

Brain science people “love” traumatic brain injury cases because it can help explore what happens when bits of the “brain wafer” get damaged. We’ve learned a lot from such things.

I wonder if people are intentionally “destroying” parts of the model weights to learn more about what happens? Like could you strategically wipe a gig of the model so it’s “all zeros” and see what happens?

I have to wonder

Training is pretty much a 1x cost, and efficiency there is already on the way down with architectural improvements. Inference though is an ongoing cost which over time takes orders of magnitude more resources, so focusing on making that far more efficient means way greater gains over time.

With Reinforcement Learning, inference is very present in post-training stages now too

Inference costs are higher than training now. I think.

Nvidia is king of general purpose training chips. But inferences can be specialized.

Cerebras's Codex Spark 5.3 has been a huge flop. Small context window and old model. But hopefully they can improve so that we can benefit from 1000 tokens/second with GPT 5.5.

> early testing shows that Jalapeño will deliver performance per watt substantially better than current state-of-the-art

We're starting to see what really matters here, and though this is hand wavy the TPU makes similar claims.

I think googles memo about having no moat still stands (see: https://newsletter.semianalysis.com/p/google-we-have-no-moat... if you are unaware). It kind of makes sense that all of this is looking more like 60's to 90's IBM, DEC, Cray, Sun and the hardware race that happened then. History doesn't repeat but it often rhymes and I suspect that these efforts will follow the same trajectory.

> One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old Dell desktop.

But if you have such a breakthrough could you not also apply it and run 200T models on todays datacenters?

Interesting comment, but the comparison with hard disk drives is probably unfair.

The IBM 350 was commercialized 70 years ago; it took 70 years for someone like you to be able to compare that to a multi-TB SSD.

Furthermore, nothing says that Moore's Law will necessarily apply to LLMs, for decades to come.

I think Jevons Paradox and scaling laws will make this not the case. If bigger models are always better (which seems they are), then will always need high-end hardware.

Usually breakthroughs in computing lead to more usage of computing, not less.

> One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old Dell desktop.

I think there will be specialized hardware (beside GPUs) that would be custom made for LLMs. Yes TPUs exist, but mainly for datacenter. GPUs exist, but they are adapted from mainly graphic application. Once all the demand from data center dries up, innovation will kick in.

> I keep wondering if hardware like this will become obsolete well before it has a meaningful ROI

it will build expertise/infra/know-how foundation for next generation of hardware

True but as someone else pointed out; at that time we'd be interested in running 200T parameter model rather than 200B. Why, you might ask? Law of human laziness - a human will become as lazy as the technology allows it to. With the 200T or 20,000 T model - I'd be heavily incentivized to ask it to make the bread for me that I enjoy making now or create a movie for me (featuring myself) which will maximize the dopamine production in my brain.

I agree with you. Stepping stones are still a part of getting there, if only to be briefly useful.

Looking at the development of memory bandwidth, capacity and prices over the last 10 years there is little indication that’s likely.

Yeah, the narrative feels like pre-IPO shenanigans, and it looks like the lid on my laundry basket. I wouldn’t be surprised if this is a con.

Who's IPO? Broadcom and Google are already listed, obviously.

If GPUs have 75% margin then 50% cheaper is no surprise.

"Typical" is doing a lot of work there. That could mean much older chips than Nvidia is currently selling.

Don't forget Stargate.

Update: Somebody on Twitter said it's going to be hosted 50/50 at Microsoft and Oracle.

Can you suggest some tutorials for Verilog and FPGAs in general?

I have a spare Tang Nano 9k but I don't feel confident about blindly asking Claude to vibecode me a solution and still would like to have at-least a basic level of understanding.

I have been eyeing what Taalas is doing [1] by making pure hardware models. The speed is absurd.

[1] https://taalas.com/products/

“People who are really serious about software should make their own hardware.” ― Alan Kay

What are the other phases. Or what are you referring to in general?

Both Google and Amazon also codesign heavily with Broadcomm (Amazon also with Marvell and Alchip)

Broadcomm does stuff like physical design, provides IP blocks, managing manufacturing process with TSMC, packaging and testing. Google and Amazon work with system architecture, performance targets, and requirements but Broadcomm as consultant.

Memory bandwidth is the bottleneck in the Spark. If you replace the SoC with an optimized ASIC but keep the same 256-bit LPDDR5 the performance will be the same. You can increase performance by using wider memory but that's also more expensive.

Unfortunately Sam Altman won't be the one to deliver us at-home hardware that can run Opus-level models

Forget about it. Datacenter class hardware is getting farther and farther from desktop use. It’s not PCIe GPUs anymore.

This is just an uncut wafer - I don't think it's intended to be a wafer-scale chip.

Cerebras etch memory onto the wafer alongside the processing elements, but AFAIK OpenAI are going to be using HBM memory and a conventional chiplet design.

You are underestimating how difficult it is even for a large nation state to attract the kind of talent and investment it would take to set up a chip industry. It is out of reach for anyone outside of the 3-5 largest national economies and a few big American/Chinese multinational corporations.

28nm chips is just about suitable to run frontier models

I doubt it. 28 nm is 4-5 generations back so inferencing would need a large number of chips with very high power consumption. Maybe you're thinking more of 7 nm which is what Chinese fabs have; it seems to be OK for companies like Huawei.

And so if I was a mid-level State would it be worth while to take my nascent chip industry and push it out to build a 28nm foundry and supporting eco-system.

It never reaches breakeven so you'd have to provide billions in subsidies per year forever. The sovereign chip stuff only makes sense for the US and China; even the EU probably isn't large enough to make it work. A single country definitely couldn't.

But the energy requirements per token would be orders of magnitude worse than chips made at 3nm. So probably better for your hypothetical state to just pay the extra for more efficient chips so that they don't have (as much) of an energy problem.

> Even if LLMs don’t become AGI they will still be incredible tools

(Mostly an aside, but: LLMs have paved the way, now the problem is there, it is a challenge and a geopolitically relevant race... AGI is a goal set: not-having-reached-it will be just a stage.)

But there never really was a moat in LLM?.. I mean, I don't know where you stand, but my perception is that we all kinda knew that the whole time since 2017, and really knew that since DeepSeek. What they really care about is:

1. Customer acquisition.

2. Cheap(er) electricity/hardware.

So it's really surprising to me that them making their own chip surprises anyone at all. The electricity thing is already kinda being taken care of by earlier strategic alliances with some other evil people, the chip is a natural next step.

Definitely has that smell... At the same time though, they NEED inference cost to drop substantially, and even better for them if it only happens for their models on their hardware.

I assume they're doing everything they can to make that happen model-side, but coming at it from the other end makes sense too if they can swing it.

Maybe, but they’re also a massive company. At some point Google stopped being a startup and become a massive company with margins to look after

Nearly all those initiatives have failed though

I’m not understanding. If cost per token hits the floor that does not mean that you want a model that uses tokens.

If the Chinese are optimizing for token usage, that’s also speed.

Why use more token if few do trick?

I don’t know. I’m kind of glad that two of my least favorite companies are working together.

CA Technologies was much worse than Broadcom in its heyday.

Three of their top execs - CEO, CFO, and head of sales - went to federal prison on securities fraud, conspiracy, and other charges. The CEO, Sanjay Kumar, who was at least partly the fall guy for co-founder Charles Wang, served 10 years.

Being acquired by Broadcom could only have been an upgrade, as strange as that may sound.

> 17k tps taalas chip

It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually. And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases.

I wonder if it's being worked on in secret, if there's something about it that makes it infeasible, or if companies are really too nervous to lock in one model like that because the next one down the line could be a huge improvement. Re. infeasability, I have heard that the Taalas demonstration chip ran Llama 3.1 8B (a pretty horrible model) and that even that took a massive amount of transistors / die area. So it might just be the case that the good models are too big to fit on silicon?

The current taalas chip is for a 3.1B param model. I’m hope so much that they can get that up to the 30B range. Just imagine Gemma 4 or Qwen 3.6 at 17k tps.

Because they have the skills necessary to help bring custom designed ASICs to fruition. Google uses them for their TPU's, Meta uses them for their custom ASICs as well.

They don't have true competition, what they lose out on is market share with hyperscalers, since OpenAI would have no plans to share inference hardware with any other company right now. Plus, I don't know how does NVIDIA's investment equation pans out long terms given OpenAI will be investing in more purpose built inference stack for the future.

they're still kings for training, though I've heard Anthropic is training now on JAX+TPU setup, so might not be a monopoly in that segment.

that's not really how it works

Investors, I'm sure everyone's had the idea and they're doing it.

I assume OpenAI has been buying memory and "giving" it to Nvidia in exchange for a discount.

If you're referring to the big circle of silicon, that's a wafer, generally contains many chips (100-1000s).

I think Cerebras stock going down could also be partly caused by the lock-up period ending today for 200k shares (page 73 of their prospectus) - https://www.sec.gov/Archives/edgar/data/2021728/000162828026...

Dang, I just checked and CBRS is in free-fall since the IPO.

Sucks, I think they're a cool company.

OTOH, I was the only person back then pushing hard during my time at KAUST (back in 2019) to buy one of their systems when they were nobody, eventually resulting in a partnership between the two.

Then I joined their online discourse, very few users, I was semi-active there but they didn't care much.

Then I came to Toronto and heard they were opening an office here, tried to get noticed several times but got mostly ignored. I asked about upcoming events several times, anything to get involved, "yeah man, maybe one day". Then they made an event during Toronto Tech Week and didn't even tell me ... idk.

I don't get schadenfreude as I still think they're a cool company.

My point is they put all the eggs in one basket (AI inference) and neglected everything else. They seem to be on shaky ground now ... sad.

That's just the wafer disc. Looks like it was presented to Sam Altman for ceremonial purposes.

The wafer disc is what the CPU gets "printed" on.

Because of Micron, no? I don't think it's related to OpenAI's announcement

Although, custom HW has to be the focus right now - simply because we are dealing with a technology (big NNs) that are not the best match with Von Neumann architectures.

It's probably the same speed but cheaper.

Actually, I find the idea of using Cerebras etc. for /training/ (not just inference) surprising: I did not stumble in much data and discussion about "super-CPUs" in that area, where NVidia (with the tools focused on it) has that long-built edge...

Edit: contextually,

> Jalapeño is specifically designed for inference

You don't need a whole lot of software support if you just want to serve a single family of LLMs.

That’s exactly what this is.

9 months to production is completely impossible anyway.

9 months from design to early samples is probably impossible given than TSMC takes 3 months after tape out to produce them. Then it’s up to the customer to qualify and revise for production. TSMC doesn’t do that.

There’s no AI that makes this happen in 9 months.

No worse, I suppose, than, the obsession with Lord of the Rings that the authoritarian surveillance companies have. Palantir, Anduril. Then we have the not defense/surveillance ones: Mithril, Valar, Narya, Erebor

Strawberry was too complicated as a codename.

Don't worry, in Europe it's the same, but for insurances/lawyer stuff. Tons of companies have names based on Latin words such as Civitas/Insalus/Legalia/Legalitas or whatever which looks tacky/rancid/old fashioned kilometers away.

Jalapeño

Jalapeño

Jalapeño

Really has a… ring to it