Hacker News

by Ryan Harman

Google open-sources TorchTPU, a native PyTorch back end for TPUs (github.com)

198 points by ilreb 5 hours ago

pcmoritz 3 hours ago

For people interested in this: I've been working on an open-source libtpu implementation for https://tenstorrent.com/ accelerators as a side project, it can already run most of torchtpu, see https://github.com/pcmoritz/libtt (based on the tenstorrent software stack). It shows all the things involved in lowering torch to tile based accelerators.

asparagui 2 hours ago

Very cool! I have been using Lean to drive PJRT like so: https://github.com/brettkoonce/lean4-mlir and in theory your code could unlock another platform!

pcmoritz 2 hours ago

Thanks for sharing, this is super cool! Right now my approach to correctness it to just run as many tests as possible and fix them (torchtpu greatly expands the amount of available tests), but formal verification is also super cool. Having a look at your book https://lean.brettkoonce.com/blueprint.pdf

The Tenstorrent ISA is documented btw https://github.com/tenstorrent/tt-isa-documentation, I wonder if one could do the verification end-to-end with the hardware.

yhjc2692 2 hours ago

This is super cool!

pcmoritz 2 hours ago

Thanks for the encouragement, if you have the hardware and end up trying it out I'd love to get your feedback (feel free to drop an issue in the repo) :)

yhjc2692 2 hours ago

dalemhurley an hour ago

Google and Apple are going to move consumers and ordinary business users to edge inference. This is more disruptive to the upcoming IPOs than most investors understand.

npalli 5 hours ago

Interesting, Amazon doing the same for AWS Trainium.

https://github.com/aws-neuron/torch-neuronx

AnotherGoodName 3 hours ago

Pytorch conference is happening today for explaining the coincidental timing

sroussey 3 hours ago

That explains the 5x speed increases huggingface mentioned for Mac using metal.

mathisfun123 an hour ago

fancyfredbot 3 hours ago

Cool! XLA is pretty good + potentially a better compiler infrastructure than inductor which will lead to improved performance on all hardware.

Also I guess this is the final nail in the coffin of Tensorflow.

ismailmaj 2 hours ago

Amazing news, I remember the days when you had to fight XLA to make PyTorch run on TPUs, it seems we finally got first-party support!

Since HW specifics live mostly in the kernel layer (Pallas or CUDA), I imagine the Jax vs PyTorch question is more of a style preference now?

23agt 4 hours ago

Google is selling TPU shovels via Cloud services and also physically to select partners. So this makes sense.

There will be an abundance of shovels. Nebius already crashed by almost half from its hype peak.

airza 4 hours ago

it doesn't seem like there's an abundance of shovels coming anytime soon

juleiie an hour ago

Shovels and gold aren’t the problem, humans are the problem

either AI achieves hyper-efficiency and triggers systematic economic demand destruction due to unemployment, or it underperforms and sparks a massive tech-sector asset bubble burst

If AI is as good as valuations - it crashes. If AI isn’t as good, it crashes. all eggs in one basket economy.

Every future leads to a massive financial shitstorm

AI is priced in as a technology to end all markets. The last bull market you will ever see.

It’s a miracle markets didn’t notice this “damned if you succeed damned if you don’t” type of inevitable checkmate

I have been building a bunker

nico 2 hours ago

Pardon my ignorance, but who else besides Google has TPUs? Are they commercially available somewhere? What’s their main application right now outside of Google?

cavisne 2 hours ago

You can rent them on cloud + they sell directly to Anthropic. Its pretty well known that Anthropic is a heavy user of TPU's (along with Trainium & GPU's)

Lucasoato 5 hours ago

Are there TPUs outside Google Cloud?

boulos 4 hours ago

Yes. Google is famously selling TPUs on prem to big customers (e.g., Anthropic) and Crux AI is now a venture to do just that.

sschueller 5 hours ago

Google coral USB and m.2 devices are TPUs.

Archit3ch 3 hours ago

"Have you seen any software run on Coral TPUs?"

"No, but I have seen the house that the project built, and it is terrific." ~ some Google director

charleslmunger an hour ago

boulos 4 hours ago

Only in marketing name. They're not related otherwise.

goda90 4 hours ago

Maybe just marketing, but they call the NPUs in their Pixel phones "TPUs"

westurner 5 hours ago

TPU / neural compute accelerator / Math coprocessor;

"OpenTPU: Open-Source Reimplementation of Google Tensor Processing Unit (TPU)" https://news.ycombinator.com/item?id=44111452

From https://news.ycombinator.com/item?id=45848676 :

> How does Cerebras WSE-3 with 44GB of 'L2' on-chip SRAM compare to Google's TPUs, Tesla's TPUs, NorthPole, Groq LPU, Tenstorrent's, and AMD's NPU designs?

There are TPUs made out of silicon carbide with standard fab.

There is in-flight tensor processing with photons, but that's not quite a TPU.

hackernudes 3 hours ago

I think the commenter was asking if Google sells TPU chip hardware to use outside of Google's own infrastructure and cloud.

yipinwong 4 hours ago

Dumb question from non-AI engineer.

Is this somehow Google's answer to today's best DeepSeek open-sourcing inferencing engine?

bitpush 4 hours ago

I dont think big companies like Google react to news like that. Very likely this was meant to go out on Tuesday for weeks now. And it just happened to have coincided with another news.

JacobAsmuth 3 hours ago

Yeah they read the news this morning and whipped up a quick Torch->XLA translation layer and coordinated with four different inference engines (who also all wrote their own blog posts) and shoved this out the door quick to stay on top of the news cycle

parthdesai 3 hours ago

Why be rude on purpose? OP clearly prefaced by saying it was a dumb question

caaqil 3 hours ago

mathisfun123 3 hours ago

they're releasing this today because the pytorch conf is this week...

geremiiah 3 hours ago

No, these are two very different objectives.

Tanjreeve 4 hours ago

They do different things and sit in different parts of the stack (training vs inference). TorchTPU seems more like a play to get more people familiar with Googles TPU chips as they want to sell them to people outside of google.

beardedetim 3 hours ago

Yeah I'd assume this is to allow us to train on their TPUs with torch instead of needing to rent a GPU

yipinwong 3 hours ago

mathisfun123 4 hours ago

lol no this is a replacement for https://github.com/pytorch/xla

tuna74 2 hours ago

Is the Linux driver available?

nelaggy 4 hours ago

so if the moat isn't the model and the moat isn't the inference layer, i guess the moat is the hardware now? what does this mean for frontier labs?

badrequest 3 hours ago

hardware has always been the moat in the AI space

AnotherGoodName 2 hours ago

Yeah a lot of people bought into the idea of CUDA being a moat but we’re well past the point where it’s pretty trivial to jump to another equivalent stack on the software side.

jacquesm 2 hours ago

coredog64 an hour ago

AndrewKemendo 2 hours ago

Will always be true

JacobAsmuth 4 hours ago

Products can also be a moat.