Hacker News

https://public.meshllm.cloud/

> I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk)

Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why:

- when offloading the weights to RAM or NVMe, you need to transfer the massive weights from your slow storage to the GPU for each layer being processed for each token. And as such you are being bottlenecked by the transfer bandwidth (which is either the men bandwidth of your DRAM or the read speed of your disk)

- when using a distributed setup, the weights stay in the VRAM on each machine, the it's the GPU memory bandwidth that matters for the weights, and it's much higher than the two other bandwidth discussed above and as such the bottleneck isn't here. You need to tranfert data from a group of layers sitting on one device to the next one another device, but the amount of data is much smaller than the weights (we're talking about kilobytes of data, not gigabytes) so the network throughput isn't a limiting factor.

The limiting factor is the network latency: if you split your model between 4 devices, you'll have 3 times the network latency per token. If you're on a network with 1ms latency, that means 3ms of latency per token. Which means the theoretical upper bound for your inference speed without speculative decoding is 30tps (this theoretical limit assumes the computation itself is instantaneous).

So this is unlikely to be practical over the internet (too high of a latency) but on a local/enterprise network with speculative decoding it could totally work.

Edit: note that all of the above is about token generation, for prefill/prompt-processing the distributed setup will almost certainly win (because in this case, the network latency doesn't add up)

This was done on my home lab simulating 5ms latency and jitter between machines. Splits work quite well if you your nodes are over WAN at metro latency’s but not super fast on global WAN.

The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others.

I’m currently working on GLM 5.2 on my lab environment with around 10 tok/s on the same split.

Perf should be fairly straightforward to ballpark. You'll need to transfer roughly 2 . hidden_size . num_shards bytes over the network per token during autoregressive decoding. And divide that number by chunk size during prefill.

That's about the speed I get on a AMD Ryzen AI 9 HX 370 (inside a Framework 13), with Qwen3.6-35B-A3B, so doing the same on that much larger model...

> A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline.

Numbers in this example are arbitrary. How does it actually work? What if the model’s number of layers is 33, or 34?

Is there a document explaining all constraints of this implementation?

Hey, this is a super cool project. It's great to see a lot of the IPFS stuff resurfacing again.

A few questions:

1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed.

2.) Any safeguards against malicious actors poisoning model activations?

What is the incentive for me to join the public mesh? Do you have any fairness guarantees, e.g. if I contribute 1/8th of the VRAM required to run a particular model, do I get at least 1/16th of the inference share, or anything similar to this?

This is super impressive, We have a lab with lots of different epycs and different models - to bring them together this way is amazing. Well done!

I have never really delved into kv cache implementation, do they run effectively separate caches per layer?

If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries.

Is anyone doing best-of-n with a n stage pipeline running each query offset by one?

Would this benefit from integrating with the Colibri project announced here just days ago?

I wonder how security is done in this engine, since it's accepting input from anyone. llama.cpp's RPC layer seems to says that you shouldn't run it in public (I assume because it is lower level and may result in RCE on your GPU)

The obvious burning question is how performance looks over different network conditions on some standard models. Have you done much benchmarking? Is it mainly latency affected or is overall throughput less than the capacity of the GPUs due to being distributed?

How does this differ from exo?

Is it a fully custom inference engine or are you reusing parts of an existing stack? (llama.CPP, vLLM, etc.)

Curious about: does it have fault tolerance if one of the machines goes down mid-inference? Can it dynamically reroute, or does it just retry?

Something like this is nice, where instead of having 1 model with X active experts, you have 10 different models, all small and dense, trained on specific information. and loaded on 10 different servers, with one router.

I’d love a bug report - we can get it working for you!

Throughput is not a problem as you just share relatively small vectors (a few kilobytes in size), the key issue is network latency.

That isn’t true. llama RPC is incredibly slow but staged splits in skippy are orders of magnitude faster.

I'm not affiliated, but yes – the main 'point' of iroh is that it's 'dial-a-key', QUIC with encryption based on the keys of the endpoints.

Just wondering, why do you care about encryption in this context?

The catch is that the token generation speed is going to be limited by network latency, making it unbearably slow to run over the internet.

It can be great on a local network though, especially if your workload is prefill-heavy (more text input to process than output tokens to emit).

Is it? I don't see anything on the website about splitting a model across multiple devices, only about putting local models on the internet, a wholly orthogonal problem (which is already easy with existing tools, since models use an http API).

Cool, always good to have more in the ecosystem. I love Iroh and hope this continues to succeed.