Hacker News

It's been annoying to me as an end user that so many basic things require nightly. I use nightly as my main toolchain, but enabling unstable features makes a project nightly-only, which is undesired for crates that don't already revolve around the unstable feature.

I most often encounter unstable features when I reach for a basic common-sense utility method and discover that it's not stable. Like just earlier today I would have reached for bool::toggle which not only is unstable, but is also newly added as of like a month ago! but some unstable methods have been sitting around for years.

And now that IntelliJ-Rust is proprietary, I can't even make a feature request anymore for the ability to exclude unstable features from the autocomplete. So they will taunt me forever, perfect little helpers just locked away.

> GPUs aren't really SIMD, they're SIMT (single instruction multiple thread)

False. If they were threads they'd have their own PC. They do not - only the warp has a PC.

> You just write plain scalar code and the hardware model dispatches it efficiently to SIMD execution units.

Absolutely not. If you don't write coalesced loads, bank-conflict free, predication-free, cooperative code you will get worse than CPU performance.

32 wide - only AMD has a 64 wide mode

64 what? Bits/bytes/something bigger?

.NET and Java have three levels of SIMD support, Go's ongoing efforts, and does the upcoming C++ standard.

Autovectorization, depending on compiler's cleverness, really portable SIMD operations, and then the CPU specific SIMD ones.

So this should be perfectly doable in crate that advertises as portable, while leaving the non portable stuff to another crate.

Except you can't use this in actual code, because either, as is the case in this example with f32x32, you run out of registers and spill all over the place. Or you aren't using your full vector register or could've gotten better performance by "unrolling" more often for the larger vectors.

If you use f32x16 (the avx-512 wisth), SSE now effectively has 4 registers to work with and will spill when doing anything beyond the most simple stuff.

The default should imo be relative to the native register width, so you can do 1x, 2x or sometimes 4x the native width, depensing on your register preasure.

Sure but there's no real way to use that in a portable way, at least not a way that maximises performance on every CPU you run it on. That's pretty much impossible at the moment.

The create is called portable_simd.

There is no reason a portable_simd relu_dot implemention should need to specify the SIMD width.

But the design and documentation of portable_simd makes the fixed size syntactically easy/the default and the width agnostic code harder.

Counter question: why shouldn't it be portable?

It's definitely a 80% solution where you occasionally need to drop down to intrinsics (at zero runtime perf cost) for CPU specific instructions.

But just having vector types, arithmetic, swizzling, loads and stores will go a long way for basic tasks.

And with generics you can write code that is type and width agnostic. No need to rewrite your code of you want to go from SSE to AVX512, just change from f32x4 to f32x16 (or use generics) and you are done.

The tentative plan is to open source all the compiler and `std` bits with our products built on top (compilers are not good businesses). More about our products coming in the next couple of months!

Didn't want to go into crazy detail in the post.

Each family of operations is a trait parameterized by the operation itself:

  pub trait EvaluateReduction<Operation, T>: LaneEvaluator {
      /// Reduce one distributed definition to an ordinary uniform scalar.
      fn evaluate_reduction(&self, value: LaneValue<Self, role::Distributed, T>) -> T;
  }

  let one   = evaluator.splat::<Splat, _>(1_u32);
  let two   = evaluator.splat::<Splat, _>(2_u32);
  let three = evaluator.binary::<Add, _>(one, two);

  let total   = evaluator.reduce::<Sum, u32>(three);   // a uniform u32
  let running = <Executor as EvaluateScan<Scan<Sum, Exclusive>, u32>>::scan(&evaluator, three);

As mentioned in the post, execution shape is typed too. A static shuffle takes its control as a type-level constant, and the shuffle mode constrains which controls are expressible:

  // Shift down one lane, keeping our own value where the source is inactive.
  let down  = <Executor as EvaluateShuffle<Shuffle<Down>, DownOrSelf<1>, u32>>::shuffle(&ev, v);
  // Broadcast from lane zero.
  let bcast = <Executor as EvaluateShuffle<Shuffle<Broadcast>, WarpLane<0>, u32>>::shuffle(&ev, down);
  // Butterfly exchange with the neighbor one bit away.
  let bfly  = <Executor as EvaluateShuffle<Shuffle<Xor>, Butterfly<1>, u32>>::shuffle(&ev, bcast);

  <ScopedWarpExecutor<'_, WarpUniform> as EvaluateBarrier<Barrier<Device>>>::barrier(evaluator)
  // error[E0277]: the trait bound `Device: NvptxBarrierScope` is not satisfied
  //               help: the trait `NvptxBarrierScope` is implemented for `Warp`

  // Six work items across four active lanes: two chunks, based at 0 and 4.
  <Executor as EvaluateStripMine<StripMine, (WorkItems, ActiveLanes<StripMined<4>>), i32>>::
      for_each_strip_mined(
          &evaluator,
          (WorkItems::new(6)?, ActiveLanes::new(4)?),
          |index, active| {
           // ...
          },
      );

I'm confused too. How does this fit between these approaches for paraellization:

  - CUDA kernels and Tiles (e.g. Cudarc, cuda-oxide, rust-gpu etc) - SIMD on the GPU. (E.g. as in the title...)
  - CPU SIMD using avx or SSE instructions (And probably thin wrappers for vectors so you can have sane syntax). Or the maybe-upcoming core simd which should abstract over architecture-specific instructions. Magic floats etc which do 4-16 computations at once, but are a bit clumsy to work with
  - Rayon thread pools - arbitrary parallel computations, including SIMD, one per CPU core.

It is not currently available but we intend to make it available after we launch our products.

The goal of this work is to run existing unmodified CPU libraries (which may use core::simd) on the GPU. If you are manually writing ML-shaped workloads, it doesn't add any value over writing with tech like torch/tensorflow/jax which are custom built for those use-cases (except maybe familiarity if you are a CPU programmer).

Part of our thesis is that decent GPUs are in every shipping device and most software doesn't use them and should.

The goal is to use similar abstractions and code across both the CPU and GPU where it makes sense.

The main problem with SIMD is it is a complex subject, even if not that good, autovectorization wins over what most common devs know about SIMD.

Well now you could in theory AI generate SIMD, which will be vibe coded, as those devs have no idea of its correctness.

I did see the article about that too. Don't know about "always" since there are applications like web backends where you're never going to add arrays of floats or something. Even if it's data science stuff, if that's in Python, Numpy is doing the SIMD for you.

Yes, this kind of thing seems to happen quite often. Popular posts spurring further posts on a theme.

It's just a way for us to add minutia and details that most don't care / need to know about. There are three audiences we try to make the posts accessible for: Rust people who don't know about GPUs, GPU people who don't know about Rust, and non-Rust non-GPU people. The toggle lets knowledgable readers go "wait, what about..." and hopefully the toggle answers it.

Could also just be trolls attracted by the Rust topic.