Hacker News

There are few paths we can take here. The CCM's primary responsibility is to implement the cloudprovider.Interface[0] which has node, route, and service controllers. However, the CCM can run arbitrary named controllers as well, giving us a fun extension point for the future. The in-tree CCMs are being phased out in favor of out-of-tree CCMs. The AWS Cloud Provider[1] is a good example of what that's starting to look like.

My colleague demo'd Karpeneter internally. We haven't committed releasing it yet but we're discussing it.

[0] https://github.com/kubernetes/cloud-provider/blob/master/clo...

[1] https://github.com/kubernetes/cloud-provider-aws

Yes this would be great addition! Also it would be awesome to see proper load balancer, ingress controller with `gateway api` and a `api gateway` (two separate things).

I’m not an oxide customer, just a fan. I have lots of ideas around how something like this should look, being disappointed with the complexity and also shortcomings of products on the market for this stuff, both in the cloud and on prem.

You can run the control plane at home. I have a video on how to do it. Until we make a smaller footprint that's the only way you'd get Oxide at home.

Every single Oxide article has comments about people wanting one at home.

A great example of completely misdirected marketing and/or engineering. Oxide should have developed home microcomputers, and they would have sold like cupcakes, with their ASCII art marketing. Instead they do million dollar "mainframes" that no one (except VCs) wants to buy.

fwiw, I am very satisfied with talos and k8s at home.

Are you referring to our RFD system?[0][1] Because if so, good news![2]

[0] https://rfd.shared.oxide.computer/rfd/0001

[1] https://oxide-and-friends.transistor.fm/episodes/rfds-the-ba...

[2] https://github.com/oxidecomputer/rfd-site

Good question. We use our own hypervisor[0] that's not KVM/QEMU. We don't have nested virtualization today so we don't follow the KubeVirt model, though we are discussing what CRDs like OxideInstance would look like for those that want to operate solely in Kubernetes manifests.

The core primitive on Oxide is the instance (virtual machine). We could support some container primitive, but that's a larger product direction discussion. Our host OS is Helios (Illumos) so there are details to iron out there regarding what abstractions we would build and expose to the users. Not impossible but not something we're currently pursuing either given that we have other immediate product asks.

If you're at a scale where compute density, power efficiency, security, and rack-level API management matters then that's where Oxide makes sense for you.

[0] https://github.com/oxidecomputer/propolis

You would use Oxide if you hate Dell/HPE/Lenovo/Supermicro. Also most people won't run k8s on bare metal because they want to dynamically provision a bunch of clusters.

Oxide is shipping the baremetal server as well, with integrated hypervisor - so running kubernetes with kubevirt on it makes less sense than integrating with the host hypervisor

Yeah me too. Like I kinda compare oxide to coreweave and coreweave is built on top of k8s afaik. I'm wondering what the underlying tech oxide leans on. If I was doing this I would be using talos and k8s on baremetal and building on top of that. Container first rather than VM first seems like a big advantage imo. Most workloads will be containers.

When oxide provide your metal?

Yeah me too, unfortunately I cant justify huge 100K USD servers so ill continue with cluster api provider hetzner and metal3. The whole CAPI ecosystem is quite cool.

Reported internally. Thank you!

We're discussing what "native Kubernetes" looks like on Oxide in the limit. There's a bunch to build, some of which is blocked on product gaps. We'll get there though!

I don't necessarily want to match the public cloud experience if there's an opportunity for Oxide to exceed the public cloud experience. Eliminating the overlay is a good example of this. We have customers using external subnets to eliminate the overlay but we haven't integrated that into our controllers yet.

> So you’d be attaching a new volume to the running worker VM for each PVC?

That's what we prototyped before local disk was released and before we started disk hot-plug work.

> Could you attach a single large volume, and do path-based provisioning on that?

Possibly. We'd still want disk hot-plug first. Otherwise, customers would have to create their cluster in a certain shape before using PVCs.

Does seem like that could be an issue for RWX PVCs, unless those volumes can be mounted to multiple nodes. Depending on the workload, you might be better off using NFS.

I cover this a bit in the post. Today they are running something like Longhorn backed by Oxide storage. When we release our CSI plugin they can use that instead.