Hacker News

First of all, thank you for presenting a succinct take on this viewpoint from the other side of the fence from where I am at.

So how can I learn from this? (Asking very aggressively, especially for Internet writing, to make the contrast unmistakable. And contrast helps with perceiving differences and mistakes.) (You also don’t owe me any of your time or mental bandwidth, whatsoever.)

So here goes:

Question 1:

How come "speed", "performance", race conditions and st_ino keep getting brought up?

Speed (latency), physically writing things out to storage (sequentially, atomically (ACID), all of HDD NVME SSD ODD FDD tape, "haskell monad", event horizons, finite speed of light and information, whatever) as well as race conditions all seem to boil down to the same thing. For reliable systems like accounting the path seems to be ACID or the highway. And "unreliable" systems forget fast enough that computers don’t seem to really make a difference there.

Question 2:

Does throughput really matter more than latency in everyday application?

Question 3 (explanation first, this time):

The focus on inode numbers is at least understandable with regards to the history of C and unix-like operating systems and GNU coreutils.

What about this basic example? Just make a USB thumb drive "work" for storing files (ignoring nand flash decay and USB). Without getting tripped up in libc IO buffering, fflush, kernel buffering (Hurd if you prefer it over Linux or FreeBSD), more than one application running on a multi-core and/or time-sliced system (to really weed out single-core CPUs running only a single user-land binary with blocking IO).

Sorry, complete noob here. Why didn't you just cd into $(yes a/ | head -n $((32 * 1024)) | tr -d '\n')? Why do you need to use the while loop for cd?

EDIT: got it. -bash: cd: a/a/a/....../a/a/: File name too long

Probably a dumb question, but is GNU Core utils interested in / planning on doing its own rust rewrite?

To be fair, Vec::set_len bug in Rust was in 2021. And even then it had to be annotated as `unsafe`. It was then deprecated and a linter check was added: https://github.com/rust-lang/rust-clippy/issues/7681

Reading that Canonical thread was jaw-dropping. Paraphrased: "Rust is more secure, security is our priority, therefore deploying this full-rewrite of core utils is an emergency. If things break that's fine, we'll fix it :)".

I would not want to run any code on my machines made by people who think like this. And I'm pro-Rust. Rust is only "more secure" all else being equal. But all else is not equal.

A rewrite necessarily has orders of magnitude more bugs and vulnerabilities than a decades-old well-maintained codebase, so the security argument was only valid for a long-term transition, not a rushed one. And the people downplaying user impact post-rollout, arguing that "this is how we'll surface bugs", and "the old coreutils didn't have proper test cases anyway" are so irresponsible. Users are not lab rats. Maintainers have a moral responsibility to not harm users' systems' reliability (I know that's a minority opinion these days). Their reasoning was flawed, and their values were wrong.

More than that: it seems that Rust stdlib nudges the developer towards using neat APIs at an incorrect level of abstraction, like path-based instead of handle-based file operations. I hope I'm wrong.

Someone once coined a related term, "disassembler rage". It's the idea that every mistake looks amateur when examined closely enough. Comes from people sitting in a disassembler and raging the high level programmers who had the gall to e.g. use conditionals instead of a switch statement inside a function call a hundred frames deep.

We're looking solely at the few things they got wrong, and not the thousands of correct lines around them.

Seems pretty impressive they rewrote the coreutils in a new language, with so little Unix experience, and managed to do such a good job with very little bugs or vulns. I would have expected an order of magnitude more at least.

Shows how good Rust is, that even inexperienced Unix devs can write stuff like this and make almost no mistakes.

Memory safety catches buffer overflows. CI catches logic bugs. Neither catches the Unix API gotchas nobody documented.

I expect nothing less from the creators of unity, upstart, and snap.

What's even harder is doing that while trying to avoid the GPL, so doing that without reading the original source code.

uutils would be so much better imo if it was GPL and took direct inspiration from the coreutils source code.

good example from the article: the chroot+nss CVE. the rule that nss is dynamic and dlopens libraries from inside the chroot isn't anywhere obvious. it's encoded in 25+ years of sysadmins finding it out. clean-room rewrites end up re-learning that, usually as new CVEs. and LLM ports of the same code inherit the problem: the function signature is what they read, but the scars are what they need.

> The code gets silently encumbered with those lessons, and unless they are documented, there's a lot of hidden work that needs to be done before you actually reach parity.

It should be stressed that failure to document such lessons, or at least the bugs/vulnerabilities avoided, is poor practice. Of course one can't document the bugs/vulnerabilities one has avoided implicitly by writing decent code to begin with, but it is important to share these lessons with the future reader, even if that means "wasting" time and space on a bunch of documentation such as "In here we do foo instead of bar because when we did bar in conditions ABC then baz happens which is bad because XYZ."

hmm... maybe a 'write lock' on the directory? though this will become more hairy without timeouts/etc...

My understanding is the uutils development process involved extensive testing against the behaviour of the original utilities, including preserving bugs.

Because it's Ubuntu policy to replace some foundational part of the system with some janky unfinished experiment in every release.

I agree with you that that's more the story here than "OMG, somebody wrote Rust code with bugs in it".

> and I'm striving to keep it 100% Zig with no libc calls anywhere. Which may or may not turn out to be possible, we'll see.

Probably will depend on what platform(s) you're targeting and/or your appetite for dealing with breakage. You can avoid libc on Linux due to its stable syscall interface, but that's not necessarily an option on other platforms. macOS, for instance, can and does break syscall compatibility and requires you to go through libSystem instead. Go got bit by this [0]. I want to say something similar applies to Windows as well.

This Unix StackExchange answer [1] says that quite a few other kernels don't promise syscall compatibility either, though you might be able to somewhat get away with it in practice for some of them.

[0]: https://github.com/golang/go/issues/17490

[1]: https://unix.stackexchange.com/a/760657

Nope! But basically, expect anything that resolves usernames, or host names, to be done in the userspace by NSS.

    Sun engineers Thomas Maslen and Sanjay Dani were the first to design and implement
    the Name Service Switch. They fulfilled Solaris requirements with the nsswitch.conf
    file specification and the implementation choice to load database access modules as
    dynamically loaded libraries, which Sun was also the first to introduce.

    Sun engineers' original design of the configuration file and runtime loading of name
    service back-end libraries has withstood the test of time as operating systems have
    evolved and new name services are introduced. Over the years, programmers ported the
    NSS configuration file with nearly identical implementations to many other operating
    systems including FreeBSD, NetBSD, Linux, HP-UX, IRIX and AIX.[citation needed] More
    than two decades after the NSS was invented, GNU libc implements it almost identically.

> The root cause of some of the bugs seems to be the opaque nature of some of the Unix API.

Seems and smells is weasel words. The root cause is not thinking: Why is root chrooting into a directory they do not control?

Whatever you chroot into is under control of whoever made that chroot, and if you cannot understand this you have no business using chroot()

> To me such a get_user_by_name function is like a booby trap

> I'd say, either split getting the user data and loading any shared libraries in two separate functions, or somehow make it clear in the function name what it is doing.

You'd probably still be in the trap: there's usually very little difference between writing to newroot/etc/passwd and newroot/usr/lib/x86_64-linux-gnu/libnss_compat.so or newroot/bin/sh or anything else.

So I think there's no reason for /usr/sbin/chroot look up the user id in the first place (toybox chroot doesn't!), so I think the bug was doing anything at all.

> The root cause of some of the bugs seems to be the opaque nature of some of the Unix API.

Some, maybe, but if you've decided to rewrite coreutils from scratch, understanding the POSIX APIs is literally your entire job.

And in any case, their test for whether a path was pointing to the fs root was `file == Path::new("/")`. That's not an API problem, the problem is that whoever wrote that is uniquely unqualified to be working on this project.

Rather, I think that using a functional safe language tricks people into thinking that the data it deals with is stateless. Whereas many many things change in operating systems all the time.

Until we have a filesystem that can present a snapshot, everything has to checked all the time.

i.e. we need an API which gives input -> good result or failure. Not input -> good result or failure or error.

Yes thats one thing Musl libc removes.

I think a lower amount of training data for Haskell might be a reason.

For core system functionality maybe. But for most applications Rust slow compiler iteration speed becomes a bottleneck when the likes of TypeScript (with Bun) and Go have sub second iteration times.

Plus AI is also good at catching, in other languages, errors that Rust tooling enforces. Like race conditions, use after free, buffer overflows, lifetimes, etc.

So maybe AI will become to ultimate "rust checker" for any language.

The way Haskell handles memory is weird and can be unpredictable.

I recall the last time there was a massive bug in the uutils project, it was because the coreutils tests didn't cover some crucial aspect people relied on. Running these tests is useful for compatibility and all, but it won't necessarily catch security issues.

Looks like they've been doing at some kind of automated comparison against the GNU test suite since 2021 or so [0]?

[0]: https://github.com/uutils/coreutils-tracking/commits/main/?a...

> Pretty shocking to see the lack of basic thought going into writing what is meant to be critical infrastructure

uutils did not start off as "let's make critical infrastructure in Rust", it started off as "coreutils are small and have tests, so we're rewriting them in Rust for fun". As a result there's needed to be a bunch of cleanup work.

So yeah, their implementation of chmod checked if a path was pointing to the root of the filesystem with 'if file == Path::new("/")'.

How the f** did this sub-amateur slop end up in a big-name linux distribution? We've de-professionalized software engineering to such a degree that people don't even know what baseline competent software looks like anymore

I love Rust, but I wonder if this is an example of the idea that its excellent type system can lull some people into a false sense of security. Particularly when interfacing to low-level code like kernel APIs, which are basically minefields inadvertently designed to trick the unwary, the Rust guarantees are undermined. The extent of this may not be immediately obvious to everyone.

> the uutils authors are well experienced in Rust

I'm not sure that they were all that experienced in Rust when most of this code was written. uutils has been a bit of a "good first rust issue" playground for a lot of its existence

Which makes it pretty unsurprising that the authors also weren't all that well versed in the details of low-level POSIX API

It's not designed to completely eliminate other bug classes but it is designed to reduce the chance that they happen.

In this case the filesystem API was perhaps not as well designed as it could have been. That can potentially be fixed though.

Some of the other bugs would be hard to statically prevent though. But nobody ever claimed otherwise.

Few things to note

1. uutils as a project started back in 2013 as a way to learn Rust, by no means by knowledgeable developers or in a mature language

2. uutils didn't even have a consideration to become a replacement of GNU Coreutils until.... roughly 2021, I think? 2021 is when they started running compliance/compatibility tests, anyway

3. The choice of licensing (made in 2013) effectively forbids them from looking at the original source

> It seems like these supposed coreutils replacements are being written by people who don't know anything about Unix, and also didn't even bother looking at the GNU tools they were supposed to be replacing.

They're a group of people who want to replace pro-user software (GPL) with pro-business software (MIT).

I don't really want them to achieve their goal.

They are deliberately not looking at coreutils code because the Rust versions are released as MIT and they don't want the project contaminated by GPL. I am not fond of this, personally.

What's the point of a "rewrite in Rust" when it introduces bugs that either never existed in the original or were fixed already?

> I'd be interested in a comparison with the amount of bugs and CVE's in GNU coreutils at the start of its lifetime

The point is, those bugs had been discovered and fixed decades ago. Do you want to wait decades for coreutils_rs to reach the same robustness? Why do a rewrite when the alternative is to help improve the original which is starting from a much more solid base?

And even when a complete rewrite would make sense, why not do a careful line-by-line porting of the original code instead of doing a clean-room implementation to at least carry over the bugfixes from the original? And why even use the Rust stdlib at all when it contains footguns that are not acceptable for security-critical code?

i don't think CVEs were a thing at the start of the GNU rewrite

You’re right, but it’s gonna be hard to stop them from raging. In many ways people want to be justified in a „see, I told you so, Rust is useless” belief, and they’re willing to take one or two questionable logical steps to get there.