Hacker News

Assuming Andy (the mailing list post) is right, no it didn't.

I know this because he said so here in this comment section: https://news.ycombinator.com/item?id=49134550

That said the AI was sort of directionality right... it identified roughly the right code.

That's good! But I wonder why it wasn't then enabled for that configuration or why the override wasn't "global" enough and the default allocator was still partly used.

Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator.

mallocng should not be used at all.

It's maybe a kernel bug.

The ISO C definition of opendir() requires an allocation in practice because it returns a DIR* and it's bad practice for the library to arbitrarily limit how many of those an application has at a time.

Maybe a C library could preallocate several DIRs and only use the heap when those are exhausted, but this ripgrep use case (lots of threads running in parallel on a large tree) would still be likely to trigger that.

> All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it's hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page's set of objects.

In theory, that's what git's packfiles are for: loose objects get packed together into a file, and if the file is well-arranged then objects that tend to get read together will also end up getting paged-in together (see also: https://www.kernel.org/pub/software/scm/git/docs/technical/p...).

GitHub in particular has spent a lot of time and effort on packing things well (e.g. https://github.blog/open-source/git/scaling-monorepo-mainten...), so I doubt that the "billions of tiny files" problem is at the root of their recent instability.

I'm becoming increasingly allergic to the performative writing of LLMs.

The worst part is how compulsive LLMs are at writing like this. Like the system prompt instructs it to "reason" and like a college Sophomore it pontificates and quotes Nietzsche to fein intelligence and orginal thought.

This is technical writing for kernel developers who already know how memory management works. If you want to understand it better, you should study up on the x86 MMU and the kernel memory management subsystem. I'm not a kernel expert but I found it pretty understandable, so I'll try to explain it.

MMU: The memory management unit, part of the CPU designed to allow an OS kernel to flexibly control how memory addresses used by a program map to physical RAM.

Page: A 4k region of memory addresses.

Anonymous: The page is just regular old memory used by a single process for general purposes, it's not part of e.g. a memory-mapped file.

Backing: The memory corresponding to a page. The kernel can direct the MMU to map any page to any part of RAM, or delete the mapping entirely.

Zero page: The kernel keeps a single 4 KB page filled with zeros (the "zero page").

Page fault: An error that occurs when you try to write a page that's not present, or write to a read-only page. Some page faults are normal and expected by the kernel, they support features like swapping, memory-mapped files, etc. Such "expected" page faults are invisible to your program. (On the other hand, an unexpected page fault occurs when your program just straight-up accesses an address it's not supposed to, which will end your program with a SIGSEGV signal -- an all-too-familiar experience for a C programmer.)

When you allocate say 4 MB of memory, the kernel simply sets it to 1k read-only copies of the 4 KB zero page. Then when your program tries to write to a page, it faults. The kernel handles this expected fault by assigning that page an individualized memory region. If a program allocates a lot of memory, it's only assigned memory for what it actually uses, when it first uses it. This is an optimization: Every program that allocates more memory than it needs is wasteful, but the kernel can recover that waste by not backing the allocation with memory -- the program has to prove each page is needed by writing to it.

Store to a freshly-faulted anonymous page: The program stored data to its own memory, which caused an expected page fault.

> becomes invisible to that same thread's reload ~10 instructions later

The thread fairly quickly tried to read the data from the same place in memory it had just written to. It doesn't get the same data back. This is a major problem that causes the program to malfunction and crash.

> A pagemap read at the instant of the fault shows the backing is the kernel's zero page

The expected fault occurred because the program tried to write to an address that was mapped to the zero page, probably because it's the first write to freshly allocated memory. (This rules out other reasons it might be faulted, e.g. the page had been swapped to disk.)

> concurrent munmap's TLB shootdown

The page table is big and slow so the CPU caches parts of it in an area called the TLB (translation lookaside buffer). munmap is a kernel function that removes the backing for an address region. Since munmap changes the mapping, it has to inform the MMU the relevant part of the TLB is no longer valid.

> per-VMA-lock anonymous-fault fast path

This is what the kernel does when the program first writes to a zero page. Saying it's the "fast path" implies there's some (slow) special case handling in the kernel code, but the bug doesn't trigger any of the special cases, we're in the most common case, which the kernel code tries to handle as quickly as possible because it has measurable impact on performance. I would guess per-VMA-lock has to do with how the kernel synchronizes this code (a "lock" is a basic synchronization primitive, you should be familiar with it if you do any kind of multithreaded programming).

> The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interaction

The mechanism: What's actually happening to cause the program not to be able to load data it just stored in memory.

Localizes: The cause of a problem like this is a needle in a haystack -- it could be caused by the program, the kernel or the hardware. The analysis has narrowed down the haystack to a specific part of the kernel code.

Race in that interaction: Two parts of the kernel code, (1) The kernel code for munmap and (2) the kernel code to handle a program's first write to the read-only zero page for a fresh allocation. These two parts work fine individually but the problem occurs when they both try to change the page tables / TLB at exactly the same time. This is quite a small, specific haystack compared to "somewhere in the program, kernel or hardware" we started with.

Agreed, the article you linked is a great example of clear, understandable, detailed technical writing.

fwiw "backing" is standard jargon in this context, and "freshly faulted" is phrasing you'll see in Linux source comments. The writeup as a whole is indeed slop, though.

This is a pretty clear analysis if you are a kernel developer.

> a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function

A result of a machine instruction to store something in RAM got erased, when the same thread attempted to load it ~10 machine instructions later. The reason is that the physical RAM page backing that particular virutal page got replaced while the thread was running.

The fault happens in the fast path for anonymous (i.e. not backed by files on disk) pages, in the granular per-VMA (virtual memory area) locks.

> "A pagemap read at the instant of the fault shows the backing is the kernel's zero page." -> Backing?

Yes, it's typical kernel terminology. "A backing page" or a "backing file".

> This is... words strung together. Nothing more. I wonder how people read and make sense of this.

There's nothing wrong with this description. It's just written for kernel developers. It needs to be expanded and explained for people who are not.

"backing" is a Virtual-Memory related term. With virtual memory, you can have memory areas (called "pages") that are not really there but only present in the metadata (the "book-keeping", so to say). The first time, someone actually tries to access this page, that access is interrupted ("faulted") and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actual physical memory and fill it with zeroes, or crash the process).

The "backing" refers to the physical memory that might or might not be present for every "virtual" piece of memory that your program has allocated.

"Freshly faulted" means that a page of (virtual) memory has just received a "backing" by the process above and is, thus, very fresh in physical memory (even though the virtual memory might have been allocated much earlier)

"~10 instructions later" refers to the assembly- (machine-) code, which, contrary to a high-level language like C, usually has long(ish) sequences of rather simple "instructions". 10 instructions is a rather short interval in assembly code.

As for the "localize", the term used is actually "localizes to" which I read as "turns out to be located in" (probably just a bad English translation by the original author)

While this whole summary reads a bit weird, I don't think it is necessarily the result of an LLM, it's probably just that someone who is rather inexperienced at writing up technical summaries did it.

I don't care if the cat is black or white, so long as it catches mice.

Translation: if this nails-on-chalkboard LLM spew identifies a kernel bug, which is then patched, its aesthetic qualities do not interest me in the slightest.

The computer seems to have identified source code to blame though? Unless of course it hallucinated which there’s always a non zero chance of

Stopped after

> Headline. The crash is real and reproducible.

Maybe it's not going to get any better. LLMs were trained to do two things:

- Mimic human language (and logic, since that's part of what we express with language). This includes code written by humans.

- Write code to solve problems. The figure of merit here is solving the problem, not reproducing anything human.

I'm not an expert on LLMs, but it's not obvious that the human reasoning they are fitting in the first case is going to be anything like the problem solving required in the second case. It was trained to get itself out of a problem, not to do it in a way that a human would relate to.

And we're already running out of human data to train on, while synthetic data has no limit. Bug reports in the future might amount to "fix this because then I'll get a cookie".

The part that doesn't make sense to me is that trying to read through a zeroed PTE should be an immediate segfault, not something that reads back zeroes (regardless of whether the zero page is pfn 0 (it isn't)).

I think the broad strokes are that, due to bad locking in the kernel, mmap() returns a VA that a concurrent munmap() is still in the middle of unmapping. The specifics beyond that seem murky/speculative/inconsistent.

Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)

The explanation is obviously written by an AI agent.

I can read it but not sure if worth spending energy on it.

It doesn't make sense for the reader to spend more energy than the writer spent on creating it.

Why is it unreadable? I actually find LLM bug reports/breakdowns to be far more detailed and concise that classical human written ones. If you read the linked repo it clearly goes it depth where the bug was found, how to reproduce it (and in depth). Most disclosures that are human written don't do this at all, they barely even tell you _how_ to reproduce the bug. Just look at the "3.3 The self-store tear", the LLM clearly describes exactly what went wrong, so you can verify it by hand.

I had the opposite reaction. Clear, detailed, well-organized. Pretty close to the ideal writeup.

Codex bundles a ripgrep binary that is linked against musl. This is noted in the bug report: "I originally encountered this bug in the rg bundled with OpenAI Codex. That binary is byte-for-byte identical with the one in https://github.com/BurntSushi/ripgrep/releases/download/15.2... [...]"