Hacker News

I'm really surprised to see you still hocking this.

We did extensive benchmarking of HHVM with and without your patches, and they were proven to make no statistically significant difference in high level metrics. So we dropped them out of the kernel, and they never went back in.

I don't doubt for a second you can come up with specific counterexamples and microbenchnarks which show benefit. But you were unable to show an advantage at the system level when challenged on it, and that's what matters.

Maybe I'm misreading, but considering it OK to leak memory contents across a process boundary because it's within a cgroup sounds wild.

What metrics were improved by your patches?

We evaluated a few allocators for some of our Linux apps and found (modern) tcmalloc to consistently win in time and space. Our applications are primarily written in Rust and the allocators were linked in statically (except for glibc). Unfortunately I didn't capture much context on the allocation patterns. I think in general the apps allocate and deallocate at a higher rate than most Rust apps (or more than I'd like at least).

Our results from July 2025:

rows are <allocator>: <RSS>, <time spent for allocator operations>

  app1:
  glibc: 215,580 KB, 133 ms
  mimalloc 2.1.7: 144,092 KB, 91 ms
  mimalloc 2.2.4: 173,240 KB, 280 ms
  tcmalloc: 138,496 KB, 96 ms
  jemalloc: 147,408 KB, 92 ms

  app2, bench1
  glibc: 1,165,000 KB, 1.4 s
  mimalloc 2.1.7: 1,072,000 KB, 5.1 s
  mimalloc 2.2.4:
  tcmalloc: 1,023,000 KB, 530 ms

  app2, bench2
  glibc: 1,190,224 KB, 1.5 s
  mimalloc 2.1.7: 1,128,328 KB, 5.3 s
  mimalloc 2.2.4: 1,657,600 KB, 3.7 s
  tcmalloc: 1,045,968 KB, 640 ms
  jemalloc: 1,210,000 KB, 1.1 s

  app3
  glibc: 284,616 KB, 440 ms
  mimalloc 2.1.7: 246,216 KB, 250 ms
  mimalloc 2.2.4: 325,184 KB, 290 ms
  tcmalloc: 178,688 KB, 200 ms
  jemalloc: 264,688 KB, 230 ms

i don't recall which jemalloc was tested.

If you go into Dr Dobbs, The C/C++ User's Journal and BYTE digital archives, there will be ads of companies whose product was basically special cased memory allocator.

Even toolchains like Turbo Pascal for MS-DOS, had an API to customise the memory allocator.

The one size fits all was never a solution.

One of the best parts about GC languages is they tend to have much more efficient allocation/freeing because the cost is much more lumped together so it shows up better in a profile.

I remember in the early days of web services, using the apache portable runtime, specifically memory pools.

If you got a web request, you could allocate a memory pool for it, then you would do all your memory allocations from that pool. And when your web request ended - either cleanly or with a hundred different kinds of errors, you could just free the entire pool.

it was nice and made an impression on me.

I think the lowly malloc probably has lots of interesting ways of growing and changing.

In many cases you can also do better than using malloc e.g. if you know you need a huge page, map a huge page directly with mmap

Yes, if you want to use huge pages with arbitrary alloc/free, then use a third-party malloc. If your alloc/free patterns are not arbitrary, you can do even better. We treat malloc as a magic black box but it's actually not very good.

I think some operating system improvements could get people motivated to use huge pages a lot better. In particular make them less fragile on linux and make them not need admin rights on windows. The biggest factor causing problems there is that neither OS can swap a 2MB page. So someone needs to care enough to fix that.

I used mimalloc to run zenlisp under OpenBSD as it would clash with the paranoid malloc of base.

I feel like the real thing that needs to change is we need a more expressive allocation interface than just malloc/realloc. I'm sure that memory allocators could do a significantly better job if they had more information about what the program was intending to do.

Just out of curiosity are you getting 1GB huge pages on Xeon or some other platform? I always thought this class of page is the hardest to exploit, considering that the machine only has, if I recall correctly, one TLB slot for those.

If there is so much performance difference among generic allocators, it means you need semantic optimized allocators (unless performance is actually not that much important in the end).

I've been using jemalloc for over 10 years and don't really see a need for it to be updated. It always holds up in benchmarks against any new flavor of the month malloc that comes out.

Last time I checked mimalloc which was admittedly a while ago, probably 5 years, it was noticebly worse and I saw a lot of people on their github issues agreeing with me so I just never looked at it again.

Facebook had talks already years ago (10+) - nobody was allowed to share real numbers, but several facebook employed where allowed to share that the company has measured savings from optimizations. Reading between the lines, a 0.1% efficiency improvement to some parts of Facebook would save them $100,000 a month (again real numbers were never publicly shared so there is a range - it can't be less than $20,000), and so they had teams of people whose job it was to find those improvements.

Most of the savings seemed to come from HVAC costs, followed by buying less computers and in turn less data centers. I'm sure these days saving memory is also a big deal but it doesn't seem to have been then.

The above was already the case 10 years ago, so LLMs are at most another factor added on.

On top of cost, they probably cannot get as much memory as they order in a timely fashion so offsetting that with greater efficiency matters right now.

Yeah, identifying single-digit millions of savings out of profiles is relatively common practice at Meta. It's ~easy to come up with a big number when the impact is scaled across a very large numbers of servers. There is a culture of measuring and documenting these quantified wins.

With the reputation of that company, one can wonder a lot of backstories that are even more depressing than a memory shortage.

Not just shortage, any improvements to LLMs/electricity/servers memory footprint is becoming much more valuable as the time goes. If we can get 10% faster, you can easily get a lead in the LLM race. The incentives to transparently improving performance are tremendous

Oooh maybe finally time for lovingly hand-optimized assembly to come back in fashion! (It probably has in AI workloads or so I daydream)

> changing our memory allocator

they've been using jemalloc (and employing "je") since 2009.

I know this isn’t who’s hiring thread, but we are hiring in AU for low-level data processing and have interesting performance challenges.

Link in bio.

Speaking as an Australian that works on React CRUD applications because there's nothing else in the market, I've been reading through this thread thinking the exact same thing.

I hear you. Actually I read this thread because we’re using jemalloc in an embedded product. The only way I found to work on interesting problems here was to work for myself. (Having said that I think Apple might have some security research in Canberra? Years ago there was LinuxCare there and a lot of smart people. But that was in 2003…)

Some of this remote stuff might interest you:

https://www.igalia.com/jobs/open/

Not sure if it's the domain you're interested in, but there are quite a few HFT firms with offices in Australia.

The one I know of (IMC trading) does a lot of low level stuff like this and is currently hiring.

SIG is hiring for some roles in Sydney: https://careers.sig.com/global-experienced

I have a relative in Australia who was hired by some type of consultancy to work on Samba, but I don't know what work he was doing.

Usually lack of knowledge that such a thing exists, or just plain ol' momentum. Changing something long in production at established companies, even if there is a tangible benefit, can be a real challenge.

I'm pretty sure it means something like this: "Because jemalloc is used all over the place in our systems that run at tremendous scale, some hack that improves its performance a little bit while degrading the longer-term maintainability of the code can look very appealing -- look, doing this thing will save us $X,000,000 per year! -- and it takes discipline to avoid giving in to that temptation and to insist on doing things properly even if sometimes it means passing up a chance to make the code 0.1% faster and 10% messier."

Jason Evans (the creator of jemalloc) recounted the entire thing last year: https://jasone.github.io/2025/06/12/jemalloc-postmortem/

If you filter the commits to the past five years, four of the top six committers are Meta employees. The other two might be as well, it just doesn't say that on their Github / personal website.

jemalloc 5.2.1 vs mimalloc v3.2.8 in Rust software processing hundreds of Terabytes. Could not measure a meaningful difference, but mimalloc would release freed memory to the OS a lot sooner and therefore look nicer in top. That said, older mimalloc from default rust crate would cause memory corruption with large allocations >2Gb in about 5% of the cases. Stuck with battle hardened jemalloc for now.

While it may seem directly related, it's just not. These things are worked on regardless of how cheap or expensive RAM is, because optimizing memory footprint pretty much always leads to fewer machines leased, which is a worthwhile goal even for smaller shops.

There’s been shocks at hyperscaler scale, ex. this got yuge at Google for a couple years before ChatGPT

If you changed from glibc to jemalloc and that solved your issues, then you should blame glibc, not the JVM.

The URL of this story seems to have changed to a Meta press release. What are you quoting?

It’s still quite corporate-y, but other than the way of writing I agree it’s generally quite clear.

Jason Evans worked for Facebook for almost two decades, starting in 2009 - https://jasone.github.io/2025/06/12/jemalloc-postmortem/

He's doing just fine. If you're looking for a story about a FAANG company not paying engineers well for their work, this isn't it.

Allocators like that aren't the default for every process because they have higher startup costs. They are targeted to server workloads where startup cost doesn't matter, but it matters a lot if you're doing crud like starting millions of short-lived processes.

scudo has been the default allocator for Android since Android 11, and we are hoping to make it mandatory for the few remaining places that don't use it. Using an allocator without memory protections in 2026 (especially after we have closed nearly all known performance gaps with jemalloc) is really not a great choice.

Doesn't it depend on vendors/customization? Default is https://llvm.org/docs/ScudoHardenedAllocator.html since Android 11 (2020).

The repo's availability isn't related to whether it's still maintained.

I work in the space. This article would not have been published if the team responsible was on the chopping block

It's just one team with like 4 people. They can layoff a lot of staff from Metaverse.

We generally try to avoid corporate press releases for that reason, but is there a good third-party post to replace it with?

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...

That's a false dichotomy: you optimize both the application and the allocator.

A 0.5% improvement may not be a lot to you, but at hyperscaler scale it's well worth staffing a team to work on it, with the added benefit of having people on hand that can investigate subtle bugs and pathological perf behaviors.

Glad we have super slow allocators then

Exactly. No need to engineer an allocator. You only live once!

    void* malloc(size_t size) {
        void *ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_ANON, -1, 0);
        return (ptr == MAP_FAILED) ? NULL : ptr;
    }

    void free(void *ptr) { /* YOLO */ }