It's the memory latency, stupid (pivotlake.io)
192 points by ntur1337 15 hours ago
swiftcoder a few seconds ago
I remember have a very long argument with a team I worked on, who were mostly convinced our service couldn't handle requests any faster because the CPUs were already at 100%. In the end we tripled the requests-per-second by cleaning up the service's memory access patterns...
boricj 14 hours ago
I work with another team at work that uses lots of message passing, C++ virtual methods in deeply nested class hierarchies, large DOMs and off-the-shelf C++ STL containers... on a STM32MP1, which has an in-order Cortex-A7 and a pittance of cache.
On the other hand, in my team I wrote my custom C++ STL containers that allocate inline, SAX-style processing libraries and an object pool with links that use index+generations like 2000s video games... on a STM32H5 with less than a megabyte of RAM.
Mechanical sympathy feels like a lost art to people I demonstrate it to... and I had to demonstrate to the first team with benchmark numbers that I would not be the bottleneck between us by orders of magnitudes.
bunderbunder 13 hours ago
I’ve had code review pushback against reorganizing data structures to be more cache-friendly on the grounds that it was “premature optimization” even though I pre-emotively justified the decision with benchmark results indicating that the performance difference was large and business relevant, and addressed a real performance bottleneck that I observed during testing.
This was on a team and at a time where Clean Code was very popular. I somewhat wonder if the problem is that, in the 2010s, many people became convinced that certain aesthetic standards were both objectively, provably correct, and more important than anything else.
hinkley 11 hours ago
A lot of my poor opinion on developer opinions dates back to a couple projects where Product and Marketing were begging us for substantial performance improvements and the teams pushed back saying this was as fast as you could expect. I got 2x in the first six months on both projects, and delivered 25% per quarter on one for almost 2 years (that maths to around 8x) before I started to scrape the bottom of my barrel.
It’s predominantly a way to shut down thinking, which is why it’s so often misquoted. Knuth’s version is more nuanced. Theirs is not at all.
As a compromise I got really good at doing optimizations that also improved readability. Things like hoisting and better variable naming combined together can be mistaken for just code cleanup.
I also discovered that keyhole optimizations are more palatable. Which is to say, move the slow bit yo a separate function with a single responsibility. Then do the optimization in a separate commit. Then the other devs don’t have to look at your changes and if they turn out to have a bug, we can always revert the change later. They almost never do, but having the option seems to grease the wheels.
nomel 4 hours ago
rowanG077 5 hours ago
plasticchris 12 hours ago
That’s just pattern matching. Most people don’t do this level of prep so they assume you didn’t either. There’s usually no ill intent, and people are maybe a little embarrassed when you point it out, but if you are tactful it’s fine.
tomjen3 26 minutes ago
All types of architecture have advantages and disadvantages. Clean code might not be very cache-friendly, but it does a pretty good job of making sure you don't have to hunt in too many places to update things and It also makes it pretty simple to split things up so that you can write unit test for most everything.
This is not the right choice for everything, just as using Python is not the right choice for everything. The question to ask is: what are we trying to do in the first place? Then take it from there.
You did that by making sure the speed-up was business relevant. I'm mostly putting this out to force people to think about whether that also applies to their specific thing.
dnautics 10 hours ago
> objectively
I see what you did there
Gibbon1 11 hours ago
Processors I mostly write firmware for have little cache at all.
I think cache friendly code also tends to be reason about friendly and debugger friendly. None of that is remotely premature optimization.
dataflow 11 hours ago
jacquesm 10 hours ago
sick_of_slop 13 hours ago
Before AI, nobody has been responsible for more psychic damage against software developers than Bob Martin.
teddyh 2 hours ago
gjvc 11 hours ago
clivedup 11 hours ago
There's not enough context for me (or other commenters) to really evaluate this, but every change has risk, and some projects aren't in a position to be accepting that risk.
A "real performance bottleneck that I observed during testing" is a little bit unclear whether it occurred outside of test conditions, or whether speeding up that particular bit improves anything anyone cares about.
jordand 14 hours ago
I work in the games industry with exceptionally skilled system programmers that can do that too. They're getting on in age, and there's fewer programmers in the younger generations that are interested in any of it. Technology is sadly increasingly homogenising around Unreal Engine 5
MisterTea 13 hours ago
> Technology is sadly increasingly homogenising around Unreal Engine 5
I kinda get the commodification of games engines and Unreal. I am sure a lot of us saw the gaming revolution unfold, watching gaming companies one-up each other in graphics and interactivity. But once you get to the point of UE5, Unity, or Godot there is little need or value to make a game engine from scratch anymore.
Don't get me wrong, if you want to write an engine, go for it. But if you just want to make a game then an existing engine is good enough.
Jtarii 13 hours ago
Emjayen 9 hours ago
Not unique to gamedev; during the boom at the turn of the millennium the bar to entry to programming was dropped on the floor to fill demand.
The vast majority of programmers today don't even understand the basics of how-computer-work - not just microarchitecural considerations but even how to interface with the kernel efficiently / understand the performance considerations of their target platform.
Ask your average programmer how to say, simply perform I/O efficiently as possible on NT given some workload characteristics. One needs to first understand this before they can begin making concessions in favour of other demands (abstraction, flexibility, ect) and very few programmers can even get beyond the starting line.
jacquesm 10 hours ago
I recently spent three weeks paring down a piece of software until it could run entirely in the L1 cache of the processor it was running on. The performance was - to put it mildly - impressive. And this was 1970's technology running on a modern CPU. This stuff can do miracles if you use it right.
fwlr 6 hours ago
Someone asked me why Zig programmers seem to genuinely enjoy passing the allocator around as an argument. I didn’t have a good answer at the time, but I think I do now, thanks to your comment - feeling the “weight” of allocations is an example of mechanical sympathy, and having to pass around and use an allocator explicitly makes your code-effort more closely resemble the chip-effort it is asking for.
kibwen 4 hours ago
Heap allocations aren't really the culprit of the waste that the root comment is talking about. Indirection in general is, and passing around an allocator doesn't guard you against that. Virtual dispatch occurs even for objects on the stack.
fwlr 3 hours ago
immibis2 12 hours ago
If it has a pittance of cache, then mechanical sympathy might not mean cache-friendliness. Linked lists were mechanically sympathetic on uncached systems - it cost the same to access any byte of memory, so you might as well go directly to the one you wanted.
boricj 11 hours ago
Looking at the specs: 32 KiB of L1, 256 KiB of L2, in-order and partial dual-issue.
That's the least forgiving kind of chip when it comes to performance: fast enough to outpace main memory by two orders of magnitude, yet the slightest speed bump (branch misprediction, cache miss) stalls it without any ability to reorder around it.
brcmthrowaway 12 hours ago
What is a cache friendly version of linked lists?
Nevermark 12 hours ago
badlibrarian 10 hours ago
duped 6 hours ago
> mechanical sympathy feels like a lost art
Programs should express the solution, not the problem.
bugake an hour ago
Modern CPUs have hundreds of megabytes of cache. That's more than the entire hard disk space required by Windows in the 1990s.
You could install an entire Windows system, emulated in cache of a modern CPU, including the hard disk, and modern computers are still slow, partly because the CPU does nothing while waiting for RAM to fetch data, and partly because decades of people prioritizing abstractions over efficient coding have made software totally inefficient.
brcmthrowaway 12 hours ago
https://people.freebsd.org/~lstewart/articles/cpumemory.pdf
Has anyone rewritten this for 2026?
drivebyhooting 11 hours ago
I feel like the solution presented is just a clustered hashmap or “locality sensitive” if you’d like it.
Still not orders of magnitude better.
quotemstr 8 hours ago
Uh, isn't this just radix-partitioned aggregation? Hierarchical (fixed two level) aggregation is old. What's novel here?
Microcosm of Pivot itself I guess. I just don't get why this project exists when DuckDB and Clickhouse both exist.
fluffyllemon 8 hours ago
From the article: "To do this, we'll use a method called radix partitioning."
I don't think they were claiming it was novel. Seemed just like an article explaining a performance optimization
northes 8 hours ago
I didn't get the sense that this was the end-all, just a relatively simple way to show how much impact cache-friendly data structures can have for somewhat little effort over the naive solution and give readers a sense for how to think through the problem.
happyPersonR 9 hours ago
Shout out to the author, it seems coherent.
I like flamegraphs
I know folks use perf to help generate them
But in the future, now that Llm is here, get and agent to check assembly, PMUs and also see what you’re doing in assembly.
My 2c and happy coding to everyone :)