Quadrillion-parameter transformers and how to train them (qlabs.sh)
39 points by frozenseven 5 days ago
joefourier an hour ago
> Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokens, a tokens per parameter of 1:500,000.
I don’t think that’s a fair comparison since (as far as I know) humans aren’t born with randomly initialised neural weights that are setup in a very generic structure, but have at the very least extremely strong inductive biases. I think it’s fair to count the hundreds of millions of years of evolution between the first animals with a primitive neural system and humans today, and the amount of “tokens” that were “trained on” is likely enormous. I think it’s better to consider the 200M tokens as a sort of “fine tuning” to a sort of genetic “pre-training”.
ben_w 2 minutes ago
Our entire genome would fit on a CD, and much of that is used for metabolism and overall body morphology, not specifically brain structure; even though it almost certainly gives us a whole bunch of stuff (e.g. enough to recognise smiles and thereby allow them to form a positive reward we can learn from, and some approximation of how to sense a potential reproductive partner), it is absolutely not enough to give us a universal language nor intuition about any tool more advanced than a sharp rock and possibly the wheel and fire.
On the other hand, even such basics may well make a huge difference.
parpfish 4 minutes ago
there's also the fact that the brain doesn't process inputs in discrete, feed-forward steps. everything is running continuously and feeding back to earlier layers.
and then there's the abstraction of 'activation' used in neuralnets that is very useful computationally, but omits a lot of the complexity in actual neural signals. you don't get fine-detail in the timing of spikes or the fact that neurons use different types of neurotransmitters.
derac an hour ago
That 200M figure needs much more justification as well, humans recieve huge amounts of data through their senses, particularly vision and touch.
smokel 38 minutes ago
A nice source for this might be "The unbearable slowness of being: Why do we live at 10 bits/s?" [1].
A lot of people seem to think that training is limited to the 10 bits/s of processing mentioned here. But raw sensor input (mostly visual) is in the order of 21 quadrillion bits per year [2], according to the same paper. It seems likely to me that learning uses much more data than is ultimately expressed in behaviour.
[1] https://www.sciencedirect.com/science/article/pii/S089662732...
[2] Assuming 16 waking hours, and 10^9 bits/s of input that the paper mentions. 16*3600*365*10^9 = 21.024*10^15
ACCount39 5 minutes ago
throwaway89864 4 minutes ago
Learning something from a lecture, seminar or a book is close equivalent to learning from text/tokens.
ACCount39 23 minutes ago
Nah. The "huge amounts of data" humans receive are incredibly self-redundant.
Incorporating non-text data into AI training runs has repeatedly demonstrated that for general reasoning capabilities, no data type beats text in value per byte.
theK 10 minutes ago
derac 21 minutes ago
ziofill 30 minutes ago
I agree with you. Also I really doubt we can so easily tokenize the input to the human brain.
EGreg 15 minutes ago
You sound just like Noam Chomsky when it comes to this :) He always talked about the LAD!
spwa4 an hour ago
If those tokens were to be passed on they'd have to be encoded somewhere, like DNA. And in DNA, there's no room. So ... where are they?
ziofill 26 minutes ago
This is such a beautiful question. It looks like a miracle that a brain, let alone a whole body, can grow from such little data. Michael Levin has been doing a lot of research on this, you should check him out.
derac an hour ago
The data is massively compressed and pruned. Where are the input tokens in an LLM?
dingaling911 an hour ago
DNA is probably the seed for the random initialization.
nico 30 minutes ago
wernam an hour ago
zardo 37 minutes ago
Tokens being passed on would be what, you close your eyes and experience your ancestors lives?
horsawlarway 12 minutes ago
TGower 22 minutes ago
> Our new paper (Baron et al., 2026) finds that when the dataset is held fixed, the compute-optimal model size keeps growing with the compute budget.
The dataset is not held fixed in practice. It would be interesting to see an ablation study of where best to allocate compute between more synthetic data, more RL rollouts, and this paper's suggestion of cranking up the parameter count. Unfortunately that's an experiment only a frontier lap could accurately do, and they are unlikely to provide such expensive and valuable information to their competitors.
notlibrary an hour ago
The planet has 20 quadrillions of ants
smokel 35 minutes ago
And several thousand quadrillion grains of sand.
Edit: "several thousand" could be in the order of millions! (i.e. billions of quadrillions of grains of sand.) Apparently the number of grains of sand per this planet is not so clear. At least to me.
intrasight 25 minutes ago
There are more ways to shuffle a deck of cards than there are atoms in the visible universe.
smokel 15 minutes ago
frozenseven 5 days ago
ozgrakkurt 14 minutes ago
Yes synapses are weights, you know science, you smart