Hacker News

They provide an explanation for using the term "sleep":

> In animals, the transfer from short-term memory to long-term memory is thought to be supported by hippocampal replay [33], especially during sleep [41]; in this phase, short-term hippocampal memories are reactivated and consolidated into cortical synaptic weights. Sleep makes animals unable to respond to external stimuli, suggesting that it must provide enough cognitive benefit to justify this cost [41]. Inspired by these biological processes, we propose a method for transferring context-window memory into persistent weights. When the model’s context window becomes full during inference, the model enters a “sleep” in which it performs multiple forward passes over the accumulated context and recursively updates its fast weights via a learned local rule. As in animal sleep, the model receives no external input tokens during this phase. After consolidation, the context window is cleared, and the model resumes operation with updated fast weights. During training, the model is optimized end-to-end by backpropagating through the entire process to maximize task performance after sleep.

This is why I object to sleep() from unistd.h. What an anthropomorphizing notion. Didn't early unix programmers understand that a computer isn't a living creature and therefore isn't capable of sleep? They must have been really stupid!

Saying something needs sleep isn't anthropomorphizing, since pretty much all complex living organisms need sleep.

Also, even when something is "specific" to humans, it might not be anthropomorphizing to observe it in something else, it could just be an emergent pattern of high intelligence.

Anthropomorphization is not inherently wrong, and in some instances, it actually lets you reason better about about complex behavior than whatever convoluted (and often wrong, especially in the case of giant neural networks) mechanistic description one might conjure.

Here the analogy isn't without reason.

I think it's interesting that folks are suddenly taking issue with "anthropomorphizing" language used in AI as if we haven't been doing this since the earliest days of computing (see "memory", "child", "parent", etc). It helps folks understand things at the correct level without needing domain knowledge

This is the struggle of naming papers. You could stretch definitions and make your own sexy headline or you could be precise and fewer people will read it.

> Does a motor vehicle get "sleep" when it is serviced?

That's more like a doctor visit and a workout. The sleep will be the part of the duty cycle when it's not operating.

> When I reboot a computer, is that equivalent to a nap?

Yes, it wakes up completely refreshed and in good working order, usually, and if there's still a problem you know you need a technician.

I find this annoying too. "Sleep" is okay, but the quippy headlines ("need sleep"—short, snappy and vague) infiltrating journals bother me. I've seen it well before LLMs, but as an example, there is a long list of title snowclones of the famous attention paper: https://github.com/vinayprabhu/X-is-all-you-need.

If it works, it's called bionics, not anthropomorphization ;)

I assume compacting is the sleep here; so, yes

> When I reboot a computer, is that equivalent to a nap?

I mean, you do put your computer into "sleep" mode and then "wake" it.

Analogies are useful. I think we need to learn how to continue to benefit from them despite the risk of anthropomorphication.

Very much agree that while it is is useful in description of motivation and inspiration,

it is very non-helpful—or worse—to use this language, this way.

One might as well say "need neural plasticity" which is as much an analogy and equally misleading and counterproductive in shaping the right model of the system.

One might even call this pernicious, what it encourages is already a social problem; and it doesn't aid understanding, it confounds it.

The analogy is helpful, but yes we should be able to “intelligently design” something better than sleep analogues since we’re not constrained by evolution like in humans.

See also, perhaps: https://news.ycombinator.com/item?id=48273597

Just from the title, I’m assuming it refers to a period of downtime used to perform some sort of maintenance on the knowledge held by the system.

Clicking through, that’s exactly what it is. Seems like “sleep” is an excellent term to use here.

>we study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights before clearing its key-value cache

There is a strong, non-trivial connection here between what your brain does in sleep and what they are studying.

You wouldn't object to referring to robot eyes or robot legs.

... and anyway, maybe it was hungry? Or getting the sniffles?

From the abstract, it looks like it's actually doing something deeper, updating weights in part of the model?

No, they're actually training weights based on context before compaction. Context is context, this is splitting the model into persistent weights and malleable ones which are periodically updated.

Wouldn't that just accelerate collapse? How much do you trust the outputs of the llm to provide trustworthy and valuable new information? I mean I understand distillation works. But that's much more structured and thoughtful than my sessions at least.

It's a network of computers with GPUs, so there's no reason it can't sleep at the same time it's awake. Just a continuous "sleeping" process going on in the background, incrementally updating the model. No need for the "thinking" process to be "unconscious" while the "sleeping" process runs. Anthropomorphism confuses everything. There's no such thing as "offline hours" because the Earth is a sphere and the United States is not the center of the universe.

>To reach a more brain-like behavior LLMs need to integrate your inputs into their model dynamically, essentially retraining real-time based on the most salient input.

That's already possible with LLMs. The challenge is that 1. it would allow permanently jail-breaking models and 2. there'd be no way for them to efficiently transfer what they'd learned to a new model generation.