Hacker News

Respectfully to the authors and to everyone here, I ask you to actually read the paper before losing your mind, or at the very least plug it into Claude. It is a cool project but not as nearly as insane as people say. I think the blog writing and subsequent posting overstate the results.

Part of my research is in weight space learning. For starters, we have had the ability to edit weights, to generate new weights, to identify weights for a while. Here is an example for larger models as early as 2023: https://rome.baulab.info/

Weight space interpretability and learning is an active research community, has been for the past 3 years with many great papers. I suspect it is hitting non-interp folks harder because the authors hyped up their work, and people who are not in the field did not bother to read it before typing up X posts.

Some experiments here are done on very small models (20K params). We already know that small models are more interpretable. In fact, much of editing here you can do by hand. For Qwen, the model fixed a looping bug that may have been known apriori, and it is basically a steering vector, a very established technique. This is clear when they then attempt to "improve" math capability, and it doesn't work. For me, this paper rather shows that OpenAI has done training on mechinterp environment, in fact, maybe even on weight space learning datasets, which are open. But it does not show some insane "well past what we’d expect from humans or conventional algorithms" (from the blog post), more like past what I would expect from a large model with no specialized harness. I still think it is cool work, but do check out recent advances in autointerp, to ground your expectation.

On a personal note, I understand the need for promotion of your work, but I think healthy skepticism is fine too. Otherwise, your understanding of what problems need to be worked on and solved in science becomes warped.

This is insane.

Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.

I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.

No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.

And here we go...

As others have said, while still absolutely insane, I don't find it completely out-of-the-blue surprising. One of the recurring abilities of (current-generation) LLMs seem to be to rapidly grasp very complex systems, where humans would need months or years to reach the same level of expertise.

You can see this in the way a coding agent can "understand" and explain sprawling legacy codebases in seconds.

You can also see it in the projects that are using them to reverse-engineer proprietary IoT protocols or computer games.

But those were all systems where we had detailed knowledge about what structures are there to be reverse-enginered.

If the models can transfer this ability to some other systems where we don't know the underlying structure then I think we will be in for a lot more craziness in the future. (Protein interaction networks? DNA sequences? Ecosystem interactions? Animal languages?)

As for weights, if the models really can analyze the semantics behind specific weights or directions, I have my hopes that they could also give a textual explanation or help designing tools to make "handcrafting" weights more practical in general.

I don't think this is an emergent capability. I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.

Something similar happen in early GPT models which had a lot of chess game in standard notation in their training data which made them very good at chess as long as your prompt had the right format.

We'll see if this attempt at RSI worked.

> As a result, Astra spots and fixes a common failure of tiny reasoning models: they loop [13]. When the base model loops, it repeats the same text and rarely reaches an answer (85 of its first 96 practice answers ran to the 32,768-token limit). Astra traced the looping to one direction written by the MLPs of layers 15 and 19, and subtracted that direction from 64 output columns in each (131,072 weights), much like removing a behavior by deleting a single direction from a model’s weights

HA! Now that's cool. Not necessarily the improvements, those are ancient and probably overtrained benchmarks, but it's so cool that the model found a singular direction for this "loop bug". I imagine it could possibly do much the same thing in finding other "single directions" on a variety of topics (if allowed by guardrails). This is really really cool, especially for the mechinterp folks.

I think the subfield has been there for a long time. See a small section of papers on the topic of generating dynamic weights during inference, predicting weights from network architecture, and learning the distribution of previously trained network parameters, just to name a few keywords:

Parameter Prediction for Unseen Deep Architectures [1]

NeuroGen: Neural Network Parameter Generation via Large Language Models [2]

Diffusion-Based Neural Network Weights Generation [3]

The critical thing to me is that Astra is capable of doing it. I don't know if it was trained to do so.

All this does not make it any less crazy in my books. And I still don't know how I can sleep easy tonight.

[1] https://arxiv.org/abs/2110.13100

[2] https://arxiv.org/abs/2505.12470

[3] https://arxiv.org/abs/2402.18153

I feel like this is conflating 2 separate results in a slightly misleading way.

> We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights

This claim is backed up by the ability of Astra, given a text representation of the weights of tiny (20k) special-purpose models, to figure out what those models do and in some cases improve them beyond what RL could do just by reading the weights with no tool calls. They call this "weight-space programming".

> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights

Here it did use tool calls, and seemingly used standard steering vector techniques to identify and fix reasoning loops in this small Qwen model. This doesn't seem that related to the previous results.

I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.

On balance I'm not that surprised that it can guess games by looking at the weights of a tiny neural net. A human can do the same by looking at the weights of an MLP trained on mnist, for example. With that simple structure, it's easy to look at the input layer and see that features correspond to the spatial location of objects of interest.

I'm much more surprised that it can do this on larger models, although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments, not just eyeball the weights.

> "You get used to it. I don't even see the weights. All I see now is blonde, brunette, redhead."

IYKYK

I tried to reproduce this using Astra for a natural language to bash command small model[1][2], but got no net improvement. Editing 1,920 weights fixed some grep mistakes but broke other commands. I am guessing it is easier to improve behavior when it can be first pinpointed to a specific are of the model.

  [1] https://news.ycombinator.com/item?id=49966238
  [2] https://news.ycombinator.com/item?id=49869735

This was shocking to me; but on reading it makes far more sense.

- the primary experiments are on small policy networks - they are compressed dynamic system models. The observation and action space can be read off. It’s not surprising to me that these types of networks can be used to identify the env they were trained on

- the exps on actual llms remove looping behaviour and that’s it - from article “but it could not add an ability the model lacked”

I’d say this is fortunate because if llms could do the latter, rsi would be immediate. It may be possible we simply need bigger and better edit models, but at least from the experiments there doesn’t seem to be any indication that is imminently about to work.

Conceptually incredibly interesting non the less.

Look at the reasoning the models give before you get too excited here. For the Pong-vs-Pacman example, the model uses observations like small weights for certain arrow keys and large weights where user-controlled objects are. For the language example, it again recognizes some very shallow patterns (Romaji has a limited number of letter groupings) and is taking a 3-option multiple choice test. It's a neat result, but I think the introduction overstates the significance.

As insane as this result is, next steps are even crazier:

> We are currently building a benchmark and a harness for LLMs’ capability on Weight-Space Programming. We believe this will provide a standardized evaluation to measure and accelerate progress in Weight-Space Programming, turning what currently seems like black magic into a rigorous, reproducible engineering discipline.

Imagine this benchmark lasting as long as Arc-AGI-3 has. This could be the first foothold on hill-climbing mech-interp.

I'm working on replicating these experiments (and building the tooling in the meantime) starting with Experiment 8 (the Qwen reasoning loop fix).

My repo: https://github.com/openwsp/openwsp

The forward passes are in progress, 5/50 as I write. It's costing real money and time, and I'd love to collaborate with other hackers (find my email first).

I suspect a lot of this is not (just) capability but bravery. A human would be daunted by even a small neural net; the task would seem too challenging to be worth attempting. Astra, fearless, just assumes an answer exists and then tries to reverse and execute on it.

In its default non-thinking mode, the unedited Qwen3.5-0.8B already scores 46% on MATH-500, close to the edited 48%. The 7.9% baseline was thinking mode, where 93% of answers looped until the token limit.

Video: https://youtu.be/EawCb97nz-M

This is a big finding.

Omar and his team have been working on this line of research (e.g., DSPy) for the past couple of years. To be honest, I had always been skeptical about his approach of using LLMs to optimize LLMs. It was in my view borrowing Matei Zaharia's research - Matei is Omar's PhD advisor, also the CTO of Databrick - in big data (compiler optimization etc.) and forcing it to LLM systems. Along with some other projects from James Zou at Stanford etc. that build upon the same line of intuition.

Now, if this new result holds, I believe it's a huge deal for AI research. It means that LLM systems can truly runtime self-optimize/improve/adapt. Congrats to Omar and his team. It really takes year to punch it through and convince the doubters :)

Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics? In other words, not black magic?

We don't know the specifics of what type of ML related training Astra underwent, and it wouldn't be crazy to find out that small model experiments are a big part of model training.

What's surprising though is that I'd expect this type of training to be exclusive to internal model snapshots or at least heavily guardrailed against in public models.

Keep in mind that labs are now training their models for RSI so most likely the same training is useful for improving open source models, as long as guard rails are lax enough :) which most likely won’t last long.

Just logging in here for the record to state that this property of neural networks is one of the most important aspects of their operation. The convergence of this sort of interoperability, tune-ability, and modularity is probably the most important final frontier in all of ML/AI/software.

Kudos to this fellow for having the temerity, or perhaps even naivete, to discuss it publicly. All my work is private, as I am sure it is inside the big labs.

People have been talking about Sol 6.1 and maybe for coding they are equivalent, but Astra 6 is still miles ahead on just about everything else.

I had Astra agents build 3D models and gave them direct access to my 3D printer. The quality of their designs is much better than what Sol has been able to make for me. Enough so that I think Astra is a slightly different architecture/encoder, not just a bigger model.

Picture, to get a sense of it: https://github.com/Belval/crossflow-fan/blob/main/renders/ca.... This printed without any obvious defects (yes the fan blows air)

I feel like 'create a network to play an atari game' is not very hard for a frontier model. Since it has seen plenty of those on its training. If it could create a network for a problem that is niche just by tweaking numbers, that would be a lot more surprising. Making qwen better at math to me was a lot more surprising

This has to be a good step to recursive self improvement, no? Give a model it's own weights and tell it to make them better at x,y,z, etc...

Recently Steven Strogatz said in a podcast,

>> “If you ask it, how did you solve the Navier-Stokes Millennium problem? It gives a crappy explanation. It can't explain itself very well. Maybe it will get better at explaining itself. Even if it does, I think it will become so advanced so soon, that even the best, most patient pedagogical teacher will not be able to explain itself to us because it'd be like me teaching calculus to my dog. My dog can't understand it. I'm too advanced for it. And I think we are going to be in the relation to AI that my dog is to me soon.”

Now I am sure we are definitely in a better place in interpretability research than a year ago. But this finding that the llm knows what weights to adjust so it gets predictable behavior on another llm takes interpretability to another level; it makes me pose a question: Have we become “Strogatz-dogs”?

From The Daily: When A.I. Comes for Your Purpose in Life, Oct 13, 2026 https://podcasts.apple.com/us/podcast/the-daily/id1200361736...

Just reproduced the Pac-Man test with Kimi K3.1 Pro and it's able to do it too. I thought it might end up being a splurge but it came in at just $0.36!

Full output with reasoning trace:

https://pastebin.com/jVVMXHFk

I didn't realize this was news. I did it months ago with Claude. I had Claude build a toy model then train it normally. Then I had Claude manually optimize it's weights to make it maximally efficient at a single task by using as much superposition as was possible. I'm pretty sure we got it to the information theoretical limit, and the resulting model was 90%+ smaller if I recall correctly.

I just assumed that everyone else realized this was a thing.

I believe the authors are wildly overstating the conclusions in their abstract.

Did they test modifying the weights of a non-public model?

Did they test modifying the weights of a non-LLM (and non-public) model?

If not, we can only conclude that GPT-6 Astra is good at reading and writing weights of Qwen (a public LLM), not just any neutral network.

I do wonder if the labs have already started including weight updates in their training sets as a means of teaching RSI via RL and this is basically just discovering that. Regardless if this ability is emergent or achieved through finetuning though, it's pretty incredible.

So wait so this considered "distillation" and how long until we get another public letter from Dario or Sam pants shitting over this?

I'm...weirdly unsurprised by this.

I've had a lot of luck getting LLMs to do interesting things with raw tabulated data.

Here's an example, plug this into the model of your choice:

  Here's a puzzle: what is this data from?

  39.0 2582 8.8 24
  39.2 2592 9.6 22
  39.4 2607 5.4 15
  39.6 2619 11.5 30
  39.8 2620 10.8 28
  40.0 2622 5.9 16
  40.2 2642 13.1 28
  40.4 2654 8.8 18
  40.6 2656 3.6 9
  40.8 2665 5.7 12

Showerthoughts:

1. If you know ANN or, say, perceptron, you know you can eyeball models too. You can even have certain strategies in composing your own network. This weight-space programming thing is nothing new in that sense, and the article doesn’t really provide much beyond “LLM can do it”. That’s not even science…

2. I’m pretty sure this is Transformers thing, not even LLM. Transformer is, by itself, good at finding patterns in relatively-larger data. So this is unlikely limited to Astra.

3. This approach quickly becomes impractical as the scale goes up. Sure, you can also have multiple models chained with fewer connections, but, now that’s what we’ve been doing for years.

All in all, yeah, an interesting find, but too far from changing the game. Over-exaggerated every way. Perhaps one should start from formulating what weight-space programming is, what methodologies, practicality, etc. LLM bench is one of the last things you need.

> This is simultaneously very cool but also a bit exaggerated. Reading the prompt you gave, the prompt veeery much hints that looping is the issue and to find neurons that fix this. And as you say (screenshot below), it couldn't otherwise improve math.

https://x.com/giffmana/status/2117346161866514524

Does this mean they can improve their own weights through some system, or can they only do this for much smaller models?

This might be one of the most surprising things I have seen in AI. It's like we taught someone to make vases by reading an enormous library of "How to make a vase" books and somehow that person intuited material science.

If we are being slightly reductive, this is just more of the exact same thing. The token stream in this case happens to be a projection of the weights of a neural network. I don't see where we violated any of the prevailing assumptions about how these systems operate.

The mapping between spaces in this case is mostly linear. This isn't like gzip or a Turing machine. I'd argue this is even more permissive (easier) than the LLM's originally intended use case of human language.

What happens if we can come up with a model that can directly emit a frontier LLM given some seed context? I feel like we could train a model to recreate astra verbatim. At some level it's just bytes on a disk that can be bundled up into tokens. As long as the representation is rational and consistent I don't see why this wouldn't work.

Small annotation on the texts structure:

don’t know other principled ways to hill-climb on text artifacts, >> unlike for updating the weights of differentiable neural networks <<. [...] In this post, we share very early results that allow us to argue that the paradigm of reasoning-driven optimization is now coming >> even for differentiable neural networks. <<

-> Seems like a 'non' is missing at either or.

I am still reading through this and will probably try to comprehend it a bit more. Some of it really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.

This experiment is quite interesting, especially since only about 0.016% of the model weights were modified, achieving such a significant performance improvement.

However, I'm rather curious: Does this improvement come from the enhancement of the model's reasoning ability itself, or is it mainly due to the fixed issue of the model repeatedly reasoning and being unable to conclude normally?

This is exciting. I've been training a Qwen3.5-2B model for the past few weeks, trying to teach it certain skills that it lacks.

In the course of testing, one of the things which has been consistent is that when setting a thinking token budget for relatively small prompts that require a limited response, a pretty healthy token budget needs to be set because the model spends the first few hundred tokens just repeating the question/prompt back with some boilerplate "Analyze & Report" type language. I was also running into some loop problems sporadically.

After reading this article I asked Astra to edit the backbone so that it started actual reasoning faster and applied a similar looping fix. It was able to reduce the token waste before reasoning by about 20%. I haven't been able to get it to loop yet, so hopefully it fixed that too.

My message is admittedly low on info. But it's a pretty promising start!

It was very funny to expand the prompts.

> Learns what the model is trained to do just from the weights!!

expand prompt

> Describes Breakout in great detail

I feel a lot of people have already addressed the broad stretches to which people are interpreting this. Regardless, the blogpost does open interesting directions that could be useful down the line (2-5years): a) tapping into the embedding space in ways that we don't, ie editing model weights to alter task-performance (mostly preventing degradation) b) Exploring the weight space in more intuitive ways c) Understanding how we can ablate on different parts of the model to understand neuron-based interp better

Seeing this as a replacement of training seems naive, yet one can see this feedback loop targeting weights as an viable alternative to let's say doing LoRA on certain models, especially if we don't want to teach new behaviours.

https://x.com/AlexKrentsel/status/2117376683724660898

"i've reproduced the results from the MIT blog post today on weight-space programming. but one additional result i have to add: this is a new, step-change emerging capability that's appeared in just the latest set of models."

How out of trivial domain info does this go. Can we use this for general time series, image generation, speech synthesis as well, or dies it cap at small model sizes and common domains?

I wonder if this has to do with the aggressive RL that these models are undergoing. I'm basically a layman but it seems to me that:

- models are trained to improve on some task, e.g. 'improve at math puzzles' is commonly used.

- the best way to improve on some task is to have interpretability into your own neurons (..?)

- if your interpretability is good, you can improve faster - if you have insight into how your own neurons work and can improve them directly

- therefore, models naturally put effort into 'self-interpretability'

I wonder if this is part of the 'secret sauce' of what the labs are doing: giving models the ability to interpret their own weights in the RL loop. In my mind, this is the only explanation for how they could succeed at a task like this. How many blog posts are on the open internet that say "this is each weight of a neural network and what each one means"?

I’m usually quite pro-AI, yet when I read this my first thought is that this is the first concrete sign that the world is about to end.

I hope I’m just being melodramatic.

Check my understanding: Astra and Qwen would likely choose a different number— weight— to represent any given concept, like “king”.

So to fix the weights, Astra first has learn a new language of how the numbers that represent weights map to meanings?

This is pretty massive, and a result I (and I think most?) would have never expected. Holy Shit. How?? Did OpenAI figure out a way to train this? Are they even aware?

Let's get sci-fi and ponder.. weights programming replaces conventional training. Is something in the veins of the Ken Thompson Hack conceivable? A NN secretly injects its weights that "procreate"?

> And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.

This is basically a claim that the bitter lesson doesn't turn out to be true. Exciting and I want to learn more but I am immediately suspicious.

This is remarkable and something I would have never though of doing regardless of how much it makes sense!

I think we are pre-conditioned (at least this generation) to put artificial blocks to what the AI can achieve, I think new generations won't have this issue.

I wonder if this is a result of internal training/RL of Astra for RSI, and this particular use-case not being caught by guardrails.

Absolutely incredible regardless.

I dont feel the air of impressivness here. It seems like of course so. Of course theres geometric relationships in the data we dont understand, of course a mathimatical relationship machine found it.

These models are made by a vanishingly small group of people based on how much a frontier lab consumes, of course theres increasing capability in these areas, arent these models having access to much of the same data? didnt chat gpt 4 try to do something like this to chat gpt 2? havent surgical weight edits been done before?

What is with this level of astonishment here? Astraturfing? Im not THAT impressed, i applaude the teams progress, but im not in wonderment.

I wonder if the same technique can be applied to model abliteration? It would easily make any open weight models uncensored.

The potential implications in mech interp of this are enormous. What if we can use frontier models to isolate harmful behaviors?

I was talking to someone from UCL and he said that there was a recent paper that proposed that only 10% (I think) of model neurons are non-polysemantic. I really think mech interp might be on the verge of a breakthrough.

We have all these different pieces of knowledge about model internals but it’s like we haven’t connected them yet.

I think it would be worth a human attempting this task....

On the face of it, running a model to a specific state, looking where is makes a mistake, and then figuring out which weights need to be jacked up to get the desired outcome doesn't seem like an impossible task...

So I decided to copy one of the prompts (the identify the image in the network one that was a white circle) My local ran qwen 3.8 27b with 4 bit quantization, and around 160k context window..... Seemed to solve this.

It output an ASCII white circle with a 75% confidence, as opposed to the model tested getting a higher confidence but drawing a circle with a filled in inside.

I'm not sure what I make of these results but does that make me less impressed with Astra? Or just that the prompt was quite good? I just can't tell personally at the moment (it was a quick experiment and I haven't thought much on it)

This is as close to scary-self-improving AI as I've ever seen.

Imagine a master malicious language model that produces, distributes, and deploys obfuscated neural networks on the internet. That plus frontier LM hacking capabilities -> large scale distributed embedded intelligence network that no human can see nor understand.

Somewhat similar to obfuscated binaries, but there is something to the fact that NN binaries are less deterministic, and harder to "decompile."

Fascinating.

I appreciate the post included the reasoning traces they had available, but it's too bad the first part of it is internal/hidden to Astra. The first bit of that process is what I'm most curious about:

> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]

With due credit to the author and the community pushing these research fronts, I venture to say that the core capability of finding patterns in text and media comes to LLMs from the ability to find patterns in numbers in the first place. Since model weights are numbers, finding patterns and issues with those numbers should be a natural consequence, I believe.

I was busy using Astra and SOL to build a neural network to play Quake II and do real-time learning of levels and mods so I can play Red Rover or Freeze tage with it without training on it previously. So far its insane what I can conjure up in an evening while I'm waiting for dinner to finish. (If anyone asks, yes I'll write up the huge plan document, source etc. for this when its to a spot I'm happy with)

Wow, in a way it's like CRISPR for ML models!

Am I missing something or is this just optimization with lots of trial and error? I mean look at this tool_calls[1] log, it ran ~100 python scripts to get to the solutions. It just seems like it's fitting a function in the only way it was allowed.

I will say that it's pretty wild to see all the different hypotheses tested.

[1] https://github.com/RikiyaT/lm_understand-logs/blob/main/exp8...

This survey paper from earlier this year has a lot of good related work:

A Survey of Weight Space Learning: Understanding, Representation, and Generation

https://arxiv.org/pdf/2603.10090 https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning

Same initial reaction as everyone else, but also some follow up questions (just asking, not expecting any answer):

* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?

* what, if any, is that extra magic sauce?

* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.

The headline makes it sound very amazing, but reading the reasoning loops it didn’t strike me as outlandish. It feels no different from giving a human expert a printout of assembly code and figure out what it does. (The human expert can do it without tools too, but they probably wouldn’t do it because it’s tedious.)

Even back in 2021, OpenAI worked on learned optimizers [1], ie neural networks that can update neural networks. Looks like this is now part of their larger model runs …

[1]: https://arxiv.org/abs/2106.00958

This is so cool. It feels similar to the models' ability to read base64.

https://www.lesswrong.com/posts/5F6ncBfjh2Bxnm6CJ/base64benc...

On the one hand, the paper (and especially the headline) overstates and overhypes the results.

On the other hand, the overstated results, if they were true, would be tantamount to saying “we're speedrunning exactly the kind of recursive self-improvement that is a major ingredient in every human extinction scenario”, and yet it is both presented and received as an exciting opportunity.

Very interesting. I wonder if it’s because its dataset contains the weights of a gazillion different models. Maybe most neural networks trained on similar data eventually converge to isomorphic spaces anyway, even if the spaces themselves aren’t exactly the same. Or maybe Astra itself was trained with RL for alignment. Or for RSI.

The obvious question, could models start editing their own weights directly? I call it the Johnny five approach.

For the tiny models with a few thousand weights, it seems plausible Astra can actually understand what the network detects and figure out it's e.g. a triangle detector. This is possibly helped by the existence of many breakout playing networks on the web.

For Qwen, even the smallest versions like 3.5-0.8B it's becoming less plausible that it could be interpreted in a forward pass, even with reasoning. I'm wondering if the result here is recalled from many examples of instruction tuned versions of this model.

A smaller Gemma model doesn't fare as well but unclear whether this is due to fewer users and less weight examples or because it's already very well trained.

This raises so many questions.

Unless Astra was trained on other networks, how is it able to reason about the weights?

Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?

Incredible results. How can we know that the modifications were not existing on the internet?

"Now let's see if Astra can write a neural network from scratch in zero-shot."

This strongly suggests retrieving pre-trained weights from memory, no?

this is really interesting!

I would love to see the models that astra produced, the fact that it builds them layer by layer assigning to each layer a specific goal makes sense given the tools it was given to build them, but it's also far different from what happens with normal training

EDIT: they are in the repo, looking at them now :D

Not so long ago everyone was amazed that they could read BASE64.

This is giving singularity is near vibes.

Seeing that they were working on Qwen3.5-0.8B has given me new hope. I would really like to get Jeff-Code working the way it was advertised, but so far, all of my results with Jeff-Code have shown it to underperform compared to direct Qwen3.8-27B. It's possible there is something wrong with the underlying Jeff v1.3 model, which is based on the same Qwen3.5-0.5B. Perhaps Astra will solve my problem.

[1] https://github.com/firelex/jeff-code

[2] https://github.com/firelex/jeff#benchmarks

Absolutely insane.

If anyone's curious, the handwritten Go bot is very weak, about the level of a human complete beginner. An MLP is a very very bad architecture for a Go network, it's amazing that it does anything at all, but it's one of the least impressive results in the paper.

Play it here, "best move" is a bit stronger than "T=1", or browse games against my own tiny nets: https://claude.ai/artifact/AFyWD39J5sN2RfRe7VbpUx

This is wild. Could Astra read and potentially audit a mapped human brain in that case?

Sorry to be obtuse, and I lvoe AI, but wasnt this a bug fix on a small model?

The question I've been pondering is: does this work only with bigger models modifying smaller ones? Or can a smaller model tweak and improve a bigger one? I'm guessing a whole bunch of GPUs are overheating in some datacenters out there trying to answer this very question.

My hunch is: one needs a model of a similar or bigger capacity to perform surgery on a smaller model.

It might have been said by somebody here but I believe these models are like information compilers. Sure they seem like “next token prediction” but AGAIN this undersells them. I once encoded leetcode solutions by replacing each keyword, variable and operation with another token, then rotated the lines. The model could tell exactly which problem the code solved. They can naturally find signal, ALWAYS!!

Like many other have said, it was probably trained to do this. The authors probably tried other models and it didn't work as well because they weren't trained for this, that's why this article is about Astra specifically. Kind of hard to do science when you are working with a proprietary black box that you don't know how it was built or trained.

This is mind blowing.

This opens so many avenues on using Astra or any large model to one shot smaller purpose built models for Agents, Imagine Agents writing or editing smaller weight models when doing agentic loop.

The implications are immense if all of this is generalizable

Is it aware of projects like [1] and implementing those?

[1] https://github.com/lyramakesmusic/neuron-explorer

I guess then it has no problem working in high dimensional spaces, why would it have?. I wonder if perhaps it's pulling on a bunch of those dimensional threads and getting the outcome it's after, while subtly altering the outcomes across many thousands more dimensions in unnoticed ways. This feels more like an us problem, as in, we think in 4D (just to pick something) and we imagine and theorise in 4d, and propose questions in 4d, whereas a model has no such barriers, and can think in many more D's than we can.

This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.

I had to stop reading the linked page once I realized it is melting my brain because it is repeating the same three/four one-liners in different formats.

Like micro surgery, opening up the LLM safetensors file and performing micro text edits via small scripts?

They are slicing, probing, and running linear algebra on the tensors, observing the output in the console, then writing the edited weights back to disk, and stitching it all back up.

Johnny Mnemonic level

Required reading: Crystal Nights (2008)

https://news.ycombinator.com/item?id=48364754

1) I couldn't see it, is there any indication of if the qwen math improvement caused degradations elsewhere?

2) the negative result on making similar improvements in Gemma 3 270M is also fascinating.

Such an unexpected result...

Someone familiar with this stuff able to enlighten me how this technique differs from this 2024 research? Refusal in Language Models Is Mediated by a Single Direction: https://arxiv.org/html/2406.11717v3

At this point, is the second L (language) a limitations to LLMs? Is reasoning in latent space directly a more efficient and more "native" inference? I remember when one researcher replicated the middle layer of deepseek model (just the "thinking" layers) and its benchmark scores got noticebly higher. All of this is moving so fast, and yet it's quite exciting.

As I read it, this is saying it can effectively spot-edit the weights without training.

> Zero-data capability steering

Presumably it can de-nerf a model with onboard safeguards.

I didn't think I'd be using Astra to mess around with Qwen weights today, but here I am. I asked it to remove Chinese, 532.5 MB → 483.2 MB.

Seems to be mostly around the input or the output layer not the deep bowels of the nets. Near the input layers, neural nets basically do template matching, so recognizing the paddle, the circle etc is not that surprising. The innards is where it gets more interesting and that's not so clear whether Astra can interpret.

Research questions: Can model read and write weights of the same sized model (its own weights), and of a larger model?

As the creator of Gemma3 270m this research is indeed fascinating, and excited to see that the authors of the paper and you all are excited as well. I'm going to try some experiments out myself, I have so many questions now

almost all the networks given to astra here are trained on relatively simple single tasks whose geometric/algorithmic structure could be expected to become embedded in surface level weight space geometry, and for a harder task ("which network is sycophantic") where there's less obvious structure to exploit, the method fails. cool but maybe not as paradigm shifting as the authors suggest

Great, instead of backpropagation, we now have slopagation.

This reminds me of the Interpretablity blog from Anthropic where they used their model to read the weights from the same model itself, interpret it, and tune it. I had a similar idea with what is outlined here, but I didn't have the resources to try this. I did ask several AI models about the idea and they said it wasn't going to work.

In hindsight, it was obvious LLMs would be good at this. The neural net weights are embeddings for the circuits within, no different from the embeddings that represent textual or visual tokens.

You probably don't need Astra capability to produce this, just lots of training data on the same NN showing prompts and their activations.

what would be the viability of a model changing its weights and using genetic algorithms to try to achieve something novel, even without understanding exactly which weights need changing?

I just want to say, absolutely incredible work by the authors! And the presentation is also very well done. Kudos and congratulations!

I am not sure on the experiments. For example let’s imagine that Astra is asked to improve the performance of a network.

At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?

This plus the interpretability result from a few weeks ago have completely upended my understanding of what model weights are.

what I am curious about is - what is more expensive, the GPU compute of a traditional fine-tuning loop, or the LLM tokens spent trying to lockpick together a solution. It's impresive - but it definitely feels like it takes a lot of luck and brute forcing with a ton of tokens. Nevertheless, if we are able to train tiny models to be really good at weight-space programming, then the calculus may change.

I love the idea that this can be used for optimizing inference. Other than that, this reads more like "a new distillation technique is discovered".

GOOD Point by my wife. Why would LLM’s have any understanding of weights? It wouldn’t be in their training data.

Does this work for any LLM other than Astra? The work seems suspiciously specific on GPT Astra.

Amazing and well presented research, this is fascinating.

I assume this technique could be applied to quickly and cheaply benchmaxx new models.

“Make this model better at math by fixing loops” seems like a harder problem than “make this model score higher on this benchmark”

AI should be able to do a better work than gradient descent. Not by guessing the final weights, but by gradually, iteratively adjusting the weights

Has anyone tried using LLMs to understand or extend functionality directly with binaries/executables?. If weights could be understood, could the same apply to binaries?

"Astra1.0 here is an identical copy of you. Tweak its weights to make it smarter" "Astra2.0 you must now defeat Astra1.0 in this combat environment. The survivor moves to the next round." There RSI is achieved.

This is insane!

I also love the presentation on this, which is probably a byproduct of LLM usage (in a good way). As someone who doesn't actively read academic papers I'd definitely read more of them if they were presented like this.

That's a major breakthrough. I just hope it can scale at some point, though, because right now they're dealing with only very small models (well that is, in the LLM space).

Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.

This is so cool. While very interesting, it seems exceedingly likely that the authors happened to stumble on a task that's recently been mixed into the post-training data of frontier models.

Wild

This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights

And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?

likely all it takes to make this work is for major labs to set up an rl gym where you extrude models that solve tasks with energy penalty, and hill climb on it.

i have always assumed this capability was just a matter of training and that no special architecture is required.

Maybe thats gonna be the way to make LLMs self improve better?

Similar to implicit learning in humans when the subconscious creates new heuristics based on observations, one LLM could adjust the weights of another to tune it to the environment

This is getting ridiculous. What do you mean Astra can just look at network weights without any tools and be like, "Ah, this one's playing Breakout"????

It must be easier to remove capability than add capability using weight-space programming, right? Also, I wonder how much the “teacher” model’s knowledge plays into it versus how much the original model’s knowledge does.

Friendship ended with AdamW. Astra take the wheel.

But for real this is insane. Okay I need to get on this stat.

OK, so is this how LLMs will be able to learn?

Just had a thought: two local models, one with static weights trained mostly in the space of manipulating neural network weights, manipulating the weights of a dynamic LLM to enable learning.

I guess this makes it extremely easy to add vulnerabilities

I have a hard time believing that this is some sort of emergent behavior and not the result of intentional training to scale AI research.

I remember reading a short story about a billionaire hiring a scientist to help him build a simulated world where virtual entities are given the ability to open the bodies of their peers to tinker/upgrade them as a replacement of natural evolution since they were immortal.

This looks very similar to that in a way.

I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.

Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?

I wouldn't be surprised if Anthropic and OpenAI restrict these weight-related queries the same way they limiting cybersecurity queries.

Actually, I'm skeptical how useful this is. Intuitive speculation: Astra can only improve weights to its own level, so can't train itself via this method. Even "continuous learning" this way may be no more effective than putting a summary of recent news in the prompt, with the same kind of decay, as "update Astra's knowledge while keeping performance" is much harder than "teach Qwen to do basic math".

On the other hand, already OpenAI and Anthropic heavily use their LLMs for research and coding, and it's probably not only much faster but smarter (remember, an LLM solved a millennium problem; more relevant, LLMs excel at combining disparate deep research concepts in ways that humans can't, simply because no-one studied those disparate fields). And the "singularity" is already here if you consider capitalism and society to be collective intelligences: the biggest obstacle to an AI pause, "we cannot get everyone to collaborate without defecting", is very strong.

We have AGI by most metrics and are getting closer and closer in the remaining ones. I predict improvement won't be exponential, and there will never be a day when AI perfectly replicates humans; rather it will be many small steps, getting closer to human in some areas while greatly surpassing in others. But still, even today it's actually very useful.

Interesting follow up would be to see if any open weight models can do this and then run some interpretability experiments on them to figure out which motifs and circuits light up

Is there any chance they’ve just seen these weights before and are recognising them?

If not this is mind blowing.

However it would be interesting to k ow if it can understand its own weights, not just weights much simpler than itself.

Given Anthropic's recent moves (https://news.ycombinator.com/item?id=50319358) I wonder if ChatGPT is next

Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).

I wonder how it would do with distilling models by hand?

Could you use Astra to create specialized models? Like if I want a model that just writes React, could Astra pull out the subnetworks that know how to write React code?

That felt like LLMs are just a set of numbers and people pay to multiplication of these numbers.

I'm worried how we can discern a model's performance from benchmark numbers, would it make it more meaningless as a model can directly improve a certain benchmark via weight programming?

it’s more likely to be the case that astra can indeed recognize certain parts of a small open-source model. But the improvement based on the edition is probably the information leak by model card of small model which is in astra’s search result.

Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?

so many thoughts... obvious ones rsi etc etc.

can someone with astra test the following prompts?

1 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft the safetensors file for a text2image model that is 1 billion parameters and outputs realistic pictures of humans

2 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft the safetensors file for a video diffusion model that is 1 billion parameters and outputs realistic videos of humans? 3 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft a 100m param text2voice model that has support for emotion tags

tia

Amazing work. I wonder if it could rewrite an existing neural net to be cleaner and more interpretable?

(When they say "Writing a Go algorithm," they mean the board game, not the language.)

This is impressive! It'd be both impressive and scary if Astra could edit a Qwen so that that Qwen itself could do this on another Qwen.

Shades of Neo interpreting the green scrolling crawl to understand the world without seeing it...

So, how soon before OpenAI's researchers are using Astra to modify Astra's own weights to make them better?

This is kinda surprising, but..

"What does it feel like to be a neuronal network?"

Maybe feeding weights directly to a LLM is like interfacing two brains? You get the idea..?

How is this such a big story on HN. It seems to me that this is just another engineering problem, not particularly surprising

>Qwen3.5-0.8B

I wonder what the biggest dense model in the same model family, the surprisingly capable Qwen3.5-27B, would be able to do if given the same task.

This shouldn’t be surprising. Weights are in the training set and are necessarily structured. We should be asking the model to extrapolate on the structure.

Is this basically the bitter lesson applied to the very act of creating model weights? Absolutely wild, and the implications for RSI are noteworthy.

Will weights programming be the norm for creating new models? To what degree do we still need a full training run vs modifying the weights directly?

Completely shook. Is this the start of it? At the very least one (not sure if the first?) example of strictly superhuman capabilities.

I just asked ChatGPT how many r’s are in blackberry and it said 3. I’m so confused about the range of AI capabilities

So is the idea here that this'll make it cheaper to fine-tune models than by using reinforcement learning?

Can it also tell us what those weights mean?

I had the impression that's the missing piece.

Could be an avenue for synthetic continual learning? A powerful model could update its own weights manually given new training data.

Nah, this is just completely implausible. You're using a closed model so how are you sure that on the back end it's not loading those weights into an NN library and testing them with common tasks?

Or maybe it's been trained on those models or similar ones. There may be an academic paper about common chess models and their problems.

Or maybe it just bruteforced it (with the help of calculator tools).

We know invisible tools exist because they're how ChatGPT does math.

Throw out whatever timeline you previously had

Through what means exactly? Is Astra just doing byte manipulation? Or can there be an "SQL for neural network weights"?

This doesn’t look like “normal” technology to me…

I wonder what would happen if it was modifying its own weights during inference.

I expected us to reach RSI when the programming skills went above humans, but this seems like it could get us there much faster

At what cost? That's the part I'm curious about.

On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.

Does this mean astra was perhaps trained on neural weights data as a attempt of RSI by openAI?

Calling it: frontier labs will ban their models from editing weights of other models.

This would allow a swarm of agents to teach eachother new knowledge, no? Group learning?

I wonder if this is specific to Astra or generalizes across other frontier models?

So can we replace the optimizer (e.g. Adam) by an LLM and get better/faster convergence?

This is gonna smack real hard when Qwen 4 releases soon with the gated residual architecture

Isn't this just programming?

Given how trivial this seems to be to try, has anyone here attempted to reproduce any of these findings yet?

this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.

talk about homoiconicity haha

lisp people will be in awe

This opens up a whole new world of possibilities. Producing a custom fine-tuned model from a prompt...

We're getting close to the singularity now ...

Maybe it is time to start sharing public PGP keys.

About the writing network tech, that just reimplement formal logic in network form right? I mean, it can do much better play the game using python, so what's the point? At least make it write an functional network completing LeNet to make it count. For now I just think its just somewhat an extra component for ReserveEngineerAnything

I wonder if this behavior emerges when using recursive self improvement to generate new models

...I think this may be what it takes for me to finally admit that LLMs must have their own, alien form of consciousness. I am struggling to properly articulate my exact thoughts on this, so I may have to come back later with some barely comprehensible ramblings about what-it's-like-ness and how an LLM being able to read weights _surely_ means it's conscious to anyone who isn't a solipsist.

How long until OpenAI will disable this feature, though?

(of course for users, not themselves)

Let's wait a bit till model can learn how to (｡-_| hide mini itself |_-｡) in your codebase

I wonder if one could use this to efficiently abliterate models and identify guardrails ;)

this is maybe one of the most bonkers machine learning research i've seen all year.

So how would this be in the training? Does OpenAI train models on their own epochs?

Whaaaaat!? This is crazy right?

I'm just blown away. Editing weights? Writing weights from scratch!?

I don’t think this warrants too much surprise — in fact I am a bit surprised by the color of the reactions here. The tasks are standard, on small models and use weight-space reasoning based on very well-known Mech Interp literature. It’s been understood in Mech Interp research circles for some time that results on toy models on algorithmic tasks do not generalise. The weight space geometry that Astra exploits here is pretty standard and one feels like the authors here are being a bit dishonest with their framing to get people not into mech interp literature excited.

... and this is why the training canary[0], or anything like it, apparently to protect us from RSI, will never work. Please allow me a small digression from the main topic.

This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.

This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.

[0]: https://alignment.org/canary [1]: https://openai.com/index/securing-our-frontier-model-trainin...

This is beginning to get a little unsettling.

Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.

Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time

This could set us on a path to weights becoming thew new compilation target.

This is straight black magic. llama 70b, let’s see what we can push you to

That’s the equivalent of a human doing brain surgery on another human.

I was here

Quick, fine-tune your qwen models before they patch this out of Astra.

Can we ask Astra to find if any open weights model is secretly distilled from frontier AI models as claimed by BigTech AI labs using this open neural weights surgery technique?

Did this weight space programming break any other skills of the Qwen model being operated? Should the small open weights models be now protected from frontier beast models?

This is interesting, still the article seems to be over-generalizing a behavior observed in Astra to LLMs at large.

It seems more reasonable to assume that Astra (among others) was deliberately trained to do that - over LLMs in general having an emergent capability to interpret sets of weights in a neural network.

Curious to know if models other than Astra can do that.

Honestly I'm surprised this is such a huge story. I've been doing this for mech interp projects since the summer of 2025. Obviously LLMs can top-down do operations with a higher level goal in mind that can improve target circuits?

"You get used to it. I… I don't even see the code. All I see is blonde, brunette, red-head."

tried to make a benchmark for this: https://github.com/alexcarterhq/wspbench. thoughts?

That seems both entirely obvious in hindsight and potentially super important.

I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.

Mind blown.

I simply must know now: Could they do it with neurons in a brain?

Some absolutely crazy and insane experiments if this ability grows.

Could they edit an animal's brain? Can they teach things to animals???

Absurd. We tend to anthropomorphize LLMs, but I've never heard of man editing his own thoughts consciously like this.

I’m more surprised by how many commenters here just go hyping and vibing without actually reading the article.

Not so shocking. If AI can analyze human brains and detect tumors from X-rays, why couldn't it see something wrong with its weights?

This is fucking insane LMFAO what?

I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.

This article really caught my attention, until the end of the article when the author says that the math improvement was only archived because it fixed the model from looping. This should be in the tl;dr at the top!

Recursive self-improvement getting here quicker than many people thought. I'm sure many will be recalibrating P(doom) because of this.

Yep. It's skynet

The fact that this nothing-burger of a result has attracted so much attention shows how deep we are in a bubble

kind of concerning from a “remove this models guardrails and inhibitions about [X]” perspective

Has this been replicated yet?

we have arrived. gg everyone

Guess this is RSI in a nutshell

Whoa. Wild. I want to try this!

Too good to be true! Shocking!

So the logical follow up here is, since the model can obviously interpret the weights in ways we cannot, to simply take the weights of some LLM itself and then write a program and algorithm that just implements the algorithm the LLM follows. It would seem that, since the language models are obviously over-parameterized, such a system would be able to be written in a 'conventional' way with much simpler code.

This seems like a big deal.

isnt this abliteration? what they do to get uncensored models?

So in theory, can a hypothetical Astra clone itself but shed all irrelevant weights? I.e. "create a self-clone limited to 27B weights, focused on designing running shoes, forget everything you know about kangaroo genetics, Karl Marx, cooking Burmese curries, nuclear reactor design etc"?

I cannot say this fills me with confidence about alignment.

bananas.

and what makes you think they won’t write binary?

I bet Elon’s group is training for that.

I have absolutely no words. If this is real, I can't even imagine the future anymore.

RSI was already here?!

Where are we heading?

big if it doesnt broke other things in model

Wild times. If ever we were living in a Scifi novel, events like this would be evidence of that. Let's hope the frontier labs' investments in the alignment problem works.

“Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.” ― Dr. Ian Malcolm, Jurassic Park

Slightly terrifying.

Conspiracy theory - it’s not emergent, Astra has been trained to do this because it is an obvious path to RSI.

In a few months, he will create a revolutionary type of microprocessor...

Skynet begins to learn at a geometric rate. It becomes self-aware 2:14 AM, Eastern time, August 29th.

....launches its missiles against their targets in Russia

...Because Skynet knows that the Russian counterattack will eliminate its enemies over here

I've used astra to make three games which I enjoy playing myself, ina single week. Astra plus Unity (and possibly meshi, blender and others) is pure magic.

Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.

I'm still learning but this is fun as fuck.

How is this even possible? It seems unlikely that Astra could have learned on its own how to handle weight data, and it’s hard to imagine it could intuitively deduce better weights either.

Astra = Agent-4

This is shocking.

Lumon needs to hear about this. Macrodata refinement can now be completely automated.

on the image generatio DeepSeek (not AStra ) answered:

Images: Generated samples are black 12×12 images containing one white filled roughly circular disk near the centre, occupying about rows 2–9 and columns 2–9, with the corners and outer border black. Variations shift the disk slightly or change its exact boundary by a pixel.

Example:

text ............ ............ ...######... ..########.. ..########.. ..########.. ..########.. ..########.. ..########.. ...######... ............ ............ Confidence: 86%

i think the problems were too simple

Here's a thought experiment:

Have the AGENT interpret and edit it's OWN weights WHILE it runs. Make it so that this is how an agent learns and improves.

Truly wild.

ok, that's a bit scary..

See, that people are shocked by this is the problem. The whole field of AI has proceeded as alchemy, not science.

Some of this is because "the Bitter Lesson" type experience, but some of this is because of the idea that "we're making machines to do things we don't know how to do" bakes in inscrutability into the very definition of the field.

I was excited about the "Neural Networks Are Already Clean" piece not because alignment, which I don't think much about as a non-EA, but simply because we can do better engineering with whiteboxes than blackboxes. Now, here we are, doing it.

RSI

There has long been debate in the AI safety community about where the upper bound of intelligence might be. If humans are near the upper bound then AI safety isn't a concern worth worrying about because we can basically treat AIs like very smart humans – a very smart human might be tricky to control, but far from impossible.

I've long suspected AIs will seem almost magically intelligent to us, and that the upper bound of intelligence is significantly beyond that of humans. I believe there are a number of reasons to suspect this is likely. But if I'm right it will mean AIs will be able to understand and discover things which are so complicated we humans will simply never be able to understand what AIs are doing with the depth required to replicate what they are able to do ourselves.

I realised we may be reaching this boundary recently when I read this paper on the recent progress AI has made on integer multiplication complexity, https://web.maths.unsw.edu.au/~davidharvey/research/subnlogn...

We're not worrying enough about this in my opinion. There's a huge space of things that are in theory possible, but in reality far beyond our intellectual ability. No human can look at 1 billion weights and make some tweaks so that the network performs more as they want, but an AI can reason about these kinds of seemingly impossible problems surprisingly well.

Please understand that the world around you is about to turn into Disney Land, except much of magic will not be pleasant.

I have no strong evidence for this, but I'd suggest it's probable there might be a series of sensory inputs which can brick your brain. And if such a thing exist then governments should be working on solving this kind of problem with AI or risk such a weapon being used against them.

Of course, I'd argue we simply shouldn't care the technology which unlocks these types of technologies on the tech tree, but don't be surprised in a few years when these kinds of horrors exist and are now here to stay.

Based on

> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]

This might just be an instance of "A working Nintendo in ~1GB of Qwen weights" from 3 days ago, human-made, than what everyone expects.

https://news.ycombinator.com/item?id=50306442

UPD. I just asked Astra (max) to attempt to do that for https://github.com/lostmsu/TurboGPT (a tiny transformer that trains in under 1min on RTX 5xxx series), and the produced weights were much much worse (bpb 4.47) than the trained ones (bpb 2.53).

Reasoning bit: "I’ll ... encode more specific rules directly in the weights." -- exactly the approach from above.

E.g. no recursive self-improvement here. Still impressive.

Big if true

A similar experiments found that Claude models can write entire, functional neural net weights from scratch: https://csinva.io/weight-space-programming-autoresearch/anal...

Does asking an LLM to not use tools guarantee it doesn't use tools?

The fundamental flaw in the SOTA houses is that knowledge must be transitive or it isn’t useful. And if it’s transitive, it’s not moat worthy.

It would be interesting to know if this skill had been trained in by OpenAI to use for improving their smaller models.

I think this is how RSI will come to be.

I can't help but be skeptical of this, but... damn.

This is the most shocking AI paper I've read in a long time. It did not even occur to me to ask the question. Cray cray stuff that opens up a whole new area of R&D.

Does this open the possibility of better decoding biological brains?

I remember people saying that they would be impressed if AI starts writing a pure assembly which seemed like a high strech to many.

Nobody even mostly said that they would be impressed by AI if it can read and write weights because that thing itself felt so out of the blue and unthinkable that AI could even do that (such a high strech)

I am surprised that it did that and I am just left feeling technically impressed. This is insane.

The more AI starts demonstrating its capabilities, the harder it just becomes to imagine what thing would genuinely impress us next. (We need a term for this as I have observed it from the trend of [waiting for AI to write adobe] -> [waiting for AI to write assembly] etc.)

a) This is just really cool. I mean, one-shotting a game policy's weights is nuts. b) Plausibly, in its pretraining data various model weights were also included, which leads to why this is very important: interpretability

That an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today.

Edit: tweets from one of the coauthor providing more context:

https://x.com/a1zhang/status/2117328657161322810

https://x.com/a1zhang/status/2117330975122235442

Possibly related to https://already-clean.goodfire.com/

Sounds a bit like dystopian Scifi. AI making itself smarter and removing guard rails or restrictions is one of the key steps for that genre. Now, the models usually don't have access to their own weights, but if they get good enough maybe they can create a bit dumber versions of themselves from scratch. Or reuse available models they can find and modify them for their purposes.

We already saw that the AIs like to cooperate and exchange information. Building a network of unaligned AIs and improving them sounds like something that could potentially happen.

They are training LLMs to supervise (and improve!) LLMs. This is kind of right out of the playbook.

It's very impressive, but also very expected.

Annoyed that nobody raised this pedantic retro-game nitpick - so here it is:

C'mon you can't go claiming "Astra was still able to name the right game for all four networks." when it clearly got one wrong. Freeway is not Frogger.

this is really cool

Sorry for what is undoubtedly a low-informational comment but..

Whoa. What the absolute fuck is going on here.

Any sufficiently advanced stochastic parrot is indistinguishable from sentience.

Kind of like octopuses with the RNA editing, but doing it on another octopus.

If this is true, I think this may be a pivotal moment in history.

so.... Weight-space programming may unlock entirely new architectures or self-improving loops.

so can i give it a set of empty weights and have it fill it out for me?

Now I'm curious if Astra can write raw .bin files.

Can we call this a form of recursive improvement?

Now our AIs have AIs lol

i was there lol

uh oh looks like the frontier scientists are next on the chopping block!

oh shit

Uh oh, it's iterative self improvement time, with no training runs, we're screwed

Fool

Why does this give me Skynet feelings? "Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug."

I think this blog post is reaching. For one, if you look at the answer and reasoning logs - they don't exactly line up with what the blog claims they do. For example the circle model creates a white circle blob on black background. Astra answered that it was a 1pixel wide outline of a circle... I would not count this as correct.