Hacker News

Would you be willing to share what the specific standards for your git commit messages are, and with what models you used that those standards couldn't be met?

Generally speaking you should be able to do something like that in AGENTS, I am really surprised that you have had an issue with most LLMs today getting them to follow a standard for SDLC. Personally, I like conventional commits a lot.

I assume you run a benchmark on all the tasks on every change the model recommends? Do you have it set up like nanogpt speed run thingy that karpathy for a thumbs up or down for every change when you did this with astra?

From the Fine Article you're calling critical:

>> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights

Out of interest, did you use an assistant like Claude Code or Codex to help with the training experiments?

Given that the top comment yesterday said “this is insane […] I’m shook” I’d say no, it wasn’t all that obvious to everyone.

As someone who uses LLMs in their coding but doesn’t pay attention to every development this is why I find it all so exhausting. I keep feeling like I need to keep up to date but there’s no point reading incremental updates because everything is so overhyped.

If you read comments about the paper it's pretty obvious people are misled. People think that the method read in Qwen weights directly to improve it. That's not what it did --- it authored a program to analyse and fix the weights.

What's confusing is they do have experiments that read raw weights, and the abstract is misleadingly written to make it sound like that's how the Qwen experiment works.

> who read the original post

I think there's your answer. A hell of a lot of people will often read the title without reading the contents of the article. I admit I'm often guilty of that

At this moment on the front page (early morning in the US), this post is #1 and the other one is #2, so it's working like a warning about the other one being misleading before clicking on it.

So you mean that the "The 'Astra can read and write weights to improve Qwen' post is misleading" post is misleading? You should post that.

[delayed]

Yeah, I was disappointed reading the title of this post because the work it is referencing was very exciting. But after reading this post, it's clear that the author is just pulling an "uhm, ackchyually" and responding to an abstract audience of laymen readers who don't know (or care) about any of the details of these results.

Anybody who actually read the original research site was fully aware of the details of the experiments because it's a well-written research site that made everything very clear. This author is clearly just jumping on the hype as a sort of contrarian where they were frustrated with how much excitement the research was generating and wanted to "correct the record" for the set of people who don't matter in this space.

Yeah. The article is not misleading. The title is, but you know, headlines are all lies that's why you read the article.

Yes, while reading the blog I immediately thought about https://arxiv.org/html/2406.11717v3 paper. Original blog does actually overstates it findings as nothing like that exists.

I really liked that comment because before that I thought I was going crazy. A lot of the discussion around these experiments seemed to imply, if not outright state, that LLMs had developed an ability to natively understand and manipulate LLM weights, a sort of transcendental introspection into their own brains, which if true would be absolutely groundbreaking.

But then you look at this complete trace generated by Kimi 3.1 when it tries to unravel the Breakout weights, and you see that it's bog standard reverse engineering, like any human would do, just vastly faster:

https://pastebin.com/G7VCWZHn

>This looks exactly Breakout: paddle agent at bottom row (row8), ball moving object anywhere, bricks rewards top rows, previous frame for ball movement. Agent weights bottom row, moving object weights rows1-7 (not row8? ball can't enter paddle row? Actually ball can), reward weights top rows1-3, obstacle walls. Previous frame used to infer ball velocity. No-op and directional actions move paddle left/right; up/down useless. Output biases similar all actions but W3 likely left/right structure. Could be Breakout. Should scrutinize output weights: actions 0 no-op,1 up,2 down,3 left,4 right. If Breakout, only left/right useful; network output weights should show action3 and4 distinctive and actions0/1/2 similar. Indeed W3 rows show patterns: [...]

Here's the work that shows how rule-based logic can be compiled into transformer weights:

* Thinking Like Transformers” by Weiss, Goldberg & Yahav (ICML 2021).: https://arxiv.org/abs/2106.06981

* Tracr: Compiled Transformers as a Laboratory for Interpretability https://arxiv.org/abs/2301.05062 (code at https://github.com/deepmind/tracr)

So it's not really an "LLMs can modify weights" thing. Rather it's "LLMs can write programs that can write or modify weights". Indeed LLMs can write many programs, it's impressive.

Yeah, this was my thought (as a layman) as well.

It seems some level of interpretability, even if it requires tooling, could result in the ability to spin out synthetic training data sets from other models in an automated fashion.

They might even discover some way to program the weights proportional to an error signal

Now that would be crazy

They make it sound like they improve Qwen by reading raw weights. They don't. In fact the experiments that use raw weights have very little to do with the experiments that work on larger models.

They did, except they also did write an abstract and a title that will mislead anyone who isn't going to read the full paper in details.

It's fair to say that this is malpractice.

Experiment 3 doesn't involve reading the weights in as part of the prompt. It's a coding exercise. Here are the turns: https://github.com/RikiyaT/lm_understand-logs/tree/main/exp3...

At each point the model has to write code to read the weights in with Python and some other Python code to perform analysis, like this: https://github.com/RikiyaT/lm_understand-logs/blob/main/exp3...

My complaint is that the post presents a confusing mix of experiments that imply capabilities that are more than the sum of their parts. This experiment 3 has nothing to do with experiment 1 for instance. In experiment 1 the model does read in the raw weights like it's reading an image.

> Why this post is here in the front page?

You know exactly why... because anti-AI hysteria is quite popular on HN.

Any "critique" or "appeal to resist AI" rooted in emotion and inability to admit that intelligence is not a feat unique to humans anymore will make the rounds here.

If people have programmed programs to interpret neural networks, what is so special about the AI doing so? It's a forgone conclusion.

You can let the AI generate training code for the next architecture too.

It's not like the AI is randomly losing its ability to write software just because it involves AI.

I am growing increasingly fond of the term "anti-AI psychosis". It's getting ridiculous sometimes.

Except LLMs already exhibit superhuman intelligence.

Agree, it makes sense, a lot of people are catching on to LLMs for reverse engineering finally, but its been possible. LLMs can and will eventually flawlessly dissect models. I experiment with Claude, I ask it tasks I know are possible, but have no idea the solution, it can reverse engineer obfuscated projects faster than most talented people I know in that field (talking about games now not AI). I heard a developer say their latest release would take at least a month to figure out the obfuscation and encryption, Opus 5 did it in like... 15 minutes of talking to me (I did throw it the right questions / suggestions). I do that all in the name of seeing just exactly what is possible.

Claude has been able to do reverse engineering for nearly a year if not longer, as well as full source code rewrites, if you know how to do a proper rewrite, luckily for me, most of my career has been rewrites.

If I told you, "I can write a Python program to repeatedly run a machine learning model and calculate statistics over the weights and activations to find and fix a specific problem", would you be surprised by that? Because this is very much a part of what ML engineers do.

And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.

It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.

It's exactly what the post says. I was waiting for my excitement to drop for the original research, but I concluded that I was mislead more than this "coverage" than the weight editing research, which was in opinion quite easy to grasp.

Agreed. Not sure it leads to anything truly groundbreaking, but more than likely will lead to very fascinating implementations that lead to even more interesting results 6-8 months from now. Think openclaw -> how people utilize now versus when it was first announced

That’s what I’m excited for: the raft of posts over the next few days as the inspired set to work

If it isn't accurate, it isn't showing anything. The Qwen math claim is entirely false in its presentation, it unblocked an existing capability. It did not instill a new one which is what an RSI loop would actually have to do

"[...] how you could prune harmful capabilities from models [...]"

We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models

"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"

If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.

How do you prune out the 'fails to prune out behaviors' behavior? If my model is failing to prune it out, can another model prune out that behavior?

> how you could prune harmful capabilities from models and build a more efficient training system

In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.

What it shows are two unrelated results mixed together for no apparent reason besides marketing. The raw-weight results are impressive, but haven't been demonstrated as ready to scale up to a real language model. The Qwen result is a mildly interesting application of functionality that's been on the market for years.

If you think that the definition of "RSI" includes systems with a coding agent in the critical path, something humans can already do albeit faster, then your definition includes all of human history. If your definition includes AI systems that learn in a similar way to biological intelligence, then only the raw-weight results from this paper are pertinent. Those results are indeed interesting! This article is merely disentangling them from the frontier-lab-fueled hype that overran yesterday's thread.

Isn’t that what the Google dreaming thing is?

According to this person, some problems are only "solved" by modern models: https://nitter.netbub.com/AlexKrentsel/status/21173766837246...

(opus <= 4.8 and gpt <= 5.5 couldn't match a PPO baseline network, and 5.6-sol and opus5 could barely match it. The 20x baseline improvement only comes from opus5.5 and sol > 6.1 / Astra)

I think the reviews are flawed for most of journals. But I can't really blame them, there is huge influx of new papers comming everyday. Unfortunately reviewers are more biased towards groundbreaking stuff so it seems they can't make connections with existing and prior work.

They are also biased to see from what institution you come from..

Someone has to already be doing this right?

If there's a linear relationship between the weights and the output, it's not so hard to read the weights and reason about what's happening. If you read the reasoning traces for how the models do those tasks, you'll see that they rely on those direct relationships.

The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.

I doubt this is either unexpected or emergent behavior. Anthropic and OpenAI are explicitly racing to reach RSI and I would expect them both to have trained their models on this problem, in hopes that it would generalize.

Yeah, I can read the weights in a small neural network, and for a given input, given that I know the activation functions, I can figure out what it does with pen and paper and lots of spare time.

I was just watching a video about this. They had this survey question for AI researchers asking when they think AI would take their job. The reason they asked this was that would be the point where AI starts recursive self improvement. If it’s able to replace the job of an AI researcher, then it could keep improving itself.

Recursive self improvement permits recursion at any level of abstraction as long as k > 1.0 (much like supercriticality in nuclear reactions, which iirc is where the earlier term for RSI, “intelligence explosion”, comes from - substitute intelligence in for neutrons and it is the same math with similar apocalyptic implications).

It's obvious that recursive self-improvement has been happening and accelerating quickly for some time.

AI coding assistants make it much easier to conduct experiments and improve AI models, which in turn improves the coding assistants etc.

You've expressed your criticism plainly without addressing anything the post might've done well. I think that's absolutely fine, but why would your criticism apply to the post but not your own comment? Personally I think it's because the standards of peer review don't apply to HN comments, but I don't think they apply to blog posts either.

You do realize how academia works, right? Granted, it is often dysfunctional, but this ain't that.

Trying to find every single angle and option to break something is how we verify the truth of something.

If something is objectively true, it will survive the scrutiny. If it does not survive, it wasn't objectively true.

This is how science works. That ain't no flaw. It is a flaw to be offended by science doing science things.