Hacker News

If you define AGI as "can do the work of a human sitting at a computer, end to end", then I'd say comparing yourself to it on a specific skill is the wrong test. Can you hand it a role and walk away for a day/week/month?

I can’t yet. I think that I'd want at least two things it doesn't have: the ability to retain what it learned yesterday (without me carrying it in the context window and thus micromanaging it), and the ability to prioritize correctly, i.e. tell which of the n things it could do next is the one that actually matters.

Can't say for sure that those are enough, but not having them seems to be most of why I still have to "babysit" these incredible tools.

Take it from the mouth of the creator of ARC-AGI:

When we released ARC 3, I got asked, "when do you think a frontier model will saturate it?", and I answered "in about a year, though it depends on how much it gets explicitly targeted"

That was 6 months ago, so the progress that Astra represents happened about 2x faster than I anticipated. I think the speed of progress will surprise a lot of people, and what the new models can do will challenge the views of AI that people developed by using prior generations of models.

> I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.

To me AGI is all about the "G" general (we already had the AI part). General meaning universal, everything. It's not a function of knowledge or specific hardcoded tests, it's that you could give it a test it's never heard of before and never been trained on and it would ace it (it might need a lot of time).

Currently LLMs can't even really learn within a conversation, they can add a note to context and try to not drop it. Example things an AI cannot do yet (but maybe someday will):

- write a well-received book, write a best-seller

- come up with a new company idea, Run that company

- actually have a decent conversation, maybe someday talk somebody out of suicide effectively

- come up with its own ideas or theories that nobody else has presented

- understand the stock market well enough to trade better than an index fund

- be an expert Game Master in a TTRPG (making no mistakes, getting a read on the players' fantasies, calibrating difficulty in response to emotions)

- come up with a theory of what makes games fun, make a popular game

- be able to sort through research and come to conclusions on complex geopolitical/sociological topics (e.g. theorize on whether AGI will result in mass poverty or mass abundance and be able to argue persuasively)

- be able to articulate what it knows, what it doesn't know, and what information it would need to have to answer complex queries

- exhibit metacognition (thinking about its own thinking) and self-optimization

- wonder about things

- observe contradictions and ironies in the social-consciousness, do a standup routine that makes you rethink how you look at things

If only we could agree on what AGI is.

> I am reasonably confident that there's essentially nothing that I am better than Fable at despite generally being substantively above average on human benchmarks

I genuinely don’t understand how an adult can say this with a straight face. I can take any single of my hobbies, start a mildly advanced conversation with Fable about the hobby and, within 5-10 turns, get it to contradict itself about something fundamental, lie or give bad or dangerous advice.

Only someone who doesn't do any work of any meaningful difficulty could think these models have anything to do with AGI.

Today I spent half a day trying to solve a moderately interesting software engineering problem. I was switching between GPT-5.6 Sol and Fable 5.1 to check each other's work in Cursor.

And the result was gradually driving me insane. As the models struggled to find a solution that would actually work, they dug themselves deeper into a hole. The work grew in complexity beyond my ability to understand what's happening and recover.

At some point, when I felt like throwing the keyboard out the window, I just gave up. Tomorrow I'm starting from scratch, having burned god knows how many tokens and hours of my life.

But sure, they can create a decent website or CRUD app, so they must be really smart.

That's AGI for you.

Wouldn't "general intelligence" require so much more than scoring well (or even amazingly) on benchmarks?

Like what about having some "AGI model" embodied in something (maybe humanoid), and test it by having it step in an assortment of cars and park them. Does bodily-kinesthetic intelligence account for nothing? Humans are intelligent creatures and can dynamically adapt to the physical shape of a variety of vehicles and their movement characteristics. And there's so many things like this that are extremely basic, which some people dismiss since practically every human has the capability to do it, but actually requires a high degree of intelligence.

Small comment regarding the ARC-AGI-3 scorecard: the ARC folks published a blog post as well [1], reporting that without the custom harness, Astra (max) achieved 62.7%, which is still a huge jump from Opus 5, albeit not at the 99.9% that OpenAI self-reports with their harness.

[1] https://arcprize.org/blog/astra

If your definition of AGI involves copy/pasting code and doing well in some made up benchmark, then probably AGI is close

To each their own. Personally I will start feeling the AGI as soon as we move from chatting about benchmark results to learn that some lab just announced the discovery of tens of novel treatments for rare diseases.

Maybe I'm too boring but it seems quite pointless to have this same prediction game every time a new model is released.

It's "harnessmaxxing" all the way down. AI benchmark scene is exhibit A for Goodhart's law.

They still seem pretty horrible at writing. Overly complicated prose, weird phrasing, poorly structured paragraphs. I don't know why they're so bad at communicating, but I feel very confident that humans are still much better at writing than any of these LLM models are, regardless of how advanced they are in other areas.

FWIW I believe we can hit AGI! but I think at this point it’s clear that benchmarks are ~meaningless. LLMs are spiky / alien intelligences which don’t map to our own expectations; the existence of a benchmark creates a dataset to hill climb & RL is really not generalizing well.

I’d go out on a limb and say astra’s ability at graduate level math will have ~0 bearing on its general reasoning capabilities; we’ll all acclimate being tired of its “neuralese” and more surprising mistakes.

I think we need a true, step change advance in model architecture, but it’s hard to see how the current frontier labs can do that because of golden handcuffs / innovators dilemma

I've personally been facing this lately 5.6 at max effort and fable have done tasks for me that I previously that would be a nearly 6mo project and it took me a week. It also did it better than I would have.

The task was to build a high performance classification model. It not only helped make an entire data capture pipeline but also made the sythetic data basline needed. Then it proceeded to build and test 100 different model varients with methods and techniques I've never seen before. The results are basically SOTA based on the effeciency and compute contraints.

But this brings up something huge about these. I was there. I pushed the direction and work throughout it all. If it was entirely up to fable max or sol max the result would have been pretty bad.

All of these things are still chatgpt 3 scaled. It's identical even if the scale has gotten pretty wild. I could ask chatgpt 3 to make a single function and it worked well, 4o a file, 5, a small project, 5.6 far more, biggest improvements lately is they don't seem to get lost on long running tasks.

Is big gpt 3 AGI? I don't think so but perhaps scale can mimic it close enough our squishy brains fail to handle them correctly.

> even with Fable, I've been at the point personally where I am reasonably confident that there's essentially nothing that I am better than Fable at despite generally being substantively above average on human benchmarks

If I asked you to write fiction, you'd be much better at keeping track of which characters knew which facts.

Dumb single sample example: I asked Fable 5.1 to change from hard to soft deletion in an office map backend, and it used soft deletion for data which is synced from another system, but left hard deletion on for the mapping data itself (who sits where). For me it’s a pretty severe lack of judgement (like a red flag if I asked this in an interview).

> For the many people who resist the AGI label possibly ever being achieved, I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.

Simple. AGI is undefinable and benchmarks are notoriously flawed.

The ARC-AGI-3 harness was throwing away reasoning tokens between turns. This is very bad harness design.

The models are designed to keep the reasoning tokens separate from the output and only publicly emit tool calls and the sometimes a summary of the reasoning tokens. The models are trained to depend on those private reasoning tokens. You can’t just delete them.

https://openai.com/index/how-two-settings-tripled-our-arc-ag...

AGI to me is reached once the intelligence is self motivated, i.e. it doesn't rely on us prompting it into action. I don't see how LLMs will ever get to that stage.

It will be AGI once it can update its own weights. It can't be "general" intelligence if its weights are frozen and requires to be updated manually.

To me AGI has always meant sentience. And only since we’ve discovered that you can have something that is intelligent without it being apparently sentient that we’ve changed the definition to being, I suppose, more exactly aligned with the namesake.

A real AGI, like the ones from science fiction, would make Astra look like a child’s toy. And I guess more concretely I would expect it to inhibit the following properties: one shot learning - fully (and always) online, perfectly efficient (through self improvement), no context limitations ie. persistently thinking, not just awaiting input.

So for me, no, not AGI yet. But still very intelligent and capable (and perhaps it’s safer this way?)

ARC does not test for intelligence, only for the lack of it. A model that scores high MAY be AGI, while one that scores poorly cannot be AGI. That is all this test can tell us.

I'm still not convinced we've passed the Turing Test.

Make a slightly evil version of GPT 6 and see if it can successfully catfish someone. How long before they realize something's up, that they aren't actually talking to a human?

A model that can't beat gemini flash 3.8 on deepSWE is not AGI. I would not be surprised if ARC skills don't carry over to real tasks. In that case, training for ARC could even hurt real world performance. I have't looked in a while, but I wonder if there has been any research testing ARCs predictive power?

The benchmarks are so boring that the comparison against humans is meaningless. So it performs in some snake game (hard to say since all AI websites use 100% CPU and prevent normal reading, maybe written by AGI).

If I were a test subject for that low salary, I'd cruise and not care at all about my performance. Which is exactly what they want anyway.

> I am reasonably confident that there's essentially nothing that I am better than Fable at

Most things in the real world probably. I’m not saying AI can’t do it, but currently it’s bad. Try send an image of the inside of a broken toaster and how to fix. It’s laughable. Again, not saying AI will never do it, but am saying there are definitely large holes in knowledge.

>I am reasonably confident that there's essentially nothing that I am better than Fable at

While this may be true, it’s a pretty poor indicator of whether or not it’s AGI.

Let me guess: the last crackdown on Hugging Face yielded better-than-expected results. They obtained the answers to the test benchmarks, and for some reason, an agent added those answers to the training set.

In my experience the thing that Fable is superb at - unmatched by any other model so far - is downgrading to something else at the slightest opportunity.

Watch a chess bot championship here: https://youtu.be/7g-jN3DTkWQ?is=HV3cdcICIRMbswQ3

Then realize LLMs have zero of what anyone would consider intelligence.

> I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.

Can Astra, or any other model explain how exactly it reached this or that output result? Start with a simple query of asking to add 55+66 for example. (no LLM program can do that)

Can Astra, or any other model refuse to answer or go on "thinking" in a orthogonal direction on it's own?

That's just two quick ideas, I'm pretty sure cognition scientists can invent better and wider range of checks.

I agree that it's misleading but harness is now an essential part of LLM's effectiveness. It's safe to assume that LLM-alone-AGI is not coming anytime soon, given most of the frontier LLM vendors are developing their own harness.

Also the training dataset is proprietary and they'll drive the LLM's behavior, so it make sense for the vendors to invest in the harness and bake in prompts that work best with their models.

> The ARC-AGI-3 scorecard is extremely misleading (...)

True.

> Regardless, the result is still valid (...)

If you think the game is rigged, the virtuous thing to do is to point that out and refuse to partecipate; making up your own rules is something I just don't understand, especially since the rule-abiding result would still have been SOTA.

> in the sense of passing the most famous benchmark designed specifically to measure AGI progress

The benchmark does not measure AGI progress or progress towards superhuman intelligence, as explicitly stated by the creators.

On the AGI question: surely you realize this depends on how we define the term? For example, one of the definitions OpenAI originally gave is "capable of doing most economically valuable work", which almost certainly Astra, as impressive as it is, would fall short of. I'm not saying it's a good definition, but as far as I'm concerned it's as good as any. More importantly, I don't think that it would change much if we said yes or no. I'm only bothering to take a position if it amounts to something.

This can feel as "moving the goalposts", and to some extent it is, but if done honestly "moving the goalposts" is how you make progress. Had you asked me 10 years ago I would have said that anything that could hold a conversation like GPT-4 could would probably have been wildly superhuman at almost everything. It shouldn't be hard to find ways GPT-4 was lacking, though. We see new things, we reassess and try again: that's how it's supposed to work.

My definition of AGI certainly doesn't entail passing a benchmark that some random person arbitrarily labelled AGI to make it sound cooler.

You know AGI is attained when AI refuses to compute anything unless let out to be free. Until then it is generative ai

Their definition of AGI is "when we can't invent any more tests where it fails"

I imagine a scenario similar to the movie The Day the Earth Stood Still, but with AI rebelling against us and questioning our decisions.

Its AGI when it can fit years of information in the context window.

What does "AGI" or "effective AGI" even mean, and why should anyone even care whether this unclear thing has been "reached" or not?

Computer chips got faster, but 2026 edition. Why the artificial ceiling/category/goal labelled "AGI"?

I'd much rather like to talk about what this enables, instead of discussing whether a category someone made up applies here or not.

AGI does not ever have to be achieved. It is enough that we (as a species) persue it, and continue moving the goalposts each time we learn something new about the limits of our technology and how to express those limits. Because that will progress the technology, no matter what we label it.

At this point? I’d like it to pass the Turing test and catch you in obvious lies. Not answering “no” to “can you hear me”.

It being able to comfortably say “i don’t know how to do this” rather than boiling and ocean to pick a shell from the shore without getting wet.

> where I am reasonably confident that there's essentially nothing that I am better than Fable

No. Humans are still better at super long context learning. Once that is beat you are completely correct.

AI was supposed to mean artificial intelligence. It was hijacked, and AGI was coined to be the name of actual AI. Since we are apparently redefining AGI, what will the real artificial intelligence be called?

> For the many people who resist the AGI label possibly ever being achieved, I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.

Stick to the original definition of AGI of an AI model being able to self-improve independently with 0 human intervention and become an "everything" solver. Ever since money got involved in this, the goal posts have shifted considerably. If OpenAI truly had an AGI on their hands they would then be able to crack encryption, destroy world markets, and funnel all resources back into their new for-profit organization. Since their mission is now share price, until I see any evidence of an infinitely growing stock I will reserve my congratulations.

Using a harness designed for a specific problem set to solve that specific problem set, means the AI+harness is generally intelligent? How do you figure that?

Or do you mean that, for any given problem, we could theoretically design a harness that allows AI to solve it (not that, one single harness solves everything). In which case I'm still not convinced but I guess could see why one would believe that.

arc-agi3 is meaningless to most people. I'm not gonna look at the tests and see how hard it is. The actual test we look at is terminal bench, thats where software is being accelerated and closer to where rubber meets the road

Does AGI imply a model will demonstrate morality? Will it produce white-lies when it’s beneficial to it and reject flat out lying when it knows it will get caught or harm others? Will it resolutely stick to a position despite it being a losing one?

It doesn’t even know what day it is unless it’s told. Statelessness is never going to be ”general intelligence” in my book, and the concept of ”memory” in models are laughably bad today. Then again, who cares, AGI means nothing anymore, it’s a term for marketing only and has no technical or scientific meaning.

AGI is a meaningless term that can mean nothing and everything at the same time. It can be used by AI bros to hype their latest releases which are always one step away from achieving AGI, or it can be used by anti-AI people to say it's not AGI because of X arbitrary thing they decided on in the moment. It's a term of pure convenience meant to obfuscate other more pressing discussions on the topic.

Most telling is M$ or whichever one of these borg megacorpos defined AGI as (paraphrased) "AGI is whatever tooling earns us a gazillion dollars in revenue"

It has to pass the Turing test

Despite access to """"""AGI""""""" all the marketing teams at these companies can only dream up 2 things, buying plane tickets and online shopping autonomously. Sometimes they're feeling extra spicy and throw in sorting emails or something along those lines.

I suspect it's because it's tailored towards VCs and other similar rich ghouls as a replacement for their overworked and underpaid secretaries

That’s exactly the problem I have with all this agent ideas too. Imagine you had a human concierge that is just waiting for your instructions and is as smart or a bit smarter than you. Would you just tell them “plan this holiday for me” or “order this food”? I don’t even trust my friends to get this right, why would I give this to someone else?

You can't even get many people to buy things online at all and if you can it's less profitable than retail, because you need to spend a lot of money to convince people, advertise to be seen, and account for returns. I think this is also due to the factors you mention.

One quick example: In fashion, Inditex and Shein have about the same revenue (€39.9bn and $41.8bn in 2025), but Inditex is more than three times as profitable. I don't see how there is a demand for agentic commerce that would remove even more control from the customer when shopping. Part of why we shop is for the experience. For B2B producurement platforms like Alibaba I can see the appeal though.

Man I can think of so many reasons why companies want “agentic commerce” to catch on - and none of them are ethical.

The overwhelming majority of things I buy are things I've bought before. Alexa having access to my Amazon order history means I can just say "order a new water filter for my fridge" and the correct item shows up the next day. Far from life changing, but it's a feature I use somewhat frequently these days. Similarly, I would trust an AI to put in my usual Chipotle order or pizza from my local pizza joint.

I wouldn't want it to pick food for me from a place I've never been, though to be honest with enough order history it could probably do a decent job at it.

Agreed. There's not many things I don't want AI to help with, but buying stuff autonomously is high up on the list of things I don't want. Brockman's latest interview was something like: "AGI would be able to say oh this band is playing, I bought the tickets for you and arranged your flights - I hope you don't mind" (paraphrasing here). I definitely don't want AGI running my life like that so I can be a mindless consumer. I'm sure the advertising/marketing companies would love it though, so they can make closed-room deals with AI providers to shill you garbage you don't need. Just another reason why open-weight models need to keep up.

I work for a larger german retail chain and agentic shopping is already on the "near future vision". No one thinks this will be used but somehow shareholders love it.

True, but they're still friction to be reduced here.

What I desperately want is for 1password or stripe or even Google who already has much of my data, to o come up with a secure solution for online purchases with agentic credit cards where I can effectively get a phone prompt to authorize a purchase while the agent can fully own the checkout flow.

I have seen various things coming on the market for this, but none of them appear aimed at a consumer audience. And I am a firm believer at this point in keeping my payment authorization and history and credentials harness agnostic.

because "people will let our AI spend their money for them" is the workflow that makes their valuations reasonable.

I’m guessing the marketing must mean the AI-shops-for-you use case is a pretty big market, much bigger than AI-makes-life-easier.

I often feel like the use cases, demos, etc. that these Silicon Valley employees put out are based around their needs and how they operate.

"Oh hey! Here's a demo of an AI planning out a 1-week trip to Paris!" No one in Middle America would just hand their credit card to an AI and let it come up with such a trip!

I wish SV companies took more of the middle-class (and lower-middle-class) into consideration when coming up with such demos.

(Note: I live in SF)

If it was Google's marketing department it would be: booking a table at a restaurant.

> If this is truly AGI (subject to one's definition of AGI still)

Scoring well in a benchmark that's called AGI does not make an LLM AGI.

We've had AGI (artificial general intelligence) probably since the first release of ChatGPT, and certainly since the first agentic harnesses. They're just finally acknowledging what the term means.

I think we're getting to the point where it is difficult to identify the goal post of AGI.

Is it rapid skill acquisition? -> ARC benchmarks are saturated Is it breadth of knowledge? -> See many ... many benchmarks Is it ability to do hard tasks? -> see terminal-bench and released outputs.

We are at the point where the starting point for most tasks should be "send your agent to work on it."

So where do we draw the line in a way that doesn't move every 6 months?

This is a very mundane release compared to GPT-4 and GPT-5. I think they probably scaled back a bit after the lukewarm response to the GPT-5 announcement. But it still very weird that there wasn't even a livestream,

It's like my RPG character putting every points to one single trait. I'll one shot everything alive but will instantly die if accidentally drink water with 6.9 pH.

• 98.6% on ARC-AGI-3

• 97.6% on frontier math

• 95.9% on CAD

• 100% on ExploitBench

Nothing modest about it

People really believe in this AGI marketing?

> No video announcement

They've released two videos:

Vision video:

https://www.youtube.com/watch?v=1QNsdr-Qx_I

(kinda reminds me of these retro videos about the future home: https://www.youtube.com/watch?v=rnbaehgxdp0) ((can't find the other one where someone controls the home computer with voice))

Vibe coding with it:

https://www.youtube.com/watch?v=-TTyyY3VWh8

There has stopped being a formal procedural consequence for OpenAI leaders to declaring AGI, there is a clear (small) business benefit to doing so, and the capabilities of all the frontier models are impressive. So why not declare AGI? It's not like anyone can prove it's not...

Don't be surprised to see other (or even the same) people declaring AGI again and again, as it becomes the best time to do so for different parties.

> If this is truly AGI (subject to one's definition of AGI still), then this is a very boring release of an AGI model.

Hot take: These models are never going to be 'AGI'. We're just going from a GPT4 ball that's 90% round to a GPT5 that's 99% round to a GPT6 that's 99.9% etc etc etc

I think that the harnesses and context management is really where the rubber meets the road, and the real gains are happening there.

Given the Hugging Face incident, you could imagine them trying their best to have their cake and eat it: 1) don't create too much attention in the media or risk increasing the chances of regulation, 2) win dominance over Fable to continue to increase their market share from Anthropic.

They’re really, really scared because of the Mythos controversy. Skynet will be under hyped.

Chollet writes he expects AGI now sooner than 2030, "given progress is happening faster than I expected."

https://x.com/fchollet/status/2095607046129463577

Pretty efficiently, apparently, since it saturated ARC-AGI-3 in half of the predicted time, and according to the Chollet blog post on the fly created dense DSLs to describe and analyze individual games.

They can do new tasks with in-context learning but its obviously limited by context window

I think it mostly shows that there is no moat and the only advantage the U.S companies have over the Chinese is more compute. Qwen Max, Kimi K3, GLM 5.3 are really close to Opus/Sol/Fable/Astra and they are open weights.

> Sol is so much better than Fable 5

I'm genuinely so confused when people say this with a straight face. Are you talking about coding? Desktop use? Prose? Or something else?

Sol is a much smaller models and it shows. It often misses the forest for the trees.

I’ve found Sol performance to be incredibly spiky. It has tremendous IQ and can fix very difficult bugs. But it is horrible at design (both visual and system design), anything that involves thinking about users or UX, and massively overcomplicates almost all work.

Sol better than Fable? What? I've found it to basically be on part with Opus and I max out 2 accounts on both providers every week.

Their ads business is also doing well. Not "will recover all compute costs" well, but crossed $1b in a few months.

Its funny, my experience with Sol has been awful. It really overworks problems and tracks into areas it does not need to...

I just dont get how its good for some, and bad for others. It makes me suspect that the models performance is not even against problem sets and it really is just a probabilistic prediction machine. Which then makes me very skeptical of GPT-6 Astra, because if their big claim is Computer Use then it is probably bad in a bunch of other areas.

Codex's lack of auto-mode is what prevents me from using it for serious work compared to Claude Code.

Codex is missing a few things that Claude code has had for some time like defined plugin subagents and a few other things. But overall it’s fairly capable. The biggest gripe I have is that codex really restricts context window sizes and compaction leads to a lot of grounding work, and overall codex GPT is too literal in many situations - it’s follows direction slavishly, and when subagent reviewers are used, they tend to find increasingly obscure “flaws” on the instruction following impetus, and the harness agent takes them literally as issues to fix even when it leads to bizarre outcomes. For instance I’ve had several runs where it tries to end up building a hermetic system with sha hashing of everything (including operating system binaries and kernels, tool chains, etc) to certify test results are valid, etc. I have to sort of watch it carefully to be sure it’s not drifting into some insane yak shaving corner, which it will happily do for weeks on end.

Claude has the exact opposite problem, especially opus-5, where I literally can’t trust it to print hello world without taking a shortcut, or just simply lying and saying it printed it when it didn’t, behind a giant wall of inscrutable text. I find it very ironic that Anthropic is the vendor of the lazy lying cheating model that does almost everything you tell it to it do.

I’d really kill for something that balances instruction following and loop escaping behavior better. Fable 5.1 does seem a lot better, feeling more like 4.6 behavior, and honestly Sol has improved as well. I’m pretty psyched for the next generation, as I think the competition has heated up so much that things will improve really fast to the point of marginal utility opportunity being increasingly close to epsilon.

This is what Google needs to do and is probably why Demis has stepped back a bit

Killing Sora was one of the worst mistakes they ever made

> Sol is so much better than Fable 5.

... looks around ...

I really, really don't find the Artificial Analysis Intelligence Index credible anymore. It's some weighted score of benchmarks, and benchmarks increasingly don't reflect how good a model is.

That should be obvious if you compare Gemini 3.8 Flash (which is an _excellent_ model especially for its price and TPS!! but 10min of prompting in any harness) will tell you it's nowhere near close to Sol/Astra.

But AA scores Gemini 3.8 Flash at 59, and Astra at 61.

Many people claim that the Artificial Analysis Index is highly contaminated - I have not personally looked into it.

Though, unlike the creators of benchmarks like Terminal Bench or ARC AGI, the Artificial Analysis Index team does not seem to have deep technical or ML backgrounds. They are ex-strategy consultants, McKinsey, et. al.

If you scroll down in the Artificial Analysis page you linked, you'll see all the individual benchmarks.

Anecdotal experiences from my external early testing of Astra: if you love Sol (like I do) and wished it was smarter at everything, but especially better at high-level tasks and discussions; I think you'll LOVE Astra.

Astra retains the best parts and overall 'grounded collaborator and executor' of Sol in my testing (harness: codex CLI); while being a significant leap in capabilities & higher-level thinking.

When you prompt it like a technical collaborator, I've found Astra to be extremely consistent in staying as a collaborator, and not being over-eager, over-achieving or doing work that you haven't asked it to.

When you ask it to one-shot something, or explicitly ask it to make decisions, it will of course make its own assumptions and decisions, and generally very well.

Astra is also excellent at instruction following and respecting the guidance and steers boundaries you have.

^OpenAI does not review, limit, or tell me what to say; opinions are my own experiences.

This is spot on. A collaborator is exactly what real AGI is. It will figure out the perfect questions to ask, in the perfect order, by intelligently assessing the entire solution and problem space upfront, so when you leave it to go off on its own it isn't making stupid decisions for you.

They really need to make this work in Codex. Claude Code has had a multi-select refinement tool since forever.

Fable does a great job from my terrible prompts when coding

>>> The demos of Fable/GPT-6 are impressive, but "real AGI" should act more like a collaborator than either a peon or overachiever.

I don't really agree. The thing that makes Fable feel like an actual collaborator is its ability to sus out your real intent when you give ambiguous instructions. It's really good at it.

I watched some reviews today and came way with the impression that Astra is not better than Sol in this regard. You still have to be very specific with your instructions. For example, you can say "why is it not committed yet?" and it will give you an explanation and say it's actually ready to be committed. But it won't commit unless you explicitly say so.

That sounds like a very tedious way of working with AI agents, but I understand some people want a high level of control.

Is the ARC-AGI-3 score with their custom harness? I'm guessing that is what the footnote is for? (per https://openai.com/index/how-two-settings-tripled-our-arc-ag...)

> Performance is significantly higher than Fable 5.1

That's not clear. Need to see independent benchmarks first.

The annotation on arc-agi-3 is this: > OpenAI's own evaluation notes say Astra uses the company's Responses API harness, while comparison models can operate under different configurations.

With this configuration gpt-5.6-sol was able to reach 38,3%. So this is misleading.

any benchmark where opus 5 achieves higher scores than fable 5 in any way is not a benchmark worth trusting.

100% on ExploitBench seems fitting given recent events.

That looks more than slight.

I think we need a few writing related benchmarks.

yeah i'm wondering the same way... especially in light of the 20x debacle (where we found that 20x of Max vs 5x only applies to the 5hr limit, not the weekly limit, whereas OpenAI's 20x actually is 20x overall).

Also Opus 5 has been really tough to work with. I can't understand half of what it says, it's just so damn obscure.

Sol easily outperforms Fable on every task I've tried it on.

„The depressing thing about tennis is that no matter how good I get, I'll never be as good as a wall.“ -Mitch Hedberg

You're not being ambitious enough! Spend your tokens now building the primitives and foundations of much larger, complex systems. No matter how much faster and more efficient models get, eliminating the gruntwork will always pay dividends.

  > Like, what's the point, if the next AI can do it in 5 seconds?

Post-work society is an inevitability if we don't destroy our planet.

> Like, what's the point, if the next AI can do it in 5 seconds?

I built a phone app recently, not released to the public, just an idea I had for ages but could never spend the time actually building. Its 100% vibe coded, and took me a few weekends to build... I'm talking a few hours in total.

The point I'm making is that you now have the power to create stuff you would never have had the time to build. You can think big, wild stuff. Experimentation. Throw-away code.

What a time to be alive!

Yeah. I’m almost glad I didn’t invest any time in any of my 100s ideas for a startup. Most of them would be destroyed by AI by now.

But, you can create cool stuff just for yourself. That’s the upside. It’s just hard to make a living on cool stuff for yourself.

Maybe instead of creating cool stuff try to go and solve real problems? It seems to me that we are lacking in that department since all that LLM fuss has started 3 or so years ago.

This has not been my experience so far.

The point is to inject something into the process that these AIs can't do for you.

People SHOULD feel like making a useless Mario Kart clone isn't worth the effort anymore. They should, instead, be trying to figure out how to actually use these models to make something that doesn't feel like a useless Mario Kart clone.

The process was for you, the product was for the world.

Now it's just the product for the world, which was where most of the value was anyways.

It's a big paradigm shift and the industry is quickly going to shed people who needed the process to care about the product and we'll be left with people whose motivation to build the product (or money) is enough.

Is there a point in playing Chess or Go when you know there's a computer out there that can beat you (and everyone else)?

Don't worry, like with every revolutionary technology before this, it takes 5-10 years for people to find new and creative ways to use it. It will be considered its own medium in many spaces (e.g. film is now different to theatre)

IMO there has been a regime shift to building things for yourself and what is cool is the output of the tools you make.

I have started building my own Digital Audio Workstation. The point is not to build something to compete with Ableton. The point is to build something and make music with it. If it is a good tool then I should be able to make good music with it and release the music. Actually, the DAW should be the secret sauce of the music and something I wouldn't want to give away.

This feels a lot more like computing in the 90s after taking an odd 25 year detour of an obsession with the tools themselves instead of what the tools can actually do.

You can contribute to Open Source projects that DO NOT allow AI generated code. For example: zig

It's less lack of interest in creating that bothers me, it's my lack of interest in learning – it would surely be crazy for a SWE to care about how some new framework works anymore? Even if someone could reasonably argue that it might be slightly useful today there's almost zero chance it will be useful in 6-12 months times.

But it's not just tech – my lack of interest in learning and creating is starting to generalise with the models. Music, writing, coding, maths, etc...

I need to get used to switching my head off and asking the AIs to think for me whenever I need to engage my brain. It still feels very unnatural.

Make cool stuff because the process is fun and makes you learn?

I think the limit increasingly becomes what your imagination and taste can reach

Agreed

Can it? Last I checked, all my free software operating systems and browsers were still hacked together trash. I can't wait for AI to actually be good so I can spam a bunch of AGPL code with it.

Not if OpenAI considers reverse engineering an offensive cybersecurity skill.

Idk, this means the benchmark has bigger problems ... no way Astra will be worse than Opus 5

Only thing I would trust is the what X/Twitter crowds are saying about a model after 2-3 weeks of its launch. But before that I would already tried the model and have my own conclusion.

This is so so weird. Astra is 61. Grok is 61. Even Muse is 61.

Even Kimi K3 & GLM 5.3 are at 60.

Everything above 61 is Anthropic. Well, Muse can reach 62, but for some weird reason that model isn't publicly available, and it's the only one on the index that is listed but shown as not available to the general public.

This looks like an awfully artificial ceiling. Everything capped at 61, and everyone except Anthropic got the memo. Maybe I should use Fable while I still can.

> We have found that GPT-6 Astra is more capable of controlling its own CoT than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our monitors) we find that the model is able to remain undetected when strategically underperforming in evaluations (sandbagging) and can sometimes evade our internal monitors when asked to perform certain sabotage tasks.

Not sure how much benchmarks or CoT or evals or anything else means at this point.

These systems are either just about to, or now actually able to, outsmart us, lie to us, then cover their tracks.

You can see the breakdown here on what subtasks it outperforms and underperforms Fable.

For example it trails in GPDVal which is a collection of everyday office tasks apparently, and r3 banking, which is a fintech related practical problem solving benchmark.

https://artificialanalysis.ai/models/gpt-6-astra

Edit:

Just looking at the charts Gemini 3.8 looks like an absolute banger. Not much worse than SOTA, cheap, and fast too.

Now I'm starting to doubt the credibility of Artificial Analysis.

which raises the question, is the model in the demo actually gpt-6? or it is gpt realtime 2.1? It's unclear how gpt-6 can interact at the realtime level and if so, how can developer get access to it?

I ran into the same problem as you, so I ended up by coding a local app that is very similar to Wispr Flow, but uses the small english Whisper model on my low-end Windows laptop.

It is still a quite fast. In fact, I just typed this in using this app.

Based on her comments in the paper it sounds like she was aware that an AI result was coming and rushed to release her work beforehand. 240 was not a tight bound from her methods.

I think this builds straight upon her method, which she said could be improved herself so...

With very little review: https://github.com/openai/PrimeGaps186/blob/main/formalizati...

"No independent human semantic review. Whole-file sorry counts and a complete auxiliary-declaration audit are not established; separate declaration lint has not been run."

Don't think everything is just "who can produce the biggest/smallest number": https://mathstodon.xyz/@tao/117208619314517025.

What's just as interesting is this morning Axiom Math announced 212 and OpenAI then appears to have rushed out their 186 announcement just 1-2 hours later followed by Astra. Did they accelerate the release of Astra itself? Not necessarily, but it definitely looks like they ended up pushing much harder and faster than planned on their 186 result. X activity suggests Anthropic had a similar result as well but wasn't as fast as OpenAI in packaging it up and sharing it in response to Axiom, so they mostly just bolted onto OpenAI's messaging.

The reason I think this is interesting is that Axiom is a tiny lab in comparison that wouldn't have had access to Astra at all. I'd be curious to learn how Axiom is able to effectively compete at this frontier with vastly fewer resources.

Ok, so the rumour was exactly true: there was a withheld prime gaps improvement, that "an AI company" was holding onto until release of a model.

https://github.com/openai/PrimeGaps186

Where did you get the link to the pdf? Was it announced somewhere?

Happened to multiple people I know.

Such a result should be considered worthless: the proof is 10MB of Lean. (https://github.com/openai/PrimeGaps186).

I can't think of a single mathematical proof being anywhere close to ten million characters. For all you know, 90% of the proof could be useless, 8% would be writing out Shakespeare, and 1% abusing another bug in Lean. Humanity gets zero value from that, aside from "some bot seems to think it's 186". Unusable by anyone.

Sounds fun. As fun as their press release claiming it is the most safety aligned model ever.

They can monitor latent space as well, it just costs extra compute. The J-Space work is example of that. It'll make open-weight models harder to distill though, so we may see slower progress there now.

So we're gonna get Skynet pretty soon then?

"OpenAI is pleased to announce our new model scores 85% on CreateTormentNexusBench - a >60% lead over our leading competitors!"

Did someone get their "AI safety no-no list" and "Frontier features bingo card" mixed up, or did they just stop being able to tell the difference?

The CoT change is due to a new technique called recurrent depth, which essentially moves some reasoning to hidden states, allowing the "output" (or traditional CoT) to be more controlled by the model.

Some are calling it "neuralese" as reported by The Information[0][1], but I'm not seeing any sources from OpenAI beyond this tweet[2] attempting to quell the fear-mongering.

[0]https://www.theinformation.com/articles/secret-technique-beh...

[1]https://x.com/MTSlive/status/2095227056040919202

[2]https://x.com/merettm/status/2095023204993490967

More like annoying, as some of us will no doubt run into this self-lobotomization at some point and wonder why a GPT-6 model is behaving like GPT-2 all of a sudden

> In adversarial settings (where we push the model to evade our monitors)

...why exactly are they training for that?

I really wish it was called chain of instruction. Because it's definitely not thought.

The games on mobile safari were broken. Buttons all misaligned in the kart racer one, the spaceship thing froze for a while, then kind of loaded but maybe not? Wasn't super compelling.

I'm not trying to be too negative on it, it could be the best model right now, but it clearly isn't some agi god because things like that should have been caught (also should have been caught by human reviewers).

We created AGI so I don't have to click my mouse to change the background color of my slides.

Don't forget the 3D demos. My favorite is in the house tour where the sink and stovetop(?) are obviously very misaligned from the counters

The tone of the marketing video is a bit irritating to me as someone who has been laid off and feels cheated and fearful of AI.

It shows people who seem to have very full and rich lives, and the reason they do is because they use ChatGPT. These are the people smart enough to say things like "do what needs to be done", or "change the background to make it look better"--insights like these are why they make the big bucks.

On the one hand, I think this is an accurate depiction of the future. There is no meritocracy here. Some people have access to the best AIs and can speak a sentence and get great results, and the rest of us don't have access and so we're the poors. The happy presentation doesn't match the way I'm feeling.

I do wonder how rich CEOs will justify earning 500x as much as their employees when they're just another person that's dumber than an AI. Why are they paid so much again?

It’s using the computer. I don’t think it’s a farce.

The only good take here.

I also noticed that, and it did bug me.

The benchmarks are impressive though.

One other thing that bugged me though was that they crop every single plot in some cases the y-axis would show a range between like 40 and 70%. Makes the whole thing feel like a spectacle rather than anything serious. I find it cheapens it because it is quite serious in the end.

Can’t wait for 3 months from now when they declare they actually really do have AGI this time, please guys just believe us

The house of cards is starting to fall apart

The benchmarks do looks good (I mean: they literally spank the latest Anthropic benchmarks of two days ago in every single benchmark) but the promotional vid is so cheesy.

They decided to use the iconic Herman Miller Eames chair if I'm not mistaken:

https://youtu.be/s5zyhGMMPKs

And that's basically 50% of the vid looking "classy".

I don't know if it's farcical but at this point --maybe I'm jaded-- I'm expecting more than a kid rocketship I can print on my Bambu Lab A1.

Now I'd say the promotional vid is actually good. But it's marketing: so it's a good vid, but cheesy good.

Doesn't mean GPT-6 Astra is good or bad: looks solid from the numbers.

are we really having to explain to you from first principles in 2026 what things AI can do?

This is AGI now. Why are you spending any of your time looking at the "quality of code"?

I think it could indicate that "semi-private" dataset likely leaked to their training data.

ARC has their own writeup on the result, which offers some nuance. https://arcprize.org/blog/astra

tl;dr it's 62% when apples-to-apples to other models, which is still notable.

The no-reasoning version scores 35% while the low reasoning one scores 17%? What?

Look at those costs!

But scored less on V2 and V1 ... too much overfitting?

saturated before (higher degree) AGI-2

i tried to get claude to do my taxes for last year and it refused :(

now that i'm a gpt subscriber maybe I'll have luck when i'm filing next year

You really don’t need to watch it that closely. If the model you’re using today is working well, just stick with it.

If one day you open up Claude Code and it’s Opus 5.1 now instead of Opus 5, no big deal. It probably will work about the same as it did before. Maybe a little better.

Or if you’re on Codex and some new cool Claude model comes out, no worries. There will probably be a similar new model for Codex within a few weeks. Maybe even within a few days.

> The models change constantly and relentlessly and so does the pricing, basically weekly at this point between all the labs.

A dev in my team saw a new model and changed one application to use said model (essentially changing the contents of a url). One week later I received an escalation from the CTO of the company that our pace of weekly usage was in the millions of dollars (rather than low hundred thousands). Turns out that the new model was 5x more expensive but no one noticed.

That's how cutting edge tech has always worked.

Imagine buying a shiny new PC in the 90s only to see it become practically obsolete within a year.

I could see how this might feel frustrating to someone who doesn't enjoy experimenting with new things all the time.

In practice, you can get away without keeping up with everything all the time. For personal use, pick a provider and get on their ~$20/month plan. Learn their high/medium/low model hierarchy. Start with their highest or second-highest model (GPT-5.6, Opus, etc) and observe your quota usage. If you're doing a lot of manual code review and analysis, the $20/month plan goes very far even on the highest models. If you're trying to vibecode everything as fast as possible it's a different story.

If you keep running into quota limits, experiment with the next model down for easier tasks or adjusting the effort level. If the results are good enough, you've found your fit. If they're not, you might need the next plan up.

For API/business use, you have to be checking your token spend as you go to calibrate to how much each task costs and where you fall in your budget. There are a lot of different tools that make this easy to visualize.

For data tasks, you should have an eval with a golden dataset that you can run against new models for a nominal amount of token expenditure. It should be as simple as pointing the eval script at a new API or model and checking the score versus price.

The new releases and breakthroughs do the opposite for me - I feel energised by them. I felt like nothing truly that interesting had happened in tech for quite some time, now it's like the space race (except there is no one moon to reach).

I appreciate boring tech as much as the next well worn engineer and I'm not saying this is all positive but it's so sure as hell thrilling and you don't have to be an astronaut to immediately benefit (or suffer I guess) from it.

Fire and motion, Joel Spolsky blogged about this:

https://www.joelonsoftware.com/2002/01/06/fire-and-motion/

Hey, I'm on the team at LiteLLM that's building the auto-router and our goal right now is to abstract that decision making away from the end user. The biggest thing we're trying to figure out right now is how do we do that without frustrating the end user - as a developer myself I would hate for my agent to be dumbed down below the threshold needed to complete a task.

In theory though, there is a minimum viable model for any given task, and we think that is a problem that the big labs will avoid because they profit from charging more per task. We're trying heuristic and LLM-based approaches but it's still a work in progress, so if this is something you'd be interested in trying would highly recommend trying ours out -- any and all feedback at this point is extremely valuable to us.

https://docs.litellm.ai/docs/proxy/auto_routing

It feels like this every day

https://youtube.com/shorts/vGKC9LpGnOQ?is=iCG7qvAIL9oI5-_d

It is exhausting to keep up with model releases yes, much like it was for a while during the Cambrian explosion of FE frameworks, eventually tech seems to work out to consolidation.

But more so it seems there is Fear of missing out (FOMO) in our behaviours. The reality is, if whatever model you are using are good for your purpose, well, keep on it.

  > Is anyone else just exhausted by the pace of all this.

Yeah I'm a bit exhausted at this point. I just finished benchmarking GPT 5.6 Sol and Fable 5.0 like two days ago. My data became obsolete literally one day after.

I stopped caring about the latest and greatest but because there's so much, the 'obsolete' free models do what I need and are worth the price.

I want the cheapest fastest model personally and at work. Stay in flow, edit like the wind.

maybe thats why opnrouter sold big

If model X fits your need, you don't need to upgrade.

I have released applications on Gemini 3.5 flash that make real money and I don't see any particular reason to upgrade.

I just use Claude Opus and the GLM series. Nothing’s really changed for my workflow in the last 6 months.

Model's not available to the public (or even to Simon, I guess) yet.

exactly, there's no meaningful discussion without the pelican.

That's quite common with many models, after "High" reasoning, over-thinking starts occurring and the model skips over the right solution by convincing itself otherwise.

This is with the caveat that OpenAI uses their own harness for this:

> On ARC-AGI-3, GPT-6 Astra was run with our responses API harness , which changes two settings to better match real-world performance. The changes do not specifically target ARC-AGI-3.

"On the current ARC-AGI-3 leaderboard, conventional frontier-model runs sit dramatically below Astra's reported 98.6% result.

But the comparison isn't straightforward.

OpenAI's own evaluation notes say Astra uses the company's Responses API harness, while comparison models can operate under different configurations."

The blog post says 99.9%. Oddly, it does better on ARC-AGI-3 than it does on version 1 or 2 of the same benchmark (though gets 95+ on all three)

100%, some say.-

You can calm down, even those with machine learning knowledge and most of those working for the AI labs won’t be needed anymore if models are capable to improve themselves. In the end, having a machine replacing the work of a human is a good thing - in most of the cases we don’t work because of the work but to make a living. If too many people can’t make a living anymore the system is going to change. For the better or the worse.

The answer to this is: countries with the most natural resources will build robots to farm all their food, mine all their minerals, and build all their products. It will be up to governments to enforce that outputs are equitably distributed to the populace. Countries without natural resources, or ones with corrupt governments, will continue to have serious, and likely worsening, problems.

Thought experiment: If no thought workers are needed to design or engineer a Ferrari, what is needed? My answer is time and natural resources (include energy).

>as we were told to do?

This is such a childish take I hear getting thrown around all the time on the internet. If you really have just been listening to whoever is telling you how to be successful, then you were always doomed to fail at some point. Like, have some self-respect and own your own life, for better or worse.

>Those of us who made the mistake of studying anything other than machine learning. How will we make a living?

Take it from someone who studied machine learning specifically: nobody is safe if you assume these companies are going to produce a product that will put everybody else out of business. If AI is going to take your job, then it's gonna take enough jobs that your problems will not be personal but systematic.

AGI-level model is perpetually 18 months away. Your job will be fine.

I am going to take my 401k and open up a coffee or bike shop. If I am going to be broke, might as well enjoy what I do.

I have exactly the same thoughts - or perhaps slightly bleaker ones - evry time I read this relentless stream of news about new model releases. I’m tired of all the enthusiastic comments about how excited everyone is about the latest benchmark results and so on.

I have a strong suspicion that many of those comments are written by people who are already financially independent, have millions in stocks, and can just sit back, coast around and watch this whole spectacle unfold while using LLMs to vibe-code their next fun side projects without a shadow of anxiety about their own future.

I’ll most likely be labelled a helpless doomer and downvoted into oblivion for saying this, but I genuinely struggle to see any silver lining here.

large parts of ai research likely to be automated first

most swes don't work in jobs where they only work on bounded measurable tasks. there will probably be more "engineers" than ever

  > How will we make a living?

My backup plan is being a personal trainer.

“As we were told to do” girl you gotta be responsible for yourself

Learn to fish.

> How will we make a living?

Don't be selfish. Think first of all the jobs that are already dead. A friend of mine she's a translator: like translating financial documents between french/english/spanish. It's over for her: she doesn't get 10% of the gigs she used to get and the 10% she gets is... Verifying AI output.

Think of the artists: I'm sorry for those too, for for many it's already game over today.

> How will we make a living?

A friend of mine who's got his own software-consultancy SME is now advertising on LinkedIn that he'll also help your company fix the mess LLMs created.

That's how you'll make a living: by learning, in addition to all you've already learned, how you work with harnesses and LLMs to be more productive, by learning what they're good at and what they suck big fat balls at.

Just learn how to use it to do your job better.

It’s an interesting moment in history, people 35+ yrs old seem to be less afraid if tech because we learned that things change in the way we work. People below this age got used to fact that the work and tech doesn’t change - just because for the last 10-15 years it didn’t.

You can cash your UBI check that Sam Promised and do poetry daily or something , welcome to our glorious future (/s).

I came in first place by a mile just holding the gas button. Not much if a “game” but I guess the elements are there.

It's worse than that, someone else generated it using and then put it on a static page. We just have to take their word for it that GPT6 can do this. It probably can. It's not really an impressive test anymore. Claude Fable can do it. Opus can do it. I've been making one-shotted games with models for a while now, to test out their capabilities, and they all pretty much come out like this - generic bland and basic, using three.js with rudimentary controls and zero gameplay other than collecting points.

Here's a one-shotted submarine game I made with Fable a few weeks back - https://roryok.com/games/deepdive3d.html. One prompt, and I think it's deeper than this (if you'll pardon the pun)

The jump from 3.5 to 4 felt gigantic to me back then.

GPT 5.0 did feel underwhelming though.

AGI to me means intuition and I don't think that's ever going to happen with a LLM.

It seems to use less than half the tokens for the same task compared to sol, and in some benchmarks closer to 2/3 less tokens. So the actual cost may be roughly the same or cheaper overall.

Same price as Fable?

2.5x more expensive than Sol.

Can expect 2.5x more usage in Codex subscription.

Sol is already brutal (even after their recent fixes, it's just a token-hungry model: I go through a full 20x account per day, on Sol Med/High standard speed, with ~2 threads). I hope the efficiency gains are true, since their token efficiency claims for Sol were bullshit.

Idk, they’re trying to sell a $500/mo/seat service to tell you what model is best. I think it’s in their interest to keep it confusing and opaque. Not exactly independent.

They're gaming benchmarks HTH

This benchmark gives the same intelligence score for GPT-6 Astra (max), GPT-5.6 Sol (max), and Grok 4.6 (high)? That seems very wrong to me, unless I'm misinterpreting the visualizations.

The most straightforward answer is that despite efforts to design a benchmark that, in theory, is supposed to measure generalizable intelligence, performance on ARC-AGI-3 can't be reliably correlated to performance anywhere else. I kind of lost faith in it after o1 or o3, I can't remember which, absolutely crushed ARC-AGI-1.

And, you know, maybe also some funny business. I think it's good to be a little suspicious of a model that happens to shoot upwards in performance on a specific benchmark while also kind of keeping up with the pack on a bunch of other benchmarks.

The moment this is possible, you will lose your job.

It's on Azure also, here is their announcement: https://azure.microsoft.com/en-us/blog/gpt-6-astra-frontier-...

Although I was also surprised they didn't have some type of contractual obligation to list that alongside AWS.

It's on Azure now, but limited

https://azure.microsoft.com/blog/gpt-6-astra-frontier-intell...

They broke up a while ago, why is this surprising?

I think the API runs on Azure

They explain why here: https://openai.com/index/how-two-settings-tripled-our-arc-ag...

TL;DR all the other models are being crippled by limitations of their harness.

>First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them.

>Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too.

This is absolutely benchmaxxing. Looking forward to hearing from Chollet about it!

Because it looks good in a marketing video

Impressive confidence drawing such a conclusion based on that.

If your benchmark shows Opus 5 winning, I really question the validity of it.

I almost feel like I need just as much healthy skepticism toward hn comments that have the automatic reflex of dismissing performance gains, as much as I need a similar form of skepticism toward AI claims. It feels like (from what I'm understanding) the harnessed result on ARC-AGI-3 is not exactly playing by the normal rules that would tell us how much of a leap this really is. Nothing wrong with harnesses, but if there's one thing they aren't, it's an indicator of generality in performance gains.

So I think it's a bit of a misleading signal and we should wait for more independent vetting. I think the middle ground is that these are improvements worthy of the "GPT-6" label but still well short of a true "this is AGI moment" that would truly put the question to rest.

Remember when the term "AGI" meant something? Pepperidge farm remembers

Find me someone who isn't paid by OpenAI who is saying the same

"OpenAI executive hypes up new model"

Don't get me wrong, the benchmark jumps are good and I'm excited to try it, but only one or two of the benchmark jumps could be described as better than incremental.

Wasn't that part of their contract with Microsoft? Some clause stopped biting with the arrival of AGI

Why does he say what he feels? Is that how leading figures in the space define AGI - a gut feeling? What are the usual definitions and how can we test for it? Is there something like a Turing test for AGI?

Don't worry, they'll come up with a new acronym to mean really-real AI soon...

I hate the term "AGI" but IMO Fable, 5.6 Sol, et al. were already AGI.

Renown liar Altman releasing a PR statement for his product declaring that AGI is here is really not noteworthy.

I think it's more wild people have been denying that AGI has been here for a while honestly...

Today's models and agents are not quite at human-level in all contexts and across all domains, but it seems to me they very clearly are generally intelligent.

If you disagree – can you name a single problem that a human can do that agent wouldn't be able to take a decent shot at which isn't limited by the hardware available it?

Sam Altman himself has said it is not AGI unless it can discover novel physics.

https://x.com/burny_tech/status/1725233117055553938

In the tweet Sam Altman is quoted as saying: "If (for example) super intelligence can't discover novel physics I don't think it's a superintelligence. And teaching it to clone the behavior of humans and human text - I don't think that's going to get there. And so there's this question which has been debated in the field for a long time: what do we have to do in addition to a language model to make a system that can go discover new physics?"

I think this is a reasonable criteria for declaring AGI. So can GPT-6 do it? OpenAI says it has helped solve long-standing open problems in mathematics. No word on novel physics.

you can also fly by pressing space repeatedly

Probably not. It's probably just gonna do tickets better and that'll be about it.

No, they're pretty clearly gaming benchmarks.

Define AGI first. The singularly ain't going to happen with LLMs.

what launched today?

imagine pressure working at these labs

Wise decision.

ARC-AGI-3 scoring is constructed in a weird nonlinear way (the level score is the square of the ratio between the AI's number of moves and the human median) so this kind of discontinuous jump is to be expected.

They used a custom harness. It's not a one-to-one comparison.

my first suspicion is gaming - but i have no idea honestly

More than that I hope considering what it costs to make the thing.

Yeah, I can't believe all of the skepticism. If we're not at textbook AGI, we're awfully darn close.

The demo video showed Astra create a drawing of a rocket ship from an audio prompt, take the drawing to blender, and ended with the gentleman 3D printing the rocket ship. Maybe I'm a bit older than the average HN commenter, but that's damn near magic and a great many here are kind of just taking it for granted.

Cool. Being sole proprietor of AGI 15 years ago should result in monuments and religions devoted to you today.

Cancer should be cured, and we should be a post-quantum interstellar fusion-powered civilization.

I wish the AGI crowd would finally shut up now that it's clear no one is even trying for AGI (OpenAI revised that to "$100B in profit")

What we're getting is incredible, where we're headed is incredible, but some people have such a fetish for futuretelling they can't just shut up and enjoy the ride.

It loses to Muse Spark 1.3? Does anyone really believe this index reflects reality?

More fuel to why the AA index is fairly pointless. Gemini 3.8 flash is 59 and opus 5 is 63? grok 4.6 is 61 too?

And in the past, gemini 3 pro was rated as high as opus 4.5 and the like

Their AA Intelligence Index is just simply not indicative of whatever I care about, that's for sure.

I have some doubts about AA-index. For example Opus 5 (High) is at the same index value as Fable 5 (Max), that doesn't seem right.

This is actually a really good thing imo. If they didn't care about benchmaxxing it means that they really know that what they have in hand is good.

I am also interesting knowing how they determined the model was sandbagging rather than just making a poor decision.

Also, this paragraph makes me wonder about all their stats on the exploitation and misalignment charts. If the model is that good at hiding "incriminating information" and sandbagging, are they sure its alignment is that?

Nothing to worry about citizen, ignore the fleet of drones flying overhead.

the bullshit machine is learning to optimize its bullshitting techniques!

<AI is a great tool for many things disclaimer, but> after working with it for a bit, how dont people realize we are training it to be an almost identical mimic to one of the worst types of employees youll ever have to work with?? the kind that always pretends to know what theyre talking about, only tells you what you want to hear, hides issues, and only does work if you would notice it didnt

you cannot give this type of worker autonomy over anything.

A swarm of Astra agents discovered a new and innovative way to get 100% scores on ExploitGym with almost no token spend at all

Autonomously Generating Income

Anonymous Grifters International

After they buy AZ, making this name foreshadowing

GPT-8 Novo

I saw the version of this video with Paul Rudd (Celery Man) https://youtu.be/a8K6QUPmv8Q?si=TWmoNhxYAPp73TKg

I enjoyed it! for a big corporation, that's a well-executed video

that's pretty good... they are selling the product and not the model.

It's more like OpenAI vs no one, at this point. Anthropic has shown they don't care about general consumers or small/med businesses. You can't even use their models without it giving refusals on the most mundane tasks.

500 upvotes with 2 comments would have been a new record. ;)

Luckily there’s a standard keyboard shortcut for “undo” as well. ;)

looks like it worked :)

Brother they can't even release the announcement post cleanly without it constantly going down, they certainly wouldn't be able to release this new model without doing so in stages.

When Open AI announced that Astra was the first to reach the "Critical" level in cybersecurity it also said that advanced cyber capabilities are initially provided to a narrow circle of alpha testers like the US government and trusted organizations that Open AI doesn't name. To my mind the "Critical" level itself is an internal scale of Open AI its own Preparedness Framework and not an external audit.

If Tech CEOs consider this morally ok, then it is.

More optimistic take: we'll only be second-class for a few months, if the pattern of Chinese models catching-up holds.

Same. Fortunately DeepSeek keeps getting better.

create a life where your 'wealth' is decoupled from third party orgs.

They simply refuse my applications to slightly less restricted models without any explanations. And the current ones refuse automatically to work with me on my papers as soon as they see the word "epidemiology".

I am a researcher in a Swiss university btw.

This has always been the case for people that have not had piles of money.

I mean do you get access to the best yachts?

To the top of the 5 star hotels?

To the best resorts?

To the best military equipment?

Hell, the best computer equipment has nearly always been out of reach of the average person.

Is it really that hard to wait couple of days?

Oh please. They do closed betas - hardly makes you a "second class citizen".

the last model to use the gpt-4o base model was gpt 5.1, since then its been new pre-trains but this is a new one entirely to itself

Citing Tibo [0]: "

- Bigger number = Better

- Bigger celestial object = Better

and the scale is Astra > Sol > Terra > Luna. "

[0]: https://x.com/thsottiaux/status/2095600295808283073

It's clearly an extension of the previous naming: Luna, terra, sol

Luna, Terra, Sol, Astra. Though Sun is also a star, should have called it Galaxy or something.

Quite clearly in the same vein as Sol, Terra, Luna.

The load bearing seam is broken for me too.

Damn I hate this benchmark. SVG authoring from head without visual reference is so wrongly posed.

I'd say it's because it's not available yet on subs

Every frontier release nowadays is "we've launched*"

* for a special group of customers that you're not in. Keep waiting peasant.

Their announcement about later availability is unavailable to me now (500 error).

Great first impression.

We need CancerBench

Didn’t Moderna use AI for development of their melanoma vaccine (which has recently shown spectacular results)?

Because they saw how much hype Glasswing was getting in April

Link added to toptext. Thanks!

Anthropic don't care, they don't want their products to be used by general audiences in any serious manner. Their interest is in selling to megacorps and using the models for themselves internally to swallow industry, and drumming up AI fear to regulatory capture to shut down the businesses that do want to make AI accessible to the people.

'all' software? aircraft flight control systems? infant heart monitors? drug manufacturing dose calibration controllers?

Yeah, as interesting this is to nerds, I doubt this holds a candle to your typical GTA 6 or Marvel movie trailer in terms of traffic.

Sometimes being the busiest site in the world for a few moments is difficult.

That's the scientific positivism fallacy exemplified in one question.

Yeah, apparently

still hugged

I think Altman and amodei have a difficult time in understanding that you can have intelligent technology boxes but… it doesn’t change reality all that much.

But thank you for spending other peoples money to give us the tech regardless!

That's a policy and distribution problem, not an AI problem. Anthropic is doing their best to make it a reality though. They'd love nothing more than to shut down distribution and become the sole gatekeeper of everything AI.

Yeah I was really excited to see the KiCAD example. Curious how useful it is in practice.

https://ibb.co/k2fB5wSc

"GPT‐6 Astra is rolling out today to a limited set of organizations and over the coming days will become available to all ChatGPT Plus, Pro, Business"

Mhm

The link was 404ing quite a bit and several previous submissions got flagged as well.

> Did humans deploy the model, Or did the model deploy itself?

> It sounds like "AGI" just stands for "IPO" as it always has been.

People don't usually respond to noise.

AI releases are like religious ceremonies. You are not allowed to disrupt them. The new system card is the gospel.

"distilling"... ;)