Hacker News

I should have been more clear. While distillation is part of how we got lucky here, what I really think is that it's just surprising and lucky that such a heavily data-driven approach ended up being so powerful here.

Transformers are like just a step or two removed from being fancy convolutional neural networks. I guess I'm just surprised that it didn't turn out to require more 'special sauce' with extremely elaborate internal architectures, and less of a big-data approach.

Because the data is so central in building these LLMs, rather than some special insights or ideas in the model architecture, or very special hardware requirements, the field is much more open than I would have guessed some years ago. And it's the fact that the data is so central that makes distillation possible in the first place.

They'd still have strong models without distillation, but strong enough to challenge frontier models and to claim the meaningful market share that they have? Probably not.

China is still quite far from monopolizing hardware production

Well aside from quality it kind of seems like those other things are getting skirted by. We are very much in a phase of let's see if this is possible and exploit it rather than should we actually be doing this

And I'm saying this as somebody that's made millions selling AI software in the last few years...

Im sure that's well on its way.

I guess it’s more like asking the paid genie to give you a new cheaper genie.

And for a lot of non-frontier purposes these days, you can bootstrap via LLM-as-judge so your hyperspecific wakeword model or whatever can be trained with little to no human input, that aspect of it is fully terrific.

The frontier models are a replicator that can give you another replicator which specifically produces tea, earl grey, hot, when you push the single button, and does nothing else.

Yes, but 10 or 15 years ago, I would have thought that there'd be more to it than just a slight modification on the ideas behind a CNN to get this level of AI.

There were somewhat good reasons to think it needed more than just this data-driven ML approach.

Distillation a pretty well documented technique that actually pre-dates LLMs https://arxiv.org/pdf/1503.02531

Here is a project that guides you through it if you want to prove to yourself that it works https://github.com/arcee-ai/DistillKit

It turns out you can train a 1b model at almost 1000 tokens/s on a m5 max laptop. As a personal experiment, I've been asking Sol for synthetic training data and synthetic agentic training data (model distillation in it's purest form), plus modified opencode, codex transcripts etc for training data, and nobody's even paying me to do it. If I'm doing it has a hobby, you can bet industrial users are doing it.

Been using a lot of Kimi K3 lately and the answers have been… „load-bearing“ to the point of hilariousness. It‘s obvious from where they distilled, even if sceptics rightly point out it can‘t have been the only source of their secret sauce, as it‘s been better than the current Opus 4.x at the time of release.

Why does Kimi insist its name is Claude?

Musk confirmed in federal court that xAI does it: https://techcrunch.com/2026/04/30/elon-musk-testifies-that-x...

It's also how providers build their smaller models out of their larger ones; they publicly talk about the process.

The evidence is Anthropic's own reporting [1]. You may doubt that they're telling the truth, but that's what they're reporting.

[1] https://www.anthropic.com/news/detecting-and-preventing-dist...

Here's an example: https://github.com/microsoft/Build25-LAB329

there is no evidence. it shortcuts post training by a huge margin this is true. but that is all.

Funny you mention Chinese Room and LLMs in the same response, I would say LLMs proved Searle wrong, agents now make cutting edge discoveries and meaningful problem solving. They not lookup tables though and you need to pay for inference, so the intuition of syntax doing the work of semantics without understanding was wrong.

Even if it were possible it wouldn't change the outcome. China is capable of training frontier models even without distillation. Distillation is only an accelerant.

The primary resource you need to train LLMs is money and China has plenty of that.

The frontier labs have competing goals in mind. They want high growth (which means little friction for account creation), API access (because enterprise money is the best money to have), and distillation protection.

Besides, identity verification that actually works at scale is a much harder problem than identity verification which is good enough to satisfy your compliance people and regulators. Especially if the fraudsters have a major world government standing behind them, and if their aim is to be identified as a real customer, not one customer in particular.

At the end of the day, while you can do your best to obfuscate your reasoning tokens, it's a losing battle to hide actual user-visible output tokens. The very nature of API offerings is that you can't do KYC on where that API's output is going - there's a rich secondary market that's not going away.

And with the sheer volume of data created from that, coupled with benign-seeming prompts like "plan out your reasoning in a document before implementing" that could never be patched without breaking existing customer workflows... there's more than enough for someone to distill on. Even if that only gets them to not-quite-frontier, if you're pushing the frontier every few months, they're only ever a few months behind you.

The nature of LLMs is that you give them input, they give you output. That allows for distillation. They do try to make it harder by hiding the chain of thought stuff, but fundamentally, if your model is publicly available, its distillable.

> Hmm, don't people think that if the frontier labs really put enough engineering effort into preventing distillation that they would be able to do that, or at least diminish it significantly?

Distillation was big news a year or even 6 months ago, but as far as we can tell it's not really a moat anymore. Now that multiple players have trillion+ parameter models and the capacity to post-train them, there's no putting the genie back in the lamp.

Well, do you think perfect copy protection is possible?

Same problem: if humans can see the output, it can be copied.

Only the Chinese authorities can stop Chinese labs from distilling from western labs. And they won’t do that, for obvious reasons.

i trained another AI on all my codex logs... it's pretty good actually

> OpenAI could still have a significant moat. ChatGPT occupies most consumers’ minds when they think about AI and has become a household name.

ChatGPT is AI for the average non-techie the world over, but the average non-techie isn't eager to pay for it. The more progress that's made, the less incentive to pay - most people are happy with the total garbage spewed by google AI overview. They'd be happy with google's 30b MoE gemma, whose performance will likely be squeezed down to something that can run on a phone in 2-3 years. Why would they pay $20 a month?

It's why OpenAI is pushing a variety of things such as ads and offer a more polished ui/ux than the competition, I think. The models are already good enough for people who just want to know how much sugar to add to their cake or when's the next basketball match their team plays - it's OpenAI's game to lose those people, by annoying UX and whatnot. If they can make a few bucks off of every one of their non-paying users it'll stretch their runway immensely. Those users will never go to Antrophic or some cheap Chinese model, but they might defect to Google because a popup on Android / in Chrome told them to.

Difference is that it was free to google/bing search. Ai prompting costs money.

If I run out of tokens on ChatGPT of course I will try Claude. I never ran out of Google searches so no reason to try Bing

Mass adoption isn’t where the money is, though. It doesn’t matter if ChatGPT has 1 billion users if they won’t pay for it.

> If OpenAI and Anthropic are smart, they will maintain similar pricing rather than aggressively undercutting each other, allowing the market to resemble Home Depot and Lowe’s, or cloud computing, where AWS, Google Cloud, and Azure coexist as highly profitable competitors.

Wouldn't that just be price fixing? If they arrive at their prices independently and they all happen to be similar, fine. But if they're all "smart" and coordinate so none of them undercuts the other, that's probably illegal.

Google also spent heavily setting up deals with other platform owners, invested in Chrome and Android to establish Google search as the default option which most users accepted. If integrations of good enough AI features are made within existing platforms most users will probably accept using those and not think too much about whose model is powering it.

Never in my life have I bought any of the household names when it comes to household equipments. Of course, I use the household names’ names when I speak about wipes, washing powder, glue, etc., but never ever would I pay the 2x to 5x premium. Same will be true for most people with regard to “intelligence”, as long as it’s suffices their needs. Thats the crucial part. Cloud computing, on the other hand, has a massive lock-in. But more importantly, their ideosyncratic ways of doing things leads to tribalism. It’s so much harder to move running processes from one provider to another without interruption than just changing an API key or the URL for your chatbot.

Their current strategy is "make the product affordable in tiny little bursts here and there." That is not a great way to build a moat.

That's just the first mover moat, OP was worried about intelligence itself being monopolized.

Because it breaks the TOS.

Even if the transformer paper wasnt published, the info would have diffused out eventually.

Its not that wild of an idea. It's not like e.g. chipmaking where even just knowing how things are done doesnt mean you can copy it.

I hope it's a good bit cheaper than that, I pay close to $400/mo for electricity and I'm in no way interested in paying anything like that for AI.

He also wanted to do a non-profit.

He even raised money on that premise.

He is a pathological liar, so is Dario. Don’t rely on the benevolence or truthfulness of these people.

They will say whatever is beneficial to say in the moment.

Altman of *Open* AI? No idea why I would trust him without very convincing proof.

And that it will be just another utility, like electricity.

Yeah, he sure does lie about a variety of things! He doesn't have the name Scam Altman for nothing.

Lol, of course what he left out is this will happen by inflating the cost of electricity rather than driving down the cost of AI.

"Who knows" is the right answer, I think.

If (a) demand for AI continues to increase, and (b) SpaceX can get to ~$100/kg to orbit, then they will have a ridiculously deep moat. Probably more like 10 years, though.

But as you said, who knows.

I'd take a market crash over a monopoly in the hands of a ghoul like Altman.

The economy he and his ilk want to build is infinitely worse.

This is funny because the stock Market has been ahistorically high. My portfolio went up over 20 percent in the last 12 months.

A major correction would be a bummer but we were never entitled to these abnormal gains in the first place.

The market (s&p500) crashing 40% puts us at levels we haven't seen since 2024, well into the creation of LLMs. Probably a worthwhile trade if it was either/or!

ChatGPT subscriptions do the same weekly allowance, which the discounts apply to.

Nemotron Super is sort of open source in the sense that Nvidia provides almost everything you need to replicate it from scratch. Of course it’s performance is not exactly stellar but it could be a good starting point for other research teams.

This viewpoint doesn't make any sense to me. The weights + inference code are the "source code" for AI. I literally don't know what else you are demanding for the "open source" label.

Was about to object, then realized you linked Allen ai and had the useful caveat. For what it's worth the OpenMDW license, which Nemotron and a few others have adopted, does say model weight.

That said, I've noticed the training procedures and corpus size of more useful open/available weights models are settling down more than I expected. Wonder if crowd sourcing good training data, even if it's just expensive model coding session transcripts, has potential to level the landscape some.

Can you explain what's missing for the "open source" label that open-weight models like DeepSeek/Quen/GLM/etc don't release?

Is it just the supplementary data/code for how they were trained, not just the final product?

There is also https://apertus-ai.org/ but yeah, "not useful yet" if you were looking to replace your coding agent. Very useful if you are doing LLM research.

I really like molmo 2

It's common for different models to find holes in another's work. There are various good reasons for that.

FWIW, we use ChatGPT for our primary model and use Claude to do the reviews. This works better than ChatGPT doing it's own review even with a clean session/context.

Did you try to say "think more deeply about this problem" to fable after getting your solution, having one model focused on creation then blaming it for not doing proper review when the other model was told to focus sol-ely (pun intended) on review is not a fair apples to apples compaision

> Just the other day I had Fable draw up a solution for me, and then I fed it into 5.6-Sol and said how does this look…

You should be doing this for every solution.

Even Fable reviewing itself will find issues, unproven assertions, etc. Same for Codex models. A review loop is critical.

The fair comparison would be to also do the reverse: start with Sol then have Fable clean up. Then compare the Fable-Sol and Sol-Fable outputs side by side.

I used to review each others work, Sol is amazing at review and finding what’s missing.

It's a funny comparison. Comparing the empathy of some software to the empathy of a company. It's like saying my car was more empathetic than my school. How can those two objects even be compared is what i am wondering

It varies, the researchers absolutely do have humanity’s needs in mind, it’s why they founded the companies and are doing work everyday. However the issue is that it’s got so much money involved that the heartless soulless billionaires are getting involved. I think the actual literal people doing the real work are doing it because AI could cure every cancer and every disease, make us a multi-planetary species, outlast humans by millions and millions of years, potentially create actual organic life and make direct upgrades to humans.

I think that AI has an insane level of upside, it’s just that the greedy dumbfuck billionaires are getting their greedy little grubby paws involved. If left to researchers I think the sky is the limit, but unfortunately thy need assets, so there isn’t a clean solution to that.

Ideally, we could somehow separate AI from funding from malicious entities like billionaires, but right now that doesn’t seem possible. Hopefully in the future researchers with genuinely good intentions can have far more direct control than dumbass greedy old fucks, but we’ll just have to see.

I think the future can be bright in theory but we’ll have to see, making insanely powerful open-source models is the direct way to get around the billionaires so I think that’s our only option. Make open-source ASI you can run on a consumer computer.

Fable is available for $100 a month. If you're a working developer, you can pay that. I wouldn't really say it's "reserved for the richest customers".

Fable is indeed larger than Sol. OpenAI is developing Astra which will be more of a Fable-sized model.

If you can train a larger model then you can distill smaller models from it. You don't need to necessarily serve the larger model publicly. Distillation is much more effective when you have unrestricted access to the original model.

I don’t think anyone said anything about either being less evil? Just having more consumer oriented products..

The naming schema actually tripped me up for a week or so.

My brain's initial conception of the concepts was earth-relative, so I mapped it as:

Sol = big, it's the sun Luna = medium, in-between sun and earth, space Terra = small, terrestrial

A simpler explanation is that it's just a better naming system.

Calling something "small" might make it sound inferior to competitors. And S/M/L gets awkward as soon as you have more than three sizes.

This naming system can get near-infinitely bigger or smaller.

Yes but with gemini specifically they said that pro was still in training. And the comparison isn't really atrocious unless Gemini 3.5 Pro is worse than Gemini 3.5 flash

It's possible to drive them hard enough to really read everything, but you have to go really hard on them with many many cycles.

Maybe you missed deepseek v4 Pro. Try it in minimal mode, where it gets up to around Kimi or Opus level.

It still occupies a place on the Pareto frontier.

I'd be very surprised if that were the case, unless they have severely devalued how much "100% usage" is worth – which, as I understand, they could at any point, given that they don't publicly specify how many tokens (or at least "credits" [1]) are included in each plan per month.

It really reminds me of pay-to-win games at this point: Two currencies (credits, tokens), both with a floating, intransparent exchange rate between each other and real money, random airdrops...

[1] https://help.openai.com/en/articles/12642688-using-credits-f...

What about companies purchasing enterprise contracts? Most contracts are minimum 12 months. At a minimum, to secure enteprise requirements like zero-data retention, you'll need to lock into a single provider.

These price reductions are mostly targeted towards self-serve customers on individual or small team plans, where individual choice matters and the friction of changing models/providers is low.

Yes, they haven't advertised a pending price jump in the near future.

> Why do you think they want less people subscribing?

Losing money on each subscriber?

very easy to lose money on subscription, very easy to make money on api pricing

ChatGPT already notifies the authorities if it thinks you’re doing something illegal. Fable downgrades itself if it thinks you’re doing something even vaguely suspicious.

Above average levels of paranoia here, but one way you can prevent that is by not connecting the machine in question to the internet.

The company may be losing money but the people are getting enormously rich and cashing out religiously

Which is not that great for people using less than 50% every week, because the next reset date moves forward too. In essence, it is redistributing compute from people who haven't used their quota much to those who have.

Though I think they gave a banked reset this time.

Do they take them back? Codex a few weeks ago said I had 2 resets. I didn't use any, and now it doesn't say that.

Looks like you’ve confused clarification for begging

ChatGPT Work and Codex use that. Normal chat has a different, unspecified limit.

Because he is an egomaniac. It is clearly a low effort method for farming upvotes. Majority of the time he's clutching at straws to find "dupes" with 3 comments threads.

Read the site regularly. Saw it come up. Remembered it from earlier. 2 days old news now.