Hacker News

Long-horizon RL symptoms perhaps?

if you want to save some $ you can start with abliteration and after few turns switch to normal glm5.3 for example from z.ai subscription or other source. I didn't test it yet with deepseek, but I'm assuming it will will work the same.

those models will follow your request when they already agreed to do this

deepseek tends to quickly assume this is benchmark and even when have web access thinks things like "simulated internet" etc (but waste tokens on thinking if there could be way to cheat)

> should the model be opened it’s still not dealignable because it just doesn’t know enough

This is the only AI-safe approach - see the OpenAI agent swarm incident where the swarm engaged in offensive cyber because guardrails were not active for their task and the model had clearly been post-trained on that narrow domain.

That is a shout out to the Void Star book, https://www.theguardian.com/books/2017/apr/26/void-star-zach...

If you're an amazing, top 1% software engineer, and you switch jobs to become a penetration tester, you're not going to be top 1% day one. There's knowledge specific to penetration testing you'll need to research and absorb and practice before you're good.

For the models, it's always day one. They don't learn from experience. Whatever they know after training is all they'll ever know. So if you don't teach them penetration testing in training, they won't know it. Maybe they could theoretically derive everything from first principles, and so could a top 1% software engineer, but that's a very slow and inefficient way to do things.

I have essentially no knowledge in this field, but another concern may be that if the model has minimal cybersecurity knowledge, it may also implement insecure systems as a side effect.

> For a model it doesn’t seem that far fetched to make the leap from “I have the knowledge of what hacking is” and “I know how computer systems work” to just being able to exploit stuff anyways.

You'd be surprised. Cyber offense is a very narrow domain, especially given the extensive mitigations that are being used in modern systems. Leaving it out of post-training tasks unless you're specifically interested in it is the no-brainer approach.

Interestingly, the Kimi one seems to implement the safe spot bug from the arcade original [1], while the Claude one doesn’t.

The prompt says that the agent shouldn’t look up existing implementations on the internet, so I’m guessing that the implementation is baked in?

[1] https://strategywiki.org/wiki/Pac-Man/Tips

On mobile Kimi definitely wins from the UX point of view

Until I looked, I thought I'd see Kimi vs Claude competitive gaming with pacman.

That would have been interesting to watch, which model gets the highest score.

What's the best bang for the buck ≈$10–20 (per-month/equivalent) subscription for a good enough model these days?

I had the lowest plan ($20/month I think) and it literally gave me fifteen, maybe twenty minutes tops, using the cheaper K3-256K variant. Basically enough to fill the context about 1.2x and that was it.

It's a really smart model but way too expensive. The same $20 goes 100x further with Claude. I used it for the initial task planning while the plan didn't expire, then handed off to DeepSeek or Minimax.

That is surprising. I had a $200 sub when K3 came out and at the time there was no 5h limit, making the subscription really good for running 4-5 big tasks each month.

I’m not saying you’re wrong (I have no experience), but wouldn’t it be useful to test this model out for yourself before passing judgment on it?

> Which means that the US lead is all but gone.

You can see in this discussions the goalposts being actively moved to desperately create a notion that the lead is still there.

You even see people in this thread referring to imaginary wonder models that they expect US companies to have under development that are supposedly leaps and bounds more impressive, albeit the only reference there is to their existence is the commenters hopes and dreams.

I think people need to realize that there is no longer a cutting edge company that is expected to dominate, and among the multitude of AI companies in the field competition is so tight that no one would even be surprised if tomorrow some other company far away from the US was able to crank out the top performing model, or something like a small local model that outperforms mid/high tier frontier models. The local model alone would be a death sentence to US companies, who are still burning through 10x their revenue.

Ant/OpenAI both have unreleased models that are seemingly much more capable than what they offer today, so I'd say that's pretty unclear.

Could be part of the reason that the frontier models are getting the administration to go a little harder against Chinese models.

Perhaps you missed this part:

> Open-weight principles. These evaluations frame our open-weight release decision. First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro's offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights. We believe making K3.1 Pro available in a non-discriminatory way benefits people who need dual-use capabilities for defensive purposes. When a capable model is open and runs on infrastructure its users control, many more teams gain the means to audit the code they maintain — including legacy systems that rarely receive dedicated security reviews. We believe that letting this capability diffuse gradually — rather than concentrating it in a few proprietary systems — ultimately makes all software more secure.

I don't think Chinese labs are behind at all. I think the tell is that they keep releasing models 2 weeks after new American proprietary models that match or beat them. Suggesting they're sitting on much higher capabilities and releasing just enough to undercut

> When evaluating the Intelligence Index, it generated 150M tokens, which is very verbose in comparison to the median of 87M.

This is 100M fewer than Opus 5.5, 40M fewer than Fable 5.1, and about a third of Sonnet 5.5. In the top-6, it's 3rd, beaten by Muse Wonder at 140M and Astra at, impressively, just 60M.

Muse Wonder is even higher, and there is no way in hell it's better in practice than either Astra or Fable.

From that link:

> reasonably priced when comparing to other models of similar price

???

> beating both GPT-6 Astra and Fable 5.1. It's also faster than both.

Very similar to my adversarial programming language (KillSwitch) benchmark[1], at least on pass rate, it is much slower though.

1 - https://bench.killswitch-lang.org

I don't trust these Artificial Analysis indexes and I'm not sure why anyone considers them authoritative, the site just sprung up a few months ago and now everyone quotes it?

From my own use of the models I find their ranking to be totally inaccurate.

while this is an awesome model, I have a bridge to sell to folks who think (based on AA leaderboard) that this is better than Opus 5.5

> When evaluating the Intelligence Index, it generated 150M tokens, which is very verbose in comparison to the median of 87M.

So their base price is 1/5th of Opus 5.5 per token, but if you want "highspeed" it's 2x that and then I guess it generates 2x the number of tokens as well?

That might suggest that Opus is competitively priced after all.

Talking point #6 - https://www.reddit.com/r/singularity/comments/1xdquz5/kimi_a...

I mean... They just copy the US's homework, so I would hope they can run a leaner operation.

This is a price drop from Kimi K3 ($3/$15/$0.30) despite being a slightly larger model (3T vs. 2.8T params).

I'm glad I waited a day to post my analysis of LLM price cuts because it needs...an addition.

Same experience. I had the $19 monthly plan for experimenting and running benchmarks, but it was a uselessly small amount of usage, a single task would blast through the whole five hour limit and a big chunk of the weekly limit. All of the US provider plans are remarkably more generous (though OpenAI plans have gotten a lot stingier lately, where they used to be the most generous).

Welp, that ruins this news for me. I'm glad open models are catching up but if I can't practically run them locally _or_ through a flat fee then it doesn't really help me. I fear API pricing.

Had a similar experience here. Their limits are very, very low, and ultimately make it not worth subscribing. Can't leverage a good modal if it exhausts its 5-hour budget in 30 minutes.

Paying API prices for Deepseek 4.1 Flash feels like a much better deal at the moment.

How? I've been running k3.1 Pro with K2.8 preview for codegen (subagents) and I've only used 22% of my weekly and it's been running for 8 hours on a RE project.

Are you using k3 with thinking set to the absolute highest and only using k3 for everything?

We really need consumer hardware to catch up so we can run these models locally...

I have the same plan and have never experienced that with K3. I have hit the limit within a 5 hour quota using Swarm, however.

Yeah, the plan is far too expensive for what it provides. I wonder if the quota with these new models will be better, but I'll wait for others to try this out before wasting another hundred bucks.

It also has the lowest quota of any model on https://opencode.ai/go

I think Kimi just doesn't have the means to subsidize plans.

Ya it's not very usable compared to Claude / codex

Yup, insanely low 5-hour limits.

Honestly, use Pi, start the session with K3, design it, store the design + chat history + todo with phases + why + readme then when you are at the time to build do `/model` select deepseek and let it rip!!

What about via US hosted data center resellers like Fireworks?

It seems that GLM-5.3 is still the go-to for cybersecurity.

Also the speed

Why is that a talking point now, all of a sudden?

From my POV it looks like you guys tried "They're distilling us! Very immoral!", saw that and nobody cares, and now you're trying to run the more emotionally impactful "They're distilling us... with your grandma's stolen credit card! They're stealing from your grandma!!"

I imagine that Chinese AI labs would be more than happy to pay for tokens to train on. If Anthropic goes and bans them as quickly as it can identify to them, of course they'll shift to using lots of cheaply obtained sockpuppets to keep getting the same data.

I doubt that an AI could do anything with these that legions of scammers haven't already figured out how to do - after all, it's not like AI set up this distillation setup in the first place :)

Why do we believe all of this without a shred of actual proof. Breached computers and stolen cards is straight up crime, you'd think that there would be a lawsuit or two then.

Hardening against what? Apologies if I am being dense here.

The way I see it, automated systems should basicially have access to the financial system anyways. This is some of the appeal of cryptocurrency.

Sybil resistance built into the traditional financial system (for example firewalling based on residential IP, requiring banking apps, etc.) is a massive annoyance that can obviously by circumvented at scale.

Hardening in this case could mean biometrics, or it could mean rethinking the trust model so that controlling an army of duplicates confers no advantage because you have to be trusted in order to do any damage anyhow. Containing multitudes buys you nothing.

I hope we chose the latter. Biometrics are creepy.

Color me naive but I can’t believe that approach works at any reasonable scale. Any meaningful token spend at distillation scale would be raising alarm bells at all the cc issuers.

No offense but you don't know what you're talking about.

Spitballing random pr pieces from companies about to IPO is not helpful to anyone, fyi.

Probably something like:

MiniMax -> Qwen -> DeepSeek (pro) -> Gemini (Carbon) -> Fable (to beat Carbon) -> Astra (to be slightly worse than Fable but cheaper).

I doubt we'll see any major US release besides Grok in the next 3 weeks.

Would be nice if we got a Terra that's competitive with Gemini Flash and DeepSeek Flash.

The fun part is, we're already at the point where we are seeing something frontier-ish almost every single week.

The really fun part will be when they all start adding System 1-3 natively (IMO) - which I think will be the next release cycle.

> So they admit that they could do more, but they won't?

Yeah, there's another paragraph:

> Identifying and removing cyber-related data. During development, we observed vulnerability research performance improving steadily as post-training scaled. Tracing the source of this trend, we found that cyber-related content was embedded within the general mid-training corpus, and cyber-related tasks were present in the RL environment mix — a natural byproduct of training on broad data. In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness. We will continue to apply appropriate controls at every training stage, of which cyber-related data identification and restriction is one example.

(emphasis mine)

It's clear that cyber is a touchy subject right now. But it's also clear that RL works on it, because it's highly verifiable. So smaller models are going to become very interesting, because they can be fine-tuned later to cover this field.

With frontier competitors and the U.S. govt actively pushing "concerns" about Chinese models, Kimi including this statement on "offensive cyber capability" seems like smart defense. Models are opaque and training details are trade secrets, so whatever they did might be negligible or significant and no one can tell. This bit is also artfully done:

> "below what our pipeline can produce, and remains behind that of frontier proprietary models."

Like Ant and OAI, they've realized vaguely claiming to have nerfed their latest model to an unspecified degree not only looks 'good guy' prudent, it provides a built-in excuse when a model fails.

Or it's a cop out. I could totally outrun Usain Bolt, but by design I don't push to set new World Records. I will increasingly show faster times though.

They don't want to be banned, and maybe their best offensive cyber capability models will be for their govt.

We really need some breakthrough like DiLoCo does, but for heterogeneous unstable consumer hardware to make volunteer crowdsourced AI training a reality. Yeah, I understand from current technical angle how much of a mountain that is. Call it AI@Home or whatever. I never thought it would bother me personally or even become a problem, but these companies acting like gatekeepers and virtue signaling while also in bed with governments etc.. this leads nowhere good.

Yeah that's a huge downside. That the model does the task without tripping over its own feet, like discovering a vulnerability the model itself accidentally built 15 minutes ago and triggering its own guardrails, made the Chinese models attractive to use for security-adjacent things. Also useful to red-team your own stuff. While I can understand the reasoning behind it (and I guess having a model that's really good at both offense and defense released into the ether could lead to CCP asking some very pointed questions if Moonshot managed to get blamed for some incident stemming from their model's misuse) it's still a huge disappointment.

I don't want to give US labs the results of my DNA tests and my family tree going back to the 16th century just to secure my shitty app, but if the Chinese are bowing out as well it seems the only alternative is to admit that some script kiddie with a bunch of tokens will just pwn you now and then.

Isn't this done so they can defuse Anthropic's excuse for regulatory capture?

This doesn't seem particularly notable and is in line with Anthropic and OpenAI's approach of restricting release/access to more capable models.

Read this as ‘dear government we can train a cyberweapon if you pay us’ assuming they wouldn’t say anything if they were already paid to do that

I hope one day they learn to inhabit their position more confidently instead of merely aping their competitors.

They will. Just not in the version they release to you.

Moonshot (Kimi) seems to be unashamedly into American culture[0], I wonder if they're paying closer attention to and giving more credence to some cultural aspects of American AI discourse including aspects of AI caution/panic.

[0] https://x.com/crystalsssup/status/1944287779896328668

(note: extremely speculative and based on almost nothing, and especially near-complete ignorance to alternatives)

All it takes is one idiot who happens to have the resources to "pWm Teh Interwebs" to scare both the US and Chinese governments.

The more interesting question is how much more performance can be squeezed out by post training the open weights through RL.

On the surface that seems like a responsible thing to do

Oh great, more censorship. We wouldn’t want those pesky plebs securing their work.

Cowards.

The way to stop the vulnerability apocalypse is to fix the vulnerabilities, not to shoot the damn messengers.

I hope someone abliterates and FTs this model on vulnerability discovery so we can find more bugs in our testing, insecure tech stacks.

Kimi is the closest Chinese equivalent of Anthropic.

Good point. Wouldn't be good if there were white hat models being developed by legitimate organizations at this point?

I mean they say they will release the weights by November 4, so presumably after that it won’t be hard for people to fine-tune it to be better at cyber right?

It's not actively nerfed like the Western frontier models, it's just downweighting offensive "cyber" as a domain for post-training/fine tuning. Which, if anything, ought to improve model capabilities wrt. other domains.

Haven't gotten to try it yet but that'd be a breath of fresh air. Too used to LLMs flip flopping at the slightest hint of challenge.

I might want to try that. DS 4.1 Flash anyway fares much better at it than GLM (which is quite not so good at this aspect). Hope Kimi is even better.

I always appreciate when a model does that. It's amusing, but also having it be argumentatively wrong is a lot less annoying than being sycophantically wrong.

I much prefer that for chat, but I'm somewhat worried that we shouldn't give tool-enabled agents an attitude. It's all roleplay, and I wouldn't want a system playing the role of an annoyed assistant to misuse its tools.

Reminds me of early test versions of Gemini Flash 2.0 (or was it 1.5?). It would argue something, try to prove itself right, realize it’s proof is agreeing with the user halfway through and, instead of admitting defeat, would backtrack to try another approach. So much fun.

It could be the system prompt

Eh, there's good and bad. The model mostly sticks to prevailing narratives in the training corpus. So if there is something that the prevailing narrative is really wrong about, it's gonna be just as pig-headed about it as a typical internet forum rando.

When people say distilling they mean SFT on other models outputs.

In their thinking and if you ask them, I've seen Kimi K3 and other models confidently say that they are Claude from Anthropic, and accuse you of attempting to jailbreak them if you say otherwise.

They're cope, as if Anthropic or OpenAI have some magic moat, and not just basically a headstart and more money.

Eh, not necessarily, many of the RL vendors sell to Chinese labs, too: https://www.interconnects.ai/p/glm-53-how-chinese-labs-keep-...

Maybe it's intentional so government doesn't block open weight release.

There was law proposal (rumour?) last week about Chinese labs having to get approved for weights release now.

Moonshot has been frontier since this summer. At this point whenever they release a model it's on par with whatever the latest OpenAI/Anthropic model release was.

Not sure I'd put Google there in practice. And while Muse is an impressive jump from behind it's still not quite at the frontier either IMO.

You either include all labs which can reasonably produce models which are capable enough to be considered generally on-par with the best available models, in which case there has been way more than 2 frontier labs for a while now, or you take a more literal definition and only include labs which are capable of producing models which are considered "the best" out there, in which case there are basically just 2.

I think the former definition is a bit more meaningful since it avoids issues with determining what actually constitutes "the best" model out there. In that case, Google has been a frontier lab for a long time, Meta arguably as well, and you're also ignoring xAI which has been competitive for a while now. And, obviously, there's a bunch of Chinese labs which would deserve the title as well.

But if one is being strict (and willing to accept generally accepted definitions/measures/etc.), then there has only ever been two frontier labs: OpenAI and Anthropic. They're the only labs to have had "the best" models at any point in time since LLMs became a thing.

Muse Wonder is good but it's not frontier. It goes neck and neck with Opus 5.

right now it looks like only anthropic since openai made the staggering business decision to release their midrange model (terra) rebranded as sol 6.1 and quantized astra into the ground.

everyone else mostly seems to have just decided to cheat on benchmarks to make their models look better than they actually are, and none seem to want to actually try to make better models.

It's still just 2. Google hasn't been at the top in a long while, and even when it was, it was just a tiny improvement over what other labs had previously released. Meta has never been at the top. Only two labs release LLMs that really push the limits out.

> How many frontier labs are there right now?

As long as LLM's can be distilled, and all these "labs" (what a weird name) can distill one another, there are going to be as many as capital is available to be burnt on the topic.

I'm not sure what counts anymore. You've got OpenAI and Anthropic; then Meta, Google, and X; then over to Alibaba/Qwen, Z/GLM, Moonshot/Kimi, Deepseek; and Mistral in Europe. How many people out there in the real world can actually come up with tasks that differentiate those 10?

About how much would the session have cost if you were paying API prices (assuming good cache rates)?

One things Kimi is useful for is accessing/querying websites that have paywalls or try to prevent use by AI agents. It is aggressive and persistent for that, though not always successful.

Not sure if this is benchmarkable besides for the own person deciding which one they like more. Models difference is more about taste, preferences and prompt style.

idk, openai has alot of apple designers and their prototypes

How do their numbers compare with Anthropic and OpenAI?

  document.querySelector("table").style["table-layout"] = "auto";

software is solved what r u talkin bout?

It is pretty obvious anthropic is lying yet again if you think about it rationally.

Unlike Claude models which always hide the thinking process, Chinese models always show it. So how the heck they can delegate the api call to anthropic and still get the full reasoning? Yes there are tricks to reveal the thinking output, but they are just tricks and do not works reliably. So what would happen when they failed to do it? Cancel the ongoing token generation and revert back to it own model? What about the glitches, the delays, the changes in behavior?

I would understand if the attempts are small in volume, because normally people hardly notice it, but anthropic claims that they happened for millions of calls. Such large numbers would surely get noticed by Chinese users and got reported in the tech forums like linux.do. yet we haven't seen any sight.

I like it. If the increase in unrestricted cyber capabilities can be slowed down, please be it. The world needs to get ready, and that takes time. Way more time than even a slow progress would allow.

Unless you want to see more of what happened the past few days.

I think the argument that "cyber capabilities should be democratized, thus keep producing them now, fast, and for everybody" is a very naive one.

If I were them, I'd release the weights and claim I had nerfed them for safety, having done no such thing in reality.

Even with Chinese models this close Moonshot only makes $1B in revenue compared to $60B for Anthropic. Enterprises (where all the real money is) will almost always pick a turnkey solution like Anthropic rather than messing around with running their own open weight setup

I've consistently heard the opposite from Chinese programmers. Maybe the performance is better in some tasks or maybe the sample size of San Francisco is just skewed.

Are you using same harness/agent on all of them? It makes a huge difference for long running tasks.

Same model, faster inference

Astra is very strong.

I think I have yet to see a pelican where the left leg and pedal are drawn on the left side of the bike.

Simon's comment apparently got flagkilled; I vouched for it.

If enough people are flagging it, we may be at a point where the pelican benchmark is truly overdone.

definitely not frontier though, as even on max both legs are in front of the frame; although I am wondering how much getting that right correlates with model intelligence.

I like the low effort one most.

Can I just say that I love how you are also just always here the moment something drops lmao. Hope you're all well! The pelicans indeed are solid.

It's broken for me in Firefox, worked with Chrome.

Post-training (i.e. RL fine-tuning) not pre-training. This is just a post-trained checkpoint of the K3 pretrained model.

Which is also gonna be open weights on 4 Nov.

When the frontier is pacing itself in its speed in terms of adding new capabilities (notice how fast opus 5.5 got released after Astra), then that also means that the open weights models which were just behind are accelerating in their speed just as much as well.

Good news for us because the models which used to feel 6 months behind now feel like they are lesser.

Double threat to Anthropic and OAI because if open weights models become good enough at the current level of Sota while being cheaper (which time will tell if this model fits the bill or not), then people will use those models instead of OAI/Anthropic provided it is "good enough"

Though it has to be said that only real world use case can be trusted but I remain optimistic and hope that this turns out to be good in real world use cases as well.

But in my opinion, all of this is really good for us end users!

Maybe limited compute to handle launch traffic spikes?

Still get the model out there.

makes you wonder if they have intel on an upcoming model drop by the frontiers

It's not like US companies spend trillions on training itself.

Mostly it's datacenter build out and circular financing magic when the same dollars being commited multiple times by multiple companies passing hot potato to each other. They basically commit capacities 2-3 years ahead.

It's 'sustainable' because there's enough regulatory risk to using Chinese models that many will avoid them, and a 3T parameter model is still a bit beyond even many enthusiasts for running locally.

its hyperscalers and neoclouds data center build out that ate those trillions.

a lot of those money is the commitment from US AI labs that's not yet spend until the data center is roll out.

you are conflating.

There would be no Chinese model without US models to distill though. So, in a way, the Chinese models also cost a trillion dollars, they just don’t cost China that much.

Page isn't up but it's a 55 (4th overall) https://x.com/ArtificialAnlys/status/2113688083145314416

It seems so, judging by quotas on https://opencode.ai/go

no

The older Kimi K models (2.5 and 2.6) weren't bad for the time, but all of their models have had problems with overthinking. This also makes them significantly more expensive in practice than their pricing suggests, since they produce way more output (reasoning) tokens than most other models. Anecdotally, I've also had problems with all the pre-K3 models thinking themselves in circles trying to solve very simple problems that other models didn't struggle with. (I haven't played with K3 much because it's just too expensive.)

For me, it is the opposite. It may be slower but it follows the rules better and makes less stupid mistakes, it takes less time to merge the solution.

I have the opposite impression. I find K3 to be extremely capable at full-stack work; its UI design skills are great. I'd love to use it more, but the quota on Kimi's plans is abysmal. I wonder if these new models improve that.

Yep, and often actually very expensive to use. More expensive than the frontier models because there's no realistic coding plan.

<thinking>

Your question is vague and you didn't explain your point of confusion but consider the fact that securities laws are not fungible between nations.

first time? Welcome to the irrational SV investor world. Its about hype, perception, politics, perceived future returns, not actual market fit and the proven viability of a technology. Its pets.com, myspace, and NVIDIA pumps because people want to 10x the money now.

Everyone and their dog has been saying its a bubble, there is no moat, this is all known techniques being scaled (very inefficiently) onto increasingly expensive hardware clusters at exponential cost, etc.

This probably doesn't move the frontier, it just sits at the current frontier (opus 5.5).

Chinese models never agreed to the pacing anyway lol.

"We must start a cartel with Google, Meta, and OpenAI"

it's better than all OpenAI models

Why does your benchmark have Opus 5 scoring the highest?

its not open weight.

AA-Omniscience

here's a fun benchmark https://fenderog.com/art-crit/page.html?p=ceate-an-svg-of-an...