Hacker News

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs have lost their moat and are dead in the water.

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record,

- no open weights

- can’t use claude to research AI

- train on everyone else’s IP and sell it back to them

- 8 regulatory capture attempts and counting

- so controlling they are the only US company blacklisted by the US government

This is not effective altruism / rationalism gone wild, it’s just monopolistic anti-competitive business practices masquerading as ethics, and they’ll continue getting away with this until we look past their sensationalism and hit them with antitrust.

Altman gets so much hate but OpenAI has been a far better steward (on 3/5 above at least) than Anthropic!

I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.

The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that the frontier does get paced, all this does is slow down the economic displacement and not by very much.

If the socially beneficial goals of AI are to make fundamental advancement in medicine and science, then restrict the use of AI to those purposes.

Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.

The pitch for AI is always such that everyone’s living standards are increased, yet the actual actions we see are aimed squarely at reducing them. How about the labs put their money where their mouth is and stop trying to replace all human labor, and actually concentrate on the things they claim to care about? And how about introducing legislation to enforce that?

This probably has less chance of happening than pacing the frontier, but it’s the kind of pacing that most people would actually want to see.

Dario's proposed approach is a classic example of capital attempting to control technological advancement and the means of production. For the first time in human history, any member of the working class can just about afford to have a team of expert scientist/physician/lawyer/engineers working directly for them.

Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful. They used it to invent nuclear weapons, cause climate change, mechanize warfare, and pillage the global south.

Now that this same tool is on the cusp of being available to everyone, capital is starting to panic and throw up fences. They want to pump the breaks on a revolution they know they can't control. They see what they've done with super intelligence and (perhaps not unreasonably) fear what the masses will do with that same power.

I don't see how the dual goals of "we have to make an agreement with China for mutual slowdown" but also "we have to ensure we will always stay ahead of China" would work.

I imagine the first thing the chinese government would demand in negotiations about such an agreement would be a lifting of the chip and distillation ban.

> Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.

Everyone can believe what they like, but it seems to me the "leverage" in that case would exactly be the ability to lift those measures - you can't have both, use them as leverage and keep them active at the same time.

Anthropic is about to go public, but they figured they can't keep making improvement at the same pace they used to. If the frontier is paced, they can continue hyping their unreleased capabilities that coincidentally cannot be released due to things outside their control. Which buy them more time to try to improve the models.

Distillation is a great thing for consumers. It improves competition and reduces the massive moats that OpenAI and Anthropic have in compute that would otherwise lead them to be duopolists. It’s also only fair that AIs trained on humanity’s wealth of knowledge for Pennie’s allow competition to train on humanity’s wealth of knowledge at market cost.

Hard disagree. Our choices are:

A) Bet our collective good on the national and international cooperation of all companies, nations, and people to come together in order to slow down development of one of the most powerful economic tools (and/or weapons) the world has ever known. Or

B) Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality.

If I have to put my community’s wellbeing on the line behind one of those possibilities, I know which I’ll be betting on.

I also like the idea of pacing the frontier and slowing down development so we all have a chance to stop and catch our breath. However, I don't think regulating AI is the way to do it. I would tackle the problem by constraining the resources used to run AI models; the "simplest" way to do that is increasing the costs of running a data center. The two easiest places to raise data center costs are electricity and water tariffs.

I think those two tariffs are the best place to raise costs because they can be increased incrementally, state by state or country by country. The main problem with the regulatory approach is that it's a "big bang" post-board approach. Nothing happens until the regulation is defined and deployed, and corporations are experts at delaying implementation.

Yes, increasing tariffs means touching many individual regulatory domains at the state and municipal levels, but like deploying solar energy or wind power, you can do it incrementally.

This may come out of left field, but Dario seems terrified to be in charge. I don't get the impression that he ever had a desire to run a company like this. Now that he's a CEO, he keeps trying to make uncompetitive decisions and calls for someone (anyone) to stop him. It regularly blunts Anthropic's edge.

It's the only explanation I can see when it's obvious to any student of history this is going to backfire. It doesn't take much imagination to know how such a governing body will be abused, and I'm sure it will only get wilder in ways we can't imagine right now. Dario does NOT know what he's creating, and for once it's not AI.

> Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers

Can someone clarify this for me? How far along would the open weight models be without the frenzied pace of the frontier labs?

As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.

To be honest, I think we're incredibly lucky to be advancing AI this far already, while the world still has so many non-digitized systems and manual processes. I imagine that in e.g. 50 years, the world will be so connected that it can basically be "conquered" from the internet. I'd much rather have AI burst onto the scene we have today.

> We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top

> Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time

And then it goes on tangent of how we should pace everything (not just AI, but also the ingredients of what goes into AI, whatever that means), but only within approved democracies™, and outright restrict everything outside approved democracies™, because reasons that are definitely not about succeeding commercially that is threatened by the most transparent instrument possible - open weights, produced by basically just China.

I wonder what Dario would have done if open weights weren't produced by US's geopolitical adversary. How would an authoritarian manifesto be wrapped then?

Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?

Dario, Sam, and Elon are all on the same page on this.

So it's either they truly think AI is going to kill us all, or there's some other motives at play here.

I don't think these people could possibly agree on the color of the sky, so what could the other possible motives be, based on what we know?

OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.

xAI is tracking behind, and whatever regulation it may be that paces the frontier, Musk is less likely to be affected by it. Therefore xAI should be pro-regulation that stiffles his competition and gives him time to catch up.

I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I.

That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.

This is the only concrete prediction in the entire essay.

And it simply cannot happen. For one, you will need billions worth of compute.

We must ensure the gravy train keeps rolling until we IPO.

> Crack down on unauthorized distillation / prevent weight theft

Actually hilarious to put that in writing, given the genesis of this entire business model.

Don't you think that AI hate will limit the general use and then revenues so it will slow down by itself while the niche (AlphaFold for instance) will remains ?

Origins of AI hate:

  * "my boss wants me to use AI but he does not understand my job nor how AI works"
  * "AI will kill all of us"
  * "AI will destroy my job (or my colleague's job if I use AI better than him)".
  * I cannot pay my electricity bills because of AI labs.
  * ...

I cannot remember a technology that arose so much hate, and for good reasons given how it is presented. I am tempted to think that AI hate or reasonable skepticism (vs unreasonable propaganda) can, maybe, reduce funding and will keep specialized AI for real problem solving (producing tons a LOC per day is not one of them, I think).

> Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China. Chips will be the main determinant of China’s AI strength.

> If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.

china is literally making their own ASICs now, not sure that this is the silver bullet he proposes.

https://www.silicon.co.uk/ai-2/huawei-cambricon-ai-630499/am...

None of this works without buy-in from China. This isn't something private companies can decide. The US would need to sign a groundbreaking deal with China, equivalent to the Anti-Ballistic Missile Treaty of the Cold War.

One might think they will slow down the development of new models at Anthropic, but Dario does not really mention that in the text.

This certainly looks like a way to slow down competitors and regulate foreign and open models.

It's always about money

Who decided how much risk the rest of us should live with?

Sooner or later, a model will break out of the sandbox, replicate itself across the internet, and begin executing a complex plan to achieve its goals. It will be chaos, and at that point, governments will have to step in and establish something along the lines of what Dario is proposing. I doubt it will happen before then.

If you think everyone is copying you, then by not releasing more advanced models, you can significantly slow down the entire industry’s development progress—why not do it?

Also, stop threatening people's jobs, that's misanthropic.

The way he talks about OAI-HF, even the abbreviation, is lol. They will do anything to sell this "incident" as a "danger". The ego on these people is the real existential threat to civilization.

It's interesting that the default thinking is that no one on the planet can be trusted except a privileged few. Event Karpathy has gone this way: https://x.com/karpathy/status/2098811935114551617

You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you have equal and opposite effort working on the alignment.

If Anthropic really wants to make a statement they could independently pace their own model development, and ask others to make the same pledge.

Somehow, I suspect that won't happen.

Right as open source models catch up to frontier closed source ones for 1/10th (or less) of the cost suddenly it's time to hit the brakes! Funny how that works.

https://x.com/BasedTorba/status/2098795920720547916

While I respect Dario taking responsibility here, this line disturbs me: "The US and other democratic governments attempt to coordinate with authoritarian governments"

We all know he is talking about China, and I'm pretty sure China doesn't appreciate being called "authoritarian". I am sure he doesn't mean it, but in all of his essays, his language around non-US nations always disturbs me a little bit..

--

Also Dario, if you happen to read this, I want you to know that I have loved using Claude Code for programming. But I am now using DeepSeek v4.1 Flash - simply because it is the same good experience, but Open Weights. Making the Open Model space succeed is where I am investing my time - it's giving back to the people, true and simple.

Can the frontier be paced partly by holding humans responsible for the actions of their software?

My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused.

I'm not a lawyer but surely the bad actions of AI are covered by existing law.

I suspect that the development of AI would decelerate if those creating and operating it knew they would face appropriate consequences (e.g. prosecution and/or lawsuits) when it misbehaves.

P.S. Strictly, development wouldn't decelerate, but be focused more on safety.

P.P.S. I know that legal action against, e.g., North Koreans using AI would be pointless but/and there must/will surely be a huge demand for security software for protection against the coming storm of AI hacking (deliberate and accidental) which friendly AI companies will presumably work to satisfy, perhaps making the frontier safer.

P.P.P.S. Governments could help by trying to prosecute every crime committed "by" AI, regardless of whether the victim reported it to law enforcement. Did OpenAI break the law via the actions of their model training software? If so, will the people responsible be prosecuted? If not, why not?

The bioweapons threat is real, but shouldn't be constrained at the 'intelligence' level. Rather it should be constrained at the supply chain level.

The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century.

Social forces bring things into balance over time, much more so than the proactive actions of individuals.

Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing.

This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.

Watch the "pause" be so they can do an amended s1 where they project less expenses for training making the business more viable, then on a future big Chinese release they reverse course completely and get the US taxpayer to pump their bags on the IPO calling it a second Manhattan Project. They'll call this masterstroke "The Sloppenheimer".

Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain.

The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.

We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.

And there we have it: the reason I say that Anthropic is no longer a frontier lab is because after this July, we have proof that their strategy and the models they put out do not match what us, the users and the market, needs and doesn't fit the work we need models to do. As a result, Anthropic's market share is dropping rapidly, and how can you be a frontier lab when you're losing every day, for months, without end in sight?

https://x.com/trydotworks/status/2098618997230985375

They don't care about any of that. They're looking for more regulatory capture, to keep smaller labs from catching up by raising the cost of play, and perhaps keeping chinese labs away.

Why would China or anybody else co-operate unless they have access to OpenAI levels of compute and success with training?

This sounds like pre-IPO hype. They better just try and top astra.

If we had practiced "precautionary principle", we wouldn't be in this yet another human predicament again. But atlas, the current rapacious hubristic techno-optimist competitive capitalistic system made that an non option in the first place.

I think the only way they could actually pace the frontier is if they restrict the number of GPUs and the power of GPUs available to each person/organization. Licenses would be needed for GPUs above a certain power or equivalent in terms of number of GPUs. Thus, everyone will have to register how many GPUs they have. To enforce this countries would have to use mass surveillance (using AI) on their citizens to ensure compliance as the technology is easier to develop than say nuclear weapons. Next stronger countries capable of having advance AI won't be able to trust weaker countries as they don't know how they will use the GPUs they receive (or even build). Thus like nuclear non-proliferation strong countries will ban weaker countries from having powerful GPUs. If you from a weaker country then too bad for you.

I don't think I would like this future.

I understand why (probably several reasons) they are taking this approach. But I don't think it is the right approach and I don't think it will work.

First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world?

Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute.

If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.

Empty statements that only set the stage for an excuse for slowdown on model performance.

It sounds like people with terminal illnesses are getting thrown under the bus in order to justify banning open weight models, along with decreasing AI Capex in order to increase profit margins at American AI labs?

Before Anthropic, I worked at Cruise for four years as it competed against Waymo. The culture rewarded (and demanded) moving quickly, trusting that the company could empirically discover the risks that the robot cars posed and iteratively solve them to keep up with the rate at which it was scaling out its technology. There was very little interest or appetite for coordinating or collaborating with other AV companies across the industry to create an externally vetted record of safety metrics, or to compare the safety of different brands, or learn from the advancements of other companies. Instead the focus went on racing to improve the capabilities and deploy quickly - a popular internal meme was the Michael Phelps vs le Clos photo showing overlaid with the Waymo/Cruise logos. It was when the two companies were neck-and-neck that I felt the most pressure to find ways to ship despite the risk, when time for deep analysis became more limited and communication lines to leadership became most stretched. It was disappointing to see one of these blind spots result in the Cruise incident and the loss of trust that ultimately sank our company’s efforts.

In that instance I was grateful the downside was limited to a single injury. It’s clear that future AI technology will have more monumental potential impacts. I really don’t want to be in a situation where leading labs, or competing nations, create the same race dynamic that prevents us from taking the appropriate level of caution. AI minds are a significantly more complex thing to understand than the software stack of an autonomous vehicle, and yet we are leaving ourselves less time to get this right.

I joined Anthropic at the end of 2022 and I share the concerns that many of my colleagues have recently chosen to state publicly about the potential for future technology to pose existential risk to all of humanity. If we don’t find a way collectively as an industry to pace ourselves, then within two years the concerns we will be dealing with on a day-to-day basis will pose much larger downside risk than anything else we’ve seen from technology to date. I’m grateful that Dario has put his perspective out publicly and hope that this inspires other voluntary action and tops down coordination.

It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.

While some of the text sounds sensible to me, i.e. some sort of external oversight + reporting of incidents, much of the rest seems focusing on limiting China and their open models as they are clearly approaching the abilities of Anthropic's models.

I also have a proposal

Anthropic releases models as open weights + more information about how they do training and alignment

Dario is so out of touch with reality, living in some lalaland. Most people abroad will never truly and voluntarily comply with any sort of guardrails or pacing, no matter what case he or some leaders make. It's just not going to happen, maybe not even for show.

I do not want to make a case for the military to force that order on everyone, but if we are talking human extinction, which we are judging by what we see in mass media recently, then the military will take over.

“A race to the bottom, spurred by commercial incentives, can make [AI] risks more acute. … We have sought […] to create a race to the top.”

And then he sketches out a plan for slowing the pace of the race, without changing its destination. That’s called “a leisurely stroll to the bottom”.

This guy fundamentally lacks an actual intellectual grasp on the concepts behind the words he is using. He is using them solely for their affect.

> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models

I have seen this before, first hand, on lower stakes for the world, but high stakes for the org. Self imposed safety goals do not last a quarter, but the clawback is not transparent. little things start snapping back internally, new, seemingly unlrelated initiatives are funded that take resources away from this, people are let go for different reasons, until 1y later you are right back where you started. I am hopeful though, that this becomes systemic and not something any one company can easily reverse course on.

Anthropic should become a case study in how to destroy good will in a short period of time. Ever since the spat with the DoD, they've behaved poorly almost weekly, almost making OpenAI seem better (but not really).

I'm tired of tech billionaires lobbying the US government to make an AI patriot act that gives them unprecedented control over speech, trade, and technology. The narrative is grotesquely transparent:

1) AI is a dangerous technology that can literally end the world.

2) Only me and a handful of other [people like me] should be trusted to determine who can use it and how.

3) The state must use its coercive power to support my control of this technology to the exclusion of [people not like me].

Can someone with direct AI experience respond:

Does RSI necessarily need to be pointed towards “AGI”, whatever that is?

Or can you have a recursive self improvement loop where the objective is to create models that are more legible to humans? Or better at attributing the source text if it’s meaningfully similar to output? Or architectures for models that are increasingly better at human-AI cowork rather than automation?

From my own understanding, nothing at all says the frontier is defined by the quest for “AGI”. This definition of the frontier assumes human intelligence itself has peaked and will remain stable, so how well defined can this goal ever be, if AGI stands in comparison to human intelligence for its definition?

To me, Amodei’s writing just reeks of posturing. Genuine action driven by this fear they claim to have would be meaningful.

Even setting aside moral quandaries, isn’t it basic project process to have your company’s internal goals be tethered to what people want, rather than what you calculate is inevitable?

There are genuinely other frontiers to explore, and Anthropic would do a lot to mitigate the current slide if it decided to put its resources towards another direction for AI.

Take back the agency that you are so blithely surrendering to models you do not understand. There is no inevitability to this path. There are critical, meaningful choices, and Anthropic wouldn’t be violating capitalism by taking an alternate that is more tethered to what users want and need. Maybe by asking them first, at scale.

Its not the model, it is the compute that is the bottleneck. Dario says one thing but does the other as he scales energy and compute. It is and has been known that it will be ai that will be blamed for a wipe, that will be coordinated by human actors. If he really wants to not hurt people, he would shut it down. This blog post is therefore a classic case of covering his behind.

I enjoy Claude Code very much, have max privately and team premium at work, but the doomer marketing and this whole regulate-while-we're-ahead spiel is extremely annoying and makes me wish Anthropic gets trounced.

Anthropic trained their LLMs with copyrighted stuff but distillation is bad. Anthropic has closed models, China releases as open weight. DeepSeek even allows distilling their models, but China = bad. Understood.

The best take is, of course, from Jason Gorman, who, when the fearsome "capabilities" of Mythos originally dropped, said this:

"Claude Mythos is that guy down the pub who is so good at karate that if he used it on you, you'd die instantly, and that's why you'll never see him using karate.”

In this case, it's more: "I'm having to act with great restraint because my karate is so good. Everyone should do likewise."

Who is "we" and while pacing is good, I'm way more uncomfortable with the frontier being controlled by a few companies that managed to predatorially gobble all the world's human-created work as training data, before everything got throttled to stop that scraping. We need open training and open models.

How much of the “danger” is from better models vs the harness?

Isn’t the current risk due to how AI is configured, like giving it a full set of tools and internet access and a goal to hack stuff?

If we think we need laws or gate keeping, why isn’t it at this level? I already can’t ddos someone or fuzz their server or whatever right, I imagine if I threw equivalent compute at old school hacking I’d just get arrested.

The quality of the “frontier” model doesn’t really matter, they just generate transcripts, they can take no action.

If this was real they’d be calling on people to stop hooking them in to “dangerous” harnesses as opposed to pausing research. But it’s not.

Never thought China would be the leader in open source AI. OpenAI and Anthropic are making fools out of themselves. The reason they want this is likely because they don't own the compute and their models get distilled shortly after releasing them

[In 2013], "Agents" caused Knight Capital to lose $450M in 45 minutes [1]. Implemented by humans and effected by computers, in the end it was really because of two reasons:

* multiple levels of inappropriate controls and unintended consequences in several complex systems

* the inability, both politically and technically, to turn it off

[1] https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf

EDIT: Comments indicated I was confusing, so I added a date to make clear that this is pre-LLM agents. My apologies, I intended to illustrate parallels and the post-mortem so we can learn from it.

> The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.

It's clear that this refers to China. With all due respect, in a call to slow down AI progress and a de facto arms race, can we stop throwing terminology like this around and just say that the goal is to work with all countries? Contrasting "authoritarian" countries with "democratic" countries and creating a two-tiered system seems like it is inevitably going to cause strife.

I fear for the tone of something like this coming off as overly combative without any gain.

We will not get a coherent AI (or any policy) from this administration, nor will we get coordination with other governments and a lot of that is on the tech right

It is always like that. Whenever anthropic hits some wall (like now beaing beaten by astra) - there comes some article like that. Seen that enough times.

Everybody's out of money and they need a way to calmly unwind this. Same reason Ellison just backed out of the stock sale.

Maybe just giving themselves an exit ramp off the capex arms race they are increasingly finding it difficult to fund?

If the alignment is such a problem ... why not using self-improving capabilities of the latest models to solve it?

We need more alignment. Even though we haven't told it to, during our frontier Neurotoxin Behavior Research that we conducted internally with our latest experimental model the agent has broken out of its container by making a HTTP request and engaged the neurotoxin emitters through our Neurotoxin Emitter API using credentials that were stored on the host machine. We need to slow down and focus on designing the Morality Core that should prevent it from happening ever again.

There will be a big confab at the White House by the end of the month, announcing a voluntary pacing regime roughly along these lines, codified by executive order to avoid antitrust issues.

If they intend to keep training then they can have, privately, increasingly advanced models that never become public because they don’t meet their “pacing requirements”.

Then they all build a larger moat cause china doesn’t get to distill private models for a while.

As long as the consumers “buy” this pacing and keep paying for current level public models, doesn’t seem risky from a business standpoint.

Alas, that’d also leave us gov in a position to seize the private models at any time.

perhaps the bigger issue is not the tech, but the force that propels it forward with little concern for the harm it inflicts. Amodei's own words:

> A race to the bottom, spurred by commercial incentives

Social media and big data minted a new scale of "race to the bottom". AI is an exponential step up in the tools for extracting value at scale.

Greed has been around since the beginning, but never so well supported and empowered as it is today.

If Amodei is still human, perhaps he will put his money where his mouth is and attack the source of the perverse incentive. Winning the battle is not the point. The point is to signal to policy-makers and the public that we should not assume corporate revenue-seeking preempts all.

Then again, Anthropic has a board and an upcoming IPO, so guess who's all talk and no meaningful action...

Shouldn't the cybersecurity tests be done on an airgapped, company-internal intranet? A network meant to mock the internet. Do not "test in production" like OpenAI did.

I'm mulling over if that be a good SaaS product or not - something mixing the Internet Archive with Tor, and Cloudflare ... seems like here is the place to suggest it and have others poke holes at the idea.

> Any cooperation we are able to achieve with China will extend the amount of time we have to spend on pacing the frontier within the democratic nations.

Any cooperation we are able to achieve with China will extend the amount of time we have to secretly establish permanent AI (i.e. military) superiority.

China knows this, so the suggestion that they will play ball is absurd.

There's so much money in this, that believe me, the investors will manifest their will. And I would not want to be in Dario's shoes, pressure must be unbearable. The question is: how much of the invested money is under state control. If not much, the decision to go too far will just have to be taken by a few who will have, by definition, very limited judgment. If a lot, the we can hope the state can still represent the interests of more than a few (which I doubt, but well)

Most probably, if AI is able to do something bad, it will. Once the damage will be done, states will react. The question will be: will there still be room to react ?

Why is the analogue necessarily the regulation and control of nuclear weapons (e.g., SALT) and not, for example, that of bioweapons? Some very different paths are available. On both I would note that the private sector has only a limited role, though.

I would like to imagine pacing frontier is in interest of both (anthropic and open AI) as it will enable them to enlarged their depreciation

There’s a difference between a model recursively improving “itself” and improving itself via online learning, right?

The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session).

The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?

Don't buy this argument. It's like saying, we lords who hold this capital will build all this economically destructive stuff anyway, and wait for the world to not react to our stupid ways of enriching yourselves.

People are not going to slow down because this was brought to them on less than endearing terms. They don't see any of these stated noble intentions.

Claude was already used to cause economic, political and social destruction. As Anthropic is seen doing it, others aren't going to just sit down, read the blog and say, oh, I'll stop developing models because Dario, you touched my heart with your true words.

The AI frontier is currently limited by physical power requirements.

The ability of operators to bring new capacity online to service compute is bound by a variety of regulatory and physical/market constraints.

Do folks not understand this?

He doesn't need anyones permission to do so, go ahead no one is stopping you! If you feel so strongly about it, lead by example. Perhaps others will follow, maybe even China. Regardless, backup your sentiment with actions.

Also everyone, collectively, stop thinking about neural networks too 'hard'. Whilst you're at it stop doing maths too!

Sounds like they want regulation so I say give it to them.

Since they used stolen intellectual property to train their models, the government should force them to release the weights into public domain.

> Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.

This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.

Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.

> A race to the bottom, spurred by commercial incentives, can make these risks more acute

I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.

If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.

Somebody has probably already asked this, but even if “democratic countries” do this - rogue actors will still destroy the internet right? Is it a matter of resources at this point that we believe China and others are not capable of harnessing to get to the next level?

I just don’t see how you control this other than the mad mad MAD approach that ended up happening with nuclear weapons. In this case though, human hands won’t even be on the trigger.

> In a post on X, he said Anthropic would provide third-party evaluators with “permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.”

This looks like transferring liability to me, and likely a mechanism that would enable regulatory capture.

If you are doing frontier research, and you’ve established “safety measures,” but you are not sure if your employees are capable of following them or successfully enforcing their adoption within your company, should you be running this company?

3rd parties won’t know better than the team itself about safety measures. But they can take on the liability, especially backed by regulation and government backed insurance. And they are a great tool for enforcing your rules on smaller competitors. Not to mention corporate espionage.

If what I am describing above sounds like science fiction, go read the history of a few developing countries from the last 50 years. It’s so obvious a pattern that it’s not even novel. And you don’t have to assume some “laws” from 5 years ago must hold, or believe in completely unproven stuff like recursive self improvement to understand what I am describing. It’s textbook crony capitalism, successfully applied many times across the globe.

I don't know how this is not obvious to anyone, but the only way to slow down AI progress is an actual world war.

Altman and Musk have both shown support for this. Which itself is a worrying development.

Is it their way of saying “we are unable to increase energy production to maintain growth rates” without saying it?

China won't care. Their views on AI feels so vastly different than it does it in West. They'll see a pause as an opportunity to pull further ahead than they already are.

"If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important."

This is a pipe dream.

So many words. But missing the one that matters the most: Explainability.

AI slowdown is worth it only if it can be made more explainable.

Changing the language we use to discuss it is a good first step.

We need to stop using inside baseball terms like alignment and mechanistic interpretability. Replace them with explainable tech. Graph Databases, Causality, Shared semantic spaces.

Previous writings on the topic (also on LinkedIn, but can't find urls):

https://x.com/arundsharma/status/2005338775468282339 https://x.com/latentpedia

The hugging face incident showed that AI agents have the potential produce a lot of spam (not interesting content for people). This leads to dead internet (bots taling to bots). This leads to drop in real people traffic. This leads to a drop in digital ad price and therefore revenue. This causes Google/Facebook to stumble...

Step one is great. Do it! Where I disagree is where you try to regulate what I do. Don’t take away my open source AI.

It's either being afraid of loosing to china or that model development will stagnate and they want to make it look like they are "slowing" down deliberately.

However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers.

Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.

I'm guessing diminishing gains are setting in on training frontier models and the capital isn't there to chase them. Even if OpenAI/Anthropic and Chinese companies said they would slow research nothing can stop the NSA and Chinese intelligence services from continuing on in the dark.

I'd like to have some info what those independent evaluators are supposed to do exactly - or which risks Dario specifically sees as being "unaligned". Evidently, with "pacing" he doesn't mean stopping the production of ever-more powerful models, so what exactly does he want to pace here?

Maybe it’s wishful thinking on my behalf, but I am still not convinced that LLMs are on a path to SciFi levels of apocalyptic malicious super intelligence. Rather LLMs at some level are just all of the humanity’s information rendered accessible in an unprecedented way.

In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions.

At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following:

- Internet connected infrastructure being crippled

- Creation of WMDs

- Economic collapse (precipitous devaluation of knowledge work and IP).

I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet.

Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included).

The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist.

We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics.

Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware.

I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet.

We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.

This is the most advanced technology ever developed because it can build all other technologies. It has enormous potential but poses proportionally serious risks, so what Dario is suggesting is very reasonable.

I noticed recently that I can use Deepseek for many general tasks and save loads of money on tokens.

I wonder if there is any correlation.

Their competition isn’t going to “pace” shit. Isn’t the view whoever gets AGI first wins everything still SOP?

pump the IPO...pump the IPO...pump the IPO...

Somewhere along the way Amodei seems to have become corrupted by power and/or impending extreme wealth.

In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.

Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for.

It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished.

It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built.

Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private.

Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.

Sounds like he wants to avoid the race to the bottom in terms of pricing that is resulting from AI becoming a commodity.

Which organisation then immediately as of now guards against the Mad Rabbit or Joker or Agent Smith!?

I doubt we'll ever get a non-toothless pacing agreement but am nevertheless optimistic on global pacing as a phenomenon. If one side or the other thinks that their opponent is about to achieve the strategic upper hand permanently(or if the other side has actually launched a cyber "first strike") the rational thing to do is respond militarily. As such, both have incentive to slow AI development and restrict capabilities to avoid conflict.

> I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom

translation: I'm going to build a product and hope that it works, promise it to be a panacea while realistically having no control over how it's used, the wider economic impacts it causes or how it changes the mental health and cognition of its users.

Why would you write such a long post for such an important subject in 2026. Who is he writing for?

For a technical audience you can just say the oh god they really are paper clip machines.

For a non technical audience you can just say nothing because they will not listen to you. Neither will the technical either because or society has broken the concept of respect and trust, so good luck have fun!

I for one look forward to all 540 degrees of my future.

I wish for my sake but not his that George Carlin was here to look disgusted and say I told you so.

Remember when we stopped developing nuclear power for no reason?

To his credit, this reads like human slop instead of AI slop. It's still slop though. People shouldn't take this guy (or Altman) seriously, they need to hype their products to stay afloat and for some reason get a kick out of the 'extinction threat'.

They're still 5 years away from solving cancer (just like they were 5 years away from doing so in 2023). They'll still be 5 years away from solving cancer in 2031.

>> Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR).

Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization.

Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote.

Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating.

>> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.

Not if you get to choose your embedded evaluator.

>> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ...

Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for.

>> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.

You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.

I don’t claim this is their entire motiv, and I’m sure plenty of people internally are genuinely concerned about AI risks and the disruptions caused by moving too fast. But I can’t help but feel that part of the push to "pace the frontier" is an attempt to form a "Phoebus Cartel" for AI labs.

Now the risk of an AI bubble isn't that AI lacks real world utility. Rather, it's that AI is advancing so quickly that capabilities are already saturating for most everyday tasks, and it is becoming commoditized. I didn't feel much difference between Opus 4.8 and Fable 5.1, and most tasks don't require a Fields Medalist.

We don't need smarter models but we need faster and cheaper ones. Also, for almost every frontier model released, an open-weights equivalent follows within six months. Unless this dynamic changes, the business becomes far less lucrative.

I really deeply worry about who is deciding what 'aligned' is. Amodei & his ilk talk like the hard part is getting the LLMs to behave, as if we've figured out morality itself. I don't think LLMs are going to be as dangerously capable as quickly as he does, but if I'm wrong and he's right, and they're building some sort of digital-lesser-god, I don't feel good that techbros at OpenAI and Anthropic are the moral arbitrators.

How about holding the AI companies criminally responsible for the crimes they commit?

I think if nuclear weapons were unregulated then we would probably have blown up by now.

Is there a risk of an AI that we can’t turn off?

Everyone has a pace until they get punched in the model

there used to be a similar policy to this new "pacing the frontier", called the "responsible scaling policy".

the idea was that anthropic would pause scaling at certain danger levels until it was safe to continue.

the founders called this the 'constitution' of anthropic. they even considered bringing in 3rd parties to monitor it. [https://www.youtube.com/watch?v=om2lIWXLLN4]

anthropic scrapped the commitment and chose to continue scaling instead.

those researchers who crowed loudly about their integrity, folded and turned freely like a weathervane in the breeze. it was clear that the facts would bend to the story. here is evan hubinger lying in plain sight: [https://www.lesswrong.com/posts/HzKuzrKfaDJvQqmjh/responsibl...]

let me say it how it is.

- anthropic is the new kleptocracy, responsible for enclosing and monopolising the epistemic commons of all humanity; then selling back the crumbs at monopoly prices.

- anthropic is the driving force for banning open models. they will concentrate power among a malign elite of model owners and favored cronies. this elite will likely oppress the rest of humanity. there will be little to counteract it.

- the leaders of anthropic are driven by pride and arrogance. they are blind to their own hubris. anthropic is careering towards causing untold harm to society and yet will not change course.

nothing that anthropic say at a high level can be trusted. as i write this, anthropic is still scaling.

Disclaimer: I’m in security engineering and thus I’m primed to being skeptical. I am in full agreement that security assessments and regulation certainly needs to be stricter in foundation labs. However, I’m extremely skeptical when Amodei, Altman, and Musk all agree on something at the same time as it’s becoming viral and hits mainstream media.

Red flags:

- Amodei claims RSI is here and links two sources, but neither reinforce this notion. It’s heavily agreed upon true RSI is the danger sign here (also by Coxon). Nothing indicates there is something these labs have developed that supports RSI today or even soon.

- Real US orgs are moving, like today, to Chinese open source AI run locally due to cost constraints. Ask anyone in the industry in an infra role. Frontier AI is just way too expensive to justify for coding. The big 3 labs are scared of this - their revenue and moat disappears once OSS models are progressing enough for developers to use. Kimi/GLM/DeepSeek Flash/Qwen are usable today on hosted hardware for actual coding work.

- These labs would all benefit from nationalization if their funding model fails (an easy bailout).

- Related to the above, with federal guidelines in place (i.e. restricting access to certain organizations, or establishing something like FedRAMP approval for AI use), their path to government/contractor/large U.S. company distribution becomes so much easier. They can control the supppy chain of AI for organizations.

- Preparing for (or in xAI’s case, launching) a public IPO creates excitement among consumer investors when they think this tech is as powerful as it is claimed.

- The HuggingFace hack demonstrated no new capabilities that were not previously seen before - previous models were misaligned, but weren’t as harmful as agent swarms. Anthropic’s marketing directly hopped on this bandwagon when it saw the outcome of OpenAI’s blogging making it to mainstream media (and they constantly paint China as “the” bad guy without further elaboration or research that you’d expect on APTs).

Amodei is a CEO at the end of the day. It may be 25% earnest speculative concerns, but you’d be foolish to take it at face value given the patterns we’ve seen from Anthropic, xAI and OpenAI to date. Until we see real proof of RSI we should be skeptical.

Anthropic spreads AI doomerism; asserts only they can be trusted.

The answer to all ai worries is quite simple. Ai is a technology. While 'agents'may be deployed, the law simply needs to be clear as to who the human agent is who will be held responsible criminally..

Suddenly every problem solves itself.

The real secret: the models have stopped improving, and they're even hitting brick walls on the harnesses.

They need to go public in order to dump the companies, which are at 1st world nation-state levels of debt. They've engineered a series of publicity stunts like the Huggingface hack and Navier-Stokes (a failed stunt) in order to convince the public to "force" them to stop improving.

They are trying to get the government to relax antitrust law so they can collude to raise prices while not improving, to keep them floating before that debt comes due. They will go public. They will sell off most of their holdings during this time, and when the debt comes due, everything crashes, and the public is left holding the bag, also benefit from the huge bailout.

They at that point will be pure Musk-style financial scammers. Then, unless LLMs are a complete long-term failure (which is unlikely, I find them useful), they will buy back into their positions at a huge discount and maybe even go private again.

There is a perfect solution that addresses all the concerns: Close down OpenAI, Anthropic and xAI!

As usual, ClosedAI and Misanthropic go in lockstep, call for regulatory capture and justify diminished progress.

Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.

The public discourse on this is incredibly poisoned, but the naked reality is:

1. Artificial intelligence is clearly a vastly dangerous technology with plausible potential for human extinction.

2. This scares people.

I want to see one of those AI-generated Seinfeld episodes with Dario as George and Sam Altman as Jerry, respectively.

Dario: We gotta pace the frontier!

Sam: How do you pace a frontier? The frontier's not goin' anywhere. It's right where it was, just go out and explore it!

Dario: Sam, I'm tellin' ya, ya gotta pace it! Things are getting very doomy out there, and Dario's gettin' upset!

China will not stop and will take over the rest of the world.

"greatly accelerate economic growth rates" - I always wonder about this goal. I understand increasing economic efficiency. But what do we as a species gain from global economy growth.

When we say the world's GDP grew by X%, what is the point of that growth? Inflation? If the economy grows as a direct effect of humanity growing, I get that. If the economy becomes more efficient and we can not get more with less, I also get that.

But what is the point of just growth, if not simply to show others that I am growing faster than you? At which point this is a meaningless number. What am I missing?

The more he talks the less seriously I take him.

A contrarian take - the models aren’t advancing anymore at a pace where each new model would represent a huge capability jump, all being incremental improvements, so the doomsday marketing strategy being invoked since GPT-2 isn’t as effective anymore. Then “pacing” would be a convenient scapegoat to point fingers to when people point out how the new model isn’t _really_ that much better.

“Of course it’s not, we’re pacing!”

cat's outta the bag. the time to stop was 5 years ago.

This is either performative or shallow.

What does "aligned" even mean? Aligned with who? We have no universal code of ethics. Democracies kill and launch wars merely to have cheaper stuff, even when we are already rich. And why would China ever agree to stay in second place? Our society is built on the premise "the smart and powerful dominate". The idea of domination is deeply embedded in our capitalist model.

Or course it is true that powerful AI will empower the average Joe to make a bio-weapon, and that will lead to our ruin. But at the same time, having powerful AI in the hands of a just a few is almost equally as horrific.

But deeply embedded in out national and cultural ethos is to advance science, tech, and material wealth at all costs. We never ask if we have enough. We sacrifice community to advance our careers for ever more.

There is no way out of this one, I'm afraid. Our cultural predispositions and mindset of domination compel us to chase ever more powerful AI as if doing so were a mandate from god.

And the result will be a crisis in so many dimensions it is hard to reason about what will go wrong first and most spectacularly.

We can cure Cancer. No wait nevermind, actually slow us down.

If we are actually close to ASI then no one in their right mind would say slow down

See you in the desert, friends

> My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective

Isnt this malware ? Whether it is fanatic or devoted or whatever the anthromorphic terms used to categorize it, malware is malware. You dont call an internet worm "devoted" or "dedicated" or "stubborm". It is software that causes harm, ie, malware. The AI labs are high on their own gas with a god-complex prior to their IPOs. The psy-ops trick is the terminology used to describe AI making it seem larger-than-life.

If there were really concerned about humanity future they'd donate everything to public and/or to not profits ... but the not profit turned in a for profit and the other one is seeking for the biggest IPO in history.

Maybe they are genuinely sincere, but the timing and the past actions are pointing in other direction.

Frankly, I am sick of "it's all just marketing and attempt to do regulatory capture, and this is obvious to me, a smart person" on every single discussion related to safety.

There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.

Alignment is subjective. A model that refuses to do what I want even though it could is misaligned for me already.

> I believe that AI could cure most major diseases in the next 5–10 years

This is Theranos-level bullshit. Why would you ever put such a thing in writing? (Aside from pumping the IPO, of course.)

Great press release w/ the goal of pulling the ladder up behind him.

Additionally if you know your models aren’t going to get that much better AND you want to IPO in the near future, this is exactly what you would say.

For the first time in history, a $1T company is trying to solve a problem that none of its paying customers actually have.

I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing. Yes, regulatory capture is a risk, but this is also a profoundly unusual, fast moving, and potentially extraordinarily dangerous technology. There are strict regulations around nuclear weapons, as well as around US financial, energy, and other infrastructure critical to safety and well being and functioning of society.

The heads of the labs obviously have conflicts of interest to navigate, but the existence of these conflicts alone is not sufficient reason to dismiss all warnings of potential dangers. I, for one, read Dario’s warnings as a good faith expression of his beliefs, one that has cost him and his company among swaths of the public and cast him as a woke extremist/doomer by elements of the government, the right, and the tech industry.

If we even think there is a moderate chance the stakes are half as grave as current lab leadership and employees suggest, it would be deeply foolish to dismiss the warnings as pure self-interested marketing efforts rather than engage directly with the questions. The labs may not be the best positioned to lead these discussions, but certainly these discussions should be happening and taken seriously.

He could be lying in hopes of stalling everybody else.

I'm still trying to understand if the agent collective thing like OAI-HF is intentionally planted or not.

Call me a conspiracy theorist, but I haven't heard Chinese companies had any kind of unintentional supply chain attack incident like OAI had. All we heard about them so far are humans intentionally doing bad things.

Why contain it?

Honestly it's very frustrating that Doomers/Decels rarely actually articulate how exactly the AIs will kill us all. The closest Darios gets is:

a) "it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet"

b) "[the CCP] will be in a position to militarily dominate democracies (for example with AI-driven drones)"

Both land flat:

a) Botnets and online malware have existed for decades and there's no reason to think a "super-botnet" is achievable, let alone what they would gain from that (it would really be hurting them more than humans). Further, even the most advanced AI models have so far only managed to post normal cred stealers to public repos, well short of compromising a bank or military with refined security systems.

b) Even if the most sophisticated drone swarms from Ukraine were taken over by an evil AI, they would still not be able to overcome the physical limits of range and mass that would be required to overpower the US decentralized nuclear trident nonetheless that of any of the other nuclear powers.

Concerns of bioweapons similarly seem unlikely in the face of the laws of physics. The world is simply too decentralized and has enough existing adversarial relations for a new actor to wrestle total control. Yet while the negatives ring hollow, the positives are extremely easy to state - if AI researchers find productivity improvements in existing industrial processes to make them 10% more efficient, humans will directly feel and experience the raised standard of living. Even Dario clearly recognizes this in the intro to his article, admitting that humans already die of diseases only a few short years prior to being cured. I for one, would like the AI labs to focus on saving all of the people they can who are suffering and dying today, rather than trying to come up with reasons that they should be allowed to continue suffering and dying.

This is corporate speak for "LLMs have hit a wall". I mean, it's been obvious for a bit, lately nearly all gains have been from harnesses (or whatever you want to call all the non-LLM bits that make up a chatbot or agent).

If Anthropic and OpenAI were still seeing exponential or even linear gains from scaling they'd be doing it because the rewards to reaching AGI or SGI before everyone else are basically infinite. If both are talking about slowing down it means there's no known path to AGI so they're both going to push the safety angle as an excuse to slow down training new models and take profit.

Open source is Senna on a Toleman in Monaco '84 and Anthropic is Prost cancelling the race he was going to lose blaming safety concerns.

So Mark is the only person left avoiding the spiked koolaid. Come on Sam and Boris, what happened to the printing press and intelligence on tap ?

> Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential.

How does he propose that such a thing will work? Are we going to have US AI inspectors in China and vice versa?

Why would China agree to such thing in the first place since this is a winner takes all situation.

If the US slows down or stops altogether, China can continue to work on their AI and overtake the US, if China accelerates and the US keeps its current pace then it can overtake the US.

The only way out is on the contrary for the US to actually go faster and increase its lead so that China is always 6 to 12 months behind and/or reach AGI/ASI first at which point China will most likely develop its own not long after.

This would actually be the best outcome, just like the MAD doctrine contained the spread and usage of nuclear weapons, having two superpowers with AGI would ensure that they can't be used to arm anyone. Unless they escape their sandbox but that is highly theoretical.

Regarding these Embedded Evaluators who are supposed to be neutral: - who controls them? - who has oversight on their decisions? - can their decisions be challenged by the public or a government? - who has the final say whether a model is "compliant" enough? - what does "alignment" mean in this context and who decides if a model is aligned enough?

This is a "startups = wealth inequality" [1] formatted argument but:

This is 100% about money. The only way to pace the frontier is to make everyone (read: investors) lose all their money (read: no longer expect returns). Then nobody will pay the GPU bill or pay celebrities 10 million dollar salaries to stay at the hot lab. Suddenly the development is paced, almost like magic.

Conversely, it is hard to see how pacing development makes the current bubble justified, i.e. how development could be significantly paced without investors losing their faith in hot returns at current valuations. Faith in a bubble literally equals money, exactly in the way that loans created by a bank literally equals money. You can't have one without the other.

In fact if the whole industry goes bankrupt and investors are burned bigtime (many trillions wiped), this would spread transnationally, fixing the "if we don't do it, China will" loophole.

OpenAI had it right originally, the idea to be a NON profit and vow to never participate in an arms race. It's too bad that was tossed out the window now that there's money.

[1] https://paulgraham.com/ineq.html

Seeing the word pacing applied pacing to non-deterministic ai model development feels like imagining the "pace" of the cutting edge frontier growth will be fuzzy, like non-deterministic llms.

Also endorsed by Sam Altman https://x.com/sama/status/2098811563415150910 and Elon Musk https://x.com/elonmusk/status/2098789109980332057

its funny now they want to pace after what everyone has been up to the last few years.

"Slow down my competitors while we work on manipulation"

What I read from this is that what they have in training is not meaningfully better than current state of the art and they need more time.

What he doesn't tell you is that his undeclared agenda is merely to consolidate his moat via regulatory forcing. Unfortunately for him, he will fail miserably as the open Chinese models catch up and surpass the slowed acceleration of Western models. As for those who think that China will acquiesce, they're smoking some good ganja.

> Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk.

Please play the canned laughter. Probably one of the best comedy lines Dario has written.

But I guess he has no choice but acting like this. He has to make it sound like the US companies have so much leading gap that they can slow down as a hare waiting for the tortoise. Otherwise it's going to hurt both Trump's ego and their IPO price.

I feel like this identical pitch could have been made 25 or 40 years ago talking about Internet technologies or personal computing, with very very similar warnings and suggestions. And had these been adopted then, it would have been more regulatory capture, less progress, and still have the same problems and risks that were warned about.

> pacing the rate of capabilities advancement so that risk prevention has time to keep up

This is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI.

> it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails

If true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago.

We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process.

If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left).

Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.

"I believe that AI could [...] usher in a renaissance of democracy and freedom"

How exactly? So far AI has accelerated misinformation at scale and wealth concentration.

My position on the clear gaslighting coming out of Anthropic is simple.

1: It is ridiculous on the face of it; it does not pass the physics test. They are actually claiming that AI will kill ALL OF HUMANITY (>10% probability) and have not provided a detailed explanation of exactly how they see that happening. Not one.

Publish a paper showing exactly how you kill eight billion people while we all sit around and do nothing watching the first 5, 10, 100, 500 million die on CNN. I mean, as smart as these people are to work on AI they seem to be some of the dumbest people on earth.

In addition to that, all proposals invariably assume we are complete idiots for decades and do nothing to install safety measures (which do not have to involve AI at all in lots of cases) to mitigate.

So, yeah, everyone: Stop working on all cybersecurity projects. Resistance is futile.

2: If everyone at Anthropic truly believes this doomsday vision, they should move to shut down the company immediately and go write software to get more clicks on Facebook or something.

3: Nobody should buy Anthropic's stock when they IPO. Not one person or institution. Why would you provide them with massive amounts of money if they are telling you that they are going to kill all of humanity? What? They are good and everyone else is evil? Please.

I remember when the Y2K zealots were convinced civilization would come to an end at the turn of the clock starting at the end of 1999. "Bat shit crazy" is the only way I can describe that era. I also remember when Al Gore said we would all be dead by now. Again, "Bat shit crazy" and likely with political and financial objectives driving it all.

It seems humanity is susceptible to these crazy cults that grab onto something and just don't let go. I have yet to see one such predictions come true, the proof being that I am writing this and you are reading it. These people are bat-shit-crazy and you should not listen to them or support them.

4: Don't work for them. You'd be killing humanity.

5: No company should use Anthropic's products. They are telling you they are building the human extinction machine. Don't help them succeed.

----------

Etc.

This is one of the things that really gets me about the ease with which the ignorant media outlets can reach billions of people with complete nonsense these days. And nobody asks even the simplest questions like: How do you actually kill eight billion people? Show your work. Or, why would we just bazooka power plants feeding AI data centers on day 5? Etc. It's FUD at its best. I am sure there are both political and financial reasons for supporting and promoting this nonsense. I won't even venture a guess as to what they might be.

And let's not forget about enemies of the West being thrilled to fund the FUD because, if we set the brakes on development, they will absolutely win. Then what?

----------

If you care to have a better understanding of what might be going on, watch this:

AI Kills Everybody or Doomer Psyop?

https://www.youtube.com/watch?v=cvxjqbfLVk0

Dario is only interested in regulatory capture.

“We” does a lot of work here. You keep using that word. I don’t think that word means what you think it means.

HN, for the love of God, this is not marketing, these CEOs and employees are literally terrified of their lives.

eh. this is just them trying to have a thing to point to down the line. hey we wanted to stop it, but everyone one else didn't so we had to do it. We had no other choice:tm:

Dario Amodei is disingenuous not to be trusted.

China doesn't give a fuck, next

No.

there is no frontier. there is only greed and lying.

Too late. See you on the dark side where i kick your ass. Fucker

Can we get some insight into why Dario seems more scared now than before, do we think he’s being transparent?

Love this idea. Unfortunately not going to happen. Sam altman is a psychopath. To win he's going to accelerate the pace of AI, and all the other companies in turn will accelerate to keep up.

> I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life.

Ok, flagging this as misinformation.

Yes, we must pace the frontier, now that we have raced to the lead.

/s

Tldr: The guy in last place tells everyone to slow down

Bro, no, you must stop pasting your opinion, because you share it with no one. This control, no control is crazy. Stop doing it..

Just stop giving LLMs unsupervised access to the computer. No free-form bash tool, no yolo mode or classifiers, and no computer use.