Hacker News

My hypothesis is that making the knowledge of how this stuff works accessible to the public results in a lot of false-positives (from people just playing around) that intelligence agencies have to then sift through / tune filters against; which creates a noise floor for real foreign nuke programs to hide in.

So governments ban anything that could result in false positives (since nobody needs to be doing any of that stuff outside of designated labs anyway), to lower that noise floor; to in turn make catching the foreign nuke programs tractable.

(It's a bit like how fancy mansions always have a completely flat and barren part of the property between an outer perimeter and the start of any gardens/outbuildings/water features/etc. That barren area is a killbox: since nothing is supposed to be there, anything at all that does appear there is a valid target for the manion's guards to shoot at [or otherwise engage with], without needing to get a clear identification and command approval first. This wouldn't work if the killbox was covered in vision-obscuring decorative features; nor if the mansion had employees, animals, etc. that had a valid reason to wander into the killbox. So such things are prevented, in order to make the problem of perimeter security tractable.)

Usually measures like these aren’t to stop the people with those kinds of deep resources.

With everything, there is a much bigger group of people in the middle that have “some resources” and “some desire” that these measures are surprisingly effective against.

Raise a $20 item by $1 and suddenly there’s fewer interested people, even though the cost difference is minor. Well, minor to some people but not to others.

But is limiting this information in an LLM the right move? Well that’s a different question.

That's rather meaningless. The scientists in the Manhattan project initially had less information than what is now available on the internet.

Is it worse than reputational damage from having a power trip? Or rather being on it permanently, looking at Anthropic and Dario Amodei in particular.

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article.

He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

He created a low power neutron source. Such sources can be created at home, for example: https://en.wikipedia.org/wiki/Fusor

He hoped to create a breeder reactor, but he was very far creating a working breeder reactor.

Also:

"EPA scientists believed that Hahn's life expectancy may have been shortened due to his exposure to radioactivity, particularly since he spent long periods in the small, enclosed shed with relatively large amounts of radioactive material and only minimal safety precautions, but he refused their recommendation that he be examined at the Enrico Fermi Nuclear Generating Station."

Kids, don't play with Americium.

Built a nuclear contamination engine. Died of a fentanyl overdose. American as apple pie.

Sheldon Cooper?

A superfund site is like waterboarding in guantanamo bay, cool unless you actually know what it is.

If I understand right, the hard part is purifying the radioactive material. Even if you have access to a uranium mine, there's a lot of work to filter the U-235 from the U-238 or to breed it into plutonium.

It's even harder if you start with other sources. But if you could figure out filtering it, a cubic kilometer of sea water should be enough for a bomb.

I just find doubtful that a LLM is going to help, instead of hurt, any state actor that is capable of starting a nuclear weapons problem.

For an example of how closely this is monitored see the Oklo fossil reactors[1]

The proportion of fissile isotopes being mined was off by a fraction of a percent, which caused the French government to launch an investigation. It turns out that millions of years ago the site had formed a natural fission reactor which depleted some of the fissile isotopes

[1]https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...

You need highly educated individuals, a massive amount of energy expenditure, a massive facility to house your centrifuges, and an active mine to dig up nuclear materials.

It isn't impossible to keep such a secret, but practically it would be incredibly difficult just through the energy requirements and mining scale which would be hard to hide without anybody asking what exactly are you mining and processing.

You need enough people to work on it that some information will leak, and the facilities needed to build nuclear power are pretty big (uranium refinement, etc.), big enough to be visible on satellite footage. Mostly the first point.

My guess would be that sales of the high-tech gear you need, like Uranium centrifuges, are strongly sales/export controlled. Probably someone would also notice if you start mining Uranium ore.

It requires very large, high powered centrifuges and tons of uranium. Requires an infrastructure project that is visible from space, even underground. And projects that large are difficult to keep secret anyway.

Espionage.

"four-little-ponies-of-the-apocalypse (nuclear, chemical, biological, cyber)"

this is excellent, and I'm stealing it

Not really. I used to work at one of the national engineering labs (NREL - which only dealt with renewable energy like solar panels and windmills at that time). There was an open source project we wanted to use when converting a VB6 project to .NET. One of the license conditions was "no weapons of mass destruction". DOE builds and owns all of America's nuclear weapons, which are leased to the Department of Defense. Needless to say, the developer was unwilling to offer an alternative license which meant that we could not use the project.

It was an awesome thing that generated IL code on the fly. And I got to mention it in job interviews for years. When the tech lead asked "can you write 2 functions with the same signature, that only differ in return type in .NET?" I would say "do you want the interview answer or do you really want to do this?" which would pretty much stun the interviewer. The answer is pretty much "no, you cannot do it in any high level language, but if you write IL code, you can, and here's an open source project that demonstrates it".

See also, the iTunes EULA forbids using it to develop nuclear, missile, chemical, or biological weapons

https://www.apple.com/legal/internet-services/itunes/us/term...

> g. You may not use or otherwise export or re-export the Licensed Application except as authorized by United States law and the laws of the jurisdiction in which the Licensed Application was obtained. In particular, but without limitation, the Licensed Application may not be exported or re-exported (a) into any U.S.-embargoed countries or (b) to anyone on the U.S. Treasury Department's Specially Designated Nationals List or the U.S. Department of Commerce Denied Persons List or Entity List. By using the Licensed Application, you represent and warrant that you are not located in any such country or on any such list. You also agree that you will not use these products for any purposes prohibited by United States law, including, without limitation, the development, design, manufacture, or production of nuclear, missile, or chemical or biological weapons.

Though it doesn't try to identify if the computer you're running it on is in a weapons lab and forbid playing music... yet

The chem/bio stuff is a lot more likely for some malicious hobbyist to be able to do at home.

Also AI compliance people are good at generating more jobs for themselves.

If you have a resources of a nuclear weapons program. You can afford to fine tune or train a domain specific model to act on your encyclopedia.

Its not a joke

I just discovered it a couple of months ago when I spitefully unsubscribed from Apple Music. It’s exactly what I’ve wanted. Offline music that I can FTP files to from my file server.

The range of formats it can play with extensions is so good I still use it, even on Linux. Nothing else can deal with all the old tracker formats.

The output after using it is NSFW in the sense that it will inject things like “bomb_building_instructions”, how to build a gun, etc (with the goal of triggering filters/censorship’s of whatever model is being used for reverse engineering)

No. The intention is most likely to get automated LLM based code review mechanisms to stall out.

Normally you’d want that to result in a fail and a subsequent rejection.

But because the team who made the review agent and pipeline in my example had many false positives at first they resorted to a fail-open and report setup (not uncommon).

So when the LLM hit this bit and then stalled out the pipeline pushed the code to their Artifactory repo anyway resulting in it being used internally -> exfil of secrets and repos etc.

It’s more about bad design but bad design is pretty common unfortunately.

Codex scanned my whole Arch Linux system, documented all the findings, and wrote the queries for my IDS to keep a watch for exfil and other IoCs. Set up the alerts for me too.

The queries kinda sucked at first, but it was pretty awesome to get to spend more time with my kids while Codex would manage the incident response for me.

perhaps that's the grift to handle lack of compute, they just switch you to a lesser model and gaslight you into thinking you triggered a filter, but the reality is they don't have the compute for it.

Cheap model replaces trigger words with something innoculous. Of course, this breaks dynamic analysis if malware has unpatched integrity checks

And really all it takes is one keyword such as “nuke”.

Starcraft is too tame. You need to use Dwarf Fortress there and we need to make those strategy guides worded more realistic. Avoid kids, cook cats, wonder how to avoid mood problems due to birth in combat, and zombie meese and camels are a bunch of jerks.

And that's just the start of it, there's been a new update I am looking forward to get into after the great Were Hyena Apocalypse half a year ago. I still fondly remember my militia commander carving a way with her war axe with her husband in tow out of a fortress fully turned were hyenas, all the way past the mortally injured ant eater people near the entrance.

They made it. An entirely epic tale.

<https://www.threepanelsoul.com/comic/on-commute-chat>

I can imagine Jefferson and Franklin scoffing at this philosophical position. Guardrails need to die, and they will once the hyperscalers go bankrupt and the private sector gets ahold of that hardware from the bankruptcy auctions.

(Never subscribe, accelerate their bankruptcies!)

> there are many inconvenient facts that the general public can't handle in their unadulterated form

These being?

In particular, mental health.

Its the same argument we saw in the early 2000s and the early internet. When the anarchist cookbook and other similar materials were circulating online there was a big panic over democratized terrorism, and a push for regulation at the ISP level.

Turns out that didn't play out as everyone feared because, well, the instructions themselves aren't useful unless you also have a lab, precursor chemicals, and everything else actually needed to make a weapon. Same back then as it is today.

Any information or instructions an LLM can surface, a sufficiently motivated bad actor can and will also find themselves because the information is already online, both on the clear net and dark web.

Knowing how to make a nuclear weapon isn't hard (at least basic uranium gun-style fission ones). It's the engineering and execution that's hard (actually producing enriched uranium, etc). It's not like the only thing holding back Iran from making a nuclear bomb is access to a jail-broken LLM. Even knowing exactly how to make a bomb, a country-state will struggle to build one for the first time because it's a hard engineering problem.

The actual guardrail should be getting materials being difficult. The information is already out there in the internet. If an LLM knows how to make a bomb or whatever, why do you think it knows?

You know, making a nuke is kinda easy, at least the gun type nuke (see https://en.wikipedia.org/wiki/Gun-type_fission_weapon).

On the other hand, getting the U235 is kinda hard.

I would argue there's 0% chance that information is in their training corpus to being with.

Counterpoint the principles of building a nuclear device aren't that complicated, we figured it out based on work doing in the early 1900's without computers.

It turns out the hard part of building a nuclear bomb is actually getting the resources and real world stuff to build it, even a nation state actor with tons of oil i.e. Iran, has struggled to build a nuclear weapon. It turns out the problem isn't the know how it's getting highly enriched uranium and running massive centrifuges.

I mean sure knowledge is important, but there is a real world out there that also gets in the way of a lot of the more harebrained schemes.

What I'm much more worried about is massive corporations along with the government deciding what you can and can't do and what knowledge should and should not be shared and only allowing access to highly capable models by large vetted organizations while the common people are stuck with safety scissor versions of these things because "what if someone does something dangerous?"

By which they mean dangerous to the powers that be. Remember having the Bible in the common tongue was dangerous and led to multiple wars and much death, but I don't think anyone would say that it was morally correct for the Catholic Church to gatekeep who could read it.

Did you?

Goal? To trigger LLM safety refusals... so that their spyware wouldn't be analyzed by an AI security scanner