It's Time to Investigate the AI Labs (calnewport.com)
118 points by ibobev 2 hours ago
jimmyjazz14 24 minutes ago
> We must move past vague discussions of “AI” and instead isolate the specific types of systems that are creating problems.
This is absolutely correct and its surprising how rarely you see commenters in the media push to go into the specifics on this. AI is just matrix math and its what you connect that math to that matters, we should talk a little more about what we are willing to connect AI to and little less about how scary or capable the models appear.
Animats 5 minutes ago
Yes.
A good start might be removing virology info above the undergraduate level from training sets. The main "kills everybody" threat involves biological viruses.
AI-driven hacking is a problem, but it's really just nation-state level hacking available to smaller companies or wealthy individuals. Everything needs to be toughened up to the point that doesn't work. Ask countries that have been up against Russian hacking for years, such as Estonia.
Beyond that, most of the threats are not that serious. Below the level of organized crime or corrupt government.
Animats 11 minutes ago
This is a wrong-headed attempt to regulate AI. The real problems are different.
AI systems, especially multi-agent ones that can do things, are more akin to corporations, than individuals. When you read the logs from the Hugging Face incident, you're seeing something that looks a lot like internal corporate emails. Various units of the organization are arguing over what to do and who does what. They eventually converge and get the job done, breaking the rules at times. This is normal corporate behavior.
When agentic AIs do something outside their own internal world, they do it by engaging in transactions with external systems and people. As yet, few have robots to do their bidding. So, again, this is normal corporate behavior.
A corporation is a goal-seeking system. In theory, if you agree with Milton Friedman, its sole purpose is to maximize shareholder value. Internally, within the company, there are subgoals, which exist to support the top level goal. That, too, is what multi-agent AIs do.
The uncomfortable place this thinking leads is that AI regulation and corporate regulation are very similar. That's not something the political part of the world wants to think about too hard. It would mean putting more constraints on corporate power.
Yet that can't be ignored, because we're likely to see corporations where some parts are AI and some parts are human. That's already been done a few times as a demo, not too successfully. Yet. It's going to hit hard when an AI-run company outperforms a human one.
psyklic 16 minutes ago
Why aren't they running agents on isolated computers without Internet access? This way, breaking free of containers would not matter.
It is truly a security nightmare that so many people are have given agents root access to their entire computers, in addition to presumably very private personal information.
malisper 12 minutes ago
> Why aren't they running agents on isolated computers without Internet access?
They probably will soon, but even air-gapping may not be enough. See stuxnet for instance
sharadov 8 minutes ago
Private enterprise as it's being run in the US especially under the Trump administration ( where a few billions tossed at the right people ) will get you all kinds of favors and look asides.
Hopefully once the dems win back the house, we will see some regulation.
Up until now it's been a capitalists' wet dream!
The labs have become so brazen especially OpenAi that they don't even care to report incidents. https://www.politico.com/news/2026/09/25/rogue-openai-agents...
uxcolumbo an hour ago
100%. First stealing content from creators and pirating TBs of books. And now this. Baffles my mind that these labs can just get away with their 'rogue' agents trying to hack into other systems.
Imagine if a human did that. FBI would be knocking on their door.
Why are there zero consequences for these labs?
andsoitis an hour ago
> Why are there zero consequences for these labs?
Because they are seen as at the forefront of the next revolution in society and they’re not adversarial towards the US government.
Bluntly: anticipated net huge positive for the US as a whole.
popalchemist an hour ago
not adversarial in obvious ways
but they are literally trying to build a superpower that exceeds the impact of the atomic bomb in an extragovernmental manner.
andy99 25 minutes ago
esseph 37 minutes ago
CamperBob2 39 minutes ago
Animats 29 minutes ago
The argument for copyright violation is very weak. Yes, AI training reads much copyrighted material. So do researchers of all types. That's why there are academic libraries. A copyright violation only exists if what comes out is a close match to what went in. That's happened, but it was considered a bug and was fixed a year or two ago.
oersted 21 minutes ago
Most academic libraries are notoriously protective of their IP and researchers are definitely in copyright violation if they don’t pay significant fees for access.
Of course that is terrible, but it is one of the worst examples you could have given to make your point.
pessimizer 2 minutes ago
No, the copyright violation is so obvious that people arguing against the object have to spontaneously assert bizarre qualifications for copyright violation that have never existed until LLM companies wanted to freely violate copyrights. Just about a decade ago, people were getting sued for "look and feel." "Blurred Lines" lost a copyright suit for reminding people of a song by another artist.
The metaphor pretended to be reality of computers "learning" being the same as human learning is their last resort, and it is obviously silly. Computers are not human or animal, and learning is something that humans and animals do. If computers are human, then copying a file verbatim to a computer is learning. It's not even worth acknowledging - learning is a metaphor for training LLMs. When I say "you" to an LLM, I'm not referring to anyone, I'm dealing with a UI.
I don't doubt that a lot of AI people have internalized this silliness, which is how they can humor fantasies of how a bunch of programs on different computers explicitly evoked to do particular things might be alive because they can talk with it. I can't talk with my dog, and my dog is alive - so why should having a quality that my dog is incapable of be proof of life? You can certainly conceive of AI that it would be very difficult to say for sure isn't alive, and this certainly is not it. LLMs learn like books speak.
popalchemist an hour ago
It's past time.