Frontier Labs Are Selling Garbage to Fools in Washington (deadneurons.substack.com)

75 points by nr378 2 hours ago

pliny 44 minutes ago

This is an AI written post and the details are wrong (the description of the HF incident as involving Irregular is wrong and the description of the incident as only involving stealing public credentials is wrong, per the technical report the agents got access to internal HF infrastructure).

nr378 28 minutes ago

Please see below, one detail was incorrect and has been acknowledged and amended.

pliny 15 minutes ago

Your description of the HF attack as being merely "the elite task of discovering 14 Hugging Face API tokens that careless developers had committed to public GitHub repositories" does not match the description in the technical report[1].

[1] https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78... - page 9

nr378 6 minutes ago

DalasNoin an hour ago

"Every single one of these catastrophic breakouts happened inside the testing environments of the exact same vendor."

This is incorrect, the HF incident for example (the most well known) had nothing to do with irregular. I know there has been a news site pushing inaccurate articles (effort.news) on this topic but these are the facts.

https://openai.com/index/hugging-face-incident-and-the-road-...

nr378 an hour ago

Thank you, you're correct. Effort.news was one of my research sources, but you're right that although OpenAI use Irregular, they were not involved in the specific HF incident (although the failure mode was otherwise identical). I've updated the post to make that clear.

aesthesia 4 minutes ago

The failure mode was _not_ identical. The HF incident agents were not directly connected to the internet and had to compromise an internal package registry in order to access the internet.

kalkin an hour ago

As of writing it still says:

> For Anthropic, Google, and Meta, the catastrophic breakouts happened inside the testing environments of the exact same contractor.

If this is the level of understanding you have of the relevant incidents, there's a lot of chutzpah in saying that other people are "selling garbage", carrying out an "extraordinary confidence trick", etc.

nr378 32 minutes ago

DalasNoin 13 minutes ago

thank you for this reasonable reaction

hackernews682 2 hours ago

Politicians aren’t “gullible”. They know the game.

skeledrew 24 minutes ago

Let them cry, I don't see anything changing unless they can somehow get China to agree. And I doubt China will drink any of that kool aid especially while they're being disadvantaged by export controls, so the ever-improving open weight models will continue to rain. This is something the US Big Tech oligarchs will NOT win.

anigbrowl 2 hours ago

Agreed, but they're getting paid with our money.

jgalt212 2 hours ago

There is really is no excuse for Washington here. Even for the layperson+, it doesn't take much use of an LLM to figure out where they produce good and usable results and where it's of specious of value.

+ every layperson is an expert in 1 or more areas.

tracerbulletx 27 minutes ago

I mean the military was buying dowsing rods as bomb detectors not that long ago so I don't have a ton of faith in them not being hoodwinked.

iLoveOncall an hour ago

> + every layperson is an expert in 1 or more areas.

What do you think 70 years old career politicians are an expert in that would allow them to weigh whether a chatbot's answers are bullshit or not.

sublinear an hour ago

I thought the whole point of this discussion is that they don't have to care if it works. All that matters is that the majority believes it does, or at least doesn't make a fuss about the continued spending.

That's their expertise.

randallsquared an hour ago

> every layperson is an expert in 1 or more areas.

This isn't even remotely true, unless you're willing to go as far as "being this person is an area of expertise".

basch an hour ago

I dont have to be an expert to recognize when something is said with authority and confidence. Any time a model says something with conviction, my instinct is to double check.

Now if they taught them to express uncertainty and speak in terms of probability, I might be more likely to be blindly fooled by some kind of uncertain conviction.

sublinear an hour ago

Neither of you is wrong? Every lived experience does produce a unique expertise, not in "being" that person, but navigating their experiences. It would be very unusual that all of someone's experiences are completely worthless.

I suppose you've never had a unique perspective on something? Maybe that reflects on your self esteem? How unfortunate.

bigstrat2003 an hour ago

That is the entire business model of AI companies: selling garbage to people foolish enough to buy the hype.