NSA Internal Network Down (reuters.com)

201 points by nodoodles 3 hours ago

lucascdotnet 2 hours ago

I wonder (and have no idea) whether the increased reports of services being down is a case of the Baader–Meinhof phenomenon, or a actual increase in failure rate.

btown an hour ago

If I was asked to armchair-pattern-match, I'd say that there's a combination of two things: (a) a wider fear-based environment where rapid modernization is being pushed heavily, and (b) an increase in tools that give false confidence on change management.

AI is tuned to speak confidently and ship independently; it is rare that it will suggest that one go back to a broad survey of stakeholders to ensure no Chesterton's Fence is being torn down (https://fs.blog/chestertons-fence/). "I've read the documentation and logs, and nothing suggests this system needs these legacy capabilities/components/capacity" is something we hear all day, and it is a siren's song if there ever was one.

Couple that with a general (and valid!) consensus that modernization of legacy systems must be accelerated for a variety of reasons - to mitigate this new and continuing "Black October" of hyper-agile agent-driven cyberattacks (whether the network is public or private); to have more flexible infrastructure for new features to stay ahead; to allow human teams to show their efficacy in a world where technology jobs are increasingly at risk - and you have a perfect storm where change management is no longer what it once was.

joe_the_user 20 minutes ago

I don't necessarily agree about the warnings. AI often gives me "now be careful" warnings about it's output along with the caveat at the bottom of each ChatGpt page.

I suspect the situation is that AI's caveats are just easier to ignore than a person's warnings. Plus all the other factors you list. Also, AI tends to give a person the kind of communication they ask for. I always ask for options but I suspect other people ask for just answer and so they get them.

cyanydeez 26 minutes ago

I think he's just pointing out that AI has made news of IT failures "more interesting" and thus, more reports are getting out where in the past, they'd not be interesting.

I think you went off somewhere else.

Someone1234 an hour ago

That's my impression; a total of zero of the popular recent ones have been related to "AI" or Cyber attacks, but every comment in every thread, talks about those two as if they were the underlying cause.

The FAA's recent outage for example, at its core, seems like people trying to rush needed upgrades that were slow for a specific reason (e.g. reducing the testing window for 90 to 30-days).

The Microsoft outage seems like three different things linked together (and poor "full failure" recovery prep, lengthening the outage).

The bank "outage" may not have even existed, 2/3 of the banks claimed they didn't have an outage at all, and the other said it was a limited technical issue. The articles were based on DownDetector spikes.

rkozik1989 an hour ago

You need to be more skeptical of what organizations say when they have a lot to lose. Nobody takes responsibility for situations unless there's way to offload some or all of the blame. They have every incentive to lie about the situation to make it go away. Politics often plays the biggest role in these situations.

StilesCrisis an hour ago

Someone1234 27 minutes ago

swasheck 38 minutes ago

> The FAA's recent outage for example, at its core, seems like people trying to rush needed upgrades that were slow for a specific reason (e.g. reducing the testing window for 90 to 30-days).

The explanation for this one still doesn't ring true for me and that's exacerbated by how many times the story has changed after telling the public that they didn't need to know the reasons.

Agree, though, that the rest do have reasonable explanations

egypturnash 20 minutes ago

Given how many companies seem to be working hard to have all programming work done by AI, and pushing out everyone who says this sounds like a bad idea, is "AI" really not going to be a significant part of "rushing out needed updates"?

Is "Claude says it looks good so let's skip the usual testing window and push to prod on a Friday" really not a sentence you can imagine being said in this current environment?

Someone1234 9 minutes ago

kesslern an hour ago

This situtation reminds me of the East Palestine train derailment in Ohio. A bad train accident caused a big chemical spill, made national news. For months afterward, any issue with any train in the state was reported as news. People thought the trains were breaking down. In reality, train derailments happen pretty often but it's not usually a big deal so nobody hears about it.

mmooss a minute ago

Someone this morning told me there might be a 3 month blackout in the US, somehow connected to other recent technological failures. I later learned the blackout theory is based on a The Simpsons episode, and even there it was a much shorter time period but has been exaggerated.

So certainly there is a big social-psychological phenomenon. Or it's gonna be a cold winter! :)

NitpickLawyer 2 hours ago

100%. Everything is rage-bait nowadays. And AI is the newest toy they can use. Every time we've had something go down in the past ~6months, "AI did it" was all over the comments.

rsoto2 2 hours ago

https://www.optimizely.com/field-notes/articles/how-we-use-a...

https://www.bleepingcomputer.com/news/security/optimizely-su...

Is it possible that the false of productivity is leading to worse products and massive loss in valuable headcount? Instead of AIs directly writing bad buggy code? Or is "everything rage-bait, and AI is all sunshine"

ragequittah an hour ago

cochne an hour ago

wolfinterval an hour ago

matt3210 an hour ago

what was your system prompt again?

burnte 15 minutes ago

In the case of government, we have an administration that works on whims and guesses, and moves too fast, breaking a lot of things. We have absolutely seen more human-caused-issues.

kilroy123 an hour ago

Possible, but let me ask you and others. What does your gut say? Mine is screaming that something just isn't right right now.

I can't put my finger on it, but something sinister feels afoot.

eterm an hour ago

UK government recently launched a "Be Ready" preparedness campaign.

https://beready.campaign.gov.uk/

It's nowhere near the kind of readiness and preparedness campaigns of the cold-war of course, but it's a step up in alertness that's higher than anything I can remember.

rwmj 41 minutes ago

kilroy123 41 minutes ago

scottyah 19 minutes ago

My gut is screaming that everything is mostly normal, but the kids aren't alright.

Too many people are addicted to their smartphones and don't feel the same level of pride they used to in their work. All systems are shared, instead of one person being the ruler of a certain part of the stack, it's a revolving door and it's rare that anyone stays somewhere for even 5-10yrs. It leaves a wake of modifications that nobody else can keep up with. Also, the tech industry is moving fast just to move fast, a lot of old infra is still C++ and COBOL but good luck hiring anyone to use that, so they have to convert the systems to use the Latest Tech of the Day. The old, stable people don't have any interest in learning the new thing, and would rather retire with their knowledge. The new people want to rebuild from scratch in the New Thing and are convinced they need to job hop immediately after.

trinsic2 37 minutes ago

Yeah, agreed. I look at patterns. When there are long term outages across multiple disciplines, something systemic is being changed for good or ill. In this case, it being right around the midterms, I'm thinking its ill. Organizations (not sure which, Peter Thiel?) are trying to push us into a total surveillance system, I'm thinking some of these things are related to that.

digitaltrees 27 minutes ago

Like maybe we are at war with a country that doesnt have the ability to fight with conventional methods and is engaged in cyber warfare?

0cf8612b2e1e an hour ago

My personal barometer: when do government insiders sell all of their stock? They have already done all of the insider trading. Fleeing the sinking ship the second they hear of Bad News would be in line with previous behavior.

dualvariable an hour ago

Most of the rest of them I can wave off as probably incompetence and maybe augmented by LLM-psychosis in decision-makers, and the general level of chaos in IT these days. The NSA internal network being down is pretty suspicious. Maybe the Trump administration rot has just spread that far, but that is making me think there could be actual cyberwarfare going on.

[To be clear: I mean exactly what I said, which is "there could be" which is not the same as "I think there is" -- so call it a 5%-10% chance of real cyberwarfare -- which means I'm not dismissing those claims out of hand, and find the claims at least plausible, even if I don't currently believe them]

le-mark 30 minutes ago

dmix 2 hours ago

The media will certainly try to give that impression either way. They serve the doomscrollers now.

causal an hour ago

Yeah "cyber thing went down" has certainly become a reporting trend. I'm not sure we would have even heard about NSA internal network issues a year ago. May or may not be significant.

matthewbauer an hour ago

Must be a little bit of both right? Like FAA grounding + Microsoft outage are both notable on their own, but the subsequent things would usually be background noise IMO.

I also think generally that AI is probably both the cause and solution to these problems. We're certainly getting more code churn than otherwise, and in lots of cases that's really good! But you're going to have to deal with inevitable instability too.

EvanAnderson an hour ago

Aside: I forget the name of the Baader–Meinhof phenomenon when it hasn't been discussed in awhile. When I'm reminded of it I begin seeing it happening frequently for the next few days. I wish I was joking. I assume it's the Baader–Meinhof phenomenon phenomenon.

smolder an hour ago

The term "frequency illusion" is a lot more memorable and intuitive. It's what I prefer.

ihumanable an hour ago

220hertz an hour ago

I think it lies at the intersection of Occam's and Hanlon's razors. The simplest explanation is incompetence brought on by a lack of skill from cuts, and therefore stupidity not malice.

warkdarrior an hour ago

It's probably both stupidity and malice, especially on the government side. Project 2025 makes sure that malice is there as a starting point and then the current loyalty-driven hiring practices means that stupidity is a key factor in the execution.

losvedir an hour ago

joe_the_user 34 minutes ago

Technically, the "Baader–Meinhof phenomenon" is people personally noticing X once it's been reported. What you are describing is technically a "news wave" or "Media Feeding Frenzy".

But I'd note, these "named after a thing" phenomena (eponnyms) are kind of terrible for exact description since their definition tends to be fuzzy, there's a temptation to fudge their borders and they create an insider jargon (if only I could a device that could reach of the computer and slap everyone who says "steelman" when they mean strengthen an argument, etc).

arisAlexis 39 minutes ago

It's an actual increase in offensive AI cyber capabilities

redwood 42 minutes ago

With the trilateral summit scheduled for Thursday it's possible some showing off is going on between the three letter agencies in the three countries... after all they can be first movers on vulns that are likely to be patched soon

lettergram an hour ago

Everyone is being pushed for rapid updates, the AI is doing the updating (error prone) and there's actually an increase in CVEs and potential risks.

The combination is a cycle

1. AI finds bugs

2. Engineers us AI to try and fix

3. AI testing approves, minimal engineering review

4. Engineers push release rapidly, it's critical after all

5. Bugs introduced

6. Cycle starts again

b112 10 minutes ago

My suspicion is that there are discovered zero days so severe, that the banks and NSA and so on, are being patched first.

With all the recent asshattery of leaking kernel exploits, I would presume trust is gone, and it's cover the big boys first.

(a linux kernel bug vulnerability could be patched without lkml involvement or distros even knowing)

stronglikedan an hour ago

This is all just to distract us from something much, much worse! /s

dustfinger 23 minutes ago

I may not understand expectations, or the law around this, but I find it a little surprising that the NSA would publicly announce that their internal and highly classified network is down. At least not before getting it back up, running and secure again.

netsharc 18 minutes ago

In a spy movie world, someone deep in enemy territory would be relieved to see the Reuters headline, after failing to reach the NSA network and having no other channel to ask (picking up the phone to call the US embassy would get your door busted by the local men in black).

I imagine, for the paranoid spy, not being able to call home could also mean "the enemy has severed our comms, we've been compromised, burn everything and run!".

qphe95 a minute ago

Thanks for the explanation! Good thing spy movies are just movies and not real!

qphe95 3 minutes ago

So how are people getting their work done right now?

tgsovlerkhgsel 2 hours ago

That's a LOT of things going down, and I haven't heard of some major infrastructure outage likely to affect many sites in this manner.

Coincidence or is there some common explanation that I missed?

bhouston 2 hours ago

There is a lot of things going down.

I think the opportunity right now where we have powerful AI hacking systems but not enough protections is to establish long-term presence in all networks of interest. Not bring them down.

colinrand 2 hours ago

Completely agree. Things are moving too fast and so are breaking.

For example, we updated a GCP managed base image of Ubuntu that was released with 350 CVEs patched, some were as high as 9.8. In doing so, they (GCP) changed the DEFAULT setting for hardware GRO from off by default to on. That's a massive networking change packed into a security update.

We only caught it AFTER it started causing issues in production. We didn't have the OS pinned because (decently valid reasons) or catch it in non-prod (decent reason + human mistake).

This is on a modern highly automated tested resilient well architected platform.

I can only imagine these other platforms are dramatically more exposed that we are, hence these common large scale outages.

pianopatrick an hour ago

That assumes that people did not already have long-term presence in networks of interest.

More plausible to me is that spy agencies had previously found and used some of these hundreds or thousands of vulnerabilities to hack into those networks just fine. But now other people like cyber criminals are using these vulnerabilities via AI. That is messing up both the attackers who had already gotten in quietly and do not want the attention and the defenders who now have to respond to a noisy intrusion they cannot ignore.

cj9 an hour ago

Also, anyone anywhere that horded/designed exploits will be wanting to use their backlog of these "assets" before systems are patched. From criminal networks to nation states.

Outages could be grey-hat testers and emergency updates.

A temporary period like we are experiencing is to be expected when moving to a time where a single engineer can perform the equivalent work of a team of excellent engineers working for weeks. Security has always been a game of who has the capabilities, means, and requirement for any given attack surface.

daheza 2 hours ago

None of these have been due to hacking? It has all been just standard issues probably increased by cost cutting measures.

bhouston 2 hours ago

Florin_Andrei 36 minutes ago

We also have powerful incentives to deliver more "value to the shareholders", while using AI code generators we don't fully understand.

postsantum an hour ago

Appstore connect is also down

chabes 2 hours ago

What do people expect when top government offices are occupied by former Fox News hosts?

winddude 7 minutes ago

Elon Musk and Trump with DOGE. They cut ~300k federal workers, cut governments contracts across all departments, including the FAA, NSA, home land security... those organizations have been raising alarms since that the cuts to civilian contractors etc would lead to outages and issues since, including pentagon, DoD, etc. But you know Idiocracy.

lacoolj 17 minutes ago

It would help if the moment the service came back online the title of the post could be changed to (FIXED) or something

Like the United Airlines thing yesterday, the moment I got to the article it said "was down for a few hours" but the post was still just something about United Airlines network being offline

gnabgib 2 hours ago

CNN Source (27 points, 5 comments) https://news.ycombinator.com/item?id=50532849

Consermo_AI 38 minutes ago

its getting spooky

wankerrific 19 minutes ago

It’s absolutely not ai

swalsh an hour ago

I have this distinct memory in I think Jan 2020 where I looked at the front page of CNN or NYT or something. Big picture of Trump impeachment, and proceedings. Small picture at the bottom of the page of China dealing with this virus. Worthy news, but not top page news. Nobody was paying attention. Feb, more people started paying attention, but it wasn't top page yet. By March, people were starting to realize that this is a serious crisis that's going to severely impact their lives.

I'm getting Feb 2020 vibes. I don't know what's going on, but this feels like more than just some routine outages going on.

ssl-3 an hour ago

I was sicker than I'd ever been for ~all of February of 2020.

So far this fall, none of my systems have gone haywire (yet!), so I forecast that we'll be OK.

techmunky 43 minutes ago

you probably had covid mate

soupfordummies 10 minutes ago

sanex an hour ago

Well the good news is that even though 2020 and 2021 were weird years we're still here. Humanity is resilient, we always figure it out.

GolfPopper an hour ago

The fantasy/cyberpunk TTRPG Shadowrun depicted a world in upheaval after a global computer crash in the late 2020s. In addition to the pandemics and the AIs.

ericcholis 2 hours ago

There's a part of me that doesn't want to know this....

ragle an hour ago

I have also been curious about frequency bias / Baader–Meinhof - but I am also wondering: what are the chances that a state actor is responsible for the last week of issues?

Trump is headed to China for a meeting with Xi and Putin.

We have 3 carriers off the coast of Iran and (despite Trump's pleas) NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil infra.

I've seen a few takes from L3 Harris / FAA employees calling the official line about the ATC issue bunk and there are 3 other major issues in the last week, now this NSA outage - curious if these incidents could be the BRICS axis flexing their own cybermuscles (independently in multiple separate attacks with carefully placed assets) ahead of their meeting with Trump in China?

An interesting geopolitical Mexican standoff, if true.

I'm not claiming this is true. I'm literally pulling it out of my rear with little context beyond a bunch of random news stories and X posts, it's pure pedestrian speculation / curiosity.

Would love someone steeped in national security / geopolitics (like: professionally as their job, not armchair analysts) to offer their take on this.

Is this a dumb, naive theory?

GolfPopper an hour ago

>NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil infra.

I know it's not the thrust of your post, but I see this or similar framing a lot, and I feel compelled to point out that Putin can end the strikes on his oil infrastructure at any time by ending the war he started by invading another country.

ragle an hour ago

Oh, I couldn't agree more. Long time supporter of Ukraine in many ways here.

Historically he's chosen to go about things differently, which I guess is part of what is tickling the pattern matching part of my brain here.

rsoto2 16 minutes ago

rsoto2 17 minutes ago

Washington was leading covert operations in Ukraine since 2014. Russia has stated that Ukraine joining NATO was a red line for decades.

It's a lot more complicated than "Putin is a bad guy." You pro-war guys are literally sending young Ukrainians to be massacred against a state they will never win against, in the guise of being "good guys."

Do good guys escalate to the point of full on war and massacre? Or do they try to find a political solution?

https://www.cato.org/commentary/washington-helped-trigger-uk...

whynotmaybe an hour ago

> NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil

I'm not sure about naive or dumb but you might review your POV regarding this.

ragle 22 minutes ago

Very open to that.

My complete POV, FWIW:

* From a purely moral / ethical standpoint: it is good that this is happening, Russia "deserves" this and brought it upon themselves.

* This nonetheless has geopolitical implications. The less oil Russia has to sell to China, the more buy pressure there is elsewhere, the higher the price of oil is generally and thus the higher competition for resources / tensions are between great powers.

* Ukraine is doing a lot of the heavy lifting in many areas themselves these days, NATO could honestly be doing more. However: what Ukraine is able to do themselves does owe something to the financial and material support they have enjoyed (historically from both the US and NATO, currently more from NATO only) in that they are able to allocate their own efforts with according relative "ease"

* It makes sense to me that, given the US' power within NATO, Russia and its BRICS partners would try to apply pressure to the US to encourage NATO to provide even LESS support to Ukraine (or outright become antagonistic against Ukraine, to the extent we aren't already) in whatever way they could.

What revisions to my POV can you suggest?

Waterluvian 30 minutes ago

Yeah. We're not giving Ukraine enough credit. It's Ukraine pushing its boot into Putin's neck with a frankly underwhelming amount of support from NATO. The Russian humiliation at the Kerch bridge yesterday was all Ukranian munitions.

rsoto2 14 minutes ago

What is wrong about it? Do you know what NATO is and relations with Russia and NATO pre-2014? Why not come up with an actual point instead of "America good, Putin bad" drivel that we could get from any major news-network.

https://www.cato.org/commentary/washington-helped-trigger-uk...

ranger_danger 44 minutes ago

I suppose rather than straight up 'cyberattack', that some kind of sabotage is also possible... but I don't see how that could ever really be proven without massive state-level investigations... at which point people will also question whether they are themselves involved somehow.

ragle 16 minutes ago

Right - sabotage was my thought as well in some cases (with the placed assets I'm suggesting).

I don't know how it could be proven either, I'm just curious if it is plausible or even likely from the perspective of someone who knows more than me about NatSec / geopolitical issues.

ColinWright 17 minutes ago

bayat an hour ago

Following the series of such events over the past couple of days, I have to say I’m a bit surprised by how quickly some in this community are brushing them off as AI-related pain or simply another example of the clickbait-driven media ecosystem.

I understand the skepticism, and I certainly don’t think we should go down a rabbit hole of conspiracy theories. But shouldn’t we also be a little skeptical of dismissing these events so quickly? Is it unreasonable to at least consider the possibility that they’re not entirely coincidental, or to wonder why so many seem to be taking place in the US?

arrowleaf an hour ago

IMO it's unreasonable not to think they are all related in some way, even if that relation is just bad actors seeing what other bad actors have been capable of recently. This all started with the freezers in Europe and seems to be snowballing since.

jonathanstrange 21 minutes ago

If there is a connection, then my guess would be that some AI systems became ready. However, I would only consider that credible in case of the NSA if they have decided to pull the plug on their network itself, for example to stop data exfiltration or as a general protective measure against newly found exploits that require large changes. Other than NK and Iran, I see no nation state currently having a good reason to attack US intelligence destructively, and I don't find it likely Iran or NK have the ability. It certainly doesn't make sense for Russia or China to attack the US right now.

stevenhubertron 2 hours ago

It’s been a hella few weeks. Hard to believe it’s coincidence even if you want to blame AI.

rglover 2 hours ago

It's AI. Or more specifically: rushed, careless usage of AI under the pretense that it's already "solved" software and you can haphazardly push junk to the world and expect real systems to run on it.

londons_explore an hour ago

Traditional software development normally involves at least a few veterans on the team familiar with almost every line of code.

However, AI works like a speedy intern who hasn't yet got to grips with the whole codebase, and sometimes makes assumptions. That's mostly because the codebase is usually bigger than the context window, so it is 'writing blind'.

ranger_danger an hour ago

Do you have any sources for this claim?

nope12123123 an hour ago

classified 32 minutes ago

AirStreamer27 26 minutes ago

Coding is solved. Bugs not solved yet. Fix incoming

dist-epoch an hour ago

What if they were post training a cyber model and got hacked from inside by the model, like OpenAI got hacked by it's own HuggingFace swarm.

gumby 27 minutes ago

NSA seems to have deprioritized information assurance over the past decade or so and now the chickens are coming home to roost.

Someone who wants to launch a Y2K-style hardening business (LLMS to look for problems, humans+LLMs to fix them) could really make bank right now. Maybe there will be a cuple in the next YC batch.

joshka 2 hours ago

> The NSA did not immediately return a message seeking comment about the reported outage.

Did you try sending it via outlook.com?

Semiapies an hour ago

Just whisper the question to your phone.

andy_ppp 20 minutes ago

Is someone running chaos monkey across society?

chinathrow an hour ago

Well, on the other hand, a bridge is down as well.

redwood an hour ago

"The NSA did not immediately return a message seeking comment about the reported outage". There's something humorous about that framing

ranger_danger 28 minutes ago

I don't like it when they use that phrasing because all it really means is "we called them and nobody picked up", or they simply didn't have a response ready at the time... which shouldn't imply anything at all. Most people don't immediately return messages.

kinabalu76 an hour ago

I see the shoe is now on the other foot NSA.

MadrasThorn 25 minutes ago

Cant imagine how many people that will 'effect'

throw7 an hour ago

Memo to the NSA... pulling the plug won't work. ;)

reedf1 3 hours ago

Anyone have knowledge of the backend of this system?

cucumber3732842 an hour ago

aeonik 2 hours ago

Are you implying that you DON'T have detailed docs and architecture of one of the most classified and secretive organizations in the world? ;-P

infinite_spin 2 hours ago

php and spit

grep_it an hour ago

doublerabbit 2 hours ago

You telling me you don't? Bro just download it off Limewire.

SketchySeaBeast 2 hours ago

Havoc an hour ago

My money is on something absolutely antique but wrapped in modern post quantum encryption.

Not unlike Bloomberg terminals…dos era UI but does the job well

bix6 an hour ago

Just join Hegseth’s signal chat and he’ll give you the deets. Ideally pick a username like WAREAGLE.

Zambyte an hour ago

Surely

demo4567 an hour ago

More fear mongering about day to day incidents: The network goes down every time I mess with DNS, VPC peering, TGWs, route tables.

howmayiannoyyou an hour ago

https://chinaselectcommittee.house.gov/media/reports/strange...

Allowing our infrastructure to be manufactured elsewhere, maintained by companies who struggle to retain top talent, and overseen by folks out of touch with cutting edge threats, and funded on annual budget cycles by legislators even more out-of-touch ... is a seemingly insurmountable set of obstacles.

swozey 3 hours ago

Just in time for the FEMA national emergency test this week

classified 40 minutes ago

Does that mean we can shitpost Trump jokes without getting arrested at the border?

vrganj 2 hours ago

There's some speculation that all of these recent outages might be related to attempts at taking Genesis Mission online - the Trump / Oracle project to have AI tie into government and private services:

https://en.wikipedia.org/wiki/Genesis_Mission

https://substack.com/@thedreydossier/note/c-389611426