Hacker News

especially if everyone else is also down with you

at least when us-east-1 goes down, Amazon stock prices go up (outages => stockholders remember that aws exists).

Props to anyone that can build a system mature enough to withstand a chaos monkey element running loose in prod. Brilliant.

This is a good argument for encouraging employee vacations as well. Makes a more resilient company when nodes occasionally turn off.

To add to the speculation. The error messages we were observing along with the Microsoft status updates read like connection pool exhaustion. The ticket update history started with Microsoft thinking a 'restart of affected servers' fixed it and troubleshooting 'health check failures'.

Anyone who has seen connection leaks/pool exhaustion in practice, it looks like an otherwise healthy low utilization server right up until it stops replying, and a restart appears to fix it until it runs into the limit again. Easy to think that your health checks are the problem. Per MS updates, they rabbit holed on this issue and built 'an alternative health check method' which worked in testing (i.e. low number of test connections); but when they tried to apply this change, servers would go unresponsive anyway.

My guess is a rushed update with a connection leak somewhere in the stack. A call that opens, but does not close. Just the kind of thing that AI can miss because it's only trying to hit the goal and not necessarily clean up after itself.

> But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.

I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?

Exchange and Active Directory are easily my favorite MSFT products. I've always wished they would port them to Linux, even though I know that OS dependencies make that a non-starter (though they surprised me w/ SQL Server for Linux).

I'll concede that the the biggest Exchange environment I ever ran was around 1,000 mailboxes. I've never seen how it fails in really large environments. I was also fortunate that I only ran the earlier versions (pre-2013) in sub-500 mailbox environments. In those settings, though, I found it worked well and wasn't bad to administer.

It was very reasonable to use in small business environments, too (sub-100 mailboxes). I put a lot of little Exchange 2003 boxes out in small offices. It plugged-away very effectively.

Yes-- there were updates here-and-there that had issues, for sure. By-and-large it was a stable and predictable product.

I always tried to be mindful of maintaining free disk space, following MSFT's recommendations for speeds-and-feeds for I/O, and running on solid hardware (ECC memory, redundant power, UPS). I feel like I would have had more problems if I'd scrimped on that stuff, for sure, but I was unwilling to let Customers cheap-out.

With Exchange 2019 and high-availability (database availability groups) I was easily able to do zero-downtime maintenance. That "big for me" Exchange Customer I worked with had more downtime in the last 24 hours than they did the entire time they used Exchange 2019 in production.

I haven't seen the code, so I can't speak to the "nightmare of code". Large parts of it were ported to the .NET Framework in between the 2010 and 2013 releases. The 2013, 2016, 2019, and current Subscription Edition releases appear to be an evolution of the codebase with no signs of major rewrites. I question how much pre-2013 code is even present in the codebase anymore.

I dunno. My experience just hasn't been that bad. I don't want to be an apologist for MSFT. I've never fully gotten over the suspicion I have for their motives going all the way back to the Halloween documents. I don't like how they're forcing everyone to hosted offerings and killing their on-prem products. I've been pleased with the value my Customers realized from Exchange.

> The postmortem on this will be interesting.

Any bets on whether it's DNS?