Hacker News

I personally operate a fairly advanced closed-loop chlorination system, and this is beyond screwed up.

These systems can easily fail, and they tend to fail in a way that causes them to measure low or zero chlorine and thus overdispense chlorine. More below on why.

First, this means that an operator must always ensure safety, both against excessive chlorine and against insufficient chlorine. In a commercial context, that means that a real human (or maybe a totally different sensor design, e.g. a colorimetric sensor) need to measure the pool chlorine level, preferably with a high quality titration (drop test) every day before opening the pool. They should probably record the result in pen on water resistant paper, but this part is icing on the cake.

Second, hoping that an email gets noticed is ridiculous. If you do not have positive confirmation that the pool is safe to open, don’t open it.

Third, the systems should be sized so that they cannot overdose quickly. The maximum dispensing rate should be such that hours of continuous operation would be required to reach an uncomfortable level, let alone a dangerous level. (Hot tubs are sort of an exception here.)

Fourth, everyone should be using a reasonable amount of CYA in their pool. Even at a lowish level of 30ppm CYA it takes a lot of chlorine to become dangerous unless the operator has also lost control of pH. I realize that some codes are confused about this.

Fifth, chlorine is smelly. Someone should have noticed, even just casually. Admittedly, lots of swimmers may be conditioned to stinky pools, and checking for safety isn’t their job. But the staff should notice.

Sixth, the systems I’ve seen beep when there’s an alert. It’s easy to add flashing lights, too. (In fact, the system I’m familiar with automatically shuts down if it feeds for more than X minutes without reaching the setpoint, and you cannot turn this feature off. You can, however, defeat it by switching the power to the whole controller.)

—- sensors —-

Actual free chlorine sensors intended for pools use an amperometric probe immersed in iodide solution or gel, separated from the pool water by a fancy membrane. Free chlorine oxidizes iodide to iodine, and the sensor measures the iodine concentration. If the membrane fails (or general wear and tear or excessive pressure swings) cause the iodide to be lost, then the chlorine can’t react and the sensor reads zero (or low if there’s tiny bit of iodide left).

Interestingly, these sensors are rather expensive devices that appear to mostly come from France and Germany. The Chinese vendors seem to be rebranding European parts (!), and the major US vendor is very clearly reselling a French company’s product.

There are some vendors who sell a cheap (or absurdly marked up) potentiometric ORP-based closed-loop system. In my considered opinion, these are worse than useless.

This is 100% fault of the hotel, not of Microsoft. It’s unacceptable that the communications regarding such a dangerous thing totally rely on email, without any backup or double check.

I'm not even going to read the article. If your hotel depends on some cloud service to prevent chlorine gas poisoning you're doing something terribly wrong. Full stop. I don't know anything about the laws in Germany but this is dangerously close to libel in the US.

Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.

I know it’s fun to blame Microsoft for things, but that is a poor and dangerous design.

Anything that could result in that sort of harm should have a more resilient and redundant architecture.

https://www1-wdr-de.translate.goog/nrw/rheinland/rhein-erft-...

> 15 guests injured at Phantasialand

> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.

Chlorine gas is no joke. In high school I was making it by running electricity through salt water and then diffusing the gas through pool water to chlorinate my parents pool. My cousin took a whiff when I was distracted and had to go the emergency room for breathing problems. Fortunately he recovered fully.

Maybe the information is being lost in translation, but I'm trying to understand where the "chlorine gas" is being talked about. The articles I found just say "too much chlorine", which is hazardous but not the same as chlorine gas. Can anyone shed some light here?

Perhaps as a proximal cause, but not a root cause.

Core services must be as minimal and stable as possible, with safe fallbacks and never be part of a huge internet smart device big tech clusterfuck.

its interesting in the sense that emails have sort of become the de-facto standard for sending alerts and keeping paper trails. Even for systems like this. However, I agree with what other folks are saying in that email should never have been the ONLY way someone gets notified about an issue.

Here's a US Chemical Safety Board report involving something vaguely similar: https://youtu.be/35FkCArjLU0 https://www.csb.gov/assets/1/6/yenkin-majestic_report_for_pu...

> During the incident, some of the gas monitors installed within the Yenkin-Majestic facility detected the flammable solvent release approximately one minute after the release began and sent an email to an offsite employee, but the monitors did not sound audible alarms to warn employees of the hazard.

1 dead, 8 non-fatally injured.

Anyone else notice that the date given by the hotel doesn't match the date of the Office 365 outage?

It appears that the hotel is trying to duck liability by latching onto the outage as the root cause even though this happened before the outage.

screen by pool door and hotel closed circuit pool is open or pool is closed readout of current water conditions, temp, ph, chlorine, turbidity, whatever is possible. simple logic to dissalow abrupt changes without an override. anything goes to zero or infinity, closes the pool.

Buncha Microsoft bootlickers in here.

Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoft’s service failed. It doesn’t matter HOW it should be engineered.

C'mon can we really blame Microsoft for this? This is stupid.

Microsoft bullies enterprises into their products and targets tech workers who are against it. This is 100% Microsoft's fault and the amount of corporate goons on YCombinator is ridiculous. I don't think this will change at all when the designated response is gaslight and deny.

@ Risks Digest

I'm not microsoft shill but that is such a lazy dogshit title for an article about what can easily be seen as dogshit hotel infra management.

For anyone that doesn't read the link they were hurt but seriously.

This was bad system design. Hopefully people learn that safety critical services can't rely on Microsoft which they also tell you not to do.

It is 100% fault of New York Times. They've distracted everyone in the IT industry with their "reportage", when Outlook stuff started failing - engineers were reading the news - so no one at Microsoft paid attention. And that wiped out Outlook for everyone. Outlook wipeout caused "no email" and "no office" for everyone for 24-48h, I imagine that people were complaining to IT. So we've got attention spreading thin for the whole IT industry. The result - cascading failures.

Posting a bunch of semi-fake out of context images on New York Times should be investigated, really. Same goes for that "classified" leak they've reported. This wiped out billions from the market, disparaged multiple woman (who likely wanted nothing to do with the reporters), affected travel for tens of thousands, now this gas poisoning.

Since this is Germany, of course there will be no learning because we can do no wrong!

https://how.complexsystems.fail/

Who needs a horn or a flashing light, when you can just send an email?

Therac ‘26

Yeah, and everyone who dies in a hurricane was killed by a particular pattern of wind

I suppose if you really think about it, butterflies are responsible for everything.

I assume the backup pigeons didn't arrive in time either.

MICROSOFT TERMS OF SERVICE

Use of Microsoft products and services confer no warranties. Written explicit warranties are for entertainment purposes only. Use at your risk. Use of Microsoft products are dangerous to your mental and emotional health. If you experience thoughts of deep frustration, call 1-800-MSCares. All inquiries will be promptly processed in the order they will be received or when the sole contractor can get to it.

Sure, trust on a company that’s 10000% on AI, nothing bad will happen

BRB, off to read the Therac-25 paper for the third time this month.

Employee finds pool water has dangerous amounts of chlorine, yet instead of locking the door to the pool they just send an email into the void?

This time someone absolutely should be fired for buying Microsoft.

Dumb as shit, sue the hotel

Hot take.

Surprising this happened in Germany of all places; don't they have a huge emphasis on Linux and OSS?

> This was because the automatic dosing system had pumped too much chlorine into the pool.

What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.

lol Germans and the internet. Of course it's the internet's fault, not theirs.