Hacker News

> holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.

That's a common bias in management. Heroic fire fighting gets rewarded while silently preventing a crisis with proper planning isn't. Nobody ever gets credit for fixing problems that never happened.

> in 2012, I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible

This anecdote hits differently for me. I grew up reading about Fog Creek and consuming all of their writings (and that one film) as an example of what a software development company should look like. It feels strange to have the curtain peeled back to reveal that something like backups, which would be considered basic requirements today, were not part of their early operations.

I don’t mean to glorify the poor planning or risk management, but still have a ton of respect for the people who organized the response and who did that hard work, just because it was what people needed.

My company during Sandy was operated in both NY and SF, and our experience was a bunch of us on the west coast filling in for those who were not only unable to get to the office, but without power themselves in NYC. It was all unfamiliar stuff for us (different business functions were done there, but the NYC office’s work was time critical) but did build camaraderie.

Nit: the fact that an event occurred does not mean that the chance was 100% (although I guess this does depend on how one understands probability).

Speaking of Fog Creek, I didn’t realize that they renamed to Glitch and are totally out of business now.

It was sometime in the mid-2000s when I went from an admirer Joel's writings and what Fog Creek was doing to suddenly realizing it was all just very slick marketing on top of very mediocre stuff.

It really was a shocking wake up period for me in the realities of the real world vs. how we often perceive things.

IIRC it was the whole "Wasabi" thing that kind of opened up my eyes that there was a lot more smoke and mirrors involved than I had realized before.

It's hero culture and it's a curse people beg to keep.

Your comment brings back the times I've been on teams that worked double the hours a full time job should be, or spent 18 hours in a data center because a migration/data transfer was failing, and, and and.

So many times it's taken herculean effort to make things happen, usually because of poor planning and poor leadership.

Why these are viewed as "successes" or "heroic" baffles me as well.

The probability of a rare event does not go down after one happens. Especially with weather.

Oh, hi Gecko! Wild to see you in writing again.

Anyway, I was the first Fog Creek employee on site at 55 broadway (one of our founders got their first, and two peer1 employees were also on site) and also part of the first wave of the "bucket brigade". Another dozen of us showed up over the coming hours as we assessed and fixed things.

The media coverage on this was never super accurate-- we were carrying (and later, relaying) 5 gallon covered plastic buckets (like the home depot buckets with sealed lids) up to the landing where the generator was to top up its reserve tank. At one point SquareSpace (who was also running exclusively out of this DC) hired a group of muscle to bring a 55 gallon drum up, but that wasn't us.

I don't view this as lionizing failure, though I can see your perspective. Across our products (then FogBugz, Kiln, Trello) we had different backup strategies in play, but we were definitely putting a lot of eggs in the local-datacenter-that-claimed-to-have-resilient-power basket. Kiln in particular had decided to use SANs for data resilience, and that proved particularly troublesome for trying to find a workaround once the storm hit-- FogBugz on the other hand was able to spin up an EC2 version and start a slow-but-would-have-worked-eventually migration (and we even had an employee hit the road for us-east-1 location with a bunch of encrypted backups so that we could get a fast data restore online), and Trello fast-tracked-and-completed its local-to-cloud migration. While it was true that the DC had a backup generator, the fuel tanks for the generator being in the basement and becoming unavailable was not part of their resiliency plan. I don't fault us for not catching that.

...although I did have to later write the dreaded "I'm sorry, we've been hacked" blog post as a direct consequence of us not properly deprovisioning a "temporary" AWS credential that we created during that sprint. I will fault us for not catching that.

Shortly after Sandy, I switched my role from core dev on FogBugz over to DevOps, and I built out the backup and restore resiliency we had been missing. It wouldn't have kept us online during Sandy, though.

It was dangerous, and a couple folks slipped and fell. That's not a good thing and I won't celebrate that. Many employees opted out of the bucket brigade for family, safety, or personal reasons. I threw out all the clothes I wore that week as they reeked of diesel... not great.

I wouldn't wish that experience on anyone, but having come through it with fortune and no slip-and-falls, I still think about:

- Walking around FiDi post storm and seeing the flooded tunnels, clogged with cars that had been washed into them from the street parking.

- Dropping off big armfuls of coffee and bagels in the lobby for the building work crews. The folks who were essentially dredging out the basement and getting the building back online did much harder work than us.

- Going shopping for aquarium supplies in order to best jerry-rig the pump that our founder has repurposed from the office fish tank in order to transfer diesel from the drums on the curb into our 5g buckets.

Something that kept me going was that I knew that one of our FogBugz customers was the emergency response service in Haiti, and they got absolutely walloped by the storm. They were using FogBugz to coordinate relief, so there wasn't any way I was going to let FogBugz go offline if I could help it. Turns out that meant climbing a lot of stairs.

> holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.

Firefighters are heroes who bravely enter burning buildings, beloved by children and adults alike, muscular figures who get pictured on sexy calendars and thanked by tearful families.

Officious paper-pushers who make a fuss about a few cardboard boxes stored under the fire escape stairs (like me!) seldom get pictured on sexy calendars.

To play devil's advocate, paying some dudes to haul diesel for a few days every decade might be a lot cheaper than maintaining a fully redundant failover site 24/7.

It's such a familiar engineering story. Heroic effort means someone, somewhere, screwed up. I aim for boring competence so nobody has to plugs gaps in systems and processes with their literal body sometimes.

> I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible.

Quitting a job because of this is so funny to me. 2012 was truly a different time.

I'm surprised they didn't put a window washing crane on the roof as DR to simplify the raising of the diesel. The strange ass shit I've seen implemented to make the failure mode easier instead of eliminating the failure has limited my faith in our system as a whole.

Seems to me it is more likely that someone did the calculations and figured it is cheaper to send a bunch of unlucky morons up the stairs with jerrycans in case of flooding.

Startup bullshit. People at "scrappy companies" make cutting corners look like heroism and practical-mindedness. Meanwhile the little people suffer and prop everything up as the business almost implodes.

I can understand your frustration, but let’s consider:

The chance of a hurricane within 24 months was never 100%. Looking back, it did happen, but you can’t judge future-facing decisions on post-hoc information that you didn’t have at the time.

There is a difference between “no one wanted to admit it wasn’t 0%” and “no one else thought the risk-adjusted return was positive.” When people decline to take some preventative measure, that’s usually what they’re saying implicitly.

I’ve been where you are—frustrated that you predicted and told people exactly what would happen, they didn’t listen, and you were right—many times in my career. It’s infuriating. Part of learning from it is figuring out how to get people to change, not merely figuring out how to be right.

My guess is that this kind of black/white Strong Opinions approach has bitten you throughout your career.

We had the fuel filter issue at 75 Board street as well. After a few days we had to do a planned generator shutdown (the data center batteries could last a couple hours to give us coverage...) as a diesel mechanic flown in by Peer1 could service it. Props for handling that yourself!

Definitely a memorable week. I wonder just how many folks in Fidi were learning the finer points of emergency backup generators.

I would have thought that backup generators would cumulatively run for maybe a few dozen days over their lifetime. How did the fuel filters get dirty so fast?

Sandy was wild. you couldn't get computers for love or money, everything was jammed up. we ended up with whatever hardware configs had been returned and sitting in a dell warehouse just to have enough machines so for a long time after you'd have a rack of the standard build and then a random machine with a fibre channel card.

shocking how much engineering you can get done when the normal process becomes impossible and waiting isn't an option.

"unwittingly reinvented datacenters" What? does he mean for JS or The Street?

This was in 2012, not 2002. Not having your DC in your office was well known by then to be a bad model especially for a DMM, NYSE itself knew this.

I worked for a MM, we were ready to open the next day because we were in actual datacenters.

Generators go on the 17th floor so they can vent out exhaust and heat. Pumps need to go in the basement because you can only suck a liquid up by one atmosphere, which is something like 30' for diesel, but you can push a liquid up basically as high as you want. (Also, stating the obvious, the diesel tanker trucks are gonna be street-level, so you'd need pumps there anyway if you put the actual tanks higher.) This does, are you're pointing out, kinda stink if the pump itself is flooded, but the design's common and does make sense

Probably on the 17th floor to put the generators closer to the loads and reduce transmission losses. Also reduces street level air and noise emissions (and avoids street level air intake)

With fuel storage on the same floor for a reasonable runtime, larger fuel storage in the basement and a pump (or two) to move fuel, you're well prepared for lots of things. As long as the fuel throughput on the stairs is better than the fuel use rate, then you're ok for as long as you've got people to move it. Which is ok for exceptional incidents.

More than likely that's where the available space was. Data centers many large generators and there may not always be space in the lower levels to fit them, especially if the data center was not one of the original tenants.

Ground floor risks flood damage presumably? But this doesn’t explain why the 17th was chosen.

You put them up there so you get a voltage boost from the gravity pushing the electricity down. Put the generators on the ground floor, and by the time the electricity reached the top, there would be a voltage drop.

385 lbs, with a dolly, not too bad.

The "superhuman folks" above mentioned are the 75 Broad street bucket brigade crew, and uh... we made startup wages. We weren't quants, we made productivity software.

Though the "55 gallon" drum lift is greatly exaggerated, that was a crew of guys hired off the street who were walking around doing odd jobs in the disaster zone. Kinda wild. For us, we just carried the 5 gallon buckets.

It's Jane Street... they probably gave them a backpack of gold bullion for their trouble.

It was a bunch of quants so very well

Carrying 5 gallons of fuel up a dozen or more flights of stairs is hard work on its own.

I've done this job. I only had to do it a few times per day, but it sucked each and every time.

(When I showed up due to a report of an outage for our client's gear at the top of the building, the street was flooded. Water was flowing into a giant metal grating in the sidewalk like something from a cartoon. This grating lead to the basement.

And that basement was where the electrical switchgear was.

It was bad. It stayed bad for a long time.)

5 gallons is a standard size of literal buckets, they can have lids but I might prefer carrying a bucket of diesel up 18 flights of stairs with no lid over trying to get one of those goddamn lids off.

It was another time. Only big companies had dedicated data centers; the rest had an air-conditioned room in the office. That was before the "cattle, not pets" era, too: they would have a database server called "Debbie", a DNS server called "Denise", a file server called "Felix", etc.

Not the person you asked, but I've learned that if you're not careful about what you tell your employers and/or coworkers that it will almost invariably come back to haunt you in the form of unpaid labor.

It took me an embarrassingly long time to realize that (my 30's), and now I keep my conversations with my coworkers very utilitarian and divulge minimal information about myself. I try not to be a dick or anything, I'm polite enough, but pretty much everything I talk about is either directly relevant to work and/or so surface level as to border on "vapid".

ETA:

Just as an example, I used to have really bad sleep issues in my 20's that would make it so I wouldn't be able to get to sleep until around 4am, if at all. I made the mistake of bringing that up at work to my manager. Two weeks later I got a phone call at 3am because there was something he wanted to look at and "he knew I would be up". I was up, but I didn't really have much desire to get my laptop out and do work. I liked that job at the time and I liked my manager, so I did it anyway, and it wasn't until years later did I think about that incident and how utterly weird it was that I was sort of being punished for my insomnia.

My sleep issues have gotten a lot better now, so if someone calls me at 3am I probably won't hear it because I'll be asleep, but I remember that incident now and have it constantly spinning in my brain when I chat with coworkers.

No because I imagined a future where instead of me being asked for emergency help I'd be expected to perform a list of tasks every weekend as the defacto facilities manager. They were already paying me a ridiculous salary at the time. I thought about bringing it up if they ever said no to any requests I had and they never did so simply accepted it was my mistake broadcasting too much information at work which I never repeated.

I received many strange requests like to come in Friday night while drinking at a concert and ransack the offices looking for a laptop or possible phone left behind that they suspected was triggering alerts by repeatedly trying to connect to a foreign server. "You want me to drunkenly rob the offices at night and walk out the lobby carrying a stack of laptops? Ok"

If I would have 5min to work, I would probably be in the office most days. A 1 hour commute is something I want to avoid, not being in the office itself.

Yep, it is part of the design. The sprinklers are sometimes pre-action like described in the article, and sometimes clean agents. The only time we shutdown CRACs is if we EPO the whole room because otherwise some variant of this article happens.

I believe it's Halon, just like what's deployed for in-flight jet engine fires

They have masks in the room. They should, at least.

Yeah I didn't read the article in detail. It's the NYC that threw me off. And I didn't feel like it was that weird an concept, after all the real estate in NYC is super premium priced, not exactly prime datacenter country. I'm surprised there are many (like I said 33 Thomas St used to be just a phone exchange which used to be much bigger than they are now due to the mechanical tech involved).

> The headline is confusing.

Nitpick, but the headline is not confusing at all, even though a couple of people are apparently confused. I suspect that is the fault of a hasty read through, not the headline itself.

I think it looks cool. I used to work in buildings like this a lot. They're kinda cool. Dark, humming and clicking etc. I worked in telephony at a time when some (not many) of the old mechanical switches were still around and those buildings were especially spooky. It sounded like you were in the middle of a wild storm of crickets around 7pm when people finished dinner and started calling their friends. And all the little lights flashing all over (sometimes I used to keep the lights off for the cool effect if I could)

Some of them have half-floors for all the cable routing, where you can't really stand up.

I think this is why I like them. To me they're not just an ugly box but I always wonder what's inside.

Halon systems are banned, at the international and federal level, and have been since the 1990s; so, modern DCs don't and can't use them.

Ahh... the Vax War (bit hard to search for it now... gotta search for the first line). VAXen, my children, just don't belong some places.

https://www.hack.org/~mc/texts/vaxen-joke.txt

Realizing that I'm spoiling some of it... but we're here, so some of it is inherently spoiled...

           The fire alarm klaxon went off and the siren warning
           of imminent halon gas release was screaming.  We
           started to panic but the data center manager shouted
           over the din, "Don't worry, the halon system failed
           its acceptance test last week. It's disabled and
           nothing will happen."
 
           He was half right, the primary halon system indeed
           failed to discharge. But the secondary halon system
           observed that the primary had conked and instantly
           did its duty, which was to deal with Dire Disasters.
           It had twice the capacity and six times the discharge
           rate.

They don't have to blow that fast. It can be regulated.

But yes, I've seen low-O2 installations also. These have other issues though, like no protection when humans need to be in there for maintenance. And the need to be very very careful when reducing it again, making sure everyone is out. It happened to my colleague once that this went wrong and while it's not low enough that you'll die, he wasn't exactly feeling great for the next few hours.

Yeah, like the infamous Nasdaq Nordic outage in 2018 due to errant fire suppression gas release.

Maybe because it was in a normal office building? The gas extinguishers might be too hazardous there

Halon is not exactly fun to breathe, either.

We had it at work at a datacenter right beside offices and reception.

You just have to reinforce the ceiling panels and the windows so they don't get blown out, but it's not really an issue otherwise. It was all certified and regularly checked. Water is indeed a bit counterproductive as it's not really much better than fire for the equipment in there.

PS: Halon itself is of course gone but it's other less harmful gases these days.

What also happens is low-oxygen environments (they only raise the O2 when maintenance is being done).