Hacker News

In Google Cloud, that's a huge no no and you'll be put on a naughty list facing serious scrutiny during the postmortem review.

Disclaimer: I work for Google Cloud.

People who don't work on cloud services are usually very sloppy with their wording which I think is leading to some disconnect here. A "zone" is not really the unit of availability for most things people consider the "cloud" for hyperscalers. A zone is like 1-3 DCs and you are very vulnerable to stuff like localized power events. For that reason most services are not offered at the zonal level and if they are you get big disclaimers saying so and lower 9s of SLA.

Most things are at a regional level which is multiple zones. Your AWS us-east-1 is one such region. A lot of datacenters spread across a decently large physical area have to have an issue for a whole region to start having failures.

For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)

1. AWS has no cross-regional or cross-zonal data plane[1] dependencies. Customers might couple them, in violation of best practices, but AWS itself does not. It's strictly forbidden.

2. That article you linked to does not support your claim. That is a report of a single-zone incident impacting us-east-1 only. Five other AZs in the same region were still fully operational.

[1] These are not the same as control plane dependencies: you might not be able to make certain mutations, but everything should still work in the steady state.

Worked at a major ISP in the 90s with many critical infra customers. True diversity is a lot harder than many imagine.

We set up our applications and services to be multi-AZ natively. It's not particularly difficult with how many AWS services are regional or global (Admittedly, I still expect a potential DNS failure to cause issues if and when that occurs). However we are just part of a workflow, we integrate with other applications that are not multi-AZ and would be impacted. That's just out of our hands.

It's like our on-prem systems were ordered to plan out and exercise DR plans to account for a data center going offline, but we can't exercise a DR plan fully if part of the plan is 'Adapt to how application X, which is hosted in the same datacenter, also chose to fail over.' We do not own application X. There is no appetite in the business to co-ordinate the DR exercise across the entire company at the same time.

We can't even do it all in a lower environment because there are dependencies on systems that are shared across environments. Some massive, redundant filesystem that is sync'd across our data centers, but the sync is a boolean. One datacenter instance of it is the primary and the others are read-only until they get promoted. We have one of these and lower environments are just a different root folder on the system with different access patterns, but they can't make DataCenter A the primary in prod and DataCenter B the primary in UAT.

I'd imagine (maybe optimistically) most companies do think about using multiple zones. And even test for DR. Regularly. And successfully. Problem is, all that works just fine on a sunny Monday morning. Then shit hits the fan and all of a sudden everyone is forced to, at the same time. It's a different kettle of fish

Other clouds are similarly odd (or even worse). I have experienced in Azure that if you provision a resource group with some basic SKU for a given service, the only way to later upgrade to the latest and greatest SKU is to tear down the whole resource group and re-provision everything from scratch. All attempts at scale-up in place will fail with cryptic errors. Seems like you are assigned in an old corner of the DC and can't get out of it.

Extra fun if there is a key vault involved, then you have to manually delete every key in it, otherwise you can't create new keys with the same names in the new vault.

Yes. The area on the Venn diagram where services, hardware and regulatory compliance intersect is surprisingly small.

I had a client that was contracting with a very large bank and there were only something like four zones globally that we could possibly deploy in to.

Google also is unlike AWS in that their zone abstraction is very virtual, and two zones might be in the same physical building. If you have requirements that your deployment zones be physically separated by some minimal distance then it can make things pretty challenging.

> infrastructure code for GCP that interrogated hardware availability

How do you to do that ?

It comes across as really uncaring to the probably 99.9999% of people outside of the US who have no idea offhand what UTC offset it is, it almost feels hostile.

If they at least included “(UTC -7)” after PDT, while still a bit annoying, at least people could work out what time that is without having to lookup PDT from somewhere first (as I just had to).

When I started at a large company in 2020, I began to notice that they advertised all times: meetings, deadlines, events, in terms of "EST". Even when it was Daylight Savings Time.

So, being in Arizona where it's never DST, I called them out on it, and I continually bothered them when they used this "EST" notation. I told them that they were simply wrong to use it during Daylight Savings Time. I told them that MST and Eastern Time did not have a year-round relationship. They didn't care.

It turns out that a lot of very large corporations use "EST" (and other "standard time" notation) all year round. Including FedEx and UPS, for some reason.

It is obviously ridiculous, when you could simply use "ET" instead, and always be right, no matter what your DST status is. But they insisted on using this "EST" thing.

I had a conversation or two with some LLMs about it, and they rationalized it. But it is crazy to subject people to this "corporate fiction" that is obviously wrong/inaccurate for an entire half of the year, and it leads to a lot of problems.

And what about the fact this give the US government?

1) the ability to view everyone's bank data, in secret

2) the ability to change EU bank data at will, in secret

(see US Cloud act)

There are much worse things you can do to data than bomb it.

The drawback being temptation not to power them indepedently, it seems.

For sure Google zones aren't the same as AWS AZs, and I remember complaining about this years ago during that fire in a France DC. But you also said their regions are firewalls across a datacenter, suggesting a multi-region DC, which I'm pretty sure they don't have.

The overall point is accurate. Google expects you to spread things more across zones. Most users would rather zones not share buildings.

Dont downvote...read their docs....and this is just the start:

"The regions Stockholm, Osaka, and Montreal have three zones housed in one or two physical data centers." - https://docs.cloud.google.com/docs/geography-and-regions

"This means the multiple zones within each of these regions may not be located in physically separate data center campuses." - https://docs.cloud.google.com/backup-disaster-recovery/docs/...

"A region is a set of buildings operated by Google in a geographic location, such as a data center campus." - https://cloud.google.com/vpc/network-pricing

Strange. My Lloyds login has never failed since I opened the account 11 years ago.

The usability of their mobile app is atrocious though. Good thing they have a website.

Do you have an account with Llyods though?

Starling also has a disaster recovery system across cloud providers, although it is implemented differently to Monzo's (I used to work there on multi-cloud).

Multi-cloud is really tough, tbh. It's easier said than done, especially if you care about data consistency at all.

> I have always got bad vibes from Revolut...

Help me understand, is the bad vibes you get from stories from the company culture? Or rather that the media put a target on their back (at at least in my opinion this is because they have a Russian born founder? Revolut is one of the best products I've ever used, I've honestly nothing but good things to say about them.

> I have always got bad vibes from Revolut...

I opened up a Revolut account a couple of years ago with the intention of switching my current account to it (i.e. wages going into it etc), but I got spooked after hearing that they weren't registered as a UK bank and thus subject to the various regulations and protections. I tried using Chase instead and have been happy with it so far.

cash is a more reliable redundancy.

Two things:

1) 4b queries a year is nothing. 4b queries is less than an hour for some of the software I have written.

2) The banner was a joke referencing the fact that the EU has a terrible track record with privacy.

What sensible reason you have to put your money in a bank?

Agreed. Just like in the old days, when banks cut down their own trees to make their own paper for their ledgers.

Exactly. "Responding to pings" is not the same as "letting me get work done".

If I had a nickel for every solid-green status page I shook my fist at while the actual site faceplanted on every request...

“Correct” here presumably means “some other people are also experiencing problems.” For companies like large banks, with tens of millions of customers, that’s not necessarily very meaningful except for the people who happen to be experiencing the issue.

That's not sovereignty.

Sovereignty is when there is a UK company with multiple data centres in the country.

With that if one region goes down, another can carry on.

Which you can't. Unless the government steps in and demands that all ties with Burgerland are cut.

They have been shocklingly competent for international transfers. Most things just work through the app, and the few cases where I had to deal with support it went so smoothly that I assumed the support agent was bullshitting me and I'd hit a wall later (I didn't). Exactly the opposite experience of what I'd expect from a "neobank" and much better than "real" banks.

Even the KYC process the first time I was making a larger transfer was surprisingly smooth.

Currency exchange fees/spread with "classical" banks are absolutely prohibitive.

> Makes me wonder why an EU bank is keeping data outside the EU (from my understanding, only the London data center is down)?

Let me tell you a story about an idiot politician who woke up one day and thought Brexit would be a fantastic idea, and then the stupid dim-witted "ultra-hard brexit" manner in which it was implemented by a certain Mr Johnson.

The same reason UK people can't open EU bank accounts is the same reason EU banks have separate operating companies and infrastructure in the UK.

Given they operate in a heavily-regulated environment it almost certainly made life easier for them to keep UK stuff in the UK because they had no idea how (or if) the regulatory environment would ever progress beyond ultra-hard brexit.

Maybe they mix it: serving fonts/pictures/etc over scalable cloud providers and hosting confidential data themself.

Where cheat = common sense / smart / good disaster recovery planning.

London - having given itself a crown of 'financial capital', has had faaaaar longer, with a lot more resources. There's no excuse.

The government should be resilience testing everything. Did Covid teach no lessons? Is the defence budget just for shits and giggles?

The whole thing has always been strange. The outsourcing of accountability for instance has always been a pain. Technically that is what you're doing, but if you work in IT operatings and GCP or AWS takes down your service, your boss still tell you to fix it.... but now you can't.

In terms of upfront costs, cloud is always cheaper, initially. There's sort of a weird sweet spot for cloud. If you're to small it makes no sense operationally, if you're to large, it just way to expensive.

I think so indeed ;-)

I see the GCP marketing staff has found HN.

I think Azure’s design is fine. Some cool ideas that work, some “cool ideas” that are a pain in the ass. The implementation is complete dog shit. Barely anything works the way it’s supposed to, and when something works, better not touch it or you can’t touch it. Many times the answer is “why don’t you delete everything and try this other thing?”

The documentation is indeed worst than that still. It’s a maze of articles, none of them actually answer your questions because in Azure the answer to everything is “it depends”. Every doc page has to also be a sales pitch for something. They love inventing skus and features that are basically “now if you want this service to work, you must enable this feature. Otherwise, this is not supported. And to enable this feature, you must delete and recreate your instance” you know, just figure it out.

IME the diferent components of Azure vary quite a bit. Some of them are great, others are pretty inconvenient or weirdly designed. I personally feel the web UI is terrible. I like that the APIs follow consistent rules and seem to follow good practices across the board. The documentation is hit or miss, kind of like the components themselves.

Honestly that's kind of been my experience with all of the cloud providers I've had to work with.