We are experiencing an issue with multiple services in
europe-west2 beginning on Monday, 2026-10-19 23:08 PDT.
Due to a power event we had suffered an outage within
europe-west2-b zone. The power event is mitigated. Partial
mitigation to infrastructure services will be observed within the
next 30 minutes and the remaining services will be mitigated there
after.
So some kind of power failure happened to bring down one zone. Unfortunate, but this kind of stuff happens. That is why all the cloud providers have multiple zones, and I'm amazed the big companies reported as down weren't using multiple zones.
(Also reporting the times in PDT is quite unhelpful for users of this zone who are likely based in the UK or EU)
The fact they didn't immediately fail over doesn't mean they didn't had a plan.
The plan might've just been "if it gets bombed, we restore backups into other zone". Just waiting for it to fix itself for few hours mightv'e been acceptable risk.
I'm not saying it's the case here, but sometimes the cloud providers themselves have inter-zone dependencies, or even inter-region dependencies that can cause outages in other zones and regions.
> I'm amazed the big companies reported as down weren't using multiple zones
Much as with backups - if you don't test them, they don't exist.
Everywhere I've worked has had multi-AZ infrastructure. I have never worked anywhere where the failure of an AZ has been simulated regularly, with second-order effects mapped out.
We set up our applications and services to be multi-AZ natively. It's not particularly difficult with how many AWS services are regional or global (Admittedly, I still expect a potential DNS failure to cause issues if and when that occurs). However we are just part of a workflow, we integrate with other applications that are not multi-AZ and would be impacted. That's just out of our hands.
It's like our on-prem systems were ordered to plan out and exercise DR plans to account for a data center going offline, but we can't exercise a DR plan fully if part of the plan is 'Adapt to how application X, which is hosted in the same datacenter, also chose to fail over.' We do not own application X. There is no appetite in the business to co-ordinate the DR exercise across the entire company at the same time.
We can't even do it all in a lower environment because there are dependencies on systems that are shared across environments. Some massive, redundant filesystem that is sync'd across our data centers, but the sync is a boolean. One datacenter instance of it is the primary and the others are read-only until they get promoted. We have one of these and lower environments are just a different root folder on the system with different access patterns, but they can't make DataCenter A the primary in prod and DataCenter B the primary in UAT.
GCP is kind of odd in that some hardware is only available in certain zones, and some hardware is technically available in multiple zones but the capacity is not evenly spread. I've had to implement infrastructure code for GCP that interrogated hardware availability by region and zone to place VMs, trading off multi-AZ availability where it made sense.
Yes. The area on the Venn diagram where services, hardware and regulatory compliance intersect is surprisingly small.
I had a client that was contracting with a very large bank and there were only something like four zones globally that we could possibly deploy in to.
Google also is unlike AWS in that their zone abstraction is very virtual, and two zones might be in the same physical building. If you have requirements that your deployment zones be physically separated by some minimal distance then it can make things pretty challenging.
Often there are irritating legal geographic limitations on where cloud hardware can exist for a given country. I can understand why, but there is this particular drawback of having all eggs baskets close together.
It's crazy. My main account is with Starling and I have a Lloyds account for a saving an account and some free cinema tickets. I don't think I've ever had an issue with Starling, it's great - Lloyds fails to sign in about 50% of the time consistently.
A lot of the information people are leveraging here comes from Downdetector which is mostly users reporting that stuff is down, not whether they are actually down. It's a really bad source.
It's a pretty good source ime (at least, massive spikes tend to correlate well with issues). Traditional organisations aren't as transparent as SF tech companies about service downtime. Plus, if the service isn't working for its users, I wouldn't say it's actually up.
For me it was Revolut. No access to my main account, no withdrawals, no card payments since the morning. App doesn't work, there's no phone number to call, nobody to talk to. Dark side of neobanking, I guess.
The stand-in system has a limited subset of features, i.e. no bells and whistles, but enough that you can get any actual banking done.
(Not an ad / I have no connection to Monzo, I just had some appreciation for a bank doing this properly as normally banking IT is a disaster. Plus the idea of keeping a separate codebase for almost an entire bank's functionality seems very invested.)
Starling also has a disaster recovery system across cloud providers, although it is implemented differently to Monzo's (I used to work there on multi-cloud).
Multi-cloud is really tough, tbh. It's easier said than done, especially if you care about data consistency at all.
I have built these system for financial institutions since before cloud providers. It is hard/impossible if not well constrained at a level that supports data availability and consistency well (ie using a proper database with the correct setup across all locations with the underlying OS configuration/patches applied uniformly - the latter lesson cost IBM a lot of money due to system downtime for a financial institution that I brought back online). Many software vendors offering on-prem versions don’t know how this works well exactly, so I’ve had to help them along the journey and get them to sign-off for their contractual obligations to be upheld in the setup.
I opened up a Revolut account a couple of years ago with the intention of switching my current account to it (i.e. wages going into it etc), but I got spooked after hearing that they weren't registered as a UK bank and thus subject to the various regulations and protections. I tried using Chase instead and have been happy with it so far.
I used to use Starling as my main and they were fine, until I started getting issues with random card declines due to "fraud prevention" and nobody to talk to. I switched to First Direct and like the peace of mind of having a human on the phone if needed (when I wanted to make a huge transfer for my house deposit it was nice to have!) and I still get perks like free foreign card transactions.
Yep, I probably won't bother switching to them now as Chase provides a good service as far as I know, though I do get a bit twitchy about its U.S. links. However, it's headquartered in the UK and is a division of J P Morgan Europe, so I hope that provides some level of distancing from the inevitable economic collapse of U.S. A.I. companies.
I couldn't pay for groceries with Revolut in a store. Staff asked if I was using Revolut, but luckily adding a card from a different bank to Apple Pay was quick and painless (I had neither my wallet nor cash with me).
Another reminder that we need redundancy for all essential services that we rely on.
I think sovereignty might be the _cause_ of the banks' outages...
This is affecting companies that put everything into a single GCP region in the UK for their entire business - aka most likely British companies or government with a data sovereignty requirement.
The UK, at least, is big enough that you can have multiple regions within the same country. There's a lot of distance between Glasgow and London, for example. Maybe not enough if both are targeted with nuclear warheads, but at that point, we have bigger fish to fry I suspect.
Most European countries have enough space within the country to do that, and, for most things, except banking typically, hosting within the EU is enough, it doesn't have to be within your own country.
Microsoft Azure has two entirely seperate regions.
A "full function" one in/around London.
And a "tickbox" one for multi-region requirements near Cardiff.
The tickbox one has constant capacity constraints and a large number of Azure services aren't in it, but if you need to tick the "two regions and in country" requirement, you kinda/sorta can.
But considering what these sovereignty rules aim to achive, and as someone mentioned, that the problem may be due to companies relying on a single region provided by a hypersclaler due to data sovereignity rules.
Then redundancy has to be considered by both the company and the rulers.
You are confusing people with your loose application of the word.
Are you advocating for redundancy (e.g multi-cloud) or physical sovereignty (e.g self-hosted)?
> Makes me wonder why an EU bank is keeping data outside the EU (from my understanding, only the London data center is down)?
Let me tell you a story about an idiot politician who woke up one day and thought Brexit would be a fantastic idea, and then the stupid dim-witted "ultra-hard brexit" manner in which it was implemented by a certain Mr Johnson.
The same reason UK people can't open EU bank accounts is the same reason EU banks have separate operating companies and infrastructure in the UK.
Given they operate in a heavily-regulated environment it almost certainly made life easier for them to keep UK stuff in the UK because they had no idea how (or if) the regulatory environment would ever progress beyond ultra-hard brexit.
The main selling points were auto-scaling and not having to self-host. Or, on a higher level, outsourcing accountability, being able to point at someone else if something goes wrong, as exemplified by the present case.
The whole thing has always been strange. The outsourcing of accountability for instance has always been a pain. Technically that is what you're doing, but if you work in IT operatings and GCP or AWS takes down your service, your boss still tell you to fix it.... but now you can't.
In terms of upfront costs, cloud is always cheaper, initially. There's sort of a weird sweet spot for cloud. If you're to small it makes no sense operationally, if you're to large, it just way to expensive.
*If you're large and have a constant, stable usage pattern it makes no sense.
Even for large companies, if you have somewhat stable usage most of the year, but periodically have usage spikes, cloud providers still remain super beneficial.
That's the whole selling point of cloud. Spins up the resources when you need them. Spin them down when you don't. Cloud thrives off companies spinning up resources and never spinning them down, even when they don't need them anymore.
It's a little worrying that banking with an institution from one of the world's financial capitals is so vulnerable. I would've thought London would've been priding itself on security, stability, safety, etc. Confidence in their systems is their bread and butter
Afaik, even Russia and Ukraine aren't having such issues while dealing with data centres getting droned, lol.
Let me draw a line between this and the Ukrainian bombing of Russian Yandex data centers. This is a sign of structural vulnerability. If one power failure can take down more than 5 banks, what you are describing is a valuable target. It's ironic that the cloud services that promised to harden systems have delivered another layer of vulnerability.
I think Azure’s design is fine. Some cool ideas that work, some “cool ideas” that are a pain in the ass. The implementation is complete dog shit. Barely anything works the way it’s supposed to, and when something works, better not touch it or you can’t touch it. Many times the answer is “why don’t you delete everything and try this other thing?”
The documentation is indeed worst than that still. It’s a maze of articles, none of them actually answer your questions because in Azure the answer to everything is “it depends”. Every doc page has to also be a sales pitch for something. They love inventing skus and features that are basically “now if you want this service to work, you must enable this feature. Otherwise, this is not supported. And to enable this feature, you must delete and recreate your instance” you know, just figure it out.
From Google's status page https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPy...
So some kind of power failure happened to bring down one zone. Unfortunate, but this kind of stuff happens. That is why all the cloud providers have multiple zones, and I'm amazed the big companies reported as down weren't using multiple zones.(Also reporting the times in PDT is quite unhelpful for users of this zone who are likely based in the UK or EU)
The fact they didn't immediately fail over doesn't mean they didn't had a plan.
The plan might've just been "if it gets bombed, we restore backups into other zone". Just waiting for it to fix itself for few hours mightv'e been acceptable risk.
I'm not saying it's the case here, but sometimes the cloud providers themselves have inter-zone dependencies, or even inter-region dependencies that can cause outages in other zones and regions.
For example, AWS us-east-1 outages have caused global outages on multiple occasions, such as https://www.networkworld.com/article/4168878/aws-hit-by-us-e...
> I'm amazed the big companies reported as down weren't using multiple zones
Much as with backups - if you don't test them, they don't exist.
Everywhere I've worked has had multi-AZ infrastructure. I have never worked anywhere where the failure of an AZ has been simulated regularly, with second-order effects mapped out.
We set up our applications and services to be multi-AZ natively. It's not particularly difficult with how many AWS services are regional or global (Admittedly, I still expect a potential DNS failure to cause issues if and when that occurs). However we are just part of a workflow, we integrate with other applications that are not multi-AZ and would be impacted. That's just out of our hands.
It's like our on-prem systems were ordered to plan out and exercise DR plans to account for a data center going offline, but we can't exercise a DR plan fully if part of the plan is 'Adapt to how application X, which is hosted in the same datacenter, also chose to fail over.' We do not own application X. There is no appetite in the business to co-ordinate the DR exercise across the entire company at the same time.
We can't even do it all in a lower environment because there are dependencies on systems that are shared across environments. Some massive, redundant filesystem that is sync'd across our data centers, but the sync is a boolean. One datacenter instance of it is the primary and the others are read-only until they get promoted. We have one of these and lower environments are just a different root folder on the system with different access patterns, but they can't make DataCenter A the primary in prod and DataCenter B the primary in UAT.
GCP is kind of odd in that some hardware is only available in certain zones, and some hardware is technically available in multiple zones but the capacity is not evenly spread. I've had to implement infrastructure code for GCP that interrogated hardware availability by region and zone to place VMs, trading off multi-AZ availability where it made sense.
Yes. The area on the Venn diagram where services, hardware and regulatory compliance intersect is surprisingly small.
I had a client that was contracting with a very large bank and there were only something like four zones globally that we could possibly deploy in to.
Google also is unlike AWS in that their zone abstraction is very virtual, and two zones might be in the same physical building. If you have requirements that your deployment zones be physically separated by some minimal distance then it can make things pretty challenging.
Often there are irritating legal geographic limitations on where cloud hardware can exist for a given country. I can understand why, but there is this particular drawback of having all eggs baskets close together.
The drawback being temptation not to power them indepedently, it seems.
Google is PDT-centric, and never grew out of it. Everything is Mountain View time internally and it's so jarring.
> https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPy...
https://news.ycombinator.com/item?id=50163858
I know some banks out-sourced most of its infrastructure to external operators, and some operator I worked with didn't seem competent.
>>Google cloud outage takes down HSBC, Revolut, Bank of Scotland, Llyods, Equifax, more
You will probably have issues logging into Llyods even when the cloud is back up.
It's crazy. My main account is with Starling and I have a Lloyds account for a saving an account and some free cinema tickets. I don't think I've ever had an issue with Starling, it's great - Lloyds fails to sign in about 50% of the time consistently.
Do you have an account with Llyods though?
Strange. My Lloyds login has never failed since I opened the account 11 years ago.
The usability of their mobile app is atrocious though. Good thing they have a website.
cannot agree more
I bank with HSBC. Everything is up and working.
A lot of the information people are leveraging here comes from Downdetector which is mostly users reporting that stuff is down, not whether they are actually down. It's a really bad source.
It's a pretty good source ime (at least, massive spikes tend to correlate well with issues). Traditional organisations aren't as transparent as SF tech companies about service downtime. Plus, if the service isn't working for its users, I wouldn't say it's actually up.
Exactly. "Responding to pings" is not the same as "letting me get work done".
If I had a nickel for every solid-green status page I shook my fist at while the actual site faceplanted on every request...
Alternatively, and from the top of the HSBC homepage
"We would like to apologise for any inconvenience and continue to monitor the situation, which is also affecting other organisations."
For me it was Revolut. No access to my main account, no withdrawals, no card payments since the morning. App doesn't work, there's no phone number to call, nobody to talk to. Dark side of neobanking, I guess.
Not all neobanks. I use Monzo, and they had an outage a while back. I was impressed that it barely affected their functionality.
They have a stand-in system that uses a separate codebase and separate cloud provider. They have a blog post about it here: https://monzo.com/blog/tolerating-full-cloud-outages-with-mo...
The stand-in system has a limited subset of features, i.e. no bells and whistles, but enough that you can get any actual banking done.
(Not an ad / I have no connection to Monzo, I just had some appreciation for a bank doing this properly as normally banking IT is a disaster. Plus the idea of keeping a separate codebase for almost an entire bank's functionality seems very invested.)
I have always got bad vibes from Revolut...
Starling also has a disaster recovery system across cloud providers, although it is implemented differently to Monzo's (I used to work there on multi-cloud).
Multi-cloud is really tough, tbh. It's easier said than done, especially if you care about data consistency at all.
I have built these system for financial institutions since before cloud providers. It is hard/impossible if not well constrained at a level that supports data availability and consistency well (ie using a proper database with the correct setup across all locations with the underlying OS configuration/patches applied uniformly - the latter lesson cost IBM a lot of money due to system downtime for a financial institution that I brought back online). Many software vendors offering on-prem versions don’t know how this works well exactly, so I’ve had to help them along the journey and get them to sign-off for their contractual obligations to be upheld in the setup.
> I have always got bad vibes from Revolut...
I opened up a Revolut account a couple of years ago with the intention of switching my current account to it (i.e. wages going into it etc), but I got spooked after hearing that they weren't registered as a UK bank and thus subject to the various regulations and protections. I tried using Chase instead and have been happy with it so far.
I used to use Starling as my main and they were fine, until I started getting issues with random card declines due to "fraud prevention" and nobody to talk to. I switched to First Direct and like the peace of mind of having a human on the phone if needed (when I wanted to make a huge transfer for my house deposit it was nice to have!) and I still get perks like free foreign card transactions.
https://www.revolut.com/blog/post/revolut-is-officially-a-ba...
They are reigistered now
Yep, I probably won't bother switching to them now as Chase provides a good service as far as I know, though I do get a bit twitchy about its U.S. links. However, it's headquartered in the UK and is a division of J P Morgan Europe, so I hope that provides some level of distancing from the inevitable economic collapse of U.S. A.I. companies.
I couldn't pay for groceries with Revolut in a store. Staff asked if I was using Revolut, but luckily adding a card from a different bank to Apple Pay was quick and painless (I had neither my wallet nor cash with me).
Another reminder that we need redundancy for all essential services that we rely on.
cash is a more reliable redundancy.
> there's no phone number to call
Perhaps this "bank" knows its phones would be down too.
Countries talking about AI sovereignty, when they forgot compute sovereignty
I think sovereignty might be the _cause_ of the banks' outages...
This is affecting companies that put everything into a single GCP region in the UK for their entire business - aka most likely British companies or government with a data sovereignty requirement.
That's not sovereignty.
Sovereignty is when there is a UK company with multiple data centres in the country.
With that if one region goes down, another can carry on.
Redundancy does not seem like an inherent component of sovereignty
It isn't but that wasn't the point. It's that sovereignty & redundancy aren't mutually exclusive
The UK, at least, is big enough that you can have multiple regions within the same country. There's a lot of distance between Glasgow and London, for example. Maybe not enough if both are targeted with nuclear warheads, but at that point, we have bigger fish to fry I suspect.
Most European countries have enough space within the country to do that, and, for most things, except banking typically, hosting within the EU is enough, it doesn't have to be within your own country.
Microsoft Azure has two entirely seperate regions.
A "full function" one in/around London. And a "tickbox" one for multi-region requirements near Cardiff.
The tickbox one has constant capacity constraints and a large number of Azure services aren't in it, but if you need to tick the "two regions and in country" requirement, you kinda/sorta can.
Ah yes, Azure is the undisputed heavy weight champion of ticking theoretical boxes.
True on first glance.
But considering what these sovereignty rules aim to achive, and as someone mentioned, that the problem may be due to companies relying on a single region provided by a hypersclaler due to data sovereignity rules.
Then redundancy has to be considered by both the company and the rulers.
That's not sovereignty, that's redundancy.
Sovereignty is just where stuff is, not how many place it is there.
You are talking about the dictionary definition of the words.
But this is supposed to be about practical.
What's the purpose of ticking a box about housing data in UK when you can't make it available after a single incident.
You are confusing people with your loose application of the word. Are you advocating for redundancy (e.g multi-cloud) or physical sovereignty (e.g self-hosted)?
I think people have been fighting Googles monopolies for quite some time
I wonder if it's somehow related to me having to re-sign into my account after clearing Google Play Services data just a few hours ago.
Most likely it knocked out the "Ing" bank in Germany.
Makes me wonder why an EU bank is keeping data outside the EU (from my understanding, only the London data center is down)?
> Makes me wonder why an EU bank is keeping data outside the EU (from my understanding, only the London data center is down)?
Let me tell you a story about an idiot politician who woke up one day and thought Brexit would be a fantastic idea, and then the stupid dim-witted "ultra-hard brexit" manner in which it was implemented by a certain Mr Johnson.
The same reason UK people can't open EU bank accounts is the same reason EU banks have separate operating companies and infrastructure in the UK.
Given they operate in a heavily-regulated environment it almost certainly made life easier for them to keep UK stuff in the UK because they had no idea how (or if) the regulatory environment would ever progress beyond ultra-hard brexit.
https://www.reuters.com/world/uk/uk-pm-burnham-says-russia-i...
Russia woke up
Maybe they mix it: serving fonts/pictures/etc over scalable cloud providers and hosting confidential data themself.
I couldn't unlock a shared city bike this morning on the Beryl system, probably the same outage.
how can cloud inf go down? that was the main selling point, it makes no sense
Cloud is just an really expensive vps with an api, you still got to leverage that api to replicate your service
The main selling points were auto-scaling and not having to self-host. Or, on a higher level, outsourcing accountability, being able to point at someone else if something goes wrong, as exemplified by the present case.
The whole thing has always been strange. The outsourcing of accountability for instance has always been a pain. Technically that is what you're doing, but if you work in IT operatings and GCP or AWS takes down your service, your boss still tell you to fix it.... but now you can't.
In terms of upfront costs, cloud is always cheaper, initially. There's sort of a weird sweet spot for cloud. If you're to small it makes no sense operationally, if you're to large, it just way to expensive.
*If you're large and have a constant, stable usage pattern it makes no sense.
Even for large companies, if you have somewhat stable usage most of the year, but periodically have usage spikes, cloud providers still remain super beneficial.
That's the whole selling point of cloud. Spins up the resources when you need them. Spin them down when you don't. Cloud thrives off companies spinning up resources and never spinning them down, even when they don't need them anymore.
It makes no sense only if you believe in magic and marketing
Ah again London region :D
It's a little worrying that banking with an institution from one of the world's financial capitals is so vulnerable. I would've thought London would've been priding itself on security, stability, safety, etc. Confidence in their systems is their bread and butter
Afaik, even Russia and Ukraine aren't having such issues while dealing with data centres getting droned, lol.
It's really quite poor performance.
> Afaik, even Russia and Ukraine aren't having such issues while dealing with data centres getting droned, lol.
They've had ~5yr to work on that.
London - having given itself a crown of 'financial capital', has had faaaaar longer, with a lot more resources. There's no excuse.
The government should be resilience testing everything. Did Covid teach no lessons? Is the defence budget just for shits and giggles?
Since there are no humans working on outages here is the test for superintelligence:
Can someone ask Claude, Codex, Copilot, Antigravity, Devin to resolve this outage?
No Googling or help from humans this time.
Let me draw a line between this and the Ukrainian bombing of Russian Yandex data centers. This is a sign of structural vulnerability. If one power failure can take down more than 5 banks, what you are describing is a valuable target. It's ironic that the cloud services that promised to harden systems have delivered another layer of vulnerability.
> It's ironic that the cloud services that promised to harden systems have delivered another layer of vulnerability.
You misunderstand.
This is a provider outage, but the services running on them by the customers (that went offline) were not designed or setup to be multi-region.
The infra outage is GCPs fault, but application outages from customers are caused by their architecture and decisions prior to the outage.
Migrate to the Cloud, they said. It will be more resilient and less expensive, they said.
Who's "they"? Someone from [cloud]'s professional services?
I think so indeed ;-)
At this point, the way I see it is:
- There's AWS
- There's Azure
- There's everything else
And I don't even use Azure, I just hear it's good (and they don't charge cross-AZ traffic)
GCP has by far had the least amount of outages and seemingly the best security.
AWS behind, but with us-east-1 outages.
Azure is extremely far behind those two, and mostly for non-tech legacy shops that got suckered.
I won't touch oracle cloud.
Azure’s design is elegant and even — dare I say it — beautiful.
Its implementation is clumsy and ugly.
Its documentation is worse.
Its support… isn’t.
I think Azure’s design is fine. Some cool ideas that work, some “cool ideas” that are a pain in the ass. The implementation is complete dog shit. Barely anything works the way it’s supposed to, and when something works, better not touch it or you can’t touch it. Many times the answer is “why don’t you delete everything and try this other thing?”
The documentation is indeed worst than that still. It’s a maze of articles, none of them actually answer your questions because in Azure the answer to everything is “it depends”. Every doc page has to also be a sales pitch for something. They love inventing skus and features that are basically “now if you want this service to work, you must enable this feature. Otherwise, this is not supported. And to enable this feature, you must delete and recreate your instance” you know, just figure it out.
Ironic: yesterday X Money invited me to open an account — with a starting gift balance of $15. Deeper currents?