There are logged in pages for people to raise tickets, see specific services/regions they have access or restricted ones(Gov/Sovereign Cloud only available to some users) and also other features only logged in users would need.
Yeah it will not work under an auth downtime or the service application downtime, but it is not designed for that.
It is for for the items which are localized at service/DC/region level. The service app pages on azure covers planned maintenance, retirement of old services, security advisories and so on.
Planned maintenance is typically for your specific provisioned infra. For example there is a scheduled kubernetes update cycle, so you as customer would be affected, but it doesn't affect most of the customers at the time so there is not much to make public, as there is always some updates
The advisories are not disclosure, they are public already, it is more like you running this version of insecure service you have to apply this patch or we will do this automatically for you by this date.
I mean... If I was operating on shared resources, this would be a good mitigation. But an almost static status page should be operated differently to core M364 services.
Or in other words: all hammering on a status page should have near zero effect on services currently affected.
Users may be unable to connect to multiple Microsoft 365 services
Issue ID: MO1550821
Affected services: Exchange Online, Microsoft 365 suite, Microsoft Copilot (Microsoft 365), Microsoft Defender XDR, Microsoft Teams, SharePoint Online
Status: Service degradation
Issue type: Incident
User impact
Users may be unable to connect to multiple Microsoft 365 services.
More info
More info: Potentially impacted services include:
- Exchange Online
- Microsoft Teams
- SharePoint Online
- Microsoft Copilot 365
- Microsoft Defender XDR
- Microsoft Planner
- Microsoft To Do
- Microsoft Purview: Multiple Purview services including Authorization, Auto Labelling, Data Loss Prevention (DLP), Communication Compliance, and Data Classification are impacted.
Scope of impact
This issue may affect users attempting to access the affected Microsoft 365 services.
Current status
We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly. We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses. Additionally, our review of recent changes has identified a small number of configuration updates that we're investigating as potential contributors to the issue.
History of updates
- While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.
- We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
- We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.
- We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.
Why do we need to login to see the status page?
How would we learn about status in case of auth-adjacent incident?
you don't need to, https://status.cloud.microsoft/. there is a public version
There are logged in pages for people to raise tickets, see specific services/regions they have access or restricted ones(Gov/Sovereign Cloud only available to some users) and also other features only logged in users would need.
Yeah it will not work under an auth downtime or the service application downtime, but it is not designed for that.
It is for for the items which are localized at service/DC/region level. The service app pages on azure covers planned maintenance, retirement of old services, security advisories and so on.
TIL. Some of this info should still be public (planned maintenance, critical advisories) though.
Planned maintenance is typically for your specific provisioned infra. For example there is a scheduled kubernetes update cycle, so you as customer would be affected, but it doesn't affect most of the customers at the time so there is not much to make public, as there is always some updates
The advisories are not disclosure, they are public already, it is more like you running this version of insecure service you have to apply this patch or we will do this automatically for you by this date.
I assume it's related to their reasoning for the outage in the first place... "increased load", which to me sounds like a DDoS.
The login redirect probably stops a lot of the traffic, and from closer to the edge of their network.
I mean... If I was operating on shared resources, this would be a good mitigation. But an almost static status page should be operated differently to core M364 services.
Or in other words: all hammering on a status page should have near zero effect on services currently affected.
Do we know it's an (almost) static status page? :)
Maybe it's doing all sorts of queries internally.
This submission says [dupe] but no indication what other submission this is a dupe of.
Users may be unable to connect to multiple Microsoft 365 services
Issue ID: MO1550821 Affected services: Exchange Online, Microsoft 365 suite, Microsoft Copilot (Microsoft 365), Microsoft Defender XDR, Microsoft Teams, SharePoint Online Status: Service degradation Issue type: Incident
User impact Users may be unable to connect to multiple Microsoft 365 services.
More info More info: Potentially impacted services include: - Exchange Online - Microsoft Teams - SharePoint Online - Microsoft Copilot 365 - Microsoft Defender XDR - Microsoft Planner - Microsoft To Do - Microsoft Purview: Multiple Purview services including Authorization, Auto Labelling, Data Loss Prevention (DLP), Communication Compliance, and Data Classification are impacted.
Scope of impact This issue may affect users attempting to access the affected Microsoft 365 services.
Current status We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly. We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses. Additionally, our review of recent changes has identified a small number of configuration updates that we're investigating as potential contributors to the issue.
History of updates
- While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.
- We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
- We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.
- We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.
I tried outlook.office.com and got an error message in the wrong language.
Ihre Hovercraft ist voll von Aalen!
Please forgive me, HN, but ymmd
At least it doesn’t say “doing the needful”. Presumably that will happen at the singularity.
Imagine if AI goes down as well, and all the engineers don’t have the AI they need to ask to fix the AI written software. Thats where we are heading.
Mike Judge's Idiocracy was prophetic.
I can't even sign in; the code emailed to me never arrived. 2FA didn't work when I tried.