Office365 - Exchange email timeout errors and mail delivery delay

Unplanned; Complete

Overview

Affected services
  • Office365
Degraded Service Time
05:00 PM, 26-Oct-20 — 05:00 PM, 28-Oct-20
Degraded Service Time
05:00 PM, 28-Oct-20 — 05:00 PM, 30-Oct-20
Description
Users may find some delays with email delivery or may see timeout errors attempting to access their mailbox but subsequent re-attempts may help and emails will be delivered. Update - 30/10/20 Microsoft are continuing to work on the issue with email delays. If you are experiencing these delays - please use alternative methods of contact for anything time sensitive eg. Microsoft Teams, Skype or Telephone if applicable.

Technical Information

Technical Information

Work summary
Microsoft are aware of an issue and are investigating. Update from Microsoft: We're finished rerouting a small number of user requests, and verified that there has been a reduction to the number of timeouts and delays being experienced. We're expanding the amount of user requests we're rerouting to fully resolve the issue. Update 28/10/20 - 13:20 Issue appeared to have been resolved overnight though now has started to re-occur intermittently since around 09:50 this morning. Microsoft have been informed and are looking into it with the relevant fix team. Update 29/10/20 - 07:30 Update from Microsoft: We've determined that the issue reoccurred because previously applied configuration changes, meant to resolve memory inefficiencies, did not reach a specific environment and resulted in higher susceptibility to traffic increases for that region. We're deploying these configuration changes to the remaining affected environment to remediate impact. In parallel, we've confirmed that restarting the affected infrastructure has mitigated impact, and we'll continue monitoring until the deployment completes to ensure impact does not reoccur. Update 30/10/2020 - 07:30 Update from Microsoft: We're continuing to investigate options for expediting the resolution to impacted customers without causing secondary impacts as our deployment progresses. In parallel, we're monitoring message queues and confirmed that the queues are lowering, which coincides with the end of the business day in the affected European infrastructure.
Technical summary
Office365 - Exchange email timeout errors and mail delivery delay
Affecting
All mail users
Servers / Hardware
Criticality
low
Impact description
Users may be unable to access email through both desktop client and outlook web resulting in timeout issues. Additionally users may experience a delay to mail being sent and received. Impact is specific to some users who are hosted out of a particular section of Microsoft's European infrastructure of which the University is one, other organisations are also affected.
Change Reference
Additional comments
P2010-032

Request, Authorise and Publish

Requested by Steve Hall on 27-Oct-2020
Authorised by Applications
Publish to IS Website (Service Status and Alerts page): Yes
Published by Steve Hall


Additional Details

Risk classification
Risk assessment
Other people involved
Notes for future reference
Final status: We've received confirmation from affected users that the issue is resolved, and our monitoring shows stability in availability. We'll continue to monitor the affected environment to ensure this issue does not occur again.