Major widespread network outage 26-27 Nov 2015

Unplanned; Complete

Overview

Affected services
  • EUCLID web
  • MyEd
  • University Website
  • Central Wiki Service
  • DiscoverEd
  • Kinetix (KX)
  • Staffmail
  • Authentication and Authorisation
  • User File Store
  • HR/Payroll
  • E-Financials
  • ID Management
  • Reporting + Analytics (BI Suite)
  • EBIS
  • Hosted Websites
  • PURE
  • eRecruitment
  • Core Firewall
  • Load Balancers
  • Ezproxy
  • Polopoly (publishing)
  • Mail Relay
  • Event Booking
  • Electronic Journals
  • UniDesk
  • Moodle
  • Learn
  • Office365
  • Timetabling
  • EdWeb
Outage Time
08:14 PM, 26-Nov-15 — 02:00 AM, 27-Nov-15
Description
At about 20:14 on Thursday 26-Nov-2015 we began to experience a major outage across the entire the University network. We have not yet been able to diagnose the initial cause of this nor the reason why the outage was so widespread. As of ~02:00 the University network has been stable and outstanding issues with individual network switches attended to between 07:30 and 09:30 on Friday morning.

Technical Information

Technical Information

Work summary
At about 20:14 on Thursday 26-Nov-2015 we began to experience a major outage across the entire the University network. We have not yet been able to diagnose the initial cause of this nor the reason why the outage was so widespread. As of 02:00 27-Nov-2015 most of the EdLAN network is now working again with the notable exception of the Business School, nee AFB. A number of switches are still polling down on skiddaw but many cases the issue may be with management access to the switch rather than with connections through it. These will be attended to in the morning. Service may be unreliable while we investigate and remedy any remaining defects.
Technical summary
Major widespread network outage 26-27 Nov 2015
Affecting
Entire University and associated organisations
Servers / Hardware
Most
Criticality
high
Impact description
Network connectivity across the entire University and associated organisations.
Change Reference
Additional comments

Request, Authorise and Publish

Requested by Sam Wilson on 27-Nov-2015
Authorised by CIS
Publish to IS Website (Service Status and Alerts page): Yes
Published by Sam Wilson to ease-admins@lists.ed.ac.uk; staffmail-alerts@lists.ed.ac.uk; ad-announce@lists.ed.ac.uk; virtual-hosting-users@mlist.is.ed.ac.uk; telephone-alerts@mlist.is.ed.ac.uk; EdLAN-announce@lists.ed.ac.uk; www-alerts@lists.ed.ac.uk


Additional Details

Risk classification
high
Risk assessment
Other people involved
Notes for future reference