Network Outage All Campuses

August 15, 2019 8:00 a.m.
Outage category: 
VoIP phone service, Wired Network, Wireless Network
Location: 
All campuses
Status: 
Closed
Resolved alert: 
07/19/2019 11:54 am

All users were experiencing intermittent network delay or outages.

Initial symptoms: 

Customers were reporting intermittent network and VoIP based phone issues.

Duration: 
07/19/2019 10:10 am - 07/19/2019 11:54 am
Impact to Mason: 

All campuses were experiencing internet connectivity (slowness, dropped connections, etc.) problems.

Affected Services: 
Network Access
ROOT CAUSE ANALYSIS
Cause: 

The Perimeter Routers in the Data Center are configured as a redundant pair. One of the routers experienced a hardware issue causing fail-over to the other router. The secondary router also had a different hardware failure occur which started causing the intermittent issues due to high CPU load.

The cause of the service outage was determined to be a known bug in the code.

Resolution: 

Hardware on both routers were replaced, resolving the issues.

Prevention: 

The cause of the service outage was determined to be a known bug in the code. We are planning to do code upgrades on the rest of the routers on July 28, 2019.

 

STATISTICS