Outage category: VoIP phone service, Wired Network, Wireless Network
Location: All campuses
Status: Closed
Resolved Alert:
Initial Symptoms
Customers were reporting intermittent network and VoIP based phone issues.
Root Cause Analysis
Cause
The Perimeter Routers in the Data Center are configured as a redundant pair. One of the routers experienced a hardware issue causing fail-over to the other router. The secondary router also had a different hardware failure occur which started causing the intermittent issues due to high CPU load.
The cause of the service outage was determined to be a known bug in the code.
Resolution
Hardware on both routers were replaced, resolving the issues.
Prevention
The cause of the service outage was determined to be a known bug in the code. We are planning to do code upgrades on the rest of the routers on July 28, 2019.