Customers were reporting intermittent network and VoIP based phone issues.
All campuses were experiencing internet connectivity (slowness, dropped connections, etc.) problems.
The Perimeter Routers in the Data Center are configured as a redundant pair. One of the routers experienced a hardware issue causing fail-over to the other router. The secondary router also had a different hardware failure occur which started causing the intermittent issues due to high CPU load.
The cause of the service outage was determined to be a known bug in the code.
Hardware on both routers were replaced, resolving the issues.
The cause of the service outage was determined to be a known bug in the code. We are planning to do code upgrades on the rest of the routers on July 28, 2019.