Outage category: Wireless Network
Location: Fenwick Library, Peterson Hall, and Thompson Hall
Status: Closed
Resolved Alert:
Initial Symptoms
Alerts on SiteScope and starting to get calls about no internet or can not reach web sites, or get emails.
Root Cause Analysis
Cause
At 1:30am after hours, Engineer was working on a new network for the UPS store, using the Wing Zone VRF. When he cloned the existing VRF to make a new network for the UPS Store, the ANY ANY rule on the Firewall was deleted, causing the outage.
Resolution
The Any Any Firewall Rule was put back into place bringing most services back, but there were a lot of stale sessions that had to be cleared, after clearing the sessions all services came back online.
Prevention
Making sure we follow roll back protocols and Managers are informed of the situation.