Intermittent WiFi Issues

March 17, 2022 10:19 a.m.
Outage category: 
Wireless Network
Location: 
Fenwick Library, Peterson Hall, and Thompson Hall
Status: 
Closed
Resolved alert: 
03/16/2022 11:48 am

Web host name were not being resolved and Wifi AP were not getting IP addresses, and no Email updates.

Initial symptoms: 

Alerts on SiteScope and starting to get calls about no internet or can not reach web sites, or get emails.

Duration: 
03/16/2022 11:20 am - 03/16/2022 11:48 am
Impact to Mason: 

All users would have been affected by the by the intermittent outage. User would not receive emails or reach website and certain applications.

Affected Services: 
Wireless (Wi-Fi) Network
ROOT CAUSE ANALYSIS
Cause: 

At 1:30am after hours, Engineer was working on a new network for the UPS store, using the Wing Zone VRF. When he cloned the existing VRF to make a new network for the UPS Store, the ANY ANY rule on the Firewall was deleted, causing the outage.

Resolution: 

The Any Any Firewall Rule was put back into place bringing most services back, but there were a lot of stale sessions that had to be cleared, after clearing the sessions all services came back online.

Prevention: 

Making sure we follow roll back protocols and Managers are informed of the situation.

STATISTICS
Service Team: 
NSOPS, CCSE, CCSO, NSENG