Network Service Interruption

March 17, 2022 10:16 a.m.
Outage category: 
Wired Network, Wireless Network
Location: 
Fairfax Campus, wired and Wireless users may have been affected
Status: 
Closed
Resolved alert: 
03/16/2022 7:50 am

Web host name were not being resolved and Wifi AP were not getting IP addresses, and no Email updates.

Initial symptoms: 

Alerts on SiteScope and starting to get calls about no internet or can not reach web sites, or get emails.

Duration: 
03/16/2022 1:15 am - 03/16/2022 7:50 am
Impact to Mason: 

All users would have been affected by the by the intermittent outage. User would not receive emails or reach website and certain applications.

Affected Services: 
Network Access, Office 365 Email
Other Affected Services: 
Email, Web sites, getting out on the internet, certain application services were affected.
ROOT CAUSE ANALYSIS
Cause: 

At 1:30am after hours, Engineer was working on a new network for the UPS store, using the Wing Zone VRF. When he cloned the existing VRF to make a new network for the UPS Store, the ANY ANY rule on the Firewall was deleted, causing the outage.

Resolution: 

The Any Any Firewall Rule was put back into place bringing most services back, but there were a lot of stale sessions that had to be cleared, after clearing the sessions all services came back online.

Prevention: 

Making sure we follow roll back protocols and Managers are informed of the situation.

STATISTICS
Service Team: 
NSENG, CCSE, CCSO, NSOPS