Network Service InterruptionMarch 17, 2022 10:16 a.m.
Web host name were not being resolved and Wifi AP were not getting IP addresses, and no Email updates.
Alerts on SiteScope and starting to get calls about no internet or can not reach web sites, or get emails.
All users would have been affected by the by the intermittent outage. User would not receive emails or reach website and certain applications.
At 1:30am after hours, Engineer was working on a new network for the UPS store, using the Wing Zone VRF. When he cloned the existing VRF to make a new network for the UPS Store, the ANY ANY rule on the Firewall was deleted, causing the outage.
The Any Any Firewall Rule was put back into place bringing most services back, but there were a lot of stale sessions that had to be cleared, after clearing the sessions all services came back online.
Making sure we follow roll back protocols and Managers are informed of the situation.