Multiple Mason Websites and VPN are inaccessible

August 26, 2021 10:37 a.m.
Outage category: 
VPN, Website
Location: 
All campuses
Status: 
Closed
Resolved alert: 
08/25/2021 5:25 pm

Users could not connect to the GMU websites, or experienced network slowness or lag trying to login to applications.

Initial symptoms: 

Users could not connect to certain GMU websites, experienced network slowness trying to connect to applications.

Duration: 
08/25/2021 4:50 pm - 08/25/2021 5:25 pm
Impact to Mason: 

All Mason community affected by this intermittent outage.

Affected Services: 
Network Access, Virtual Private Network (VPN)
Other Affected Services: 
Mason websites
ROOT CAUSE ANALYSIS
Cause: 

The connection between the Head End PE or HEPE and Firewall was flapping. The reason for the flapping was from enabling port channel on the HEPE to the special multicast fusion router. Once the the port was enabled it caused a multicast loop which in turn caused the flooding of the network and the Firewall to flap up and down.

Resolution: 

The fix was to turn off the port channel for multicast, once turned off all networks, Websites and applications came back online.

Prevention: 

We have a template in place to put in the static route for multicast so this will not happen again.

STATISTICS
Service Team: 
Network Engineering, Network Operation, CCSO