Network Outage – Network Connectivity Issues – Campus Wide

February 07, 2023 7:42 a.m.
Outage category: 
VoIP phone service, Wired Network, Wireless Network, VPN
Location: 
Fairfax, Arlington and Science technology campuses.
Status: 
Closed
Resolved alert: 
02/03/2023 6:52 pm

Users could not get access to any network internet services including wired and wifi.

Initial symptoms: 

Users were unable to connect to the internet, and support center started getting calls.

Duration: 
02/03/2023 4:45 pm - 02/03/2023 6:52 pm
Impact to Mason: 

All Mason faculty, staff and students were affected by intermittent connectivity issues.

Affected Services: 
Network Access, Telephone Services
ROOT CAUSE ANALYSIS
Cause: 

A routing loop was created when research vlans were added to the Mason network. This loop caused the DATA Center Switch to HIGH CPU load, which caused the router to crash affecting the entire University.

Resolution: 

Engineers shut the VLANs off and the port channel to shut down the loop that was cause by the research project.

Prevention: 

ORC Project is going to contact Network team to make sure their network, does not cause another outage.

STATISTICS
Service Team: 
NSENG, NSOPS, CCSOPS