Network Outage – Multiple Remote Sites

January 20, 2023 7:34 a.m.
Outage category: 
Wired Network, Wireless Network, VoIP phone service
Location: 
9900 Main St., Democracy Ln., Point of View, and Potomac Science Center
Status: 
Closed
Resolved alert: 
01/18/2023 4:25 pm

Remotes sites experienced a loss of network connectivity (Wired and Wireless). This affected Voice over IP Phone Services as well.

Initial symptoms: 

Ping failure on network monitoring systems.

Duration: 
01/18/2023 3:45 pm - 01/18/2023 4:25 pm
Impact to Mason: 

Loss of Wired, wireless, and VoIP Phone systems.

Affected Services: 
Network Access, Telephone & Voicemail Services
ROOT CAUSE ANALYSIS
Cause: 

OSPF adjacencies went down between our equipment and the remote site vendor.

Resolution: 

Vendor couldn’t narrow down to exactly what happened. All hardware and connection seem to be up. There was also no indication of failure on our end as a few other remote sites were still up and they were connected to the same equipment (router) in Harris Theater.

Prevention: 

We leave the ticket open with the vendor for now and continue to monitor for any abnormal behaviors.

STATISTICS
Service Team: 
NSENG and NSOPS