Network Service Interruption

November 28, 2022 2:39 p.m.
Outage category: 
Telephone, Wired Network, Wireless Network, VoIP phone service
Location: 
All campuses
Status: 
Closed
Resolved alert: 
11/24/2022 12:50 pm

There was an unplanned network service interruption from about 3:05 a.m. to about 10:30 a.m. on Thursday, November 24.  During that time, most central IT services, including WiFi services, Internet access, Voice over IP services, access to Banner, ITS Alerts, and user authentication services were partially or completely unavailable.

Initial symptoms: 

Multiple customer groups contacted Facilities Infrastructure and Operations reporting issues with connecting in Data or Phone services.

Duration: 
11/24/2022 3:05 am - 11/24/2022 12:50 pm
Impact to Mason: 

Users experienced a lack of connectivity during the times of the outage.

Affected Services: 
Network Access, Telephone Services, Microsoft Teams (MS Teams)
ROOT CAUSE ANALYSIS
Cause: 

Unplanned work was being done by a network Engineer, that caused the outage. This work was done before with no issues, but this time it caused a major Network Service Interruption to the University.

Resolution: 

Engineers had to roll back the router configuration, to correct the issue.

Prevention: 

The Director and Manager has spoken to the Engineer. He is to follow protocol, notify his manager, put an RFC in, get team’s approval, when making any changes no matter how small at all times.

STATISTICS
Service Team: 
NSENG, NSOPS