Data Center Co-Location Service Interuption

March 30, 2023 11:44 a.m.
Outage category: 
Applications
Location: 
Aquia Data Center
Status: 
Closed
Resolved alert: 
03/28/2023 1:54 pm

All the services provided by servers in row 7 in datacenter are impacted. Users experienced intermittent outages for those services.

Initial symptoms: 

Reported to Data Center operations team by Support Center that security cameras had some issues.

Duration: 
03/28/2023 12:11 pm - 03/28/2023 1:54 pm
Impact to Mason: 

Departmental users and some researchers use services provided by servers in row 7. Users experienced intermittent outages for those services.

Affected Services: 
Network Access
Other Affected Services: 
Services provided by servers in Row 7 of the Data Center, including security cameras.
ROOT CAUSE ANALYSIS
Cause: 

Plugging Cisco SFPs into DATA-01-A-SW71 and DATA-01-A-SW72, the incompatibility between SFPs and switches caused issues to uplinks of the switches.

Resolution: 

1. Removed the SFPs from the switches
2. Disabled and reenabled the switch uplinks to the PE router
3. Rebooted DATA-01-A-SW72

Prevention: 

For datacenter devices, anytime we put in a new type of SFP, we will create a RFC and do the work during a maintenance window.

STATISTICS
Service Team: 
Advanced Technology