Call Centers Temporary Outage

April 08, 2022 8:06 a.m.
Outage category: 
Telephone
Location: 
All Mason departments using Webex Call Center
Status: 
Closed
Resolved alert: 
04/06/2022 5:06 pm

Call center users were logged off the system and could not log back in to take calls. Not all users were initially logged off. A few in University Information were still connected for a bit until they were logged off.

Initial symptoms: 

We have an MS team, WCC Group, and centers were indicating that they were logged off and could not log back in. Teresa Gibbons put in a ticket with the vendor, Veracity, and as on a call with their support center. We were told the system was down in Washington, DC but checked the “Down Detector” and found it was a nation wide outage.

Duration: 
04/06/2022 3:00 pm - 04/06/2022 5:06 pm
Impact to Mason: 

The users were the contact center call employees. They could not log in back in to do their jobs as they were logged off the system in the middle of their shifts.

Affected Services: 
Telephone Services
Other Affected Services: 
All inquiries/calls to the 22+ call centers on campus were unavailable to take internal or external calls to Mason.
ROOT CAUSE ANALYSIS
Cause: 

Impact per WCC: All applications on Release 10 are currently impacted by a network infrastructure problem. We have found the root cause of the issue linked to a network infrastructure problem on Release 10.

Resolution: 

WCC: We are working on redirecting network traffic.

Prevention: 

This was a vendor outage and we have no control as to what steps they will take to prevent it from happening in the future. (If they post an AAR, I will send it as an attachment to this document.)

STATISTICS
Service Team: 
Call center groups. telecom, ITS support Center, Vendors-Veracity, WCC-Cisco