Webex Call Center Outage

Outage category: 
Telephone
Location: 
All campuses
Status: 
Open
Resolved alert: 
01/27/2020 5:36 pm

Agents, supervisors and administrators are encountering logon failures when trying to log into the Webex Call Center system.

Initial symptoms: 

Anyone logged into the system was working correctly. Users that were not logged in at the time of the outage, or logged out during the outage, could not log into the system. Calls could be answered normally.

Duration: 
01/27/2020 2:57 pm - 01/27/2020 5:36 pm
Impact to Mason: 

Departments with call centers, such as Support Center, Admissions, Registrar, etc. experienced issues with agents being able to log into the system.

Affected Services: 
Telephone Services
ROOT CAUSE ANALYSIS
Cause: 

From the vendor:

Investigation confirmed the symptoms and showed that some of the instances on the backend responsible for storing user information were unreachable and needed to be restored. These instances were found to be out of memory which made them inaccessible. The monitoring agents responsible for alerting based on memory thresholds, were not operational. It was determined that the monitoring agent was misconfigured. The team is still currently investigating on what triggered the out of memory condition. We have a hypothesis that certain bulk configurations acted as a possible trigger. We remediated the issue by bringing back these instances and verified that the configuration was intact. Validation efforts concluded that the services were operational.

Resolution: 

From the vendor:

We have corrected configuration on the monitoring agents for memory threshold based alerts. This should help us detect similar issues proactively.

Prevention: 

From the vendor:

We are also reviewing the design to improve data resiliency against similar failures i.e. user access. We will provide further updates on  the triggering condition as well as the progress on design improvements once identified and upon request.

STATISTICS
Service Team: 
Telecom Admin / Enterprise Collaboration