Webex Contact Center (WCC) Outage

Outage category: 
Telephone
Location: 
All Webex Call Center users
Status: 
Closed
Resolved alert: 
10/19/2020 3:00 pm

Call Centers users could not log into Webex Call Center to receive calls for their respective phone queues.

Initial symptoms: 

Users reporting they could not log into the Webex Call Center system to be able to answer calls in queue.

Duration: 
10/19/2020 1:15 pm - 10/19/2020 3:00 pm
Impact to Mason: 

Call Centers such as Support Center, Provost, or Admissions could not have agents log into phones. Any agent already logged in was able to handle calls in queue.

Affected Services: 
Telephone Services
ROOT CAUSE ANALYSIS
Cause: 

Vendor timeline of issue:

10/19/2020 11:45 CDT – Multiple reports received regarding Agents and supervisors are unable to log into the platform
10/19/2020 12:20 CDT – Failover to secondary load balancer initiated; Some agents are able to log back into the platform
10/19/2020 12:25 CDT – Reports indicate that a subset of agents are still unable to log on
10/19/2020 13:28 CDT – Services are fully restored

Resolution: 

From the vendor:

The issue was traced back to an error condition on the load balancer. When we failed over to the secondary load balancer, logins began to work. However, it appeared that the sudden logon surge caused a problem on our authentication services that is still under investigation. A large number of agents received an authentication error. After restarting the authentication service, agents were able to log back in.

Prevention: 

We will provide a full root cause once the incident review has concluded

STATISTICS
Service Team: 
Enterprise Collaboration