Mason Call Centers Intermittent Issues Logging In

Outage category: 
All Call Centers using Webex Contact Center
Resolved alert: 
11/11/2021 12:57 pm

The user was unable to connect to the call center in order to login and receive calls. The call centers appeared closed even though they were open and trying to be functional.

Initial symptoms: 

It started intermittently by a few centers by comments in our MS Teams chat. Some agents were disconnected from the call center application and could not log back in. Then more centers started reporting it. This started around 9 – 9:15.

11/11/2021 9:30 am - 11/11/2021 12:57 pm
Impact to Mason: 

Periodically all call centers at different times were affected. Some not as much as others. Those affected could not log into the application to take calls and the centers appeared closed.

Affected Services: 
Telephone Services
Other Affected Services: 
Some call centers appeared closed and were unavailable at this time.

Webex Contact Center (Cisco) found that increased data storage and transport activity along with their updated Analyzer resulted in the initial issues.
Root cause was found to be combination of issues that exposed performance issues on the backend:
• Agent configuration data shared the path with reporting data. This caused intermittent disruptions to Agent logins.
• High number of queries running on older indices that slowed down Analyzer query performance.
• High number of redundant audit records on the system that resulted in additional load on the backend.


Cisco completed several corrective actions that include:
• Deletion of older indices to improve performance.
• Purging of redundant Audit records.
• Tuning the back-end parameters for increased performance.
• Enhancement to the backend service responsible for database calls.


Additional performance improvements on the backend services will be completed on or before 12/19:
• Separating the data path for agent configuration and Analyzer data.
• Code changes to prevent redundant audit records from being written in the backend.

Service Team: 
ITS, Veracity & Cisco - vendors