It started intermittently by a few centers by comments in our MS Teams chat. Some agents were disconnected from the call center application and could not log back in. Then more centers started reporting it. This started around 9 – 9:15.

Periodically all call centers at different times were affected. Some not as much as others. Those affected could not log into the application to take calls and the centers appeared closed.

Webex Contact Center (Cisco) found that increased data storage and transport activity along with their updated Analyzer resulted in the initial issues.
Root cause was found to be combination of issues that exposed performance issues on the backend:
• Agent configuration data shared the path with reporting data. This caused intermittent disruptions to Agent logins.
• High number of queries running on older indices that slowed down Analyzer query performance.
• High number of redundant audit records on the system that resulted in additional load on the backend.

Cisco completed several corrective actions that include:
• Deletion of older indices to improve performance.
• Purging of redundant Audit records.
• Tuning the back-end parameters for increased performance.
• Enhancement to the backend service responsible for database calls.

Additional performance improvements on the backend services will be completed on or before 12/19:
• Separating the data path for agent configuration and Analyzer data.
• Code changes to prevent redundant audit records from being written in the backend.