Outage category: Telephone
Location: All Call Centers using Webex Contact Center
Status: Closed
Resolved Alert:
Initial Symptoms
It started intermittently by a few centers by comments in our MS Teams chat. Some agents were disconnected from the call center application and could not log back in. Then more centers started reporting it. This started around 9 – 9:15.
Root Cause Analysis
Cause
Webex Contact Center (Cisco) found that increased data storage and transport activity along with their updated Analyzer resulted in the initial issues.
Root cause was found to be combination of issues that exposed performance issues on the backend:
• Agent configuration data shared the path with reporting data. This caused intermittent disruptions to Agent logins.
• High number of queries running on older indices that slowed down Analyzer query performance.
• High number of redundant audit records on the system that resulted in additional load on the backend.
Resolution
Cisco completed several corrective actions that include:
• Deletion of older indices to improve performance.
• Purging of redundant Audit records.
• Tuning the back-end parameters for increased performance.
• Enhancement to the backend service responsible for database calls.
Prevention
Additional performance improvements on the backend services will be completed on or before 12/19:
• Separating the data path for agent configuration and Analyzer data.
• Code changes to prevent redundant audit records from being written in the backend.