Outage category: Webex
Location: All Campuses
Status: Open
Resolved Alert:
Initial Symptoms
Users were unable to use Webex audio or PSTN call-in or call-back.
Root Cause Analysis
Cause
Webex is handling unprecedented load from our Enterprise customers, which contributed to the extended restoration time. In addition to that, there was a software failure that cascaded into a particular audio pool. Customers who were directed to that audio pool experienced a high rate of failure. Both primary and secondary pools were affected by the software failure, so our normal high availability posture failed. Since our services are distributed, the issue only occurred for the users that were landing on the impacted audio pool.
Resolution
The affected audio pool was taken out of rotation. The remainder of systems in the pool are fully functional and capable of handling the loads. Our teams will be closely monitoring services as we continue throughout the business day. Engineering continues to investigate the underlying root cause of the software failure.
Prevention
We know that this was very impactful for your teams and we appreciate your patience while we identified the issue and implemented the solution. Further root cause analysis and retrospectives will be completed to determine how we can better detect and remediate in the future.