Webex Issues Using Computers for Audio

Outage category: 
Webex
Location: 
All Campuses
Status: 
Open
Resolved alert: 
03/18/2020 2:30 pm

Webex Meetings users reported issues with joining their meeting. This issue was intermittent for some of Cisco’s customers, however a set of large enterprise customers experienced inability to join their meetings

Initial symptoms: 

Users were unable to use Webex audio or PSTN call-in or call-back.

Duration: 
03/18/2020 11:50 am - 03/18/2020 2:30 pm
Impact to Mason: 

Users had inconsistent issues connecting to Webex conferences. Audio issues occurred during PTSN call-in or call-back functions.

Affected Services: 
Webex Meetings
ROOT CAUSE ANALYSIS
Cause: 

Webex is handling unprecedented load from our Enterprise customers, which contributed to the extended restoration time. In addition to that, there was a software failure that cascaded into a particular audio pool. Customers who were directed to that audio pool experienced a high rate of failure. Both primary and secondary pools were affected by the software failure, so our normal high availability posture failed. Since our services are distributed, the issue only occurred for the users that were landing on the impacted audio pool.

Resolution: 

The affected audio pool was taken out of rotation. The remainder of systems in the pool are fully functional and capable of handling the loads. Our teams will be closely monitoring services as we continue throughout the business day. Engineering continues to investigate the underlying root cause of the software failure.

Prevention: 

We know that this was very impactful for your teams and we appreciate your patience while we identified the issue and implemented the solution. Further root cause analysis and retrospectives will be completed to determine how we can better detect and remediate in the future.

 

STATISTICS
Service Team: 
Enterprise Collaboration