Webex E2E Encrypted Outage

Outage category: 
Webex
Location: 
All Campuses
Status: 
Open
Resolved alert: 
06/30/2020 9:34 am

Hosts were able to join End-to-End (E2E) encrypted meetings after a considerable delay. The guests of the meetings would be unable to completely join the meeting as the audio would begin hanging at “Connecting”.

Initial symptoms: 

Enterprise Collaboration was notified on Monday, June 29th. Mason clients began experiencing the following behavior Friday, June 26th, but notification to the service owner (Enterprise Collaboration) was delayed. This was a global problem, affecting all Webex sites, not only Mason: Hosts were able to join End-to-End (E2E) encrypted meetings after a considerable delay. The guests of the meetings would be unable to completely join the meeting as the audio would begin hanging at “Connecting”.

Duration: 
06/29/2020 3:02 pm - 06/30/2020 9:34 am
Impact to Mason: 

Users that required E2E encrypted meetings were unable to utilize this feature. A small, important subset of the Mason population requires this to stay current with HIPAA requirements. This was a problem globally, affecting more than just Mason.

ROOT CAUSE ANALYSIS
Cause: 

On further analysis by Cisco Webex, they found that the issue is occurring due to a known issue (bug) on their collaboration bridge servers. Cisco bug ID CSCvu85326.

Resolution: 

Testing on Mason’s beta website proved successful and the update was pushed globally to all Webex sites over the course of several hours.

Prevention: 

Cisco Webex will continue to use update channels that push patches and new releases to select customers. To prevent this type of failure, they will increase the diversity of the pool of early adopters utilized for testing and resolve known issues before it manifests in the production update channel.

STATISTICS