Webex Outage

Outage category: 
Webex
Location: 
All locations
Status: 
Open
Resolved alert: 
03/26/2020 12:01 pm

Webex Meetings was unavailable or inconsistent when joining. The site gmu.webex.com was down. On-going meetings were unaffected.

Initial symptoms: 

Users were not able to join meetings, the website gmu.webex.com was unavailable.

Duration: 
03/26/2020 11:15 am - 03/26/2020 12:01 pm
Impact to Mason: 

Users attempting to  join meetings were unable to do so.

Affected Services: 
Webex Meetings
ROOT CAUSE ANALYSIS
Cause: 

Cisco Engineering identified a service latency condition that occurred when high volumes of users connected to the Webex sites for web page joins, meeting launches, and Webex desktop/client updates. The root cause was a higher than normal I/O wait on the data for the Webex web servers when querying for the meeting client update availability. Due to the combination of traffic initiators during a high traffic period, the web server pools became unable to handle the additional requests, which led to small percentages of meeting join failures.

Resolution: 

Cisco Engineering made a configuration change to add additional traffic shaping to the random client upgrade paths during the March 26th maintenance window.

Additional web service storage was deployed to production over the 3/27 weekend maintenance window.

Prevention: 

Cisco Engineering made a configuration change to add additional traffic shaping to the random client upgrade paths during the March 26th maintenance window.

Additional web service storage was deployed to production over the 3/27 weekend maintenance window.

A client code change is also being deployed to add further load balancing and randomization intelligence to the update process.

STATISTICS