Webex Teams Degraded Performance

Outage category: 
Web Conferencing
Location: 
Webex Conferencing Services
Status: 
Closed
Resolved alert: 
01/22/2021 9:07 am

A service impact occurred on January 21st affecting a set of Webex organizations. Affected organizations had users who were unable to upload files and screenshare from the Webex desktop app. The web client and mobile app were unaffected during the incident duration.

Initial symptoms: 

Webex notified us of issues with service. ITS Support Center confirmed they were experiencing the issue within their own office staff.

Duration: 
01/21/2021 5:30 pm - 01/22/2021 9:07 am
Impact to Mason: 

A subset of Mason users were unable to upload files and screenshare from the Webex desktop app. The web client and mobile app were unaffected during the incident duration.

ROOT CAUSE ANALYSIS
Cause: 

Engineering identified that the settings cache for a subset of organizations had become corrupt. A software defect was exposed when a toggle feature update unexpectedly corrupted the settings cache for multiple organizations. While the cache was incorrect, the database values were unaffected. The Webex desktop app queried the incomplete configuration in the cache, which led to file upload and screenshare failures.

Resolution: 

Engineering manually repaired organizations that reported the issue, and a code update was deployed during the maintenance window on January 22nd to automatically refresh the settings cache and resolve the issue.

Prevention: 

1) Engineering manually repaired organizations that reported the issue, and a code update was deployed during the maintenance window on January 22nd which refreshed the settings cache for all affected organizations. 2) Engineering deployed code updates to address the defect and extra code hardening to protect the settings cache which will prevent recurrence.

STATISTICS
Service Team: 
Enterprise Collaboration