Mason WebEx Unavailable

Outage category: 
Applications, Webex
Location: 
All campuses
Status: 
Closed
Resolved alert: 
02/04/2019 1:44 pm

Mason users were initially having intermittent issues joining Webex Meeting Center. After several attempts to join a meeting the service/server completely shut down and was no longer available.

Initial symptoms: 

Mason users were initially having intermittent issues joining Webex Meeting Center. After several attempts to join a meeting the service/server completely shut down and was no longer available.

Duration: 
02/04/2019 1:20 pm - 02/04/2019 1:44 pm
Impact to Mason: 

Those users with scheduled meetings were unable to access the service and hold their meetings. Anyone attempting to schedule a meeting at this time was unable to do so.

Affected Services: 
Webex Meetings
ROOT CAUSE ANALYSIS
Cause: 

Cisco engineering determined the root cause was a software defect in the network edge device code. As a result, there was a network connectivity issue on a link hosting services routed between the San Jose data center and Virginia data center. The affected device continued to take production traffic but was intermittently unable to process client requests.

Resolution: 

Traffic was drained from the affected device and removed from production

Prevention: 
Mason

Not applicable.

Vendor

Cisco Engineering upgraded the software code on the affected network devices. All Webex data centers were audited to ensure no other network devices were exposed due to the software defect. Cisco Engineering is reviewing opportunities to add additional software code resiliency to prevent a recurrence. Engineering will enhance monitoring to aid in the detection moving forward.

STATISTICS
Service Team: 
Enterprise Collaboration