Outage category: Applications, Webex
Location: All campuses
Status: Closed
Resolved Alert:
Initial Symptoms
Mason users were initially having intermittent issues joining Webex Meeting Center. After several attempts to join a meeting the service/server completely shut down and was no longer available.
Root Cause Analysis
Cause
Cisco engineering determined the root cause was a software defect in the network edge device code. As a result, there was a network connectivity issue on a link hosting services routed between the San Jose data center and Virginia data center. The affected device continued to take production traffic but was intermittently unable to process client requests.
Resolution
Traffic was drained from the affected device and removed from production
Prevention
Mason
Not applicable.
Vendor
Cisco Engineering upgraded the software code on the affected network devices. All Webex data centers were audited to ensure no other network devices were exposed due to the software defect. Cisco Engineering is reviewing opportunities to add additional software code resiliency to prevent a recurrence. Engineering will enhance monitoring to aid in the detection moving forward.