Mason users were initially having intermittent issues joining Webex Meeting Center. After several attempts to join a meeting the service/server completely shut down and was no longer available.
Those users with scheduled meetings were unable to access the service and hold their meetings. Anyone attempting to schedule a meeting at this time was unable to do so.
Cisco engineering determined the root cause was a software defect in the network edge device code. As a result, there was a network connectivity issue on a link hosting services routed between the San Jose data center and Virginia data center. The affected device continued to take production traffic but was intermittently unable to process client requests.
Traffic was drained from the affected device and removed from production
Cisco Engineering upgraded the software code on the affected network devices. All Webex data centers were audited to ensure no other network devices were exposed due to the software defect. Cisco Engineering is reviewing opportunities to add additional software code resiliency to prevent a recurrence. Engineering will enhance monitoring to aid in the detection moving forward.