Outage category: Telephone, VoIP phone service
Location: All Campuses
Status: Closed
Resolved Alert:
Initial Symptoms
Call Centers stopped receiving incoming calls.
Root Cause Analysis
Cause
This was a scheduled phase 2 move of 9 call centers off of our AACC (Avaya Aura Contact Centre) to the WCC (Webex Contact Center. Moves were scheduled for staggered times:
10:00 am = Immunization/Student Health, ITS Support Center.
1:00 pm = Housing, HR/Payroll, International Programs.
4:00 pm = Counseling Center, Admissions/Graduate Admissions.
Supervisors were on a Webex meeting for the 10:00 rollover going through last minute changes and roll outs when we were notified that other call centers were not receiving calls.
Resolution
At around 10:45 – 11:00 we realized that Verizon ported all of the numbers at once so we were completely LIVE. The remaining supervisors were notified by email and phone to join the current Webex to proceed at once with going live. After a very hectic and unexpected time change for the later centers we were able to get 5 centers running by around 12:30 with some issues but they were receiving calls. The others came online slowly but were able to receive calls well before 2:00 with issues but it was all clear by 3:00. Clean up continued the rest of the day and the vendor was online for any issues that needed attention or questions that came up.
Prevention
Better communication between Mason and vendors is needed as there was a miscommunication between the two vendors.