Call Center Move Intermittent Outages

Outage category: 
Telephone, VoIP phone service
Location: 
All Campuses
Status: 
Closed
Resolved alert: 
01/16/2020 2:00 pm

Internal and external calls directed to the following call centers:  Immunization/Student Health, ITS Support Center, Housing, HR/Payroll, International Programs, Counseling Center, Admissions/Graduate Admissions

Initial symptoms: 

Call Centers stopped receiving incoming calls.

Duration: 
01/16/2020 10:00 am - 01/16/2020 2:00 pm
Impact to Mason: 

Customers had intermittent issues reaching the following call centers: Immunization/Student Health, ITS Support Center, Housing, HR/Payroll, International Programs, Counseling Center, and Admissions/Graduate Admissions.

Affected Services: 
Telephone Services
ROOT CAUSE ANALYSIS
Cause: 

This was a scheduled phase 2 move of 9 call centers off of our AACC (Avaya Aura Contact Centre) to the WCC (Webex Contact Center.  Moves were scheduled for staggered times:

10:00 am = Immunization/Student Health, ITS Support Center.

1:00 pm = Housing, HR/Payroll, International Programs.

4:00 pm = Counseling Center, Admissions/Graduate Admissions.

Supervisors were on a Webex meeting for the 10:00 rollover going through last minute changes and roll outs when we were notified that other call centers were not receiving calls.

Resolution: 

At around 10:45 – 11:00 we realized that Verizon ported all of the numbers at once so we were completely LIVE.  The remaining supervisors were notified by email and phone to join the current Webex to proceed at once with going live.  After a very hectic and unexpected time change for the later centers we were able to get 5 centers running by around 12:30 with some issues but they were receiving calls.  The others came online slowly but were able to receive calls well before 2:00 with issues but it was all clear by 3:00.   Clean up continued the rest of the day and the vendor was online for any issues that needed attention or questions that came up.

Prevention: 

Better communication between Mason and vendors is needed as there was a miscommunication between the two vendors.

STATISTICS
Service Team: 
Telecom Admin