Call Center Digital Lines Down

Outage category: 
Telephone
Location: 
Specific Call Centers: Support Center, JC information Desk, Facilities, and some additional users
Status: 
Closed
Resolved alert: 
07/11/2019 8:00 am

Some digital phones were experiencing issues maintaining a dial tone, receiving or placing calls.

Initial symptoms: 

Digital Call Center phones were having intermittent problems maintaining a dial tone, receiving and placing calls.  This initially effected 6 phones on campus.  These phones were moved over to a working TN.  As troubleshooting continued these lines and others lost service completely.

Duration: 
07/10/2019 10:42 am - 07/11/2019 8:00 am
Impact to Mason: 

Call Center employees with these phone lines were unable to login to receive calls.  This put a burden on other employees in their areas.  Some workarounds were established.  Troubleshooting also impacted their daily routine while trying to work with the telecom vendor.

Affected Services: 
Telephone Services
ROOT CAUSE ANALYSIS
Cause: 

During a scheduled software code upgrade on Gateways 12 -0 and 12- 1 files were corrupted and caused a domino effect that brought down these lines.  Digital lines were moved off cards 12-0 and 12-1 onto the 36-0-3 but it appears the corruption followed after day one.

Resolution: 

From the vendor:

First, the failed task on the PBX was addressed by the initialization but the DSP still would not come up. A corrupt configuration file was discovered in the PBX. A copy of the original configuration file was created and then deleted the original file. A new configuration file was created and that worked. This file is retrieved by the IPMG from the call server when the ELAN registers and it tells the IPMG what type and how many DSP to bring up. Once a new config file was created the DSPs came right up.

IPMG 36-0 is now fully enabled and registered.

Prevention: 

Vendor reviewing to determine how to prevent additional occurrences.

STATISTICS