Phone Outage – Katherine Johnson Hall and Hylton Performing Arts

Outage category: 
Telephone
Location: 
Katherine Johnson Hall and Hylton Performing Arts
Status: 
Closed
Resolved alert: 
09/15/2023 2:26 pm

Phones in the two buildings have no dial tone and is blank on the screen.

Initial symptoms: 

We received TDX tickets of the phones not working in Classroom tech offices at SciTech followed by a TDX from the Hylton box office group. We researched it further and decided an alert needed to be sent just for the 2 buildings as it was not network related or a campus wide issue.

Duration: 
09/15/2023 9:00 am - 09/15/2023 2:26 pm
Impact to Mason: 

Avaya VOIP users in Katherine G Johnson Hall and Hylton Performing Arts Center were unable to make or receive phone calls.

Affected Services: 
Telephone Services
ROOT CAUSE ANALYSIS
Cause: 

These 2 buildings were scheduled to be moved to the Vonage system on Tuesday, 9/19. In preparation of this move, a list was sent to our Carousel vendor on Thursday so she would have the lines to taken out of the PBX and get these lines set up for 5 digit dialing. Unfortunately she misread the date/buildings and mistook them for the move we had earlier that morning, 9/14, as our 9900 Main Street implementation and proceeded to remove the Hylton and KJH from the PBX. We did not know this until Friday morning when offices came online.

Resolution: 

The TNs, phone lines, had to be completely rebuilt for each of the 80+ phone lines. They also had to be deleted from 5 digit dialing. This did not entirely fix it as we thought. The nodes did not come back in the usual setting. Nikolai Majorin and our Carousel vendor went to SciTech campus to troubleshoot. A trial and error took place using MASONS (a software used to interact with the Avaya phones) to try different modes. The node had to be changed in MASONS to 6001, then the devices were rebooted and came back online. This is the node that the phones move to when they are dropped off the network for inactivity. Nikolai had never had to switch it to that for a new install, but trying it out worked. They walked around and reset phones in all the areas.

Prevention: 

Make sure that lists are double checked prior to making moves and verified by several parties that we are able to make the move off of the PBX.

STATISTICS
Service Team: 
Telecom Admin, Vendor, and Network and Security Operations and Engineering