Intermittent Phone Outage

Outage category: 
VoIP phone service
All PBX Mason users on all campuses.
Resolved alert: 
09/25/2021 9:05 am

Most phones were completely down during this outage. At times service would come back up intermittently but then drop.

Initial symptoms: 

Around 2:45 pm on 9/24 phones started going in and out intermittently. It was like that for quite awhile then phones appeared to have dropped completely by 3:30 or so.

09/24/2021 3:54 pm - 09/25/2021 9:05 am
Impact to Mason: 

All users were affected by the outage. Ben Allen was in the police dispatch monitoring service there. No callers were able to make or receive calls on their Avaya PBX phones.

Affected Services: 
Telephone Services

The vendor is still uncertain but the DR failed to respond when the main system failed. There was no backup for recovery causing to lose everything.


The vendor had to troubleshoot the DR virtual trunk while another engineer had to rebuild the main node as it was corrupted. This took several man hours.


This all started appeared to begin on Monday when the auto renewal of a certificate failed. It cascaded a series of failures/issues that caused the session manager and other tools to be rebuilt. Once the session manager was completed by Friday the rest was finishing up on the main campus and that us when they found that the DR was not working.

Service Team: 
Telecom, Carousel (vendor and onsite engineer), EIS-networking, Network and Security Operations