Around 2:45 pm on 9/24 phones started going in and out intermittently. It was like that for quite awhile then phones appeared to have dropped completely by 3:30 or so.

All users were affected by the outage. Ben Allen was in the police dispatch monitoring service there. No callers were able to make or receive calls on their Avaya PBX phones.

The vendor is still uncertain but the DR failed to respond when the main system failed. There was no backup for recovery causing to lose everything.

The vendor had to troubleshoot the DR virtual trunk while another engineer had to rebuild the main node as it was corrupted. This took several man hours.

This all started appeared to begin on Monday when the auto renewal of a certificate failed. It cascaded a series of failures/issues that caused the session manager and other tools to be rebuilt. Once the session manager was completed by Friday the rest was finishing up on the main campus and that us when they found that the DR was not working.