Outage category: Telephone
Location: All campuses - ONLY those that requested any adds, moves, changes or lost an appearance (an additional number displayed on their phone) during the outage time frame
Status: Closed
Resolved Alert:
Initial Symptoms
An issue was identified on 7/17 after several tickets were received for missing or malfunctioning appearances on desk phones. After further investigation the issue was traced to Yealink desk phones on Vonage not automatically provisioning. After checking logs in the Vonage admin portal, it became clear that none of the desk phones in any account were successfully auto-provisioning. The ability to place or receive calls from the primary extension had not been affected. It was unclear what the source of the issue was. John hanks was contacted to check our network but hadn’t seen anything out of the ordinary. Vonage was also contacted for guidance and to see if there have been any changes from their end that may be the cause. A packet capture was run but additional captures and testing were need to be completed when the relevant parties were available.
Root Cause Analysis
Cause
Mason experienced a power outage on June 21st. Consequently, when thousands of our phones attempted to re-sync with Vonage, it was perceived as a potential security threat, leading to our IP being blocked due to excessive traffic. Following this event, our Telecom team did not immediately identify an issue with our Vonage/Yealink phones until mid-July, when several tickets were submitted by various Telecom Coordinators. Our Yealink phones perform an automatic update at a minimum of once daily, which enables the use of several key features such as shared line appearances and the company directory. These updates ceased on June 21st, halting any additions, moves, or changes.
Resolution
After two weeks of rigorous testing and troubleshooting involving our Network group and Vonage engineers, under the leadership of Nikolai Majorin, it was determined that the problem originated from Vonage’s services and was not a configuration issue on our network. Vonage has now whitelisted our 2 IPs to prevent this issue from recurring, and we have successfully tested the system after the change.
Prevention
Vonage has now whitelisted our 2 IPs to prevent this issue from recurring.