Systems engineers reported DNS servers in DR Site were not able to reach the DNS servers in Fairfax Data Center. Customers reported issues with Virtual Private Networks (VPN) or certain applications.

Users received different error messages while connecting to the VPN or time outs when trying to connect to certain applications.

Maintenance for the VXLAN transition affected the VPN connection due to the VPN not able to connect to the Ruth Server. Connections not going through Ruth was able to connect. Applications on specific networks also had issues due to the VXLAN transition, Network Engineers worked to determine all of the affected systems, and worked to fix issues on a case by case basis.

Part of the VXLAN change was backed out and this resolved the issue with the DNS connection between Science & Tech and Fairfax campus. Engineers are working to restore the other networks that are having issues. Work is being scheduled with affected application owners to restore services to previous levels.

In the future, for each datacenter maintenance have server admins or application owners available to test their systems right after change is made. In that case, we can fix issues on the spot or roll back changes right away. Outside the maintenance window, it will be very hard for us to troubleshooting and roll back changes, because these operations may potentially break more things.