Issues with Websites and Mail Delivery

May 06, 2019 1:38 p.m.
Network Outage - Multiple Locations - All Campus
Outage category: 
Applications, Blackboard, Website
Location: 
All campuses
Status: 
Closed
Resolved alert: 
05/03/2019 1:53 pm

DNS records pointing to F5 load-balancer were not resolving.

Initial symptoms: 

Customers could not reach specific websites that were hosted on the server Jiju.

Duration: 
05/03/2019 12:15 pm - 05/03/2019 1:53 pm
Impact to Mason: 

Websites were not available. Customers were getting page not found errors as IP addresses were not resolving to DNS names.

Affected Services: 
Domain Name Server (DNS), MasonLive Email (Alumni Email), Blackboard Courses
ROOT CAUSE ANALYSIS
Cause: 

Multiple DNS records were inadvertently removed from the IPControl system which resulted in unresolved ip addresses. This occurred when doing cleanup of old, unneeded records.

Resolution: 

DNS records were added back into the system.

Prevention: 

This occurred when doing cleanup of old, unneeded records. Going forward recommend doing a DNS push immediately following any submit. In this case, submit was done without an associated push so problem didn’t occur until an unrelated push was done. Also recommend that any DNS record delete be done as part of an RFC.

STATISTICS
Service Team: 
Cloud, Compute and Storage Operations; Network & Security Operations