Outage category: Applications, Blackboard, Website
Location: All campuses
Status: Closed
Resolved Alert:
Initial Symptoms
Customers could not reach specific websites that were hosted on the server Jiju.
Root Cause Analysis
Cause
Multiple DNS records were inadvertently removed from the IPControl system which resulted in unresolved ip addresses. This occurred when doing cleanup of old, unneeded records.
Resolution
DNS records were added back into the system.
Prevention
This occurred when doing cleanup of old, unneeded records. Going forward recommend doing a DNS push immediately following any submit. In this case, submit was done without an associated push so problem didn’t occur until an unrelated push was done. Also recommend that any DNS record delete be done as part of an RFC.