Outage category: myMason, Blackboard
Location: All users of MyMason/Blackboard
Status: Closed
Resolved Alert: Wed, December 1, 2021 2:55 pm
Initial Symptoms
Users called into Support Center that Blackboard was unavailable
Root Cause Analysis
Cause
Blackboard engineers were performing an internal development site resource cleanup and during this process the Learn security groups were deleted. The missing security group caused a break in the database which led to an outage for Learn production sites.
Resolution
Blackboard patched the security rules as a workaround and the sites started to recover. Blackboard immediately corrected the issue and redeployed, which fixed the issue permanently.
Prevention
Blackboard engineers will further audit similar imported resources. The engineers will inspect the security groups in use and validate that Learn-related security groups are not being used by any microservices.