MyMason/Blackboard Unavailable

November 30, 2018 12:00 a.m.
Outage category: 
Blackboard, myMason
Location: 
All campuses
Status: 
Closed
Resolved alert: 
10/21/2018 11:15 pm

MyMason/Blackboard was unavailable from approximately 3:35 p.m. to 3:55 p.m. ET on Monday, October 22. While Blackboard Inc. has determined the cause of the issues today, they are working to fully mitigate the problem. A final update will be sent when the issue is fully resolved.

Initial symptoms: 

Blackboard was unavailable several times on October 21 and 22.

Duration: 
10/21/2018 10:30 pm - 10/21/2018 11:15 pm
Impact to Mason: 

Blackboard and related services were unavailable.

Affected Services: 
Blackboard Courses
ROOT CAUSE ANALYSIS
Cause: 

The root cause of the incident was due to an issue with ActiveMQ where the ActiveMQ broker appears only on instances that have been terminated. This occurs with automated restarts when the broker is not terminated and AWS restarts the instances. As a result the database does not remove broker instance and when new apps are brought up, it appears the broker is running. It was also noted during the service interruptions experienced GMU was running their SIS feed file and this caused the outage. It was also observed the What’s New Module and custom language exacerbated the performance issue.

Resolution: 

Vendor working to create a permanent solution, expected in mid-December.

Prevention: 

Until the issue is fixed (expected mid-December) workarounds have been established to ensure the issue does not occur again. Nodes will not be restarted (except manually) and building blocks will not be installed or updated.

STATISTICS
Service Team: 
OLR