MicroStrategy outage

December 04, 2020 8:21 a.m.
Nodus5 Failure - 12/1/2020
Outage category: 
Applications, Hardware
Location: 
All Campuses
Status: 
Closed
Resolved alert: 
12/01/2020 9:56 am

MicroStrategy web access is currently unavailable while Engineers work to correct a server issue. Additional updates will be provided as more information becomes available

 

Initial symptoms: 

Nodus5 server offline and the currently resident VMs being orphaned.

Duration: 
12/01/2020 9:10 am - 12/01/2020 9:56 am
Impact to Mason: 

Per the ticket, only MicroStrategy users. The actual impact was broad in that 30+ virtual machines (VM) went off-line including the VM running SiteScope. However, the only ticket actual opened had to do with the MicroStrategy web service not being accessible.

Affected Services: 
MicroStrategy
Other Affected Services: 
Services running on the VM's on Nodus5. Systems were restored once Nodus5 was returned to service with a few exceptions such as MicroStrategies which had an issue restarting.
ROOT CAUSE ANALYSIS
Cause: 

The root cause for the outage was a hardware/driver failure in the Nodus5 server.

Resolution: 

Reboot Nodus5 and move orphaned VMs to a different physical host. The MicroStrategy issue was resolved by starting the associated proxy server.

Prevention: 

Working with hardware and software vendors to determine root cause of Nodus5 failure – update firmware and drivers, and potentially replace hardware components.

STATISTICS
Service Team: 
Advanced Technology and Cloud Compute and Storage Operations/Engineering