Virtual Machine Outage

July 18, 2022 11:57 a.m.
Outage category: 
Applications, Website
Location: 
All Campuses
Status: 
Closed
Resolved alert: 
04/03/2022 10:09 am
The following systems, some related to Banner, were down:
ADFSAZ02
admittop5
arubanetedit
build-1
clearpass
coligod
Corky1
Corky2
CORONA
DATA-01-A-FM01
DCOperation OLD
DEVARON
ENDEAVOR
FNDTADM3P
hestia
icsa4t
Lugnut
NALHUTTA
navigod
ptrain37
Ptrain9
shamu1
shamu2
shibdev1
STINGRAY
Trondheim
vCLS-be96ebbd-fe4f-4641-be61-346ac42e08c0
vernaculusp2
vskyline01
ZURG
Initial symptoms: 

Alerts noticed on SiteScope on multiple occasions, which were then shown to be a result of the Nodus 3 crashing.

Duration: 
04/03/2022 1:44 am - 04/03/2022 10:09 am
Impact to Mason: 

The following systems, some related to Banner, were down:

ADFSAZ02
admittop5
arubanetedit
build-1
clearpass
coligod
Corky1
Corky2
CORONA
DATA-01-A-FM01
DCOperation OLD
DEVARON
ENDEAVOR
FNDTADM3P
hestia
icsa4t
Lugnut
NALHUTTA
navigod
ptrain37
Ptrain9
shamu1
shamu2
shibdev1
STINGRAY
Trondheim
vCLS-be96ebbd-fe4f-4641-be61-346ac42e08c0
vernaculusp2
vskyline01
ZURG

Affected Services: 
Banner Admin (Service), Administrative Applications
ROOT CAUSE ANALYSIS
Cause: 

Pending investigation. NODUS 3 had a similar event in April and it is being escalated to VMware.

Resolution: 

Reboot NODUS 3 and migrated hosts and verified they were operational. More substantive troubleshooting is ongoing.

Prevention: 

Working with vendor to determine issue.

STATISTICS
Service Team: 
CCSO, Advanced Technologies, Data Center Operations