COVID Health Check Website Outage

March 03, 2021 12:16 p.m.
Outage category: 
Website
Location: 
All Users
Status: 
Closed
Resolved alert: 
03/02/2021 12:54 pm

Users were unable to log into the COVID Health Check Website. They were getting a 504 Gateway Timeout Error.

Initial symptoms: 

Users were not able to access the website to do their daily health checks.

Duration: 
03/02/2021 9:30 am - 03/02/2021 12:54 pm
Impact to Mason: 

All students, faculty, and staff were not able to submit their daily screening and receive approval to be on-campus.

Other Affected Services: 
COVID Health Check
ROOT CAUSE ANALYSIS
Cause: 

An issue with SQL caused the CPU and memory utilization to spike on the SQL database back end. Once this happened, users were unable to use this service, and their sessions would freeze.

Resolution: 

Limited the number of sessions from “unlimited” to 600, increased RAM from 20 to 32 GB, increased the number of CPUs from 4 to 8, and maxed out the amount of memory SQL could use (maxed at 28 GB)

Prevention: 

The Service Team is currently looking into the SQL responsible for the CPU and memory utilization spikes. Limiting the maximum number of sessions and providing more resources, and capping resources that the SQL can use should mitigate this issue from happening again.

STATISTICS
Service Team: 
CCSO, Advanced Technologies, DBA, Enterprise Application Support and Development