eCUI Outage

November 16, 2022 7:50 a.m.
Outage category: 
Citrix Virtual Lab, Virtual Computing Library (VCL)
Location: 
eCUI Citrix environment
Status: 
Closed
Resolved alert: 
11/14/2022 3:55 pm

The eCUI Citrix environment stopped serving desktops and applications to customers. The customers would receive an error message stating that no resources were available.

Initial symptoms: 

Customer emailed someone with ITS who contacted Computing Services at around 11:30 on Saturday (11/12). Initial review suggested the system could be restored quickly as symptoms paralleled similar outages. However, upon deeper review there was a larger systemic issue, and the outage was declared.

Duration: 
11/12/2022 1:57 pm - 11/14/2022 3:55 pm
Impact to Mason: 

eCUI secure research was out of service.

Affected Services: 
Enterprise CUI
ROOT CAUSE ANALYSIS
Cause: 

The outage was caused by Kerberos authentication errors between the Citrix resources. This prevented the VDI systems from communicating with the delivery controllers. This was caused by patches applied to the Domain Controllers on Friday (11/11). These patches included updates to Kerberos encryption levels.

Resolution: 

Registry keys were added to the domain controllers to ensure Kerberos could authenticate using appropriate encryption levels.

Prevention: 

Changes are being reviewed for the patching of domain controllers, such as expanded documentation within the RFCs.

STATISTICS
Service Team: 
Advanced Technologies, Computing Services, CCSO, CCSE