Issues with Password.gmu.edu

Outage category: 
Applications
Location: 
Users attempting to set or reset their password
Status: 
Closed
Resolved alert: 
03/22/2022 12:45 pm

Users were either unable to set a password at the password website or unable to use the password reported to be successfully set.

Initial symptoms: 

Users were either unable to set a password at the password website or unable to use the password reported to be successfully set.

Duration: 
03/22/2022 12:00 pm - 03/22/2022 12:45 pm
Impact to Mason: 

Affected users were limited to those that tried to set or reset passwords during the service interruption period. The newly set passwords would not work for impacted users.

Affected Services: 
Authentication, Authorization & Account Services (AAA)
Other Affected Services: 
Services that required SSO authentication.
ROOT CAUSE ANALYSIS
Cause: 

An incorrect host-based firewall rule was deployed by an automated configuration change process which prevented the system from properly communicating with LDAP systems.

Resolution: 

Corrected the host-based firewall rule and pushed out via the automation process again. 78 impacted users were identified and reached out to reset their passwords after the service was restored.

Prevention: 

Before re-enabling an automation agent on a system, there should be at least 2 engineers to review the configuration changes and to run the changes in a dry-run mode, before the actual rollout.

STATISTICS
Service Team: 
Advanced Technologies, CCSE, CCSO, and NSENG teams