Outage category: Applications
Location: Users attempting to set or reset their password
Status: Closed
Resolved Alert:
Initial Symptoms
Users were either unable to set a password at the password website or unable to use the password reported to be successfully set.
Root Cause Analysis
Cause
An incorrect host-based firewall rule was deployed by an automated configuration change process which prevented the system from properly communicating with LDAP systems.
Resolution
Corrected the host-based firewall rule and pushed out via the automation process again. 78 impacted users were identified and reached out to reset their passwords after the service was restored.
Prevention
Before re-enabling an automation agent on a system, there should be at least 2 engineers to review the configuration changes and to run the changes in a dry-run mode, before the actual rollout.