Initially, services were not interrupted in the Data Center, however, other parts of Fairfax campus experienced outages for their specific locations.

Some services were unavailable due to extended power outage.

Due to a circuit breaker in the generator being open or tripped, the generator was unable to transfer power to the data center during the power outage. This caused the data center to run on UPS power, and cooling systems to remained inoperable, causing extreme operating temperatures for systems in the data system.

The load on the UPS system for an extended period of time without generator backup lead to the loss of one UPS system as the batteries depleted. This left half of the power infrastructure feeds down on the data floor. This caused systems to falsely report they were operating properly as they had enough power to appear to be on/visible to the monitoring system, but not enough power to work properly. Additionally, the excessive heat may have also played a role in systems functioning properly.

Because the systems ‘appeared’ to be up and functioning to other monitoring systems, the redundant system infrastructure did not kick. If the power had shut down completely, the backup systems would have sensed that the systems were unavailable and the redundant process could have taken over the load, example VPN.

The generator issue at the breaker level continues to be investigated by facilities.

At 11:12am Harris Theater had the same issue with the generator but it was a breaker problem that is being replaced at the ATS.

Electricians are investigating on the cause, and monitoring power in both buildings at this time.