Network Outage – Multiple Services

Outage category: 
Applications, Blackboard, Hardware, myMason, Telephone, VoIP phone service, VPN, Webex, Website, Wired Network, Wireless Network, Workshop
All locations
Resolved alert: 
08/24/2019 11:00 am

Power outage by Dominion Power caused a Data Center power outage affecting temperature and equipment power due to generator issues.

Initial symptoms: 

Initially, services were not interrupted in the Data Center, however, other parts of Fairfax campus experienced outages for their specific locations.

08/24/2019 8:28 am - 08/24/2019 11:00 am
Impact to Mason: 

Some services were unavailable due to extended power outage.

Affected Services: 
Network Access
Other Affected Services: 
Banner and other services were affected as a result of server shut downs as the power outage continued.

Due to a circuit breaker in the generator being open or tripped, the generator was unable to transfer power to the data center during the power outage. This caused the data center to run on UPS power, and cooling systems to remained inoperable, causing extreme operating temperatures for systems in the data system.

The load on the UPS system for an extended period of time without generator backup lead to the loss of one UPS system as the batteries depleted. This left half of the power infrastructure feeds down on the data floor. This caused systems to falsely report they were operating properly as they had enough power to appear to be on/visible to the monitoring system, but not enough power to work properly. Additionally, the excessive heat may have also played a role in systems functioning properly.

Because the systems ‘appeared’ to be up and functioning to other monitoring systems, the redundant system infrastructure did not kick. If the power had shut down completely, the backup systems would have sensed that the systems were unavailable and the redundant process could have taken over the load, example VPN.


The generator issue at the breaker level continues to be investigated by facilities.

At 11:12am Harris Theater had the same issue with the generator but it was a breaker problem that is being replaced at the ATS.


Electricians are investigating on the cause, and monitoring power in both buildings at this time.


Service Team: 
Enterprise Infrastructure Services