That mentality has come to the forefront as the Bureau of Safety and Environmental Enforcement (BSEE) has become aware of a series of failures on industrial uninterruptible power supply (UPS) systems in offshore settings, resulting in significant power loss to industrial control systems, emergency shutdown systems, and emergency mitigation systems.
These incidents have also led to failures of capacitors and batteries, compromising the reliability and effectiveness of critical systems.
The following are a series of incidents that occurred:
Incident 1: UPS failure releases smoke, which set off smoke detectors.
An offshore production facility experienced a shutdown due to failed components in the UPS system, and as a result the emergency generator ended up activated to provide temporary power. However, the UPS system encountered further issues when the power shunt circuit breaker tripped, leading to smoke and electrical arcing/sparks emanating from the power transformer case within the UPS inverter section.
To address this situation the fire team handled the potential fire hazard. Once the energy released and arcing subsided, the breakers supplying power to the UPS ended up safely opened and locked out from all power sources. This precautionary measure aimed to protect the technicians from any unintended release of electrical energy, which could pose a risk or injury.
After isolating the failed components, workers placed the UPS in safe bypass mode. This mode allowed for a temporary bypass in the system, ensuring uninterrupted power supply to critical equipment while technicians conducted a thorough inspection. During the inspection, they discovered damage in the inverter section of the UPS.
Incident 2: UPS did not supply backup power after a shut-in.
During testing activities on an offshore production facility, an emergency safety system ended up activated, causing a loss of power generation and subsequent shutdown. As a result, a full deluge of all zones initiated, and the abandon platform alarm sounded, indicating an immediate evacuation protocol.
Unfortunately, in this situation, the emergency generator failed to start, and UPS did not provide backup power as expected. This double failure further exacerbated the power loss issue, leaving the facility without any reliable source of electricity.
The consequences of the power loss were significant. One of the major impacts was on the communication system, making it difficult for the operator to maintain effective communication among themselves and with other personnel. Without proper communication channels, it became challenging to coordinate critical operations and respond to emergencies promptly. Additionally, the control system visualization was affected, hindering the operator’s ability to monitor and control crucial processes within the facility. This disruption not only posed risks to personnel safety but also had the potential to compromise the overall integrity of the facility.
Incident 3: UPS failure due to overcharging.
During an incident on an offshore production facility, a full evacuation of personnel was necessary due to an extended electrical power loss. This incident shed light on critical issues with the offshore facility’s UPS system, specifically with UPS system B, as UPS system A was undergoing a replacement project at the time.
In this incident, the facility was relying on emergency power supplied by UPS system B. However, despite being in static bypass mode, the inverter of UPS B failed.
Unfortunately, this failure went unnoticed, and the rectifier of UPS B continued to charge batteries at an elevated voltage. The elevated voltage was a direct consequence of the failure of the voltage indicator, which failed to trip the emergency circuit breaker and stop the power supply. This oversight resulted in the batteries being subjected to overcharging, leading to their failure and subsequent loss of power supplied from UPS B. The failure of UPS B compromised the availability of backup power, which is crucial for maintaining critical operations and ensuring personnel safety on the offshore production facility.
Without a reliable backup power source, the facility could face difficulties in maintaining vital systems, such as communication, control, and emergency equipment.
During the course of its research, BSEE investigators found:
- UPS systems and maintenance guidelines did not provide adequate information to allow the continued safe and reliable operation of the industrial UPS systems. Specifically, the operating and maintenance instructions did not direct the needed replacement of all relevant components within the UPS system with parts time-limited to less than the design life. Some component examples are batteries, filter capacitors, cooling fans, internal random access memory batteries, power cards, and control cards.
- Self-healing polypropylene capacitors located in the inverter section of the UPS ended up exposed to excessive current, which caused the capacitors to fail resulting in bulging and discoloration.
- The lack of updated hardcopy electrical drawings caused confusion and impacted the ability of technicians to address issues quickly and accurately.
- The UPS system had clogged air filters, limiting air flow, and causing exposure to excessive heat. The excess heat eventually started to evaporate the solution inside the capacitor, which produced unsafe pressure causing the capacitor to fail.
After researching the incidents, BSEE recommends operators and contractors consider:
- Conducting training sessions and raising awareness among personnel regarding emergency response protocols, including the appropriate actions to take during power loss events and communication system failures.
- Evaluating the feasibility of introducing redundancy measures to critical systems, such as backup power generation and uninterruptable power supply, to minimize the risk of future disruptions.
- Evaluating communication and control system visualization infrastructure to enhance real-time monitoring and alarming capabilities.
- Considering implementing advanced monitoring technologies, redundant communication channels, and backup visualization systems to minimize the impact of power loss on critical operations.
- Reviewing emergency response protocols and procedures to ensure they adequately address power loss scenarios and including specific guidelines on communication system redundancy and backup measures.
- Training personnel on emergency response procedures and reinforcing the importance of timely communication during critical situations.
- Developing a comprehensive testing and maintenance program for all emergency systems, including power generation and control systems. Regularly reviewing and testing the emergency power systems to verify their functionality and performance and addressing any identified issues promptly.
- Ensuring maintenance personnel establish and conduct an approved periodic examination of the UPS alternating current and direct current capacitors. Examinations should be thorough and include capacitance testing, visual inspections for bulging and discoloration, and thermal scanning to detect and track potential problems.
- Ensuring all electrical one-line diagrams required by 30 CFR 250.842 are up-to-date and available to personnel working on electrical equipment.
- Ensuring equipment suppliers provide adequate documentation, including specifications, drawings, and installation, commissioning, and decommissioning procedures.

