Power failures get most of the attention in data center design. They are sudden, they are easy to recognise, and decades of practice have produced reliable answers: UPS systems, generators, redundant feeds. Cooling failures are quieter at the start and often just as damaging. Public outage reports from the past few years include several cases where a chiller plant fault turned into a service outage within tens of minutes, as rising temperatures forced operators to shut down equipment or move workloads elsewhere. In AI halls, where heat density is much higher, the time available is shorter still.
Why the clock runs fast
When heat rejection stops, the heat produced by IT equipment keeps going. It accumulates in whatever can absorb it: the air in the room, the coolant in the pipes, the metal of the racks and the building structure. How fast temperatures rise depends on the ratio of heat load to that thermal mass.
Air holds very little heat. A 1,000 m² hall with a 4 m ceiling contains about 4,000 m³ of air, which can absorb roughly 4.8 MJ per degree of temperature rise. With 2 MW of IT load and no cooling, that air alone would warm by about 25 °C in a minute. In reality, equipment and building mass absorb much of the heat and the rise is slower, but the arithmetic explains why air-cooled rooms can go from normal to critical in a few minutes once all cooling stops, and why the same room at a tenth of the density had far longer.
Liquid cooling changes the picture in two ways. The coolant itself holds much more heat: a technology loop holding 2,000 litres of water-based coolant stores about 8.4 MJ per degree, so with heat rejection lost but pumps still running, a 1 MW load warms it by about 7 °C per minute. That is a useful buffer but a short one, especially when warm-water designs already run close to component limits. And if the pumps stop, the heat stays at the chips. A GPU whose cold plate has no flow reaches its thermal limit within seconds and throttles or shuts down.
| Scenario | What absorbs heat | Rough time to trouble |
|---|---|---|
| Air-cooled, low density, cooling lost | Room air, equipment, building | Tens of minutes |
| Air-cooled, high density, cooling lost | Room air, equipment, building | A few minutes |
| Liquid-cooled, heat rejection lost, pumps running | Coolant volume in loops | Minutes, depends on loop volume and margin |
| Liquid-cooled, pumps stopped | Cold plate and chip only | Seconds |
The figures are illustrative. Real ride-through depends on the design and should be measured or modelled for each hall.
The usual chain of events
Most cooling incidents follow a similar pattern. An initial fault occurs: a chiller trips, a controller fails, a pump stops, a cooling tower loses water, or a power event knocks cooling plant offline while the IT load stays on UPS. Redundant units should take over, but sometimes they fail to start, or they were already out for maintenance, or the same root cause affects them too. Supply temperatures begin to rise. Server inlet temperatures or GPU temperatures follow. Thermal protection starts to throttle processors, and services slow down. Operators then face a choice: shut down equipment in a controlled way, move workloads to another site, or wait and hope the cooling recovers. If they wait too long, servers shut themselves down abruptly.
The window for a good decision is often the first five to ten minutes after the initial fault.
Design measures that buy time
Several design choices extend ride-through.
Pumps and controls on UPS keep coolant moving through a power disturbance. For liquid-cooled racks this is essential, because loss of flow causes failure in seconds.
Thermal storage, such as chilled water tanks, gives the plant a reserve to draw on while chillers restart.
Fast-restart chillers and controls shorten the gap after a power event.
Redundancy that is real matters: N+1 or 2N cooling only helps if the spare units are maintained, tested and not sharing a common failure point.
Load shedding plans define in advance which workloads can be paused or moved, so less important GPU work stops first and critical services keep running longer.
Signals that give early warning
Most cooling failures announce themselves, if the data is watched together.
| Signal | What it can mean |
|---|---|
| Chiller or pump alarm, even with standby running | Redundancy is now lost; next fault is an outage |
| Chilled or facility water supply temperature drifting up | Plant not keeping up; hot weather or degraded unit |
| Rising ΔT across CDUs at steady load | Reduced flow or heat exchanger problem |
| Room or aisle temperatures rising in one area | Local airflow or containment problem |
| GPU temperatures and throttling events increasing | Cooling margin running out at the chip |
| Cooling tower water level or makeup flow abnormal | Water supply problem developing |
The most useful warning is the first one: the moment redundancy is lost. A site running on its last good chiller is one fault from an outage, and that should be treated as an incident even though nothing is hot yet.
Practise the response
Cooling incidents move too fast for improvisation. A good response plan states who decides to shed load, which workloads go first, how to move or pause GPU jobs without corrupting them, and at what temperatures each step begins. Teams that rehearse it, even on paper, decide faster and more calmly.
After an incident, the review should look at the full timeline: when the first signal appeared, when it was noticed, when action started. The gap between the first two is usually where the most time is lost.
CloudSino brings cooling data into the same view as the IT load: CDU status, coolant flow, ΔT and leak alarms, per-U temperatures, server room heatmaps and GPU temperatures from NVIDIA DCGM. SmartBSM correlates the alarms from a cooling event into one incident tied to the racks it affects. CloudSino monitors only, so your cooling controls stay in charge. Ask us to walk through a past incident with your own data.
Frequently asked questions
How long does a data center take to overheat after cooling fails?
It depends on heat density and thermal mass. A low-density air-cooled room may have tens of minutes, a high-density one only a few. Liquid-cooled racks with pumps still running are buffered by the coolant volume for minutes, while a cold plate with no flow lets a GPU hit its thermal limit in seconds. Measure or model each hall.
Why should liquid cooling pumps be on UPS power?
Pumps and their controls need UPS power because liquid-cooled chips depend on continuous coolant flow. If a power disturbance stops the pumps, heat stays at the cold plate and GPUs throttle or shut down within seconds. Keeping coolant moving lets the loop's thermal mass absorb heat while chillers or other heat rejection equipment restart.
What is the earliest warning sign of a data center cooling failure?
The earliest useful warning is losing cooling redundancy: a chiller or pump alarm while the standby unit carries the load. Nothing is hot yet, but the site is one fault from an outage, so treat it as an incident. Rising supply water temperature, higher CDU ΔT at steady load and GPU throttling usually come later.