A server can be online, answering requests and showing normal CPU and memory use while it is already one fault away from failure. One drive in a mirrored pair has dropped out. One of two power supplies has failed. A memory module is logging a rising number of corrected errors. Because the redundancy is doing its job, the application keeps running and the operating system reports nothing unusual. The warning sits in the hardware logs, and in many organisations nobody reads them until the second fault takes the server down.
Redundancy hides problems
Servers are built with redundant parts precisely so that one failure does not stop the service. That is good engineering, and it has a side effect: the first failure is silent at the application level. A degraded RAID array still serves data. A server running on one power supply still runs. A fan failure is compensated by the others spinning faster.
If nobody notices, the server keeps running with no spare. The next failure, often weeks later, becomes an outage. In large fleets this pattern explains a good share of "sudden" hardware incidents. Reviews of them often find a warning in the server's own event log that had been sitting there for days.
Where the warnings live
Most of the useful signals come from the server's baseboard management controller, the BMC, which monitors hardware independently of the operating system and is reachable over the management network through IPMI or the newer Redfish standard. Others come from the drives, the operating system or the GPU driver.
| Component | Early warning signals | Typical source |
|---|---|---|
| Memory (DIMMs) | Rising corrected ECC error count on one module | BMC event log, OS logs |
| Drives (SSD, HDD) | Media errors, reallocated sectors, wear level, RAID degraded state | SMART or NVMe health log, RAID controller |
| Power supplies | One PSU failed or lost input, output out of range, redundancy lost | BMC sensors and event log |
| Fans | Fan failed, speed far above peers | BMC sensors |
| CPU and board | Thermal trips, voltage alarms, machine check errors | BMC, OS machine check log |
| GPUs | Corrected and uncorrected memory errors, retired pages, Xid errors, throttling | DCGM, driver logs |
| Network cards and optics | Link flaps, error counters, optic temperature and power | NIC counters, switch, transceiver data |
| Firmware | Versions out of line with the fleet standard | BMC inventory |
Some of these are true predictors. A memory module whose corrected error count climbs steadily is much more likely to produce an uncorrectable error later, and many operators replace modules once corrected errors pass a threshold. Drive health data has been studied publicly at scale, for example in the quarterly drive statistics published by one large cloud storage provider, and certain SMART attributes are clearly associated with later failure. Other signals, such as a lost power supply, are not predictions at all. They report that redundancy is already gone.
GPU servers fail more often
AI servers pack many expensive, hot, high-power parts into one chassis: eight GPUs with high-bandwidth memory, several network cards and optics, large power supplies and dense cooling. They fail more often than ordinary servers, and their failures cost more. Meta's published report on training a large model across 16,384 GPUs over 54 days counted 419 unexpected interruptions, with about 78 percent attributed to confirmed or suspected hardware problems, and GPUs and their memory the largest category. In a large training job, one failed component can stop thousands of GPUs until the job restarts from a checkpoint.
A worked example
Consider a GPU server whose BMC logs this sequence over three weeks:
- Day 1: power supply 2 reports loss of input. The server continues on supply 1. No application impact.
- Day 4: corrected memory errors start appearing on one GPU, a few per day.
- Day 15: corrected errors on that GPU rise to several hundred per day. The driver retires memory pages.
- Day 21: supply 1 trips during a load spike. With supply 2 already failed, the server shuts down, taking a training job with it.
Two separate problems were visible for weeks. The power supply needed a replacement on day 1. The GPU needed attention by day 15 at the latest. Either repair, done during working hours, would have prevented an unplanned outage and a job restart.
Turning signals into action
Collecting the data is the first step. Acting on it consistently is harder. Teams that do this well follow a few habits.
Out-of-band data is collected for every server, including those outside the critical clusters, and kept long enough to show trends.
Each component has a written definition of "degraded", with thresholds based on the vendor's guidance and the site's own failure history.
Lost redundancy becomes a work item with a deadline, even when nothing is down.
Alarms are read in context. A power supply alarm on a server in a rack whose PDU is also alarming may point to the power feed instead of the server.
Every alarm links to the asset record, so the ticket names the server, its rack, serial number and warranty status, and the technician arrives with the right part.
Failures feed back into the rules. After each part failure, the team checks which signals came first and adjusts the thresholds.
CloudSino DCOS reads component-level health out of band from server BMCs: power supplies, fans, disks, RAID cards, memory, CPUs and GPUs, along with hardware logs and firmware versions. Alarms are raised for the individual component, so a lost power supply or a rising memory error count shows up as its own event, tied to the server and its asset record. Ask us to run it against a sample of your own servers.
Frequently asked questions
Can corrected ECC memory errors predict a DIMM failure?
Yes, a steadily climbing count of corrected errors on one memory module is a useful warning that it is more likely to produce an uncorrectable error later. Many operators set a threshold and replace the module once it is crossed, during a planned window. Track counts per DIMM over time, since a single isolated corrected error means little.
Why do servers with redundant parts still fail suddenly?
Redundancy keeps the service running after the first fault, so a failed power supply, degraded RAID array or dead fan often goes unnoticed by the application and the operating system. The server then runs with no spare until a second fault causes an outage. The first failure is usually recorded in the BMC event log well before that.
How should I handle a server that has lost power supply redundancy?
Treat lost redundancy as a work item with a deadline, even though the server is still running. Replace the failed supply during working hours, and first check whether the PDU or feed serving that rack is also alarming, since a feed problem can look like a failed supply. Link the ticket to the asset record so the right part arrives.