Reading GPU health signals: XIDs, ECC and thermals

Xid errors

Xid messages are error reports written by the NVIDIA driver to the system log when something goes wrong on the GPU. Each has a number that identifies the type of event. Some point to hardware, some to software, and some could be either. NVIDIA's public Xid documentation lists them with likely causes.

A few are especially useful to recognise:

The pattern matters more than any single event. An Xid 13 that appears once with one user's job is probably that job. The same Xid appearing on one GPU across many different jobs points at the GPU. An Xid 79 that repeats on the same slot after a GPU swap points at the board, riser or power path instead.

Memory errors and row remapping

GPU memory is protected by error-correcting code. Single-bit errors are corrected silently and counted. Double-bit errors cannot be corrected, and the work using that memory is lost.

Recent data center GPUs handle failing memory cells by remapping the affected rows to spare rows. DCGM and nvidia-smi report how many rows have been remapped, whether a remap is pending, and whether a remap has failed. A pending remap takes effect only after the GPU is reset, so a GPU with a pending remap should be drained and reset at the next opportunity. A failed remap, or a GPU that has used up its spare rows, is a candidate for replacement under warranty.

Rising corrected error counts on one GPU are an early sign. They may never become a problem, but a GPU whose count climbs steadily deserves a closer look before it produces an uncorrectable error in the middle of a long training run.

Thermals and throttling

GPUs protect themselves from heat and power overload by lowering their clock speed. DCGM reports the GPU and memory temperatures, power draw and the reasons for any current throttling, such as power cap, hardware slowdown or thermal limits.

Occasional power-cap throttling during peaks is normal. Persistent thermal throttling is not. On air-cooled servers it often means a failing fan, blocked airflow or a hot aisle problem. On liquid-cooled servers it can mean reduced coolant flow, a partly blocked cold plate or a poor thermal interface after maintenance. A single GPU that runs several degrees hotter than its seven neighbours at the same load is worth investigating even if it has not yet throttled.

Throttling also hurts performance quietly. In a training job that synchronises across many GPUs, one throttled GPU slows every step for all the others.

Interconnect health

GPUs talk to each other through NVLink inside the server and to the host through PCIe. Both links keep error counters. Growing NVLink CRC or replay counts, or PCIe replay and correctable error counts, often come before link failures. Many are caused by connectors, risers or cables, which are cheaper and quicker to fix than GPUs.

Diagnostics and health checks

DCGM offers two kinds of checks beyond passive monitoring. Background health watches flag problems in memory, power, thermals and interconnects as they happen. Active diagnostics run test workloads at several levels of depth, from a quick check of a few minutes to longer stress tests. Many operators run a short diagnostic whenever a GPU is reset, returned from repair or allocated to a new tenant, and a longer one before raising a hardware return.

A worked triage

A training job fails overnight. The scheduler shows the failure came from one server. Its logs show Xid 79 on GPU 3 at 02:14. Looking further back, DCGM shows GPU 3's temperature running 6 °C above its neighbours for two weeks, and the server's BMC shows a fan in the same zone at higher speed than its peers. NVLink counters for GPU 3 are clean.

The likely story is a cooling problem around GPU 3 leading to a thermal or power event. The right actions are to drain the server, inspect the fan and airflow, run a long diagnostic on the GPU after the fix, and return the server to service only if it passes. Replacing the GPU first would probably have left the real cause in place.

That triage needed three data sources: driver logs, DCGM history and BMC data. Teams that keep them in separate tools spend most of their time collecting evidence.

A simple response policy

Write down what happens for each signal class. A useful starting policy: drain and reset GPUs with pending remaps at the next safe window; drain and run diagnostics after any hardware-class Xid; open a replacement case after a failed remap, a repeated uncorrectable error or a failed long diagnostic; and investigate any GPU that runs persistently hotter than its peers. Review the policy every quarter against what actually failed.

CloudSino collects GPU utilization, memory, temperature, power and ECC errors through NVIDIA DCGM, next to the server's own BMC data from DCOS, so a GPU fault and the hardware signals around it sit in one view. SmartBSM correlates related alarms into a single incident, and iDCOS links it to the asset record and the ticket. Ask us to show it on a sample of your own GPU servers.

Frequently asked questions