Flash storage wears out. Every NAND cell survives a limited number of program and erase cycles, and every enterprise SSD carries an endurance rating that says how much data can be written to it over its warranty life. For most traditional servers, that rating was a formality: drives were replaced for capacity or age long before they wore out. AI workloads change this. Checkpoints, cache offload, dataset staging and some runaway processes can write to local drives at rates that use up years of endurance in months. Developers have also reported desktop AI tools writing far more to disk than anyone expected. Operators who do not watch write volumes find out when drives start to fail or switch to read-only mode.
How endurance is specified
Drive vendors express endurance in two related ways. Terabytes written, or TBW, is the total amount of data the host can write over the warranty period. Drive writes per day, or DWPD, is how many times the full capacity can be written every day for the warranty period, usually five years. JEDEC standards define the workloads and conditions used to rate them.
A 3.84 TB drive rated at 1 DWPD over five years can absorb about 3.84 × 365 × 5, or roughly 7,000 TB written. The same capacity at 3 DWPD, typical of mixed-use drives, absorbs about 21,000 TB. Read-intensive drives may be rated below 1 DWPD.
What the host writes is not the whole story. Internally, the drive moves data around for garbage collection and wear levelling, so the flash sees more writes than the host sent. This write amplification depends on the workload: large sequential writes keep it low, small random writes raise it.
Where AI workloads write
Checkpoints are the largest source in training. A training job saves its full state regularly so that it can resume after a failure. For a 70-billion-parameter model, the weights alone are about 140 GB at 16-bit precision, and with optimizer state a full checkpoint is commonly several times that, often over a terabyte. Saving every 30 to 60 minutes to local NVMe as a first stage, before copying to shared storage, writes a great deal.
Cache offload is the growing source in inference. Serving systems that move KV caches from GPU memory to local SSD write continuously while traffic is high, as described in our article on the KV cache.
Dataset staging, preprocessing and shuffling copy large amounts of data to local drives at the start of jobs and between epochs.
Logs, swap and scratch files add a steady background load, and a misconfigured job or tool can write the same data over and over.
A worked example
A training server has eight 3.84 TB NVMe drives rated at 1 DWPD, used together as local scratch. A job writes a 1.1 TB checkpoint every hour to these drives, spread evenly.
| Item | Value |
|---|---|
| Writes per drive per hour | about 137 GB |
| Writes per drive per day | about 3.3 TB |
| Effective DWPD from checkpoints alone | about 0.86 |
| Rated DWPD | 1.0 |
| Margin left for everything else | about 0.14 DWPD |
Checkpoints alone nearly use the full rating. If the job saves every 30 minutes instead, the drives see about 1.7 DWPD from checkpoints, and at that rate the full five-year endurance is used in about three years, before counting write amplification, staging or logs. If the cluster runs that way continuously, drive failures will arrive well inside the expected server life, and they will arrive in clusters, because all drives in the fleet age at the same rate.
Two simple changes often fix it: choose mixed-use drives rated at 3 DWPD for checkpoint scratch on training nodes, and tune the checkpoint interval to the real failure rate of the cluster.
What to monitor
NVMe drives report their health through a standard log page that includes the signals that matter:
| Field | What it tells you |
|---|---|
| Percentage used | Vendor estimate of endurance consumed; 100 means the rated life is used up |
| Data units written | Host writes since manufacture (each unit is 512,000 bytes) |
| Available spare | Remaining spare blocks; alarms when below threshold |
| Media and data integrity errors | Uncorrectable errors on the flash |
| Critical warning flags | Spare low, temperature, read-only mode, reliability degraded |
| Temperature | High temperatures speed up wear and cause throttling |
Collecting these daily and keeping the history gives two useful numbers per drive: the rate of endurance consumption, and the projected date it reaches 100 percent. Sorting the fleet by projected wear-out date shows which servers need attention and when, and turns a surprise into a planned replacement.
Watching for runaway writes
The other reason to watch writes is to catch processes that misbehave. A job that writes far more than its peers, a container whose temporary files grow without limit, or a tool that rewrites the same cache every few seconds can wear drives quickly and also slow down other work on the same server. Alerting on sudden changes in write rate per server, compared with its own history and with similar servers, catches these early.
Plan replacement as maintenance
When endurance data is visible, SSD replacement becomes routine maintenance. Drives approaching their rated life can be replaced in planned windows, spares can be ordered in advance, and procurement can choose the right endurance class for each server role.
CloudSino DCOS collects disk and RAID card status out of band for every server, alongside the rest of its hardware, and raises alarms when a drive reports a fault or degrading health. Each drive is recorded at component level against its server and rack, so a replacement plan starts from the asset list instead of a spreadsheet. Ask us to look at your own fleet's drive health with you.
Frequently asked questions
What is DWPD and how does it relate to TBW?
DWPD, drive writes per day, is how many times a drive's full capacity can be written each day over its warranty period, usually five years. TBW, terabytes written, is the total host writes allowed over that period. To convert, multiply capacity by DWPD by 365 by the warranty years. A 3.84 TB drive at 1 DWPD comes to roughly 7,000 TB.
Why do AI training servers wear out NVMe SSDs faster?
Training jobs save large checkpoints frequently, often to local NVMe before copying them to shared storage, and a full checkpoint with optimizer state can exceed a terabyte. Dataset staging, shuffling, KV cache offload in inference, logs and runaway processes add more writes. Together these can consume a drive's rated endurance well before the server reaches end of life.
Which NVMe health fields should I monitor for SSD wear?
Watch percentage used, data units written, available spare, media and data integrity errors, critical warning flags and temperature from the NVMe health log. Collect them daily and keep the history, so you can calculate each drive's wear rate and projected date to reach 100 percent. Sorting the fleet by that date turns surprise failures into planned replacements.