Modern GPUs process data far faster than storage can deliver it. An H100 GPU consumes data at 3.35 TB/s from its HBM3 memory. But the fastest NVMe SSD delivers only ~7 GB/s — a 478× gap. Even a network-attached parallel file system tops out at 10–50 GB/s per client.

The saving grace: GPUs don’t need continuous storage I/O during training. Data is loaded into GPU memory in batches, processed through forward and backward passes, then the next batch is loaded. The question is whether storage can deliver the next batch before the GPU finishes the current one.

When it can’t, GPUs idle. On a 1,000-GPU cluster at $2.50/hr per GPU, every 1% of idle time from storage starvation costs $21,900/month.

Training data loading: Sequential reads of large files (tokenized datasets). Predictable, high-throughput. Typically 1–10 GB/s per node.

Checkpointing: Periodic writes of model state. Bursty, write-heavy. A 70B model checkpoint is ~140 GB; a 1T model is ~2 TB. Must complete fast enough to not stall training.

Shuffling & preprocessing: Random reads across large datasets. Latency-sensitive. Can bottleneck data pipelines if storage IOPS is insufficient.

Model loading (inference): Read entire model weights at startup. A 70B FP16 model = 140 GB. At 7 GB/s (NVMe), that’s 20 seconds. At 1 GB/s (network), that’s 2.3 minutes.

A single storage server maxes out at ~10–20 GB/s. A 1,000-GPU cluster needs 100+ GB/s of aggregate storage bandwidth. The solution: spread data across many servers and read from all of them simultaneously.

Parallel file systems stripe files across multiple Object Storage Targets (OSTs). A 1 TB file might be split into 1 MB chunks distributed across 100 servers. When a client reads the file, it reads from all 100 servers in parallel, achieving 100× the bandwidth of a single server.

Lustre (Linux + Cluster) is the dominant open-source parallel file system for AI and HPC. Key components:

MDS (Metadata Server): Handles file namespace — directory listings, permissions, file-to-OST mappings. Typically 1–4 active MDS nodes for clusters up to ~50K clients.

OSS/OST (Object Storage Server/Target): Stores actual file data. Each OSS manages multiple OSTs (typically NVMe or SSD arrays). Stripe files across OSTs for parallel access.

Client: POSIX-compatible kernel module on compute nodes. Reads/writes appear as normal file operations but are transparently parallelized across OSTs.

Scaling: Add more OSS nodes for more bandwidth. DDN’s EXAScaler (Lustre-based) delivers 4 TB/s to NVIDIA’s Eos supercomputer across hundreds of OSS nodes.

In a traditional data path, reading data from storage to GPU requires two copies:

1. Storage → CPU system memory (via DMA)
2. CPU system memory → GPU memory (via PCIe)

Each copy consumes CPU cycles, system memory bandwidth, and PCIe bandwidth. The CPU becomes a bottleneck, especially when multiple GPUs compete for data. On an 8-GPU node, the CPU must orchestrate 8 parallel data streams — often saturating the memory bus before the GPUs are full.

GPUDirect Storage (GDS) eliminates the CPU from the data path entirely. Storage devices transfer data directly to GPU memory via RDMA (Remote Direct Memory Access), bypassing system memory completely.

1. GPU application requests a file read via cuFile API
2. GDS driver maps GPU memory pages for DMA access
3. Storage controller (NVMe or network) performs DMA directly to GPU memory
4. GPU is notified when transfer completes

No CPU involvement. No system memory copy. The data path is: Storage → PCIe/Network → GPU HBM.

GDS requires: NVIDIA GPU (Ampere+), compatible NVMe driver or network adapter, and CUDA 11.4+ with the cuFile library.

NVIDIA’s Storage Compute Accelerated Data Access (SCADA), announced November 2025, takes GDS further by completely eliminating CPU involvement in storage orchestration. The GPU itself manages data placement, prefetching, and caching — treating storage as an extension of its memory hierarchy.

This is particularly important for checkpointing, where a 1T model must write 2 TB of state to storage without stalling training. With SCADA, the GPU can overlap checkpoint writes with the next training step.

A frontier model training run costs $50–100M+ and runs for weeks to months. Hardware failures happen every few hours on large clusters (Meta reported one failure every 3 hours on 16K GPUs). Without checkpointing, a failure could lose days of training progress — millions of dollars of compute wasted.

The challenge: checkpoints are huge. A 70B model’s full training state (weights + optimizer states + gradients) is ~560 GB. A 1T model: ~8 TB. Writing this to storage fast enough to not stall training is a serious engineering problem.

Production training runs use a tiered approach, balancing speed against durability:

Tier 1 — Local NVMe (every 5–10 min): Each GPU writes its shard to the node’s local NVMe SSD. At 7 GB/s, writing 1.1 GB takes <1 second. Protects against GPU failures but not node failures.

Tier 2 — Parallel FS (every 30–60 min): Full checkpoint written to Lustre/GPFS. At 50 GB/s aggregate, 560 GB takes ~11 seconds. Protects against node failures but not data center failures.

Tier 3 — Cloud/Remote (every few hours): Checkpoint copied to S3/GCS for disaster recovery. At 10 GB/s, 560 GB takes ~56 seconds. Protects against everything but is slowest.

Asynchronous writes: All tiers use async I/O — training continues while checkpoints drain to storage in the background. The GPU only stalls if the next checkpoint starts before the previous one finishes writing.

Before data reaches the GPU, it passes through a multi-stage pipeline:

1. Read from storage: Fetch raw data (text, images, audio) from disk or network.
2. Deserialize: Parse file formats (Parquet, TFRecord, WebDataset).
3. Transform: Tokenize text, resize images, augment data.
4. Batch: Collate samples into tensors of uniform shape.
5. Transfer: Copy batch from CPU memory to GPU memory (pinned memory + async copy).

Each stage can become a bottleneck. PyTorch’s default DataLoader uses CPU workers for stages 1–4 and overlaps them with GPU training via prefetching. But with large models on fast GPUs, even 16 CPU workers may not keep up.

Pre-downloaded (traditional): Copy entire dataset to local/parallel storage before training. Simple, fast reads, but requires storage capacity equal to dataset size on every node. A 15 TB dataset (Common Crawl) needs 15 TB per node or a shared parallel FS.

Streaming (MosaicML, HuggingFace): Read data directly from object storage (S3/GCS) during training. No local copy needed. Data is shuffled and batched on-the-fly.

Streaming works well for text (low bandwidth per sample) but struggles for images/video (high bandwidth). The sweet spot: stream from S3 to a local NVMe cache, with the cache acting as a sliding window over the dataset.

Most organizations store their training data in object storage (S3, GCS, Azure Blob) because it’s cheap ($0.023/GB-month for S3 Standard), infinitely scalable, and durable (11 nines). The challenge is getting data from object storage to GPUs fast enough.

S3 throughput: A single S3 GET request delivers ~100 MB/s. With 1,000 parallel requests, you can reach ~100 GB/s aggregate. But latency is 10–50ms per request — 1,000× slower than local NVMe.

Egress costs: AWS charges $0.09/GB for data leaving a region. Training on a 15 TB dataset with 3 epochs = 45 TB of reads = $4,050 in egress if data and compute are in different regions. Same-region is free.

The production pattern combines object storage with local caching:

Cold tier (S3/GCS): Full dataset, cheap, durable. Source of truth.

Warm tier (Parallel FS): Active training data cached on Lustre/GPFS. High bandwidth, shared across nodes. AWS FSx for Lustre can auto-hydrate from S3.

Hot tier (Local NVMe): Current epoch’s data cached on node-local SSDs. Fastest access, limited capacity (1–8 TB per node).

Data flows: S3 → Lustre (background prefetch) → Local NVMe (per-epoch cache) → GPU memory (per-batch). Each tier acts as a progressively faster, smaller cache.

A training cluster’s storage must handle three simultaneous workloads: data loading (read-heavy, sequential), checkpointing (write-heavy, bursty), and logging/metrics (write-heavy, small files). These have conflicting I/O patterns.

Best practice: Separate storage pools for each workload:

Data pool: Lustre with read-optimized OSTs. Stripe across many servers. Tune for sequential read throughput. Cache hot data on local NVMe.

Checkpoint pool: Lustre or GPFS with write-optimized OSTs. Needs burst bandwidth (entire cluster writes simultaneously). Size for peak write rate, not average.

Scratch pool: Local NVMe on compute nodes. Used for Tier-1 checkpoints, data caching, and temporary files. No network overhead.

Inference storage is simpler but has different priorities:

Model loading: Read entire model weights at startup. Speed determines cold-start latency. A 70B model from NVMe: 20s. From network: 2+ minutes. From S3: 5+ minutes.

Model registry: Store multiple model versions, quantization variants, LoRA adapters. Needs fast random access for adapter loading.

Solution: Keep active models on local NVMe or a fast NAS. Use S3 as the model registry with a pull-through cache. Pre-pull models during scaling events to avoid cold-start latency.

Data loading bandwidth: Calculate based on batch size, processing time, and number of nodes. For text training, even modest bandwidth (100 MB/s per node) suffices. For image/video, plan for 1–10 GB/s per node.

Checkpoint bandwidth: Total checkpoint size ÷ acceptable write time. If a 560 GB checkpoint must complete in 30 seconds: need 18.7 GB/s aggregate write bandwidth. With FSDP (each GPU writes its shard in parallel), 125 nodes writing 4.5 GB each at 7 GB/s (local NVMe) finishes in <1 second.