AI Infrastructure for DevOps People

1. GPUs and accelerators

• What they are
  • A CPU has a few strong cores that each do one thing at a time.
  • A GPU has thousands of weak cores that do the same simple math on lots of data at once.
  • Models are mostly giant matrix multiplication, so GPUs fit. TPUs (Google) and Trainium (AWS) are similar chips.
• Why you care
  • An H100 costs roughly $2 to $4 an hour in the cloud, and a training node usually has 8.
  • Supply is tight, so GPUs are the bottleneck and the biggest line on the bill.
• Example
  • 8-GPU node at $30 an hour, GPUs only 30% busy: you pay $30 for $9 of work.
  • Over a month that’s about $15,000 wasted on one node.
• Like
  • An oversized idle RDS instance with a 20x hourly price.
  • Watch GPU utilization, not CPU.
• Two workloads
  • Training: teaching the model. Many GPUs, hours to weeks.
  • Inference: answering requests with the trained model. Must be fast.

2. Networking

• What happens
  • A big model is split across many GPUs.
  • After every step they swap results (gradients) to stay in sync, over and over.
• The hardware
  • Inside a server: NVLink and NVSwitch, hundreds of GB/s.
  • Between servers: InfiniBand or RoCE with RDMA, so one machine writes into another’s memory without the OS getting involved.
• Example
  • 16 nodes on 10 Gbps Ethernet: 1 second of compute, 3 seconds waiting. About 25% utilization.
  • Switch to 400 Gbps InfiniBand: wait drops to 0.2 seconds. Same GPUs, about 3x faster.
• Like
  • A microservice that makes ten network calls per request. The slowest hop sets the pace.

3. Storage and data

• Three jobs
  • Training data: terabytes of text or images, read fast and in parallel.
  • Checkpoints: model snapshots, often tens or hundreds of GB each.
  • Serving data: vector databases and feature stores for RAG.
• Failure to avoid: the data-starved GPU
  • Data on S3, workers fetching one file at a time, GPUs idle half the time.
  • Fix: copy shards to FSx for Lustre or local NVMe, and prefetch the next batch.
  • Result: utilization from 40% to 90% with zero model code changes.
• Checkpoint example
  • Save every 30 minutes. Node dies at hour 10 and 20 minutes.
  • Restart from hour 10, lose 20 minutes instead of 10 hours.
• Like
  • Checkpoints are database backups plus retry logic for a long batch job.
  • The data pipeline is a cache or CDN in front of a slow origin.

4. Scheduling

• Tools
  • Kubernetes with the NVIDIA GPU Operator (drivers plus a device plugin, so pods can request nvidia.com/gpu: 1).
  • Slurm, the HPC scheduler still common on big training clusters.
• Gang scheduling (Kueue, Volcano)
  • A distributed job starts all its workers or none.
  • Without it: job A grabs 5 of the 8 GPUs it needs and waits. Job B grabs 4 and waits. Nine GPUs allocated, nothing running.
  • Like an all-or-nothing deploy.
• Sharing a GPU
  • MIG: hardware-isolated slices. One 80 GB card becomes 7 small ones, good for 10 GB models.
  • Time-slicing: take turns.
  • MPS: processes share concurrently.
• Node pools
  • Spot for restartable training, on-demand for everything else. Same split as EC2.

5. Training vs. inference

• Training
  • Long batch job. Goal is throughput and surviving failures.
  • Scale by adding nodes. Tools: PyTorch FSDP, DeepSpeed.
  • On failure, resume from a checkpoint.
• Inference
  • Always-on service with a latency SLO. Goal is low latency and low cost per request.
  • Scale by adding replicas based on load. Tools: vLLM, TensorRT-LLM, Triton, KServe.
  • On failure, retry on another replica.
• How training work gets split
  • Data parallel: each GPU holds a full model copy and trains on different data. Like 8 workers on different log files.
  • Tensor parallel: one layer is split across GPUs.
  • Pipeline parallel: different GPUs hold different layers, like an assembly line.
• Why splitting is needed
  • A 70B model needs about 140 GB for weights in 16-bit. One 80 GB GPU can’t hold it.

6. Serving optimizations

• Quantization
  • Store numbers in 8 or 4 bits instead of 16. The model shrinks 2 to 4x for a small quality loss.
  • 13B model: about 26 GB at 16-bit, about 7 GB at 4-bit. Fits a cheap 24 GB card.
• Continuous batching
  • New requests join the running batch as others finish.
  • One GPU serves 20 to 30 users instead of 2 or 3.
  • Like connection pooling.
• KV cache
  • The model remembers earlier text so it doesn’t recompute it for every new word.
  • It eats GPU memory, which is why vLLM’s PagedAttention exists.
  • Like an in-memory cache.
• Cold starts
  • New pod downloads 20 to 100+ GB of weights. Six minutes of timeouts during a spike.
  • Fix: bake weights into the image or use a shared cache volume. Pod ready in about 30 seconds.
  • Like a slow image pull, 100x bigger.

7. MLOps and the model lifecycle

• What gets versioned
  • Code (Git), datasets (DVC), trained weights (MLflow or a registry), and config. All together.
• Evaluation gates
  • Run every candidate model against a test set and compare with production.
  • Example: retrained support bot drops billing accuracy from 92% to 84%. The pipeline blocks the release.
• Safe rollout
  • Canary: a small slice of traffic.
  • Shadow: the new model gets copies of live traffic, but nobody sees its answers.
  • Instant rollback.
• Drift detection
  • Real-world data changes and the model slowly gets worse.
• The catch
  • Failures are silent. The endpoint returns 200, the text looks fluent, the answers are wrong.
  • Here “tests pass” means a quality metric clears a bar, not that the code compiles.

8. Observability

• Hardware (DCGM Exporter into Prometheus)
  • GPU utilization, memory, temperature, power, XID and ECC errors.
  • Example: repeated XID 79 means the card fell off the bus. Cordon and drain the node before it ruins a multi-day run.
• Serving (vLLM metrics into Grafana)
  • Time to first token, tokens per second, queue depth, p95 and p99 latency.
  • Example: users say it feels slow, CPU looks fine. Time to first token went from 0.4s to 3s and queue depth keeps climbing. GPUs are saturated, so add replicas.
• Cost
  • Cost per request or per 1M tokens, GPU-hours per team, via labels and tags.
• Like
  • The usual metrics, logs and traces, plus a hardware layer most DevOps people haven’t watched before.
  • CPU and memory dashboards show none of this.

9. Reliability

• Hardware fails all the time at scale
  • Meta’s Llama 3 paper reported 419 unexpected interruptions in 54 days on 16,384 H100s. About one every three hours.
• Failure types
  • Node or GPU failure: job crashes. Answer: checkpoints, auto-restart, health checks.
  • Straggler: one slow GPU holds back every other GPU.
  • Silent data corruption: a bad card produces wrong numbers with no error, and days later the model is quietly worse.
  • Hung collective: one worker stops responding and the rest block forever.
• Straggler example
  • 64-GPU job runs 20% slow with no errors.
  • Per-GPU metrics show one card clocked down from overheating. It drags the other 63.
  • Replace the node and speed returns.
• Like
  • Cattle, not pets. Assume nodes die and design the job to survive it.

10. Cost and security

• Cost levers
  • Right-size: don’t put a 24 GB model on an 80 GB card.
  • Spot instances: 60 to 70% cheaper, fine for training with checkpoints.
  • Reservations or savings plans for steady inference load.
  • Scale dev and test endpoints to zero overnight.
  • Keep utilization high. It’s the biggest lever.
• Example
  • A 4-GPU dev notebook left on over a long weekend: 72 hours at $12 an hour, about $860 for nothing.
  • Fix: auto-shutdown for idle GPU notebooks.
• Security
  • Model weights are valuable IP. Restrict access, encrypt, keep them on private networks.
  • Training data can contain sensitive material. Control who touches it.
  • LLM endpoints add new attack surface (prompt injection, data leakage). Use rate limits, guardrails and audit logs.
• Like
  • Normal FinOps and DevSecOps (tagging, budgets, least privilege, audit logs) with a higher price per resource.

Key Lessons

1. Utilization is the metric that matters.
   An idle GPU costs the same as a busy one, so the bill depends on how much work each hour produces.
2. Fast GPUs are only as fast as what feeds them.
   Network and storage decide whether the GPUs compute or wait.
3. Schedule distributed jobs all-or-nothing.
   Gang scheduling prevents the deadlock where two partly started jobs hold GPUs that neither can use.
4. Plan for failure at scale.
   Checkpoints, health checks and automatic node replacement turn a crash from a lost run into a lost half hour.
5. Watch quality as well as health.
   A model can return 200 with wrong answers, so evaluation gates and drift metrics belong next to latency and error rate.

Summary

AI Infra Concept	DevOps Equivalent	What to Watch
GPUs	Expensive compute instances	GPU utilization
Interconnect	Service-to-service network	Sync wait time per step
Storage and checkpoints	Cache, CDN, backups	Data throughput, checkpoint age
Gang scheduling	All-or-nothing deploy	Pending jobs, partial allocations
Serving optimizations	Connection pooling, caching	Time to first token, queue depth
Model lifecycle	CI/CD with canary and rollback	Eval scores, drift
Reliability	Cattle, not pets	XID and ECC errors, stragglers
Cost and security	FinOps and DevSecOps	Cost per 1M tokens, idle GPU-hours

If we already run schedulers, networks, caches and dashboards, we know most of AI infrastructure. What changes is the price of idle time and the number of ways a job can wait.

Read Next

• How to Run ML Training Jobs on Kubernetes With GPU Node Pools: A DevOps Engineer’s Guide
• How to Monitor ML Models in Production: Drift Detection With Prometheus and Grafana

Follow along on LinkedIn to catch the next post when it goes up.

---

Tags:
#AIInfrastructure   #MLOps   #GPU   #Kubernetes  
#DevOps   #SRE