Autoscaling on Kubernetes — GPUs, Queues & Spot
AI workloads do not scale on CPU percent. They scale on queue depth.
# what you build
The rung where this becomes real infrastructure. You will move the service onto Kubernetes, add a GPU node pool that only exists while it is needed, and autoscale on the metric that actually matters for AI — pending requests, not CPU. Then you will cut the bill hard with spot capacity and scale-to-zero.
What you end up with
handles after this
~50,000 people at once
what it costs to run
$150–600 / month
What you will be able to do
- Run the model service on Kubernetes without the YAML becoming a second job
- Autoscale on queue depth with KEDA, and why CPU-based scaling misleads you
- Add a GPU node pool that scales from zero and disappears when idle
- Use spot capacity for inference, and handle the interruptions properly
- Model loading, readiness probes, and rollouts that do not drop requests
The build, step by step
These are the chapters of the video, in order.
- 1Move onto KubernetesThe cluster, the deployment, and a Helm chart you can actually re-use.
- 2Scale on the right metricKEDA watching the queue, so capacity follows demand instead of CPU.
- 3Add GPUs on demandA node pool that scales from zero, with taints so nothing else lands on it.
- 4Cut the billSpot instances, graceful interruption handling, and scale-to-zero overnight.
The stack
- EKS
- KEDA
- GPU nodes
- Spot
- Helm
# before this one
This project continues the system built in 05 · Traffic Management — Load Balancers, Streaming & Rate Limits. You can start here, but the repo assumes the previous rung exists.
$ subscribe --new-articles
Get new articles in your inbox
One email when a new hands-on guide goes live — Kubernetes, AWS, CI/CD, MLOps. No spam, unsubscribe anytime.