Traffic Management — Load Balancers, Streaming & Rate Limits
AI traffic is slow, bursty and expensive. Normal web defaults break it.
# what you build
A request that takes thirty seconds and streams its answer breaks almost every default a load balancer ships with. You will put a load balancer in front of several instances, fix the timeouts and health checks that silently kill streaming responses, and add rate limiting and a queue so one heavy user cannot take the whole service down.
What you end up with
handles after this
~5,000 people at once
what it costs to run
$70–120 / month
What you will be able to do
- Why default load balancer timeouts kill streaming answers, and what to set instead
- Health checks that test the model, not just whether a port is open
- Rate limit per user — and why limiting tokens beats limiting requests
- Queue the expensive work instead of dropping it on the floor
- When a chat session needs sticky routing, and when it does not
The build, step by step
These are the chapters of the video, in order.
- 1Put a load balancer in frontSeveral instances behind one address, with health checks that mean something.
- 2Fix streamingThe timeout, buffering and keep-alive settings that let tokens flow through.
- 3Add rate limitsPer-user token budgets in Redis, and a clear error when someone hits one.
- 4Add a queueAbsorb the burst instead of failing it, and show the user where they are in line.
The stack
- ALB
- Auto Scaling
- SSE streaming
- Redis
- Rate limiting
# before this one
This project continues the system built in 04 · Managed Serving — Run It Without Managing a Server. You can start here, but the repo assumes the previous rung exists.
$ subscribe --new-articles
Get new articles in your inbox
One email when a new hands-on guide goes live — Kubernetes, AWS, CI/CD, MLOps. No spam, unsubscribe anytime.