Deploying ML Applications on Kubernetes

Our image runs anywhere Docker runs — which, right now, means docker run on one machine. That's fine until 3am, when the container OOMs, exits, and stays exited, because nobody's awake to restart it. Or until traffic triples and a single container melts while nine idle cores sit on the box next door. Running a container is easy. Keeping it running, healthy, and appropriately sized without a human babysitting it is a different job, and it's the job Kubernetes exists to do.

The good news repeats from every part of this series: almost none of this is ML-specific. A model server is, to Kubernetes, just a stateless HTTP service. The one place the model shows up — and it's the satisfying one — is the readiness probe we've been quietly building since part two. This is the article where it finally meets the thing it was designed for.

The Deployment: replicas that heal themselves

A Deployment is you describing the desired state — "I want two copies of this image running" — and Kubernetes relentlessly making reality match. A pod crashes, it starts a new one. A node dies, it reschedules onto another. You stop issuing commands and start declaring intent; the control loop does the babysitting.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: house-price-api
spec:
  # no replicas: here — the HPA owns the count (see below)
  selector:
    matchLabels: { app: house-price-api }
  template:
    metadata:
      labels: { app: house-price-api }
    spec:
      containers:
        - name: api
          # Pin an immutable SHA tag, not :latest (part 4).
          image: <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/house-price-api:latest
          ports:
            - containerPort: 8000
          envFrom:
            - configMapRef: { name: house-price-config }
            - secretRef: { name: house-price-secrets }
          resources:
            requests: { memory: "512Mi", cpu: "250m" }
            limits: { memory: "1Gi", cpu: "1" }

Those resource numbers aren't guesses — they come straight from part two. Each worker unpickles its own ~47MB copy of the model, so memory scales with worker count; we request 512Mi and cap at 1Gi to hold two workers plus headroom. Set requests too low and Kubernetes packs too many pods onto a node and they OOM under load. This is where "N workers means N copies in RAM" stops being trivia and starts being a number you type.

The probes: where the model finally shows up

Here's the payoff for three articles of build-up. Kubernetes needs to know two different things about a pod, and conflating them is a classic outage. Liveness asks "is this process wedged — should I restart it?" Readiness asks "is this pod ready for traffic — should the load balancer send it requests?" For a model server these are genuinely different, because a pod can be alive for a full second or two before the 47MB model finishes loading — up, but useless.

          # Give the model time to load before liveness/readiness judge it.
          startupProbe:
            httpGet: { path: /health, port: 8000 }
            failureThreshold: 30
            periodSeconds: 2
          # Readiness gates traffic: /health returns 503 until the model loads.
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
          # Liveness restarts a wedged pod.
          livenessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 15

Remember the readiness logic from part two: /health returns 503 while the model is still loading and 200 once it's in memory. That single behavior is what makes this safe. The startupProbe gives the model up to a minute to load without the liveness probe misreading a slow start as a hang and killing the pod in a restart loop. The readinessProbe keeps the pod out of the load balancer until it returns 200, so no request ever lands on a pod that would 503. The "pointless" endpoint from part two is now the thing standing between you and an outage during every deploy. (We point all three probes at /health, which is fine here because it only 503s during that initial load window the startupProbe covers. If your model could go un-ready at runtime — a live registry hot-reload, say — you'd split a lightweight /livez that only checks the process from /health that checks readiness.)

The Service: a stable name for disposable pods

Pods are cattle. They die, get rescheduled, and come back with new IPs constantly, so you can never point a client at a pod. A Service is the stable front for a shifting set of pods — one fixed virtual IP and DNS name that load-balances across whatever pods currently match its label selector. It's the in-cluster equivalent of putting a load balancer in front of an autoscaling group: the members churn, the address doesn't.

apiVersion: v1
kind: Service
metadata:
  name: house-price-api
spec:
  type: ClusterIP
  selector: { app: house-price-api }
  ports:
    - port: 80
      targetPort: 8000

Ingress: the door to the outside

A ClusterIP Service is only reachable inside the cluster. Ingress is the public door: it maps outside HTTP traffic to your Service, and on EKS the AWS Load Balancer Controller reads this manifest and provisions a real ALB to match. Notice the health-check path — the ALB checks the same /health everything else does, so readiness is enforced at every layer from the container up to the load balancer.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: house-price-api
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /health
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: house-price-api
                port: { number: 80 }

Config and Secrets: keep the image generic

The whole point of part three's image was that it runs the same everywhere. It can't do that if the environment is baked in. ConfigMaps hold non-secret, per-environment settings; Secrets hold the sensitive ones. Our config carries the model alias from part five (MODEL_ALIAS: production) and the Secret carries where the MLflow registry lives — so the same image, pointed at a different registry, serves a different model with zero rebuild.

apiVersion: v1
kind: ConfigMap
metadata:
  name: house-price-config
data:
  MODEL_NAME: "house-price"
  MODEL_ALIAS: "production"
  LOG_LEVEL: "info"
---
apiVersion: v1
kind: Secret
metadata:
  name: house-price-secrets
type: Opaque
stringData:
  MLFLOW_TRACKING_URI: "http://mlflow.internal:5000"

One honest caveat you should never forget: a Kubernetes Secret is base64-encoded, not encrypted at rest by default. Base64 is not security — it's an encoding. For anything real, back it with a proper secrets manager (AWS Secrets Manager via External Secrets, or Sealed Secrets). The manifest shows the shape; it is not permission to commit a live credential.

The HPA: scale on the signal that's actually saturating

Two replicas handle a normal day. A traffic spike needs more, and nobody should be scaling pods by hand at 3am. A HorizontalPodAutoscaler watches a metric and adds or removes replicas to hold a target. For us the honest signal is CPU, because inference is CPU-bound — model.predict() burns processor, not memory or I/O. One gotcha worth internalizing: once the HPA is applied it becomes the source of truth for the replica count, so you leave replicas out of the Deployment entirely — set it in both and a plain kubectl apply resets the count while the HPA immediately scales it back, and the two fight.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: house-price-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: house-price-api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

When average CPU across the pods crosses 70%, the HPA adds replicas up to ten; when the spike passes, it scales back to two. Each new pod goes through the same startup and readiness dance, so it only takes traffic once its model is loaded. Self-healing from the Deployment, scaling from the HPA, zero-downtime rollouts from readiness-gated updates — that's the "keep it alive at 3am" machinery, declared in a few YAML files.

Rolling updates come free

Push a new image SHA (part four), change the tag in the Deployment, and Kubernetes rolls it out one pod at a time: start a new pod, wait for its readiness probe to go green, shift traffic, then retire an old one. If the new model never becomes ready — say it can't load — readiness never passes, the rollout stalls instead of taking down the service, and you roll back to the previous SHA. Immutable image tags plus a readiness probe are what make a model deploy boring, which is exactly what you want a deploy to be.

What's next

We've got a self-healing, autoscaling, zero-downtime deployment — described entirely as YAML. But it assumes a cluster already exists, and an ECR repo, and that ALB-provisioning controller, and the IAM role the CI pipeline assumed back in part four. We've been hand-waving all of that infrastructure into being. In part seven we stop hand-waving and define it with Terraform: the VPC, the EKS cluster, ECR, IAM, and S3, as version-controlled code you can create and destroy on demand. The manifests are in the series repo under k8s/. See you in part seven.