Data Drift, Model Drift & Automated Retraining

Back in part one I made a claim and asked you to hold onto it: a model can sit frozen for six months, with not one line of code changed, and still get worse. This is the part where we deal with it. Every other failure in this series has a cause you can point at — a bad deploy, a CVE, an OOM. Drift has no such cause. The code is fine. The infrastructure is fine. The world simply moved, and the model is still answering questions about the world as it was on training day.

Remember the photograph analogy from part one? The model is a photo of the world at the moment it trained. Drift is the gap between that photo and today, and it comes in two flavors. Data drift (or covariate shift) is when the inputs change shape — incomes rise, neighborhoods densify, your users move to mobile. Concept drift is nastier: the relationship between inputs and the answer changes — what a given house's features are worth is simply different in a new market. Both quietly erode accuracy while every dashboard from part eight stays green.

You already built the first sensor

In part eight we started emitting a histogram of predicted_price_dollars. That was not incidental — it's a live view of your model's output distribution, and a sudden shift in it is often the first visible symptom of drift. But an output histogram tells you something changed; it doesn't tell you what, and it can't run the statistical test that turns "looks different" into "is different." For that we compare distributions directly.

Detecting data drift with a KS test

The tool is boring and effective: a two-sample Kolmogorov-Smirnov test. Take the feature values the model trained on (the reference) and a recent sample of production inputs (the current), and for each feature ask, "could these two samples plausibly have come from the same distribution?" A low p-value says no — that feature has drifted. It's the same instinct as a diff, but for distributions instead of text.

# drift/detect_drift.py  (core)
from scipy.stats import ks_2samp

P_THRESHOLD = 0.05  # below this, call it drift

def check(reference, current):
    drifted = []
    for i, name in enumerate(FEATURES):
        stat, p = ks_2samp(reference[:, i], current[:, i])
        if p < P_THRESHOLD:
            drifted.append(name)
    return drifted

To see it work, we compare the reference against two samples: one drawn from the same distribution, and one where we've nudged incomes up and household density up to simulate a market that moved. The test should stay quiet on the first and fire on exactly the features we changed — and it does:

=== Scenario A: same distribution ===
  MedInc      p= 0.667  ok
  AveOccup    p= 0.484  ok
  ... (all features ok) -> no drift

=== Scenario B: shifted distribution ===
  MedInc      p= 0.000  DRIFT
  HouseAge    p= 0.585  ok
  AveOccup    p= 0.000  DRIFT
  ... -> drift on: MedInc, AveOccup

That's real output from the script. Notice the precision: it flags MedInc and AveOccup — the two features we actually shifted — and leaves the other six alone. No false alarms on the stable features, a clear signal on the moved ones. The script exits non-zero when it finds drift, which is the hook that lets a pipeline act on it.

Two honest caveats, because a drift detector that cries wolf gets muted. A KS test on enough data will flag statistically-significant shifts that are too small to matter, so in practice you pair the p-value with an effect-size threshold. And this catches data drift — shifting inputs — but not concept drift, where inputs look identical yet the right answer changed. Concept drift only shows up when you can compare predictions to real outcomes later, which is why the strongest drift signal of all is measuring live accuracy once the ground truth arrives.

Retraining: a trigger, not a schedule

The instinct is to retrain on a cron — every night, every week. That's wasteful when nothing changed and too slow when something breaks the day after the job ran. Better to retrain when the data actually moved. We already have the pieces: the drift detector says "the world moved," and the MLflow champion/challenger gate from part five says "and only ship the new model if it's genuinely better." Wire them in sequence and retraining becomes a decision the pipeline makes on evidence.

# .github/workflows/retrain.yml
on:
  schedule:
    - cron: "0 6 * * 1"   # check weekly — checking is cheap, retraining is gated
  workflow_dispatch:

jobs:
  drift-check:
    runs-on: ubuntu-latest
    outputs:
      drift: ${{ steps.detect.outcome }}
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - id: detect
        continue-on-error: true      # a drift exit is a signal, not a failure
        run: python drift/detect_drift.py

  retrain:
    needs: drift-check
    if: needs.drift-check.outputs.drift == 'failure'   # only if drift was found
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python registry/train_mlflow.py    # train a challenger
      - run: python registry/promote.py         # promote ONLY if it beats champion

Read what this does end to end. A cheap weekly job checks for drift. If none, it does nothing and costs pennies. If drift is found, it retrains on fresh data, registers the result as a challenger in the MLflow registry, and runs the exact champion/challenger promotion from part five — which moves the production alias only if the new model actually wins. And because part five wired serving to load models:/house-price@production, the next time a pod starts or restarts it picks up the promoted model — no image rebuild, no code change. Drift detected, model retrained, better model promoted, and on the pods' next cycle the traffic shifts — a human only gets involved if they want to.

This is the feedback arrow from part one's lifecycle diagram, finally closed. The loop that made ML different from a normal web service — the one that meant a frozen model rots — now has a system watching for the rot and healing it.

What's next

That's the last missing piece. We've gone from a model in a notebook to a self-serving, self-scaling, self-monitoring, and now self-healing system: it notices when its own predictions are going stale and does something about it. In part ten we step back and assemble everything — API, Docker, CI/CD, MLflow, Kubernetes, Terraform, monitoring, and drift — into one coherent end-to-end platform, and walk the full path a change takes from git push to a promoted model serving traffic. The drift detector and retrain workflow are in the series repo under drift/ and .github/. See you in part ten — the finale.