When the inputs change

01 / CLOSE THE LEARNING LOOP

The model still answers. That is the problem.

A camera gets replaced. A supplier changes a surface finish. A new customer sends a different kind of document. The service stays up and the model keeps returning predictions. None of that tells you whether those predictions still deserve the same trust.

Out of distribution, or OOD, means unfamiliar relative to a reference distribution you chose. It is not a synonym for incorrect. A harmless lighting change can look unusual; a familiar-looking input can still produce a costly mistake. Distribution drift describes a change across a population. Performance monitoring needs outcomes. Keep those signals separate.

This walkthrough adds a small feature-distance monitor to a PyTorch classifier, then connects persistent alerts to automatic candidate retraining. The browser experiment is synthetic. The downloadable Python example actually trains a small network locally. Neither one is a production monitoring platform.

02 / CLOSE THE LEARNING LOOP

Give unfamiliar a reference point.

Have the model return an embedding as well as logits. Use the same preprocessing in training and serving, and record its version. Explicit outputs are easier to reason about here than a hook attached to a layer whose name might change during a refactor.

Fit one mean embedding per known class on reference training data. Estimate a shared, diagonal within-class variance, with mild shrinkage and a floor. At inference, measure the standardized squared distance to each class mean and keep the smallest. Averaging across feature dimensions makes the scale easier to read, but it does not make the result a probability.

This is a deliberately reduced baseline inspired by class-conditional Mahalanobis methods. It ignores feature correlations and uses one layer. It is not the full method in Lee and colleagues’ paper. A single mean also handles a class with several distinct clusters poorly. Compare it with alternatives on the shifts you care about before depending on it.

score(z) = min over classes c of mean_j [(z_j − μ_cj)² / v_j]

def forward(self, features):
        self._check(features)
        distances = ((features[:, None] - self.means[None]).square()
                     / self.variance).mean(-1)
        return distances.min(1).values

03 / CLOSE THE LEARNING LOOP

Calibrate the alert, not your optimism.

Reserve independent, representative in-distribution data for calibration. The example uses its 99th percentile score, with the higher interpolation rule, as the threshold. Flag scores strictly above that threshold. This targets roughly a one-percent tail on that calibration sample. It does not promise a one-percent false alarm rate tomorrow.

Keep a separate evaluation set to estimate the false positive rate, uncertainty and detection rate on known shift examples. Check cameras, sites, products and other meaningful slices. A threshold that looks fine across the whole fleet can overwhelm one site. Many repeated checks also create more chances to raise a false alert.

Use model.eval() as well as torch.inference_mode(). They do different jobs: evaluation mode controls modules such as dropout and batch normalization; inference mode disables autograd bookkeeping. Store detector statistics as buffers so they travel with the state dict. Never compare a new model’s embeddings with the old model’s feature reference.

model.eval()
with torch.inference_mode():
    _, reference = model(train_x)
    _, heldout = model(calibration_x)
    detector = ClassDistance(classes=2, dimensions=reference.shape[1]).to(reference.device)
    detector.fit(reference, train_y).calibrate(heldout, quantile=.99)
    _, features = model(request_x)
    scores = detector(features)
    flagged = scores > detector.threshold

04 / CLOSE THE LEARNING LOOP

Make the alert useful to an operator.

Emit counts, score quantiles, flagged counts, input-validation failures and latency, tagged with model, detector and preprocessing versions. Add a bounded site or product slice when it helps someone investigate. Do not put every sample ID into a metrics label. Keep an opaque event ID in a separate, access-controlled review log, with governed links to retained inputs.

The example evaluates non-overlapping ten-minute windows. A window needs at least 100 valid requests; more than eight percent must be flagged for three consecutive windows. These are illustrative operating choices, not statistically derived limits. Low-volume windows reset the streak. A one-hour cooldown limits repeated jobs. Track missing telemetry and service failures separately so silence never looks like recovery.

For a real service, keep counters and trigger state in durable storage. Use event-time watermarks for late data, deduplicate completed windows and evaluate each approved slice independently. Calibrate the alert policy on historical traffic, including its seasonality and autocorrelation. Start with alert-only operation and measure how much review work it creates.

breach = total >= self.min_count and flagged / max(total, 1) > self.rate_limit
self.streak = self.streak + 1 if breach else 0
cooled = self.last_queued is None or timestamp - self.last_queued >= self.cooldown_seconds
ready = self.streak >= self.required_windows and cooled and reviewed_labels >= 200 and contract_ok
if ready:
    self.last_queued = timestamp
    self.streak = 0
return ready

05 / CLOSE THE LEARNING LOOP

Do not feed the alert straight back into training.

Put flagged examples in a review queue. Sort out broken files, sensor faults, changed preprocessing, valid new conditions and genuinely new classes. A bad camera needs a repair before it needs a training job. An unknown class needs an explicit label-schema and product decision. This example supports the existing two classes only.

Get trusted labels for useful new examples. The model’s own prediction is not a ground-truth label. Sample ordinary traffic too, including cases below the OOD threshold, or the dataset will reflect only what this detector happened to notice. Track label quality, provenance, coverage and the age of unresolved reviews.

Before training, deduplicate related examples and split by the unit that could leak information: part, customer, site, session or time. Keep a new-domain holdout out of training and calibration. Preserve an untouched original-domain test set. Retrain on a deliberate mix of reviewed new data and original data so the new condition does not erase the old one.

06 / CLOSE THE LEARNING LOOP

Automate the candidate job.

The downloadable Trigger class requires sustained alerts, a cooldown, at least 200 reviewed labels and a passing data contract before returning true. The demo uses that result to launch training automatically. Its labels come from a synthetic oracle. In your application, the reviewed-label count must come from your labeling system and must refer to the same region, time range and dataset manifest as the alert.

Run aggregation on a scheduler, for example every ten minutes. In one database transaction, claim the completed window, update trigger state and insert a job into an outbox with a unique key built from the champion version, policy version and immutable dataset-manifest hash. A worker claims that job with a lease, validates the manifest and launches the training command. Retries reuse the same job ID; a completed job never trains twice.

The manifest should pin input and label versions, group splits, preprocessing, label schema, code revision and training configuration. Record the seed, environment and artifact hashes with the run. Bound the worker’s resources and retries, and alert on failed or stalled jobs. The small Python download keeps trigger state in memory to make the mechanism readable; it does not implement this distributed scheduler or exactly-once processing.

if not queued:
    print("No candidate queued: trigger conditions were not met.")
    return
candidate = Classifier()
candidate.load_state_dict(champion.state_dict())
# Replay original data. New labels come from our synthetic oracle, NOT predictions.
train(candidate, torch.cat((train_x, new_x)), torch.cat((train_y, new_y)))

07 / CLOSE THE LEARNING LOOP

A training run is not a release.

Evaluate champion and candidate on the same reserved original-domain and new-domain examples. Report counts and uncertainty, not just rounded percentages. Include important class and operating slices, calibration, latency and the downstream cost of errors. If the new data is too sparse to support a decision, keep collecting it.

The example gate allows at most one percentage point of original-domain accuracy regression and requires two points of new-domain improvement. Those values are placeholders. Set your budgets before looking at results, and use an uncertainty-aware comparison appropriate to the task. Repeatedly tuning against the same holdout eventually turns it into development data.

Refit the detector in the candidate’s feature space and calibrate it on a separate, approved mixture of old and newly supported conditions. Keep additional untouched shift examples to test what it still misses. Version model, preprocessing, detector statistics, threshold and policy together. Do not quietly widen the reference to make an alert disappear.

Only a passing bundle becomes eligible for a shadow run or a small canary. Compare live quality when labels arrive, operational metrics and rollback criteria. Expand traffic gradually, retain the previous bundle and switch back if the agreed limits fail. The download writes a candidate bundle and an evaluation file. It intentionally does not change a serving endpoint.

def promotion_gate(old_baseline, old_candidate, new_baseline, new_candidate):
    """Illustrative accuracy gates, not a universal acceptance policy."""
    values = (old_baseline, old_candidate, new_baseline, new_candidate)
    if not all(0 <= x <= 1 for x in values):
        raise ValueError("Accuracy must be finite and in [0, 1]")
    return (old_candidate >= old_baseline - .01
            and new_candidate >= new_baseline + .02)

08 / CLOSE THE LEARNING LOOP

Try it locally, then replace the toy assumptions.

Download the example below, create a Python environment and install the pinned dependency. Run the unit tests, then run demo.py. It generates independent training, calibration and test samples, fits the monitor, emits three versioned windows and trains a candidate if the trigger is ready. It writes output/candidate.pt and output/evaluation.json in your working directory. It sends nothing anywhere.

The code was exercised with Python 3.12 and PyTorch 2.4.1 on CPU. The seed makes this particular demonstration repeatable in that environment; it does not promise identical results across hardware or PyTorch releases. The shifted toy distribution is deliberately easy to detect. It is not an OOD benchmark.

The biggest gap is the representation itself. A network can collapse an unfamiliar input onto a familiar embedding. This distance score will miss it. It also cannot see a changed relationship between inputs and labels when the input distribution stays the same. Keep input checks, outcome monitoring and a random audit sample alongside the detector. The useful system is the whole loop: notice, investigate, learn, test and release.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m unittest -v
python demo.py