Monitoring and retraining

A model that was right in October can be wrong by December, without anything breaking. This tutorial scores drifted data, follows the alert, and decides on a retrain the way Pipewright intends: with the facts in front of a person.

1. Say what to watch

monitoring:
  alerts: { notify: [you@example.com], events: [monitoring.alert, training.rejected, run.failed] }
  drift: { enabled: true, method: psi, threshold: 0.2 }
  prediction_drift: { enabled: true, threshold: 0.2 }
retraining:
  mode: propose                           # off | propose | candidate (trains, never promotes)
  when: [drift, performance]
  decisions: { notify: [you@example.com] }
  approvals: { required: [] }             # approver groups the platform registered

pipewright alerts sync --env dev

2. Score normal data, then drifted data

pipewright run --env dev --usecase churn --pipeline inference --source-uri gs://<bucket>/scoring_customers.csv
pipewright run --env dev --usecase churn --pipeline inference --source-uri gs://<bucket>/scoring_drifted.csv
pipewright monitor status --env dev --usecase churn

Each scoring run measures every feature’s drift (PSI against the training baselines) and the predictions’ drift. The drifted file moves two features well past the threshold: you get an alert, and a retrain case opens.

3. Read the case, then decide

pipewright retrain list --env dev --usecase churn
pipewright retrain show --env dev --usecase churn --case <id>
pipewright retrain approve --env dev --usecase churn --case <id> --reason "real shift after the price change"

The decision pack says what moved and by how much, the trend over recent runs, the data checks, the champion, and what retraining would use. Nothing trains until every required group approves. You could decline (the data is wrong, fix the export) or snooze (wait for more evidence); both expire, and a snoozed case re-opens if things get worse.

4. Performance, once outcomes arrive

Drift says the inputs changed; performance says the model got worse. When outcomes arrive, point monitoring.performance.labels at them. Since 1.2.0, labels.time_column joins each outcome to the score made for its cut-off, so a customer scored every week is judged on the right score.