Promotion and CI/CD

What runs in production should be exactly what was tested. This tutorial adds a production environment fed only by promotion, then moves the whole lifecycle into GitHub, where every change is a pull request and every decision has a reviewer.

1. A production environment fed by promotion

Build a second environment with the starter, in its own project, with promote_model: manual. Its environment file names where its releases come from:

apiVersion: pipewright/v1
kind: Environment
metadata: { name: prod }
spec:
  manifest: gs://<prod-project>-pw-artifacts/pipewright/environment.json
  promotion: { from: test }

2. Promote

git checkout <the commit live in test>
pipewright promote --from test --to prod --usecase churn
pipewright model approve --env prod --usecase churn --version 1

Production’s policies apply first. Images are copied by digest, templates must match the tested ones, and the approved model’s files are verified one by one and registered in production as a candidate. Anything that differs is turned back before anything is copied. Rollback works as in any environment.

3. Workflows from one command

pipewright ci init --environments test,prod

This writes GitHub Actions workflows: checks on every pull request (validate, test, doctor, dry runs); on merge, a release to test; for an environment with promotion.from, a promotion workflow instead of a release; an operate workflow for runs, approvals and deployments by hand; and a decisions workflow that turns pending decisions into gate runs waiting for reviewers.

4. No keys

The workflows sign in to Google Cloud with Workload Identity Federation: GitHub gives each run a short-lived signed token naming the repository, and Google Cloud trusts it only from your repository, for a few minutes, as one CI service account. The starter creates the pool, provider and account when you set github = { repository = "org/repo" }. Each GitHub environment needs the variables GCP_WIF_PROVIDER, PIPEWRIGHT_SERVICE_ACCOUNT and PIPEWRIGHT_PIP_SPEC.

5. Decisions with reviewers

Add required reviewers to the production GitHub environment, and turn on "Prevent self-review". A model approval, a retrain case or a deployment with approvals.deploy_model: ci then waits for a reviewer in GitHub; the event records who reviewed it next to the CI identity that acted, and pipewright lineage shows the person behind the CI.