Concepts

Pipewright rests on a handful of ideas. Each one exists because, without it, something goes wrong for a real team. Read this once; the tutorials will make more sense.

Two sides, one contract

Pipewright splits the work between two kinds of people, and gives each a file the other never edits.

The result is that an ML team never waits on a ticket for a bucket, and a platform team never finds a model team’s service account with owner rights.

Environments and the manifest

An environment (dev, test, prod) is a Google Cloud project with everything Pipewright needs. Its manifest is a JSON file the platform team publishes in Cloud Storage: which project it is, where the registries and buckets are, which orchestrator runs pipelines, and for each team which identity runs its pipelines and which dataset holds its tables. It also carries policies: limits a team cannot override, such as manual model promotion in production, the GPUs a step may ask for, or the approved base images.

A use case refers to resources by name, never by value. Where the training pipeline needs the dataset, the configuration says ${resources.bigquery_dataset}, and each environment fills in its own. Move a team to a new project and nothing in its use cases changes.

Tenants

A tenant is a team as the platform sees it: a runner identity, a BigQuery dataset, a landing prefix for files, and the feature datasets it may read. Every step of a team’s runs uses its runner identity, so one team’s mistake stays in its own dataset.

Use cases

A use case is a folder: usecase.yaml, a config/ folder with one concern per file (data, training, evaluation, operations, triggers), table schemas, your pipeline code and components. Only a fixed list of keys may differ per environment, so production runs the same model judged the same way. pipewright resolve --env prod prints the effective configuration. See the use case files.

Your pipelines stay plain Kubeflow Pipelines v2. The contract at the edges is small: every pipeline accepts project, region, source_uri and run_id; components take their image from COMPONENT_IMAGE; and training writes an evaluation record next to the model.

Releases and contracts

pipewright release turns one commit into one release. It creates or updates the team’s tables (only ever adding), builds the component image, compiles every pipeline, pushes each to Artifact Registry named by its digest, writes a release contract that records them, verifies it, and only then moves a live pointer. The contract is written once and never changed. Rollback moves the pointer back; nothing is rebuilt.

A digest is a fingerprint of exact contents. A tag can be moved to another image; a digest cannot. That is what makes "the code that trained this model" a fact rather than a guess.

The launcher and runs

One shared launcher per environment, on Cloud Run or inside a private GKE cluster, turns files landing in a team’s prefix, schedules, API calls and Pub/Sub run requests into pipeline runs of the live release, as the team’s identity. It is deployed once and never redeployed for a model release. Run IDs are deterministic, so a retried trigger never starts a duplicate.

Gates and the champion

Every training run is judged on held-out rows against gates (metric, slice and fairness thresholds) and against the champion, the version that serves now, on the same rows. Only a model that passes is registered. Pipewright recomputes the gates from the evaluation record before it approves or serves a version, so a rewritten step that drops a gate cannot promote a failing model.

With approvals.promote_model: manual, a passing model is a candidate until a person runs pipewright model approve; then it is the default version, the champion. Serving has its own approval: approvals.deploy_model is manual, automatic, or gated in CI.

Monitoring and decisions

Scoring runs measure feature and prediction drift (PSI against the training baselines) and, once outcomes arrive, live performance. Alerts email the team. When drift or a performance drop matches the use case’s policy, Pipewright opens a retrain decision case with a short decision pack, instead of retraining. Named approver groups approve, decline or snooze it with their own identity; nothing trains until every required group agrees. Snoozes expire, and a case re-opens when things get worse.

Promotion

Production does not build. pipewright promote --from test --to prod carries the release live in test, and the model approved there, to production: images copied by digest, templates checked against the tested ones, the model’s files verified one by one, production’s policies applied first. In the closed retrain loop, production detects, decides, approves and serves, and never trains: an approved case trains in test and comes back by promotion.

Events, logs and lineage

Every key moment is a lifecycle event on Pub/Sub and a labelled log line: releases, runs, evaluations, registrations, approvals, promotions, deployments, alerts and decisions, each with who did it. An event log in BigQuery keeps them. Each training run can keep a snapshot of exactly the rows it read. pipewright lineage leads a model version back to its run, release, rows, evaluation and decisions.

Two orchestrators

By default pipelines run on Agent Platform Pipelines (formerly Vertex AI Pipelines), Ray on Agent Platform and serving on Agent Platform endpoints. With orchestrator: kfp in the manifest, the same compiled pipelines run on Kubeflow Pipelines in a private GKE cluster, with KubeRay and KServe. ML teams change nothing. See Running on GKE.