Home / Platform

One path from YAML to production, the same for every model.

Every ML team on Google Cloud ends up wiring the same services together: triggers, releases, identities, gates, approvals, monitoring, a record of who did what. Pipewright is that wiring, built once and tested. Your models and your data stay yours, in your project.

Start the quickstart Read the concepts

releases/churn/9f3c2e1.jsonwritten once

Release9f3c2e1churn · commit by r.lee · 2 min ago

verified

component imagesha256:e4e27f97c1…

training templatesha256:7a02b6d4e8…

inference templatesha256:c51f09a3b2…

resolved configsha256:3d88e1f7a0…

contract valid against the schema
templates match what was compiled
table changes additive only
environment policies pass

churn · test · release 9f3c2e1live

Configure

YAML, checked offline

ML team

Release

Pinned by digest, verified

Run

Train, judge, score

Launcher

Watch

Drift and performance

Monitor

Decide

Approve, decline, snooze

People

Prove

Evidence pack, write-once

Audit

Architecture

Google Cloud’s own services, wired the same way for every team.

The platform team builds an environment once and publishes its manifest. ML and SQL teams work only with configuration, code and the pipewright command. Every step runs in your project, as the team’s own identity.

Pipewright on Google Cloud: configuration and releases, triggers and runs, results and signals, and online serving, with the nine steps from publishing the manifest to serving a model — Nine steps, from the platform team publishing the manifest to an approved model serving with a canary. Open it full size to read every label.

The same environment with a private GKE cluster running Kubeflow Pipelines, KubeRay, KServe and CronJobs — The same environment with `orchestrator: kfp`: Kubeflow Pipelines, KubeRay and KServe run inside a private GKE cluster. Everything outside the cluster is unchanged.

For every team

One platform. Four people who need it to work.

Clear ownership is the point: the platform team runs the cloud, ML teams own their models, approvers decide, auditors read the record. Nobody does someone else's job.

Describe the model. Skip the plumbing.

A use case is a folder: your pipeline code, plus YAML for data, training, gates and monitoring. Pipewright does the rest.

Validate offline before anything costs money
Release in one command: build, pin, verify, go live
Champion comparison and gates on every training run
Drift and performance monitoring with email alerts
Features by name, point in time: from feature tables or Feature Store groups, never from the future
Ray when a step needs a cluster: one per run, made and removed for you
Synthetic data when you need it: one switch, learned and gated inside your environment

usecases/churn/config/evaluation.yaml

evaluation:
  primary_metric: auc
  gates:
    - { metric: auc, min: 0.75 }
    - { metric: calibration_error, max: 0.05 }
  champion: { enabled: true, min_improvement: 0.005 }
  slices: [contract_length_months]

$ pipewright release --env prod --usecase churn
verified: contract v3 is valid
verified: training and inference templates match
live: churn -> 67d2e88a

One environment for every team, from one file.

The starter Terraform builds an environment in one project, or Enterprise’s private blueprint builds one with every guardrail on. Teams join by name; one shared launcher serves them all, each with its own identity and data.

One manifest per environment: the names of everything a team may use
Limits teams cannot override: approvals, compute and GPU limits, approved base images
Private when you need it: CMEK, VPC Service Controls and no public endpoints, with Enterprise
CI without keys: GitHub Actions through Workload Identity Federation, let in by identity
Policies per environment: manual promotion, required approvers
Feature access per team: the datasets each team may read, granted, never copied
Synthetic data rules per environment: purposes, engines, gate floors, approvers

platform.yaml

spec:
  blueprint: private-managed
  region: europe-west2
  policies:
    promote_model: manual
    retraining: { required_approvers: [model-risk] }
  approver_groups:
    model-risk: { members: [group:model-risk@yourco.com] }
  tenants:
    retention: { landing_prefix: incoming/retention/ }
    fraud:     { landing_prefix: incoming/fraud/ }
  guardrails:
    vpc_sc: { mode: enforce }

Decide with the facts, under your own name.

When a model drifts you get a short decision pack by email. The email never approves anything: you decide with your own identity, and the record keeps your reason.

Is it real? PSI per feature, the trend, live performance
Is it the data? Data checks on the latest batches
Quorum: one approval per required group, never the requester
Snoozes expire, and cases re-open when things get worse
Synthetic data releases wait for every required group, data protection included

pipewright retrain approve --usecase churn

Case rc-20261007-c3d3ac: churn (prod), drift
Drifted: tenure_months 3.035, monthly_charges 0.853
Data checks passed. Champion v1, 1 day old.
Approvers: ml-leads (done), model-risk

$ pipewright retrain approve --case rc-20261007-c3d3ac \
    --reason "real shift after the migration"
approve sent as approver group model-risk
recorded: case is approved
training submitted on the chosen window

The record, not a reconstruction.

Every release, run, approval and deployment is an event with who did it, kept in an event log you can query, and pipewright lineage reads a model back to its data. Enterprise adds an evidence pack per model: hashed, and stored write-once.

Fifteen controls, each with the facts it rests on
Gaps are reported, never hidden
SHA-256 for every file; tampering shows
Retention locks for years, in your project
Synthetic data releases get their own pack: seven controls

gs://.../evidence/packs/churn/v4/

evidence.json       controls, lineage, approvals
report.md           for people
contract.json       the release that trained it
model_card.md       intended use, limits
evaluation.json     metrics, gates, champion
events.json         every lifecycle event
audit.json          who changed or used it
SHA256SUMS          written last

12 pass · 1 info · 2 n/a · 0 gaps

What approvers receive

A decision pack in the inbox. The decision stays with a person.

When drift or a performance drop opens a case, the people your team names get the facts in a few lines: what moved, by how much, the trend, the data checks, the champion and what retraining would use. Every step after that is sent again, so nothing is decided in silence.

@Email to the addresses in retraining.decisions.notify

⇄Pub/Sub feed for your own apps, opt-in per use case

!Escalation when a case waits too long, and a weekly digest

Inbox · ml-team@yourco.com

[prod] churn: retrain decision needed (rc-20261007-c3d3ac)

InboxML decisionsAction needed

Cloud Monitoring <alerts@monitoring.example>to ml-team, model-risk

Tue 21:17

churn (prod) drifted, and a person needs to decide what happens next. Nothing will retrain until the required groups approve.

Drifted	tenure_months 3.035, monthly_charges 0.853
Predictions	PSI 1.253
Trend	worst PSI 0.01, 1.04, 3.03 over recent runs
Performance	not judged yet: 0 labelled rows
Data checks	passed: 1,000 rows, most nulls total_charges 1.5%
Champion	version 1, trained 2026-10-06
Approvers	ml-leads, model-risk

Review the full decision pack, then decide with your own identity:

pipewright retrain show --env prod --usecase churn --case rc-20261007-c3d3ac View case rc-20261007-c3d3ac

This email never approves anything. Approve, decline or snooze with pipewright retrain; every decision is recorded with your name and reason. You receive this because you are in retraining.decisions.notify for churn.

Promotion

What you tested is what ships. Byte for byte.

pipewright promote carries a tested release, and the model you approved, from test to production. Nothing is rebuilt. Every artifact passes four checks at the border, and anything that differs from what you tested is turned back.

component image test sha256:6d978a1f617ef908a46d2703fe48bf51… prod sha256:6d978a1f617ef908a46d2703fe48bf51… identical model files test 6 of 6, the approved v5 prod 6 of 6, unchanged since promotion (EV-15) verified

Turned back at the border (real refusals from our tests)

✕ A pipeline that compiles differentlytraining: container.resources only in the destination. Nothing was promoted.

✕ A model nobody approvedv6 is not the default (approved) version in test: approve it there first.

✕ A setting production forbidsapprovals.promote_model: auto is not allowed in prod (platform policy)

The record

Every model can tell you where it came from.

Lineage is not a project before each review: it is what Pipewright writes as it runs. A score names its model version and run; the run names its release, its image and the exact rows it read; each decision names the person who made it.

A snapshot of what each run read, with a rows fingerprint, so a run can be replayed exactly
Lifecycle events with who did it: releases, runs, evaluations, approvals, promotions and deployments
An event log in BigQuery, kept by a subscription, queryable by anyone allowed to
Structured, labelled logs for every step, routed to the tools you already use
Erasure by key across a team’s tables, when a customer asks to be forgotten

Lineage in Fleet View

pipewright lineage --env prod --usecase churn --version 2

churn-classifier v2 (prod), promoted from test v10
data      entities churn_entities, features customer_activity
          snapshot churn_snapshots · rows 45,353, fingerprint a91c…
build     release 5708ac83eaa2 · image sha256:e4e27f97…
judged    auc 0.8124 · gates pass · beat champion v1
decisions approved  by model-risk     2026-10-31 09:12
          promoted  by CI, for r.lee   2026-10-31 09:40
          deployed  by model-risk     2026-10-31 10:02
in use    churn-endpoint 100% · weekly scoring

Ways to run Pipewright

Run it the way your organisation works.

Every option runs inside your own Google Cloud project: your data, models and keys never leave it. What changes is who builds and operates the platform, and how much governance comes with it.

Open source

Your team runs it

The Pipewright core under Apache-2.0. Your platform team builds the environment once; ML teams describe each use case in YAML and ship with one command. Free to use, change and extend.

CLI and shared launcher
write-once releases
retrain decisions
promotion by digest
feature tables, point in time
Ray, KServe on GKE
gated synthetic data

View on GitHub Talk to us

Managed on Google Cloud

We run it, in your project

The Pipewright team builds and operates it in your Google Cloud project, so your ML teams start on a working platform instead of building one. You keep ownership, data and billing, including your Google Cloud credits and commitments.

environments built for you
first use cases live with our engineers
upgrades and patches rolled out
platform health watched
your teams onboarded

Start a pilot

Enterprise

Private and governed

For regulated organisations: private by default, with the evidence auditors and model risk ask for. Everything in the core, plus the private-managed blueprint, evidence packs for every model, governed synthetic data releases, Fleet View across every environment and support from the Pipewright team with agreed response times.

CMEK and VPC Service Controls
no public endpoints
private GKE
evidence packs
Fleet View
keyless CI
SIEM exports
on premises on request

Book a demo Compare editions

See it run in your own project.

The quickstart builds an environment, installs the CLI and releases your first use case.

Start the quickstart Build it with your coding agent