Quickstart

From an empty Google Cloud project to a released, trained and judged model. You play both parts: the platform team that builds the environment, and the ML team that uses it.

Before you start

• A Google Cloud project with billing enabled, and permission to create resources in it (Owner is simplest).
• On your machine: Python 3.11, Docker, Terraform 1.9 or later, and the Google Cloud CLI, signed in with gcloud auth application-default login.

1. Build an environment

The starter is Terraform for everything Pipewright needs in one project: buckets, registries, a BigQuery dataset per team, identities, Pub/Sub topics, the launcher on Cloud Run and the environment manifest. It is built twice, because the launcher image must be copied into the registry the first apply creates.

git clone https://github.com/fluxorsoft/pipewright-starter.git
cd pipewright-starter
cp terraform.tfvars.example terraform.tfvars    # set project_id, your team, operators and approvers
terraform init
terraform plan -out=env.tfplan && terraform apply env.tfplan

scripts/publish_launcher.sh "$(terraform output -raw images_repository)"
# prints .../pipewright-launcher@sha256:...  set launcher_image in terraform.tfvars, then:
terraform plan -out=env.tfplan && terraform apply env.tfplan

terraform output -raw environment_file          # keep this: it is your environments/dev.yaml

2. Install the CLI

python -m venv .venv && . .venv/bin/activate
pip install "pipewright-mlops[gcp]==1.3.0"     # the command is pipewright
pipewright --help

Pin the version everywhere, CI included. The same release is on GitHub with its wheel, and the launcher image is in GitHub Container Registry with its digest in the release notes.

3. Start an MLOps repository

Use cases live in a Git repository your team owns. Releases are named after its commits.

mkdir mlops && cd mlops && git init
mkdir environments
terraform -chdir=../pipewright-starter output -raw environment_file > environments/dev.yaml
cat environments/dev.yaml

apiVersion: pipewright/v1
kind: Environment
metadata: { name: dev }
spec:
  manifest: gs://<project>-pw-artifacts/pipewright/environment.json

4. Your first use case

Copy the churn example: two pipelines, components, configuration, table schemas, tests and generated sample data. Use the team name you gave the starter as the tenant.

pipewright new-usecase --name churn --tenant <your-team> --owner <your-team>
pipewright validate --env dev        # schemas, references, cross-checks; offline
pipewright doctor   --env dev        # read-only checks against the live project
pipewright tables sync --env dev     # creates the use case's tables
git add -A && git commit -m "churn: first release"
pipewright release  --env dev        # build, pin by digest, verify, go live

5. Train it

Copy the example’s sample data to your team’s landing folder. A file landing there starts a training run through the launcher; pipewright run starts one from a file already in Cloud Storage.

gcloud storage cp usecases/churn/data/sample_customers.csv gs://<project>-pw-landing/incoming/<your-team>/
# or, from a file already in Cloud Storage:
pipewright run --env dev --usecase churn --source-uri gs://<bucket>/<path>/sample_customers.csv
pipewright status --env dev --usecase churn
pipewright model list --env dev --usecase churn

The run ingests the file, checks the data contract, trains, evaluates against the gates and the current champion, and registers the model only if it passes. With promote_model: manual it is a candidate until a person makes it the champion:

pipewright model approve --env dev --usecase churn --version 1

Where next