Google Cloud services
Pipewright runs on the services Google Cloud already provides rather than replacing them. You do not need to know them all to start; this page is here for when you meet one. Agent Platform was formerly called Vertex AI, and older articles still use that name.
Set up once, by the platform team
| Service | What it does for you | In Pipewright |
|---|
| Cloud Storage | Files of any size, in buckets | The manifest, release contracts, run working files; a landing bucket whose files start runs |
| IAM | Who may do what, on which resource | A runner service account per team; the launcher and CI have their own; no keys |
| Artifact Registry | Container images and packages, named by digest | Component images and compiled pipeline templates, pinned by each release |
| Pub/Sub | Messages between services | Landing events, run requests, lifecycle events, approver groups |
| Cloud Run | A container as a service that scales to zero | The launcher, shared by every team in an environment |
| Workload Identity Federation | Outside software acting as a service account, without a key | GitHub Actions releasing and promoting |
Every run
| Service | What it does for you | In Pipewright |
|---|
| Agent Platform Pipelines | Runs Kubeflow Pipelines v2, each step a container | Every training, scoring and synthetic run, as the team’s identity |
| Training machines | The machine, memory and GPUs a step runs on | steps in configuration, within the platform’s limits |
| Ray on Agent Platform | Managed Ray clusters | A cluster per step, created for it and deleted after it |
| BigQuery ML | Models trained and used in SQL | Use cases with a bqml section, no pipeline code |
| Cloud Scheduler | Jobs on a schedule | Scheduled runs, the retrain tick and the weekly digest |
Data
| Service | What it does for you | In Pipewright |
|---|
| BigQuery | Tables queried with SQL, no servers | A dataset per team: raw data, snapshots, predictions, metrics, monitoring |
| Feature Store | Registered features over BigQuery | Feature groups as a source, read as of each row’s cut-off |
| Online serving | The latest features by entity ID, in milliseconds | Endpoints that take a customer ID, with features: online |
Models and serving
| Service | What it does for you | In Pipewright |
|---|
| Model Registry | Model versions, with a default | Only passing models are registered; the default is the champion |
| Endpoints | Online predictions, with traffic split | deploy-model, canaries, GPU serving and scale to zero |
| Apigee and Cloud Armor | API keys, quotas and a firewall | In front of a public service that calls an endpoint; the platform team’s choice |
Watch and record
| Service | What it does for you | In Pipewright |
|---|
| Cloud Logging | Every service’s logs in one place | Structured logs labelled with the use case and run |
| Cloud Monitoring | Alerts on metrics and logs | alerts sync: email alerts per use case |
| BigQuery subscriptions | A topic’s messages written into a table | The event log: who approved, promoted and deployed what |
Private environments and GKE
| Service | What it does for you | In Pipewright |
|---|
| Cloud KMS | Encryption keys you hold | Applied to every pipeline job, model, endpoint and query, from the manifest |
| VPC Service Controls | A perimeter around your data | Enterprise’s private blueprint; production enforced |
| GKE | Managed Kubernetes | Kubeflow Pipelines, KubeRay and KServe in a private cluster |
The platform team enables these APIs in each project: aiplatform, artifactregistry, bigquery, cloudscheduler, iam, iamcredentials, logging, monitoring, pubsub, run and storage, plus cloudkms for customer-managed keys and container for GKE.