How to Build a CI/CD Pipeline for ML Models in GitLab CI: Train, Gate, Deploy and Retrain

By KP | TZoneLabs | DevOps & Cloud Engineering

One GitLab pipeline can run the whole ML lifecycle: test the code and the data, build the training image, run training as a Kubernetes Job, register the model, promote it only if it beats the current one, deploy it as a canary, and release it with a manual approval. The runner never touches a GPU. It starts the Job, waits for it, and passes the model version to the next stage. The same pipeline also runs when a drift alert or a schedule triggers a retrain.

This is a bonus Part 6 of our MLOps series from a DevOps point of view. Parts 1 to 5 each covered one stage: the lifecycle, versioning, training on GPU nodes, registry and serving, and monitoring and drift. This part wires them into one .gitlab-ci.yml, which is the file a DevOps engineer would be asked to own.

Each concept below is lined up against the DevOps task it mirrors, since reasoning from something you already run day to day is the fastest way into an unfamiliar workflow.

What Maps to What

ML Pipeline Concept	DevOps Equivalent
Code tests plus data validation	Unit tests and linting, with the data treated as another input
Training stage	The build stage, only slower, costlier and not fully repeatable
Promotion gate	A quality gate before release
Canary, then manual release	Progressive delivery with a production approval
One release at a time per model	A Terraform state lock
Retrain on a drift alert or schedule	A webhook or scheduled pipeline
Protected variables for promotion	Production credentials that only protected branches can read

Step 1: Lay Out the Stages and Decide When Jobs Run

The stages follow the order of the series. The part that is different from a normal service pipeline is the rules. Training is expensive, so it shouldn’t run on every commit to a feature branch. It runs on the default branch when something that affects the model changed, and always when a trigger or a schedule starts the pipeline:

stages: [test, build, train, promote, canary, release]

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_PIPELINE_SOURCE == "trigger"
    - if: $CI_PIPELINE_SOURCE == "schedule"

.ml-release-rules:
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        - src/**/*
        - params.yaml
        - data.version
        - Dockerfile.train

Every job from build onwards uses extends: .ml-release-rules. They have to share the same rules, because a job that lists needs: [train] fails pipeline creation if train was skipped. Merge request pipelines only run the test stage, which keeps GPU spend out of code review.

Step 2: Test the Code and the Data

test:
  stage: test
  image: python:3.11-slim
  variables:
    PIP_CACHE_DIR: $CI_PROJECT_DIR/.cache/pip
  cache:
    key:
      files: [requirements.txt]
    paths: [.cache/pip]
  script:
    - pip install -r requirements.txt
    - pytest tests/unit
    - python scripts/validate_data.py --schema schemas/features.yaml

The first two lines are a normal Python job, and the pipeline caching post explains the cache key. The last line is the ML-specific one. validate_data.py checks the dataset against a schema file: the expected columns and types, a minimum row count, and a ceiling on the share of nulls per column. A silent change in an upstream table is the most common reason a model trains fine and then behaves badly, and this is the cheapest place to catch it. It stands in for the unit test that data would have if it were code.

Step 3: Build the Training Image

build-train-image:
  stage: build
  extends: .ml-release-rules
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf '%s:%s' "$CI_REGISTRY_USER" "$CI_REGISTRY_PASSWORD" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor
        --context $CI_PROJECT_DIR
        --dockerfile Dockerfile.train
        --destination $CI_REGISTRY_IMAGE/train:$CI_COMMIT_SHORT_SHA

Kaniko builds the image without a Docker daemon on the runner. Use whatever build tool you already trust. The one rule is the tag: the commit SHA, never latest, so the Job in the next stage runs exactly the code this pipeline built. This example pushes to GitLab’s own registry, so $CI_REGISTRY_IMAGE/train takes the place of the registry.example.com/ml/train path from earlier parts.

Step 4: Run Training as a Job and Wait for It Properly

Two changes to the Job manifest from Part 3. The Job name now uses the pipeline ID, and the data version is a placeholder:

metadata:
  name: train-JOB_ID
# ...
          image: registry.example.com/ml/train:IMAGE_TAG
          env:
            - name: DATA_VERSION
              value: "DATA_VERSION_VALUE"

The name matters more than it looks. A retrain started by a drift alert runs on the same commit as the last training pipeline. If the name came from the commit SHA, kubectl apply would hit the Job that already exists and fail, because a Job’s pod template can’t be changed. The pipeline ID is unique per run.

Part 3 also mentioned that kubectl wait --for=condition=complete doesn’t notice a failed Job. Here is the fix as a small script the pipeline calls, checking both conditions and a deadline:

#!/bin/sh
# scripts/wait_for_job.sh JOB_NAME [TIMEOUT_SECONDS]
JOB=$1
DEADLINE=$(( $(date +%s) + ${2:-15000} ))

while [ "$(date +%s)" -lt "$DEADLINE" ]; do
  complete=$(kubectl get job/"$JOB" -o jsonpath='{.status.conditions[?(@.type=="Complete")].status}')
  failed=$(kubectl get job/"$JOB" -o jsonpath='{.status.conditions[?(@.type=="Failed")].status}')
  if [ "$complete" = "True" ]; then
    exit 0
  fi
  if [ "$failed" = "True" ]; then
    kubectl logs job/"$JOB" --tail=100
    exit 1
  fi
  sleep 30
done

echo "Timed out waiting for $JOB"
exit 1

train:
  stage: train
  extends: .ml-release-rules
  tags: [k8s-deploy]
  timeout: 4h30m
  script:
    - export DATA_VERSION=${DATA_VERSION:-$(cat data.version)}
    - sed -e "s/IMAGE_TAG/$CI_COMMIT_SHORT_SHA/g" -e "s/JOB_ID/$CI_PIPELINE_ID/g" -e "s/DATA_VERSION_VALUE/$DATA_VERSION/g" k8s/train-job.yaml | kubectl apply -f -
    - sh scripts/wait_for_job.sh train-$CI_PIPELINE_ID
    - kubectl logs job/train-$CI_PIPELINE_ID | grep '^MODEL_VERSION=' | tail -1 > train.env
  after_script:
    - if [ "$CI_JOB_STATUS" = "canceled" ]; then kubectl delete job/train-$CI_PIPELINE_ID --ignore-not-found --wait=false; fi
  artifacts:
    reports:
      dotenv: train.env

A few details that each cost someone a bad afternoon:

timeout: 4h30m is there because GitLab’s default job timeout is one hour, so a four-hour training wait is killed at the one-hour mark and looks like a training failure. The runner’s own maximum timeout has to allow it too.
The after_script cleanup exists because cancelling a pipeline in GitLab stops the waiting job, not the Job in the cluster. Without it a cancelled pipeline leaves a GPU node training for hours, which is the same shape as the stuck loop in our autoscaler cost story. Whether after_script runs on a cancelled job depends on your GitLab and runner versions, so cancel a test pipeline once and confirm the Job disappears. The activeDeadlineSeconds from Part 3 is the backstop if it doesn’t.
The dotenv report turns a line of training output into a pipeline variable. The training script has to print MODEL_VERSION=7, the number MLflow returned when it registered the model, as its last line. Every later job can then read $MODEL_VERSION. That one line is the whole contract between the training code and the pipeline.
The runner tag picks a runner that has kubectl and cluster access. It doesn’t need a GPU, because it only starts the Job and waits. See the self-hosted runners post for tags and scaling.
DATA_VERSION comes from the pipeline when a trigger passes it, and from the committed data.version file otherwise. That keeps ordinary commits reproducible and lets a retrain point at fresh data.

Step 5: Gate the Promotion

The comparison script is the one from Part 4. It takes the new version as an argument and refuses to move the champion alias unless the candidate is at least as good as the current model:

promote:
  stage: promote
  extends: .ml-release-rules
  image: python:3.11-slim
  needs: [train]
  resource_group: ml-model-release
  script:
    - pip install mlflow
    - python scripts/promote_if_better.py $MODEL_VERSION

The resource_group line means two pipelines can’t be in the promote, canary or release stages at the same time. Without it, a scheduled retrain and a drift-triggered one can both finish training within minutes of each other and race to move the alias. Later jobs in the same group wait their turn. It is the same protection a state lock gives, and the same reason we wrote up the Terraform lock incident. Put the same resource_group on the canary and release jobs below.

A failed gate is a failed job with a clear message, which is what you want. Don’t mark it allow_failure. The point of the stage is that a worse model stops the pipeline.

Step 6: Canary, Then a Manual Release

deploy-canary:
  stage: canary
  extends: .ml-release-rules
  tags: [k8s-deploy]
  needs: [train, promote]
  resource_group: ml-model-release
  script:
    - sed "s/MODEL_VERSION/$MODEL_VERSION/g" k8s/fraud-detector-canary.yaml | kubectl apply -f -
    - kubectl rollout status deployment/fraud-detector-canary --timeout=300s

deploy-stable:
  stage: release
  extends: .ml-release-rules
  tags: [k8s-deploy]
  needs: [train, deploy-canary]
  resource_group: ml-model-release
  environment:
    name: production
  when: manual
  script:
    - sed "s/MODEL_VERSION/$MODEL_VERSION/g" k8s/fraud-detector-stable.yaml | kubectl apply -f -
    - kubectl rollout status deployment/fraud-detector-stable --timeout=600s
    - kubectl delete deployment/fraud-detector-canary --ignore-not-found

Both deploy jobs use $MODEL_VERSION, the number the training job printed, so there is no alias to resolve. That’s the pinned-version rule from Part 4 working inside the pipeline. The manual release job is the pause where a person looks at the canary’s prediction distribution from Part 5 before anything reaches all replicas. A canary that is quiet on HTTP errors still isn’t proven.

Rollback is its own manual job. A person enters the version to return to when they run it, and the job redeploys that number, as in Part 4:

rollback:
  stage: release
  when: manual
  tags: [k8s-deploy]
  resource_group: ml-model-release
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - test -n "$PREVIOUS_VERSION"
    - kubectl set env deployment/fraud-detector-stable MODEL_URI=models:/fraud-detector/$PREVIOUS_VERSION
    - kubectl rollout status deployment/fraud-detector-stable --timeout=600s

Set PREVIOUS_VERSION in the variables form when you run the manual job. Afterwards, move the champion alias back to that version so the registry matches what is running. The dynamic environments post has more on how GitLab tracks deployments per environment.

Step 7: Start the Same Pipeline From a Drift Alert or a Schedule

Part 5 ended with Alertmanager calling the trigger API. With the rules above, that call needs no new pipeline. It just passes a data version if the retrain should use newer data:

curl -X POST \
  -F token=$TRIGGER_TOKEN \
  -F ref=main \
  -F "variables[DATA_VERSION]=e5f6a7b8" \
  https://gitlab.example.com/api/v4/projects/$PROJECT_ID/trigger/pipeline

For a model that should be retrained on a regular basis even without an alert, add a pipeline schedule under Build, Pipeline schedules, for example weekly. Both paths go through the same promotion gate and the same manual release, so an automatic retrain still can’t put a worse model in production on its own.

Step 8: Lock Down Who Can Promote

Everything that moves a model is a production change, so treat it that way:

Protect the variables. Store MLFLOW_TRACKING_URI, the registry credentials and TRIGGER_TOKEN as masked and protected CI variables. Protected variables are only passed to pipelines on protected branches, so a merge request can’t run the promote job against the real registry. The secret leak post shows what happens when one isn’t masked.
Give the runner a narrow cluster role. The CI service account needs to create and delete Jobs, read pod logs, and update two Deployments in one namespace. It doesn’t need cluster-admin:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ml-ci
  namespace: mlops
rules:
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["get", "list", "create", "delete"]
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]

Bind it to the CI service account with a RoleBinding. The RBAC post walks through that and how to test it with kubectl auth can-i. Check which namespace the Jobs and Deployments live in, because the training Job and the serving Deployments may sit in different ones in your cluster. In that case you need a Role in each.

Mistakes That Show Up Later, Not Immediately

Training on every push. GPU time gets spent on README edits. Use changes rules and keep training off merge request pipelines.
Leaving the default 1-hour job timeout. The pipeline reports a failure while training is still running fine in the cluster.
Cancelling a pipeline and assuming training stopped. The Job keeps running and keeps billing until its deadline.
Naming the Job after the commit SHA. The first retrain on an unchanged commit fails on kubectl apply.
No resource_group. Two retrains finish together and the alias ends up on whichever one wrote last.
Waiting only for the Complete condition. A crashed Job holds the pipeline until the timeout and the failure shows up hours late.
Letting merge request pipelines read promotion credentials. Unprotected variables are visible to any branch that can edit .gitlab-ci.yml.
Storing weights or datasets as job artifacts. They belong in the registry and in DVC storage. Artifacts expire and bloat the GitLab instance.
Marking the gate allow_failure. The pipeline goes green with a worse model and the release button stays enabled.

Key Lessons

The pipeline orchestrates, the cluster trains.
The runner starts a Job and waits, so it needs kubectl and no GPU.
Gate on rules, not on hope.
Use rules to keep training off feature branches, and keep the comparison gate strict.
Pass the model version as data.
A dotenv report carries the number from training to every later stage, so each deploy uses a pinned version.
Plan for cancel and retry.
Unique Job names, a cleanup step and a deadline keep a cancelled or repeated run from costing money.
One release at a time, one person approves.
A resource_group and a manual release job make automatic retraining safe to leave switched on.

Summary

Stage	Job	What It Protects Against
test	pytest plus data schema check	Code bugs and silent upstream data changes
build	Kaniko build tagged with the commit SHA	Training with code the pipeline didn’t build
train	Job on the GPU pool, with a proper wait	Hung runs, late failures, runaway cost
promote	Compare with champion, in a resource group	A worse model replacing a better one
canary	Pinned-version canary Deployment	Shipping to every replica at once
release	Manual deploy and manual rollback	Unreviewed production changes

None of this needed a new CI system. It is the pipeline we already write for services, with one stage that takes hours and one gate that compares numbers.