How to Version ML Datasets and Experiments With DVC and MLflow: A DevOps Engineer’s Guide

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering

Put the dataset under DVC, point it at an S3 bucket with versioning turned on, and log every training run to MLflow with the data version attached as a tag. That gives a model the same traceability Git gives code: for any model in production, we can name the exact data, settings and commit that built it, and rebuild it.

This is Part 2 of our MLOps series from a DevOps point of view. Part 1 mapped the whole ML lifecycle onto the DevOps stages we already run. This one covers the first two: data versioning and experiment tracking.

Each concept below is lined up against the DevOps task it mirrors, since reasoning from something you already run day to day is the fastest way into an unfamiliar workflow.

The Problem: A Model Nobody Can Rebuild

A team trains a model, it performs well, and it ships. Three months later someone asks why accuracy dropped, or needs to retrain with one bug fixed. The training data has since been overwritten by a newer export, the notebook that produced the model has been edited a dozen times, and nobody wrote down the learning rate. The model exists, but it can’t be reproduced.

In DevOps terms this is a build with no lockfile, no tagged commit and no build log. We would never accept that for a service, and the fix has the same shape here.

What Maps to What

ML Task	DevOps Equivalent
DVC remote (S3 bucket)	Terraform S3 remote state backend
.dvc pointer file in Git	A lockfile that pins exact versions
params.yaml	Helm values.yaml or a .tfvars file
dvc repro	make, which only reruns what changed
MLflow experiment run	A CI job run with its logs and metrics

Step 1: Turn On Versioning for the Data Bucket

The storage layer comes first. With bucket versioning on, an overwritten or deleted object can still be recovered, which is the same safety net we put on Terraform state buckets:

aws s3api put-bucket-versioning \
  --bucket acme-ml-data \
  --versioning-configuration Status=Enabled

Step 2: Initialise DVC and Point It at the Bucket

DVC sits next to Git. Git keeps the small text files, and DVC moves the large data to the remote:

pip install "dvc[s3]"
git init
dvc init
dvc remote add -d storage s3://acme-ml-data/dvc
git add .dvc .dvcignore
git commit -m "Initialise DVC with S3 remote"

This is the equivalent of configuring a Terraform backend. The remote is just where the real objects live, and credentials come from the usual AWS chain (an IAM role in CI, a profile locally).

Step 3: Track the Dataset

dvc add data/train.csv
git add data/train.csv.dvc data/.gitignore
git commit -m "Track training data v1"
dvc push

dvc add hashes the file, writes a small data/train.csv.dvc pointer with that hash, and adds the real file to .gitignore. The pointer goes into Git and dvc push uploads the data. The pointer plays the role of a lockfile: it names exactly one version of the data.

Step 4: Go Back to an Older Version of the Data

When the dataset changes, repeat dvc add and commit again. To rebuild the model from last quarter, check out the old commit and let DVC restore the matching data:

git checkout <old-commit>
dvc checkout

If the data isn’t in the local cache yet, dvc pull fetches it from the bucket. Code and data are now rolled back together, the same way git checkout already handles code alone.

Step 5: Define the Pipeline and Its Parameters

Hyperparameters belong in a file under version control, not in someone’s shell history. Put them in params.yaml, then describe the training step in dvc.yaml:

# params.yaml
train:
  learning_rate: 0.01
  n_estimators: 200

# dvc.yaml
stages:
  train:
    cmd: python train.py
    deps:
      - train.py
      - data/train.csv
    params:
      - train.learning_rate
      - train.n_estimators
    outs:
      - model.pkl
    metrics:
      - metrics.json:
          cache: false

Run it with dvc repro. DVC compares hashes of the dependencies and parameters against the last run, and skips the stage when nothing changed. Edit one value in params.yaml and only the affected stages run again, much like make rebuilding only the changed targets. A dvc.lock file records the exact hashes of the last run, and it goes into Git too.

Step 6: Log Every Run to MLflow

DVC answers “what data and settings produced this file”. MLflow answers “how did this run compare to the other forty”. Add a few lines to the training script and tag each run with the data version:

import mlflow

mlflow.set_experiment("fraud-detector")

with mlflow.start_run():
    mlflow.log_param("learning_rate", 0.01)
    mlflow.log_param("n_estimators", 200)
    mlflow.set_tag("git_commit", git_sha)
    mlflow.set_tag("data_version", data_hash)

    # ... training happens here ...

    mlflow.log_metric("accuracy", accuracy)
    mlflow.log_artifact("model.pkl")

Each run now records its parameters, its result and the exact commit and data hash behind it, the way a CI run keeps its logs next to the commit that triggered it. For a team, run the MLflow server once with a database for run metadata and S3 for artifacts:

mlflow server \
  --backend-store-uri postgresql://mlflow:***@db:5432/mlflow \
  --default-artifact-root s3://acme-ml-data/mlflow \
  --host 0.0.0.0

On Kubernetes this is a normal Deployment and Service with a Postgres instance behind it. Treat it like any other internal tool: back up the database and restrict who can reach it.

Mistakes That Show Up Later, Not Immediately

• Committing data files to Git. Repos balloon and clones slow to a crawl. Let DVC hold the data and keep only the pointer in Git.
• Running dvc add and forgetting dvc push. The pointer exists, the data doesn’t, and the next person to dvc pull gets an error. Make the push a CI step.
• Using one MLflow experiment for everything. Runs from different models pile into one list and comparisons stop meaning anything. Use one experiment per model.
• Logging parameters by hand in the training script. They drift out of sync with params.yaml. Read the values from the file and log what was actually used.
• An open MLflow server on the network. It has no auth by default. Keep it behind your ingress authentication or a VPN.

Key Lessons

1. Version the data the same way we version code.
   A small pointer in Git and the real data in versioned object storage gives each commit a matching dataset.
2. Keep parameters in a file, not in memory.
   params.yaml makes settings reviewable and diffable, like any other config.
3. Let the pipeline skip work that hasn’t changed.
   dvc repro reruns only the stages whose inputs changed, which saves real GPU hours.
4. Tag every run with the commit and data hash.
   A run without that link is a result nobody can trace later.
5. Run the tracking server like production tooling.
   Database backups, access control and a stable URL matter once a team depends on it.

Summary

Need	Tool	Key Command or File
Store data safely	S3 with versioning	put-bucket-versioning
Pin a dataset version	DVC	dvc add, dvc push
Restore an old state	Git + DVC	git checkout, dvc checkout
Rerun only what changed	DVC pipeline	dvc.yaml, dvc repro
Compare runs	MLflow	log_param, log_metric, tags

None of this needs a platform rebuild. A versioned bucket and DVC on one dataset is an afternoon of work, and it’s enough to make the next model reproducible.

Read Next

• What Is MLOps? The ML Lifecycle Explained for DevOps Engineers
• Structuring Terraform Modules and Remote State: What Actually Scales Past One Environment

The next post in this series covers training pipelines on Kubernetes, including GPU scheduling. Follow along on LinkedIn to catch each part as it’s published.

---

Tags:
#MLOps   #DVC   #MLflow   #DevOps  
#MachineLearning   #SRE