Git + CI/CD for Data & ML Workloads: A Practical Playbook

Git + CI/CD for Data & ML Workloads: A Practical Playbook

A model scores 94% accuracy in a notebook on a Friday afternoon. Someone commits the notebook, merges it, and calls it done. Three weeks later, the “same” model in production is quietly making worse predictions than the version it replaced — and nobody can say exactly why, because nobody can reconstruct exactly what ran. Was it the data? A silently upgraded dependency? A preprocessing step that got refactored in a later commit? Without a real answer, the team spends a day archaeology-digging through Slack threads and old notebook outputs instead of shipping the next thing.

This is the specific failure mode this guide is about. Software engineering solved “how do we ship code reliably” with Git and CI/CD decades ago, and most data and ML teams inherit those tools directly — which works, right up until it doesn’t. Data and ML workloads have three properties regular software mostly doesn’t: the artifacts are often huge (datasets, model weights), the outputs are non-deterministic even when the code is identical (different random seeds, different data snapshots, GPU non-determinism), and “correct” is a statistical judgment, not a pass/fail assertion. Git and CI/CD both need real adaptation to handle that — not a lecture on branching strategy you’ve already read, but the specific changes that make the difference between a pipeline you trust and one you’re afraid to touch.

This is written for data engineers and ML engineers who already know Git and CI/CD basics and want to know what’s actually different when the thing you’re shipping is a model or a pipeline instead of an API.

Why “Just Use Git” Doesn’t Fully Work Here

Git was built to track text files that change in small, reviewable diffs. A Python script diffing cleanly is exactly what Git is good at. A 2GB parquet file, a Jupyter notebook whose JSON representation rewrites entirely on every cell execution, or a 400MB model checkpoint are exactly what Git is bad at — and if you don’t plan for this, you find out the hard way, usually when someone’s git clone takes eleven minutes and the .git directory is larger than the actual codebase.

The deeper issue isn’t storage — it’s reproducibility. In normal software, “same code, same behavior” mostly holds. In ML, the same training script can produce a meaningfully different model depending on which exact data snapshot it ran against, which library versions were installed, which random seed was set (or wasn’t), and whether it ran on CPU or GPU. Git alone answers “what code changed.” It has no native answer for “what data did this run against” or “what model came out the other end” — and those two questions are usually the ones you actually need answered during an incident.

None of this means Git is the wrong tool. It means Git is one part of a bigger versioning story, and treating it as the whole story is where teams get burned.

The Four Things You Actually Need to Version

Most ML reproducibility problems trace back to only tracking one of these four things when you needed all four:

The practical rule: if you can’t answer “what exact code, data, model, and environment produced this specific prediction” within a couple of minutes, you don’t have reproducibility — you have a model that happens to work until the day it doesn’t and nobody can explain why. Getting all four tracked, even loosely, is worth more than perfecting any single one of them.

Git Hygiene for Data & ML Repos

A few adjustments make Git itself meaningfully better suited to this work, without needing to change tools.

Stop committing raw notebook JSON. A notebook’s .ipynb file stores cell outputs, execution counts, and metadata alongside the actual code — meaning a notebook re-run with no logical changes still produces a large, noisy diff. Two practical fixes: strip outputs before commit (nbstripout, configured as a Git filter so it happens automatically), or convert notebooks to plain scripts with jupytext, which pairs a .ipynb with a clean .py representation that Git can actually diff sensibly.

# One-time setup: strip notebook outputs automatically on every commit
pip install nbstripout
nbstripout --install

Use Git LFS or DVC for large files — don’t fight Git on this. Git wasn’t designed for multi-gigabyte binary files, and forcing it to track them bloats every clone and every fetch for everyone on the team, forever (Git doesn’t forget — history stays even after a file is later deleted). Git LFS stores large files outside the main repository object store and swaps in lightweight pointers; DVC does something similar but is purpose-built for ML, additionally tracking dataset and model versions with their own metadata files that do live in Git, giving you Git-based versioning of things too large for Git itself.

# DVC: track a dataset the same way you'd track code, without bloating Git
dvc add data/training_set.parquet
git add data/training_set.parquet.dvc .gitignore
git commit -m "Track training set v3 via DVC"
dvc push   # sends the actual data to remote storage (S3, GCS, etc.)

Branch around experiments, not around “in progress.” A common anti-pattern: one long-lived branch per data scientist that accumulates months of divergent, half-abandoned experiments. A cleaner pattern treats each meaningfully different experiment as its own short-lived branch, tagged with enough metadata (in the commit message or a linked experiment-tracking run) to trace exactly which hyperparameters and data version produced which result — so that “which of these fourteen branches actually worked” isn’t a mystery six weeks later.

Designing CI for Data & ML Pipelines

Traditional CI answers one question: does the code work? ML CI needs to answer a second, harder question: does the output still meet quality bars, given this code and this data? Skipping the second question is how a syntactically perfect pipeline ships a materially worse model without anyone noticing until a downstream metric drops.

A CI pipeline built for this generally runs through several distinct gates, each one cheaper to fail at than the next:

1. Static checks (fast, seconds) — linting, type-checking, unit tests on pure transformation logic. Identical to normal software CI, and should fail fast before anything more expensive runs.
2. Data validation (moderate cost) — schema checks, null-rate thresholds, distribution drift checks against a reference dataset. Tools like Great Expectations or Pandera let you write these as code, the same way you’d write a unit test, so a malformed upstream data change fails the build instead of silently poisoning a training run.
3. Training/evaluation (expensive, often GPU-backed) — a smoke-test-scale training run (a tiny subset of data, a handful of epochs) to confirm the pipeline actually executes end-to-end, plus evaluation against a fixed holdout set to catch regressions before a full-scale, expensive retrain is triggered.
4. Model comparison — the new candidate model’s metrics compared against the currently-deployed model’s metrics on the same holdout set, with the pipeline failing (or at minimum flagging for human review) if the candidate is meaningfully worse.

The gate ordering matters for cost control as much as correctness — there’s no point burning GPU hours on a training run whose input data already failed a schema check that would have caught the problem in seconds.

CD for ML: Deployment Patterns That Actually Hold Up

Continuous deployment for a typical web service mostly asks “did the new version deploy without crashing.” Continuous deployment for a model needs to ask a different question: “is the new model’s behavior actually better,” which is a statistical claim, not a binary one — and that changes the shape of the rollout.

A model registry, not just a deployed artifact. Every model version that clears CI’s gates should land in a registry (MLflow Model Registry, SageMaker Model Registry, Vertex AI Model Registry, or even a well-organized versioned S3 prefix with metadata) recording exactly which code commit, data version, and evaluation metrics produced it — so “roll back to the previous model” is a lookup, not an investigation.

Shadow and canary deployment, not all-at-once. Shadow deployment runs the new model alongside the current one on live traffic without its predictions actually being used, purely to compare behavior safely. Canary deployment routes a small, real slice of traffic to the new model and compares live metrics before a full rollout. Both catch the class of regression that a static holdout-set evaluation can miss — real production data drifting from whatever the holdout set represented.

Rollback triggers based on model metrics, not just infrastructure health. A model can be “healthy” from an infrastructure standpoint (responding fast, no errors) while quietly making worse predictions — infrastructure monitoring alone won’t catch that. Production monitoring for ML needs its own metrics: prediction distribution drift, live accuracy against delayed ground truth where available, and business-metric proxies (conversion rate, click-through rate) that would flag a regression infrastructure metrics never would.

Orchestrating the retraining loop. The feedback loop in the diagram above — drift detected in production triggering a new training run — needs something to actually schedule and run it. This is exactly the orchestration problem a tool like Apache Airflow solves well: a scheduled or externally-triggered DAG that pulls fresh data, retrains, and hands the candidate model back into the CI evaluation gates above. If you’re setting this up, Airflow Executors and Airflow Providers & Connections cover the specific pieces (which executor to run training tasks on, how to connect to your model registry and cloud storage) that a retraining DAG actually needs.

A Tooling Cheat Sheet

No single tool covers all of this, and reaching for the trendiest one before understanding what problem it solves is a common, expensive mistake. A reasonable starting stack:

The point isn’t to adopt every row on day one — it’s knowing which category of problem each tool addresses, so when you hit a specific pain (can’t reproduce a model, can’t tell if new data broke something, can’t safely roll back), you know which category to reach into rather than guessing.

Three Examples From Real Pipelines

The patterns above are easier to apply once you’ve seen them break something specific. Here are three scenarios that show up in nearly identical form across different teams and different stacks.

Example 1: The Churn Model Nobody Could Rebuild

A subscription analytics team retrained their churn-prediction model monthly. Two months after a retrain, a stakeholder asked why the November model’s feature importances looked different from October’s — reasonable question, easy to answer, except the team discovered they couldn’t actually rebuild the October model at all. The training script was unchanged in Git. The problem was the input: SELECT * FROM customer_events WHERE month = 'october' pulled from a table that had since been reprocessed upstream, silently altering rows that used to feed October’s training run. Git had a perfect record of the code. It had no record of the data, because the data was never anything more than a live query.

The fix wasn’t exotic. They added dvc add on a monthly data export before each training run, committed the resulting .dvc pointer file alongside the training script in the same Git commit, and pushed the actual parquet snapshot to S3 via dvc push. Rebuilding any past month’s model became git checkout <that month's commit> && dvc pull && python train.py — three commands, instead of a multi-day investigation. The lesson that generalized past this one incident: a live query is not a dataset version, and any training pipeline that reads directly from a mutable table has already lost reproducibility before a single line of training code runs.

Example 2: A Minimal CI Pipeline That Actually Catches Something

A three-person ML team wanted CI but had no appetite for a heavyweight platform. Here’s roughly what they landed on in GitHub Actions — small enough to maintain, and it genuinely caught two real data issues in its first month:

name: ml-ci
on: [pull_request]

jobs:
  validate-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Lint and unit test
        run: |
          ruff check .
          pytest tests/unit

      - name: Validate training data schema
        run: python scripts/validate_data.py --data data/train.parquet

      - name: Smoke-test training run
        run: python train.py --config configs/smoke.yaml --epochs 1

      - name: Compare candidate vs. production metrics
        run: python scripts/compare_metrics.py --baseline metrics/prod.json --candidate metrics/smoke.json

validate_data.py was a genuinely small script — under sixty lines, using Pandera to assert column types, a non-null rate above 98% on required fields, and a value range check on the target column. It caught a currency column that had silently switched from cents to whole dollars upstream, weeks before that would have shown up as a mysteriously miscalibrated model. Nothing here required a dedicated ML platform — just the same CI runner they already used, pointed at a couple of ML-specific checks.

Example 3: A Canary That Saved a Launch

A media company’s recommendation team retrained their ranking model and, per the metrics computed against their fixed holdout set, it looked like a clear win — a few points better on offline precision. Rather than rolling it out fully, they routed 5% of live traffic to the new model behind a feature flag, watching click-through rate and session length alongside the usual infrastructure metrics. Within six hours, CTR on the canary slice was measurably worse than the control group, despite the offline holdout suggesting an improvement.

The cause, once investigated, was a distribution shift the holdout set didn’t represent: the new model had been trained on a data window that under-sampled a content category that had grown significantly in the weeks since the holdout was last refreshed. Offline metrics looked good because the holdout set was itself stale. Because only 5% of traffic was affected, and for only six hours, the business impact was negligible — rolling this out to 100% first would have meant a measurable dip in engagement across the entire user base before anyone noticed. The team’s actual fix was twofold: refresh the holdout set on a regular cadence instead of treating it as fixed forever, and keep canary deployment as a standing step rather than an occasional one, since it was the only thing that caught this particular failure mode.

Common Mistakes

Treating a Jupyter notebook as the unit of deployment. A notebook is a fine place to explore; it’s a poor place to leave the actual production logic, since it resists code review, testing, and clean diffing in ways a proper module doesn’t. Extract the real logic into functions/modules once it’s proven out, and keep the notebook as exploration scaffolding, not the shipped artifact.

Versioning code without versioning the data it ran against. A perfectly reproducible pipeline that silently trains against whatever the data lake currently contains isn’t reproducible at all — six months later, that data has changed, and “re-running the same commit” produces a different result with no record of why.

Skipping data validation because “the pipeline ran successfully.” A pipeline can complete without error while silently processing garbage — a schema change upstream, a column quietly full of nulls, a currency field that switched units. “It ran” and “the output is trustworthy” are different claims, and only one of them is checked by default.

All-or-nothing deployment for models. Rolling a new model to 100% of traffic immediately removes your ability to catch a regression before it affects everyone — shadow or canary deployment costs comparatively little engineering effort for a large reduction in blast radius when something’s wrong.

No rollback plan until you need one. Deciding how to roll back while production is degraded is strictly worse than deciding it in advance — a model registry with a one-command rollback to the last known-good version turns an incident into a five-minute fix instead of an hour of scrambling.

Frequently Asked Questions

Do I really need DVC, or is Git LFS enough? Git LFS solves the “large files bloat my repo” problem well on its own. DVC solves that plus adds ML-specific dataset/model versioning semantics and pipeline tracking — worth the extra setup once you’re managing multiple dataset versions or need to reproduce “which data produced which model,” not just “store this big file somewhere.”

How much CI is actually necessary for a small team or a single-model project? Start with static checks and basic data validation — those are cheap and catch the most common failure classes. Full training-in-CI and model-comparison gates earn their cost once retraining happens regularly enough that manual comparison becomes a real bottleneck; for a model that’s retrained rarely, lighter-weight manual review before deployment is a perfectly reasonable stage to stay at longer.

What’s the actual difference between shadow deployment and A/B testing? Shadow deployment runs the new model on real traffic but never surfaces its predictions to users or downstream systems — it’s purely for safe comparison. A/B testing deliberately serves the new model’s actual predictions to a real subset of users, measuring real outcomes — it answers a different, more consequential question, and carries real user-facing risk that shadow deployment doesn’t.

Can I use the same CI/CD platform I already use for regular software? Yes — GitHub Actions, GitLab CI, and similar tools are entirely capable of running data validation and model evaluation steps; what changes is the content of the pipeline stages, not the platform running them. You don’t need a separate, exotic CI system just because the payload is a model instead of a web service.

Is all of this overkill for a small side project? For a personal project with no other stakeholders and low consequences from a bad prediction, much of this is genuinely optional — the value scales with team size, deployment frequency, and how costly a silent regression actually is. The four-things-to-version framework is worth internalizing regardless of scale; the full CI/CD machinery is worth building out once the cost of a mistake starts to matter.

Summary

Git and CI/CD don’t need to be reinvented for data and ML work — they need to be extended to cover what they were never designed to track: data versions, model artifacts, and environment state, alongside the code changes they already handle well. The teams that avoid the “why did production quietly get worse” scramble are the ones who treat all four of code, data, models, and environment as things that get versioned together, who gate CI on data quality and model metrics rather than just green tests, and who deploy models the way you’d want any consequential change rolled out — gradually, monitored, and with a rollback plan that was written before it was needed, not during an incident. None of it is exotic. It’s the same discipline software engineering already trusts Git and CI/CD to provide, pointed at a few extra things that actually matter for this kind of work.