Setting Up Machine Learning Pipelines with GitOps Principles

In the ever-evolving world of DevOps and machine learning, building scalable and automated workflows has become a cornerstone for success. This guide walks you through setting up an end-to-end MLOps pipeline using GitOps principles, incorporating tools like Argo Workflows, Argo Events, MinIO, FastAPI, MLflow, Kubernetes, and Evidently AI. By following this setup, you’ll have a robust system to detect data drift, retrain models, and deploy them seamlessly.

Architecture Overview

This architecture is designed to handle the entire lifecycle of a machine learning model, ensuring it stays accurate and reliable over time.

Pre-Deployment

We start with Data Preparation, where raw datasets are cleaned and split into training and testing sets. Next, in Model Development, we train and evaluate multiple models using MLflow to identify the best-performing one. Finally, the best model, along with the processed datasets and reference files, is stored in MinIO for version control and easy retrieval.

CI/CD Integration

The pipeline incorporates continuous integration and deployment principles to ensure that changes to the system, whether in the codebase or data, are quickly and safely integrated. This includes linting, testing, and automated deployment steps.

Deployment

The trained model is deployed via a lightweight FastAPI application, which serves predictions and continuously monitors incoming data for drift. This ensures that the model remains accessible and up-to-date.

Post-Deployment

Using Evidently AI, the system monitors the production data for drift. If significant drift is detected, a Kubernetes CronJob triggers the retraining process. The retraining uses the updated data, and the new model is saved back to MinIO. The FastAPI app then dynamically reloads the updated model, enabling seamless updates without manual intervention.

Feedback Loop

This workflow creates a fully automated feedback loop: it detects drift, retrains models, and redeploys them, ensuring that the system remains reliable and accurate over time.

Here’s a high-level diagram of the architecture:

Setting Up Kubernetes

Namespace Creation

Create separate namespaces for each service:

kubectl create namespace mlserver

kubectl create namespace minio

kubectl create namespace mlflow

kubectl create namespace fastapi

kubectl create namespace argo

kubectl create namespace argo-events

MinIO Setup

1. Deploy MinIO for object storage:
   

kubectl apply -f minio_depl.yml

kubectl apply -f minio-ingress.yaml

1. Expose MinIO services:
   

kubectl port-forward svc/minio-service 9000:9000 -n minio

Deploy MLServer

Build and push the MLServer image:

docker build --platform linux/amd64 -t measureapp/mlserver:0.0.2 .

docker push measureapp/mlserver:0.0.2

Apply the deployment:

kubectl apply -f mlserver.yaml

kubectl get pods -n mlserver

Access the service:

kubectl port-forward svc/mlserver-service 5000:5000 -n mlserver

Deploy FastAPI

Build and push the FastAPI Docker image:

docker build --platform linux/amd64 -t measureapp/demo_ai_api:0.0.40 -f docker/fastapi/Dockerfile .

docker push measureapp/demo_ai_api:0.0.40

Deploy FastAPI:

kubectl apply -f fastapi-depl.yaml

kubectl -n fastapi get pods

kubectl -n fastapi port-forward svc/fastapi-service 8000:80

GitOps with Argo

Argo Workflows & Events Setup

Install Argo Workflows:

ARGO_WORKFLOWS_VERSION="v3.6.0"

kubectl apply -n argo -f "https://github.com/argoproj/argo-workflows/releases/download/${ARGO_WORKFLOWS_VERSION}/quick-start-minimal.yaml"

Access the UI:

kubectl -n argo port-forward service/argo-server 2746:2746

https://localhost:2746

Install Argo Events:

kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install.yaml

kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/eventbus/native.yaml

Create secrets for GitHub integration:

kubectl create secret generic git-credentials \

  --from-literal=username=<your-username> \

  --from-literal=token=<your-github-token> \

  -n argo

Drift Detection Workflow

Deploy the drift detection job:

docker build --platform linux/amd64 -t measureapp/drift_detection:0.0.4 -f docker/drift-detection/Dockerfile .

docker push measureapp/drift_detection:0.0.4

kubectl apply -f drift-job.yaml

Simulating Drift

Test the drift simulation:

curl -X POST "http://localhost:8000/simulate-drift" \

-H "Content-Type: application/json" \

-d '{"drift_type": "numerical_shift"}'

Monitor logs:

kubectl -n argo-events logs drift-detection-eventsource-lgtsv-6f949fdcf7-znvh2

Final Thoughts

This guide demonstrates how to build an automated pipeline for model drift detection and retraining using GitOps principles. By combining Kubernetes-native tools like Argo Workflows, Events, and MinIO, you can ensure your machine learning workflows are scalable, reliable, and efficient.

Check out the complete repository here:


or drop a comment below. Let’s build smarter, together! 🚀