MLOps for LLM: A Case Study on Dresscode

I've recently participated in the Gemma 4 challenge here on DEV.to, but fell short compared to many amazing projects. I really liked

Step 1: Data Ingestion & Engineering (The Foundation)

Think of this step as preparing the ingredients before cooking. Raw data is usually messy, scattered, and full of mistakes. If you feed bad data into an AI model, you get bad results (aka "Garbage In, Garbage Out").

• Data Ingestion: Gathering data from various sources (databases, APIs, live streams, files) and moving it to a central storage area (like a data lake).


• Data Engineering (Preprocessing): Cleaning that data. This involves removing duplicates, fixing missing values, and formatting it so the machine learning model can actually understand it.

Sample Use Cases

Here is how this step looks across three different types of systems:

1. Sensor Data Clustering (IoT / Industrial Systems)

• The Raw Data: Millions of rapid temperature and vibration readings coming from factory machines every second.


• The Preprocessing Challenge: The data is noisy, has missing time gaps (due to Wi-Fi drops), and is way too massive.


• What happens here:
  
  
  
  
  • Filtering: Smoothing out the random "noise" or spikes in the data.
  
  
  • Imputation: Filling in those missing time gaps using averages.
  
  
  • Aggregation: Downsampling the data (e.g., converting millisecond readings into 1-minute averages) so the clustering model isn't overwhelmed.
  
  
  
  

2. Image Processing (Computer Vision)

• The Raw Data: Thousands of user-uploaded photos of various sizes, formats (JPEG, PNG), and lighting conditions.


• The Preprocessing Challenge: AI models require images to be exactly the same size and format to process them mathematically.


• What happens here:
  
  
  
  
  • Resizing & Cropping: Uniformly scaling all images to a standard resolution (e.g., 224 x 224 pixels).
  
  
  • Normalization: Converting pixel colors into a consistent scale (usually between 0 and 1).
  
  
  • Augmentation (Optional): Creating variations by flipping or rotating images to give the training model more variety.
  
  
  
  

3. LLM Systems (Large Language Models / GenAI)

• The Raw Data: Massive text dumps from PDF manuals, customer support chats, and internal wikis.


• The Preprocessing Challenge: Text is unstructured, contains sensitive information, and is often too long for an LLM to read at once.


• What happens here:
  
  
  
  
  • Anonymization: Scrubbing out private data like names, phone numbers, or credit cards (PII redaction).
  
  
  • Text Cleaning: Stripping out weird HTML tags, formatting errors, or emojis if they aren't needed.
  
  
  • Chunking: Chopping long documents into smaller, bite-sized paragraphs so they can be turned into "embeddings" for the LLM to search through later.
  
  
  
  

Applying MLOps Step 1 to dresscode

Even when using off-the-shelf Foundation Models like Gemma 4 instead of training your own models from scratch, MLOps Step 1 is vital to ensure consistency, speed, and cost efficiency.

Apply to Use Case 1: Closet Image Uploads

Because users will upload images straight from their smartphones, the ingestion pipeline needs to clean up the raw files before passing them to Gemma.

• 
  Data Ingestion: Establish an API gateway that securely uploads user photos to a cloud landing bucket (e.g., AWS S3).


• 
  Image Downsampling & Compression: Smartphone photos can easily be 5MB to 12MB each. Passing these directly to an LLM creates massive latency and high API costs. The engineering pipeline should instantly compress images and scale them down to a standard resolution suitable for multimodal analysis.


• 
  Format Standardization: Convert disparate uploads (HEIC from iPhones, WebP, PNG) into a single unified format (like JPEG) so Gemma receives predictable data inputs.


• 
  EXIF Stripping: For user privacy, strip out geographic coordinates and camera metadata from the image file during ingestion.

Apply to Use Case 2: Weather & Outfit Recommendations

This use case relies heavily on text and structured API responses. The pipeline needs to handle both data streams seamlessly.

• 
  JSON Schema Standardization (Ingestion): The external weather API will return a raw JSON payload full of data we do not need (like wind angle or air pressure). The pipeline should extract only the critical metrics (e.g., temperature, precipitation_chance, season) and format them into a clean text prompt string for the model.


• 
  Text Normalization (Engineering): Wardrobe descriptions written by users or generated by Use Case 1 might have formatting inconsistencies. The data engineering layer should clean this up—handling missing descriptions gracefully, removing trailing spaces, and capping the list size so we do not exceed the LLM's context window.


• 
  Caching Layer: To reduce external API costs and latencies, we can build an ingestion cache. If the same user asks for an outfit recommendation three times in one morning, the pipeline should pull the cached weather data from two hours ago instead of making a brand-new live API call.

Step 3: Model Training & Experimentation / CI (The AI Laboratory)

This is the phase where the AI actually learns. We take the features we prepared (from Step 2) and feed them to an algorithm so it can look for patterns and build a smart model.

• 
  Model Training: Teaching the AI by showing it examples until it can make accurate predictions or generations on its own.


• 
  Experimentation: Data scientists rarely get it right on the first try. They tweak settings (called hyperparameters), try different algorithms, and run hundreds of tests to find the best-performing version.


• 
  Continuous Integration (CI): In MLOps, CI means automating this testing. Every time someone changes the training code, automated scripts run to ensure the new code doesn't break the system, tests the model's accuracy, and verifies it meets safety guidelines before moving forward.

Sample Use Cases

Here is how training and experimentation look across the three system types:

1. Sensor Data Clustering (IoT / Industrial Systems)

• 
  The Training Process: Feeding months of historical sensor data to an unsupervised clustering algorithm to discover what a "normal machine" looks like versus a "failing machine."


• 
  The Experimentation: Data scientists test different numbers of clusters or try different mathematical distance metrics to see which setup groups the anomalies most cleanly.


• 
  The CI Pipeline: When new factory data architectures are introduced, the CI pipeline automatically retriggers the training script to ensure the clustering algorithm still converges correctly without throwing errors.

2. Image Processing (Computer Vision)

• 
  The Training Process: Training a Convolutional Neural Network (CNN) on millions of tagged images so it learns to recognize the difference between the objects it should recognize.


• 
  The Experimentation: Testing different model architectures or adjusting the "learning rate" to see which combination achieves the highest accuracy in identifying target details.


• 
  The CI Pipeline: When a developer submits new code, the CI pipeline automatically trains a miniature version of the model on a small benchmark dataset to check for bugs and ensures it outputs the correct image classification format.

3. LLM Systems (Large Language Models / GenAI)

• 
  The Training Process: For most companies, this rarely means training a giant LLM from scratch. Instead, it involves Fine-Tuning (taking an existing model like Gemma and training it further on a specific dataset) or optimizing prompt engineering frameworks.


• 
  The Experimentation: Testing different system prompts, adjusting the "temperature" parameter (creativity level), or fine-tuning the model on thousands of curated fashion outfits to see which setup yields the most realistic recommendations.


• 
  The CI Pipeline: This is often called LLM-As-A-Judge CI. Every time the prompt or model configuration changes, the CI pipeline automatically runs a test suite of standard user requests. It passes the outputs to a separate evaluation script to check for hallucinations, offensive language, or formatting errors before allowing the update to go live.

Applying MLOps Step 3 to dresscode

Because we are utilizing a pre-trained foundation model (Gemma 4) rather than training a model from scratch, "Model Training & Experimentation" in our CI pipeline focuses heavily on Prompt Engineering optimization, Prompt/Model versioning, and Automated LLM Evaluation (LLM-as-a-Judge).

Apply to Use Case 1: Closet Image Uploads

For the wardrobe digitization phase, our goal is high accuracy in clothing recognition (e.g., ensuring a jacket isn't mislabeled as a shirt) and consistent structured output formatting.

• Experimentation (Prompt & Model Tuning): We will need to test different system prompts to get Gemma 4 to output text reliably (such as a clean JSON list). We can experiment with different variants of the model (like the unified Gemma 4 12B versus smaller edge versions) to find the best balance of speed, cost, and vision accuracy.


• The CI Pipeline (Automated Evaluation): Create a "Golden Dataset" containing 100 sample photos of diverse clothing items where we already know the correct tags. Every time a developer updates the app's prompt structure or upgrades the underlying Gemma model version, the CI pipeline automatically runs those 100 photos through the new setup. It calculates a accuracy score—if the accuracy falls below the baseline threshold (e.g., 95%), the CI block fails and prevents the code from deploying.

Apply to Use Case 2: Weather & Outfit Recommendations

This agentic workflow relies on function calling. The core risk here is "hallucination" (recommending clothes the user doesn't own) or failing to execute the weather API call properly.

• Experimentation (Constraint Tuning): Data scientists will experiment with the model's Temperature setting (lower temperature, like 0.2, makes the choices less chaotic and strictly bound to the user's wardrobe list) and system guidelines to ensure the model never suggests a raincoat when it isn't raining.


• The CI Pipeline (Function Calling & Safety Verification): The CI pipeline needs to strictly mock the external weather API to test corner cases (e.g., extreme blizzard, extreme heatwaves, or API timeouts). The automated pipeline will run a suite of simulated scenarios and pass the LLM's outfit response to an "LLM Evaluation Judge" (a separate script or deterministic validator) to check:
  
  
  
  
  1. Did the model correctly trigger the API function call when given a date?
  
  
  2. Are all items in the recommended outfit strictly present in the provided wardrobe list?
  
  
  3. Is a heavy coat recommended for a 90°F day? (Failure check).
  
  
  
  

If the new prompt configuration passes all simulated weather scenarios without error, the CI pipeline automatically approves the update.

Step 5: Continuous Deployment & Serving (Taking It Live)

Now that our validated model is sitting safely in the Model Registry (from Step 4), it's time to put it to work in the real world.

• 
  Model Serving: Hosting your model on a live server so apps and users can send it data and get back answers or predictions instantly. It's like turning an offline script into a live, interactive web service.


• 
  Continuous Deployment (CD): The automated pipeline that safely transitions a model from the registry into production. Instead of engineers manually copying files to a server at midnight, the CD pipeline handles testing, scales up the necessary hardware, and rolls out the new model version smoothly without causing app downtime.

Sample Use Cases

Here is how CD and model serving work across the three system types:

1. Sensor Data Clustering (IoT / Industrial Systems)

• 
  How it is Served: Edge or Streaming Serving. Because factory machinery runs continuously, data is streamed instantly into a lightweight runtime container hosted right on the factory floor (the edge) to avoid internet lag.


• 
  The CD Strategy: When an updated clustering model passes all safety checks, the CD pipeline automates a Canary Deployment. It sends the new model to only 5% of the factory's machines first. If those machines report stable metrics for 24 hours, the CD system rolls out the update to the remaining 95% of the machinery automatically.

2. Image Processing (Computer Vision)

• 
  How it is Served: Real-time API Serving. The model is packaged inside a scalable web container (like Docker) and deployed behind an API gateway on a cloud platform equipped with GPUs.


• 
  The CD Strategy: When a new object-detection model is approved, the CD pipeline utilizes a Blue-Green Deployment. It spins up an entirely fresh server cluster running the new model ("Green"). Once the green environment is verified healthy, the load balancer instantly swaps 100% of the live user traffic away from the old server cluster ("Blue") to prevent any user disruption.

3. LLM Systems (Large Language Models / GenAI)

• 
  How it is Served: Optimized Token-Streaming Serving. LLMs are incredibly massive, requiring specialized serving engines (like vLLM or TGI) to dynamic-batch user inputs and stream words back to users one by one rather than making them wait for the entire paragraph.


• 
  The CD Strategy: When a prompt template or model adapter is updated, the CD pipeline manages a Shadow Deployment. The system duplicates real live user requests behind the scenes and sends them to both the old and new prompt configurations simultaneously. The app only shows the user the old model's response, but it tests how the new model performs under actual server load before turning it live.

Applying MLOps Step 5 to dresscode

Because we are using an advanced multimodal foundation model like Gemma 4, "serving" doesn't just mean hosting a script—it means managing heavy visual context inputs and optimizing rapid token streaming so the user isn't stuck staring at a loading screen.

Apply to Use Case 1: Closet Image Uploads

Users will upload multiple photos at once when setting up their virtual closet. This requires high-throughput visual processing.

• Model Serving (Asynchronous Batching): Processing high-resolution images through Gemma 4's vision matrix takes heavy GPU computing power. Instead of making a user wait on a mobile screen, serve this model via an asynchronous queue system (like Celery + Redis). The user uploads 5 photos, the app says "Analyzing your clothes...", and the photos are processed sequentially in the background.


• The CD Strategy (Blue-Green Deployment): If we decide to upgrade the underlying vision prompt package or swap from the standard Gemma 4 model to a more lightweight quantized version (like a Q4 or 8-bit precision variant) to save cloud costs, our CD pipeline should use a Blue-Green layout. It spins up a fresh, separate GPU instance running the new model variant. Once it passes health checks, traffic is instantly rerouted to it, ensuring zero app downtime for users who are actively uploading photos.

Apply to Use Case 2: Weather & Outfit Recommendations

This agentic workflow requires lightning-fast interaction because the user is waiting to see their daily outfit suggestion in real-time.

• Model Serving (Token Streaming + Multi-Token Prediction): We shouldn't make the user wait for Gemma 4 to think, we should run a function call, and draft an entire paragraph before showing the answer. Serve the recommendation pipeline using an optimized LLM engine (like vLLM or Hugging Face TGI) with Token Streaming enabled. This spits out the outfit suggestion word-by-word instantly on the UI. Additionally, take advantage of Gemma 4's built-in Multi-Token Prediction (MTP) drafters to dramatically slash latency during generation.


• The CD Strategy (Shadow Deploying Prompts): Since changing a single line in a prompt or modifying the weather API JSON tool schema can cause unexpected text outcomes, do not deploy prompt updates directly to the public. Our CD pipeline should use a Shadow Deployment. When a developer updates the stylist prompt logic, the live system duplicates actual user requests and silently feeds them to both the old stable prompt and the new shadow prompt. The system validates that the new configuration executes the API function call cleanly under real-world traffic scenarios before we flip it live.

Step 6: Continuous Monitoring (The Health Dashboard)

Just because an AI model works perfectly on day one doesn't mean it will stay that way. Real-world conditions change, and models can degrade, slow down, or become less reliable over time.

Continuous Monitoring acts like a 24/7 heart monitor for our live AI system. It tracks three core elements:

• System Performance: Are the servers running fast, or are they lagging and eating up too much memory?


• Data Drift: Is the incoming real-world data shifting drastically from the data the model was originally trained on?


• Model Performance: Is the model's accuracy dropping or becoming less reliable as time goes on?

Sample Use Cases

Here is how continuous monitoring works across the three system types:

1. Sensor Data Clustering (IoT / Industrial Systems)

• What is Monitored: Incoming sensor scales, anomaly detection frequency, and hardware latency.


• The Drift/Failure Scenario: Over time, physical factory machinery naturally wears down, or a technician installs a new brand of sensor that measures vibration in a slightly different baseline unit.


• The Monitoring Action: The system triggers an alert if the statistical distribution of the sensor values shifts drastically (Data Drift). For example, if the average temperature reading across the factory suddenly climbs 10% without a real heatwave, the monitor flags it so engineers can check if a physical sensor is broken or if the model needs a calibration adjustment.

2. Image Processing (Computer Vision)

• What is Monitored: Image dimensions, file corruptions, lighting ratios, and model confidence thresholds.


• The Drift/Failure Scenario: A smartphone manufacturer pushes a software update that changes the default photo compression format or camera color profile, causing uploaded photos to look slightly washed out to the AI.


• The Monitoring Action: The system monitors the model's internal confidence scores. If the model usually tags objects with 92% confidence, but that average suddenly slips to 61% over a single week, the monitor catches the drop and alerts the engineering team before bad tags ruin the user experience.

3. LLM Systems (Large Language Models / GenAI)

• What is Monitored: Cost per query, response length, token generation latency (Time-to-First-Token), and formatting adherence.


• The Drift/Failure Scenario: Users start using completely new vocabulary or slang that wasn't prominent in the model’s original training data, or the LLM encounters adversarial "prompt injection" attacks trying to bypass its safety guidelines.


• The Monitoring Action: The pipeline acts as an automated auditor. It scans user prompt logs and model outputs using smaller, targeted evaluation models. If it detects a spike in toxic outputs, formatting failures (like returning corrupted JSON strings), or a sudden surge in response latency, it flags the operational bottleneck instantly.

Applying MLOps Step 6 to dresscode

Because we are using Gemma 4's native multimodal capabilities and agentic tool-use, monitoring isn't just about server uptime. It is about tracking formatting failures, tool-calling failures, API latency, and LLM output quality.

Apply to Use Case 1: Closet Image Uploads

For visual cataloging, monitoring ensures Gemma 4 continues to parse image contents accurately and output predictable, structured data profiles.

• Monitoring Formatting Compliance: Since our application backend expects a strict JSON array from Gemma 4 to catalog clothing pieces, setup a monitor to track JSON Parse Error Rates. If a prompt update causes Gemma to suddenly append friendly chatter (like "Here is your list!") instead of pure JSON, the system should catch the parsing failure immediately.


• Tracking Guardrail and Confidence Dips: Monitor the length of the lists being generated. If the average number of clothing items successfully detected per image suddenly drops drastically (Data Drift), it could mean users are uploading images with poor lighting, lower resolutions, or unexpected camera angles that are tripping up Gemma's vision processing matrix.


• Latency Monitoring: Track the Time-To-First-Token (TTFT) specifically for image payloads. Visual tokens take longer to compute; if processing latency spikes, it lets us know if our GPU cluster (or cloud provider endpoint) is throttling.

Apply to Use Case 2: Weather & Outfit Recommendations

This agentic workflow relies heavily on live integrations and reasoning logic. Monitoring here preserves the user's trust in the app's styling intelligence.

• Function-Calling Error Tracking: Since Gemma 4 natively structures external tool calls, we must continuously log and monitor Function Call Success Rates. We need to catch if Gemma 4 makes a formatting error when trying to request weather data, or if the external weather API itself drops, times out, or returns bad payloads.


• Semantic Constraint Monitoring (Hallucination Tracking): Set up a lightweight, deterministic verification layer to monitor the final output text against the provided wardrobe list. If the monitor catches that Gemma 4 recommended a "yellow raincoat" but the word "raincoat" is missing from that user's specific inventory list, it flags a hallucination event. High hallucination rates signal that we need to lower the model's temperature or tighten the context constraints in the prompt registry.


• Cost & Token Usage Tracking: Monitor the total token count per outfit generation. If token volume climbs unexpectedly, it means the model is getting caught in infinite loops during its multi-step agentic planning. Because Gemma 4 supports large context windows (up to 256K tokens), unmonitored prompt expansions can quickly lead to expensive API bills.

Step 7: The Feedback & Retraining Loop (The Growth Cycle)

The final step of the MLOps pipeline closes the circle. This is where our AI system learns from its real-world mistakes, updates its knowledge base, and gets smarter over time.

Think of it as giving a student their graded test papers back. If the AI never finds out when its predictions are wrong or right, it can never improve.

• The Feedback Loop: Capturing signals from the real world. This can be explicit feedback (like a user clicking a thumbs-down button or correcting an error) or implicit feedback (like a user ignoring a recommendation entirely).


• The Retraining Loop: Feeding that new real-world data and feedback back into Step 1 of your pipeline. The system automatically packages the new data, triggers a fresh model training session, evaluates if the new version is smarter, and updates production.